Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply BSA diffs to the database #2229

Closed
wants to merge 9 commits into from

Commits on Dec 10, 2023

  1. Add the BsaDomainRefresh entity

    Add the BsaDomainRefresh class which tracks the refresh actions.
    
    The refresh actions checks for changes in the set of registered and
    reserved domains, which are called unblockables to BSA.
    weiminyu committed Dec 10, 2023
    Configuration menu
    Copy the full SHA
    83f6eb4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0ecc2b7 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c7ae72a View commit details
    Browse the repository at this point in the history
  4. Rebase again

    weiminyu committed Dec 10, 2023
    Configuration menu
    Copy the full SHA
    cf067fb View commit details
    Browse the repository at this point in the history
  5. Rebase

    weiminyu committed Dec 10, 2023
    Configuration menu
    Copy the full SHA
    1d77d97 View commit details
    Browse the repository at this point in the history
  6. Rebase

    weiminyu committed Dec 10, 2023
    Configuration menu
    Copy the full SHA
    1660eae View commit details
    Browse the repository at this point in the history
  7. Create diffs between BSA downloads

    Compare the latest BSA download and the previous one to discover:
    
    * New and deleted orders
    
    * New and deleted labels as well as konwn labels referenced by new orders
    
    The diff is stored on GCS as two files, one for orders and one for
    diffs.
    weiminyu committed Dec 10, 2023
    Configuration menu
    Copy the full SHA
    88dc0f1 View commit details
    Browse the repository at this point in the history
  8. Apply BSA diffs to the database

    Add a step in BsaDownloadAction to apply the diffs from the most recent
    download. This includes:
    
    - Add new labels and remove deleted labels to the `BsaLabel` table.
    - Add new entries to the `BsaDomainInUse` table if applicable. These are
      registered/reserved domains that match the new labels.
    - Generate list of domains that are not blocked and save them in a GCS
      file. These include all domains from the previous step as well as
      domains derived from invalid labels.
    
    More tests are needed but this PR successfully processed a real
    download.
    weiminyu committed Dec 10, 2023
    Configuration menu
    Copy the full SHA
    e2eb621 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2023

  1. Post rebase fix

    weiminyu committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    f140dcd View commit details
    Browse the repository at this point in the history