Skip to content

Script to transform the Disconnect block-list into Safebrowsing v2 format for Firefox Tracking Protection

License

Notifications You must be signed in to change notification settings

mozilla-services/shavar-list-creation

shavar-list-creation

Build Status Coverage

This script fetches blocklist .json from urls (such as shavar-prod-lists) and generates safebrowsing-compatible digest list files to be served by shavar.

Requirements

  • python ≥ 3.6
  • (optional) virtualenv and/or virtualenvwrapper

Run

  1. (optional) Make a virtualenv for the project and activate it:

    virtualenv -p python3.8 shavar-list-creation
    source shavar-list-creation/bin/activate
    
  2. Install required libraries:

    pip install -r requirements-test.txt
    
  3. Copy the sample_shavar_list_creation.ini file to shavar_list_creation.ini:

    cp sample_shavar_list_creation.ini shavar_list_creation.ini
    
  4. Run the unit tests (currently under development):

    python -m pytest -v --cov=. --cov-branch
    
  5. Run the lists2safebrowsing.py script:

    ./lists2safebrowsing.py
    

Usage

This is run by a Jenkins deployment job every 30 minutes that:

  1. Checks out this repository
  2. Checks out the shavar-list-creation-config repository
  3. Copies stage.ini or prod.ini to shavar_list_creation.ini
  4. Runs python lists2safebrowsing.py, which uploads updated safebrowsing list files to S3 for shavar. ⚠️ Since there is no concept of deploy for the scripts, the changes that are merged to main are immediately accessible Jenkins for both Stage and Production.

About

Script to transform the Disconnect block-list into Safebrowsing v2 format for Firefox Tracking Protection

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Packages

No packages published