web-monitoring-processing

A component of the EDGI Web Monitoring Project.

Overview of this component's tasks

This component is intended to hold various backend tools serving different tasks:

Query external sources of captured web pages (e.g. Internet Archive, Page Freezer, Sentry), and formulate a request for importing their version and page metadata into web-monitoring-db.
Provide a web service that computes the "diff" between two versions of a page in response to a query from web-monitoring-db.
Query web-monitoring-db for new Changes, analyze them in an automated pipeline to assign priority and/or filter out uninteresting ones, and submit this information back to web-monitoring-db.

Working and Under Active Development:

A Python API to PageFreezer's diffing service in web_monitoring.page_freezer
A Python API to the Internet Archive Wayback Machine's archived webpage snapshots in web_monitoring.internetarchive
A Python API to the web-monitoring-db Rails app in web_monitoring.db
A Python API to PageFreezer's archived snapshots in web_monitoring.pf_edgi
Python functions and a command-line tool for importing snapshots from PF and IA into web-monitoring-db.

Legacy projects that may be revisited:

Get Python 3.6. This packages makes use of modern Python features and requires Python 3.6+. If you don't have Python 3.6, we recommend using conda to install it. (You don't need admin privileges to install or use it, and it won't interfere with any other installations of Python already on your system.)
Install the package.
```
python setup.py develop
```
Copy the script .env.example to .env and supply any local configuration info you need. (Only some of the package's functionality requires this.) Apply the configuration:
```
source .env
```
See module comments and docstrings for more usage information. Also see the command line tool wm, which is installed with the package. For help, use
```
wm --help
```
There is not yet any published documentation.

Name		Name	Last commit message	Last commit date
Latest commit History 212 Commits
archives		archives
get_article_text		get_article_text
page_freezer_python_module		page_freezer_python_module
scripts		scripts
web_monitoring		web_monitoring
.editorconfig		.editorconfig
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
ChunkingDiffsFromPageFreezerAPI.py		ChunkingDiffsFromPageFreezerAPI.py
Data diff doc.ipynb		Data diff doc.ipynb
Data from different sources.md		Data from different sources.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
Pagefreezer diff service demo.ipynb		Pagefreezer diff service demo.ipynb
README.md		README.md
circle.yml		circle.yml
deployment.md		deployment.md
diff_config.json		diff_config.json
requirements.txt		requirements.txt
run_tests.py		run_tests.py
setup.cfg		setup.cfg
setup.py		setup.py
test-requirements.txt		test-requirements.txt
versioneer.py		versioneer.py