Build PageFreezer-Outputter that fits into current Versionista workflow #9

dcwalk · 2017-03-09T03:48:14Z

From @ambergman on February 10, 2017 8:5

To replicate the Versionista workflow with the new PageFreezer archives, we need a little module that takes as input a diff already returned (either by PageFreezer's server or another diff service we build), and simply outputs a row in a CSV, as our versionista-outputter already does. If a particular URL has not been altered, then the diff being returned as input to this module should be null, and no row should be output. Please see @danielballan's issue summarizing the Versionista workflow for reference.

I'll follow up soon of a list of the current columns being output to the CSV by the versionista-outputter, a short description of how the analysts use the CSV they're working with, and some screenshots of what everything looks like for clarity

Copied from original issue: edgi-govdata-archiving/web-monitoring-ui#17

dcwalk · 2017-03-09T03:48:15Z

From @allanpichardo on February 10, 2017 15:7

From what I've observed with the Pagefreezer API, taking a diff of 2 pages takes an average of 5 seconds. If there are ~ 30,000 pages to monitor, then pagefreezer is probably not the most appropriate diff service for this task. I think the main bottleneck in Pagefreezer is that they transcode the diff information into HTML for every request. I have run similar diffs on my machine using Git diff and it usually takes one second or less.

Here's what I suggest:

We make a command line tool that creates git diffs of 2 pages and saves them to file (or a local database). This command line tool can take optional filters so we can remove unimportant parts of the page over time. The cli tool could be set as a cron task on a server and run daily to diff the 30,000 pages in the background.
If a significant difference is found, then the CLI creates the CSV entry for the analysts as per your description.
Since the last diff has been stored as a file (or database row) the git visualizer can pull that and do the parsing at that particular time. Thus we incur the cost of transcoding to HTML only when it's necessary.

dcwalk · 2017-03-09T03:48:15Z

From @titaniumbones on February 10, 2017 15:14

I love the idea. Question : what if the HTML (DOM) context is what tells us whether a diff is significant?

On February 10, 2017 10:07:57 AM EST, Allan Pichardo [email protected] wrote:

From what I've observed with the Pagefreezer API, taking a diff of 2
pages takes an average of 5 seconds. If there are ~ 30,000 pages to
monitor, then pagefreezer is probably not the most appropriate diff
service for this task. I think the main bottleneck in Pagefreezer is
that they transcode the diff information into HTML for every request. I
have run similar diffs on my machine using Git diff and it usually
takes one second or less.

Here's what I suggest:

We make a command line tool that creates git diffs of 2 pages and
saves them to file (or a local database). This command line tool can
take optional filters so we can remove unimportant parts of the page
over time. The cli tool could be set as a cron task on a server and run
daily to diff the 30,000 pages in the background.

If a significant difference is found, then the CLI creates the CSV
entry for the analysts as per your description.

Since the last diff has been stored as a file (or database row) the
git visualizer can pull that and do the parsing at that particular
time. Thus we incur the cost of transcoding to HTML only when it's
necessary.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
edgi-govdata-archiving/web-monitoring-ui#17 (comment)

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

dcwalk · 2017-03-09T03:48:15Z

From @allanpichardo on February 10, 2017 15:29

@titaniumbones Yeah, I suspect that it will. If we do this with Node.js, then we have the option of using jQuery to parse the HTML archives. We can determine that certain HTML nodes are insignificant, such as <head meta> for example or <rel type="stylesheet"> etc etc... Therefore, at the time that an archive is loaded from disk, then the CLI uses jQuery to delete such nodes from the text blob and then execute the diff.

The diff would look like a regular git diff, but then the visualizer would have some logic that could convert the ++++ ----- syntax into <ins><del> syntax and we'll output that on screen.

When viewing the page in the visualizer, an analyst would have the option of selecting a DOM element and saying that it's insignificant, thus adding it to an ongoing list that would be fed back to the CLI on the next cycle.

dcwalk · 2017-03-09T03:48:16Z

From @ambergman on February 10, 2017 16:28

@allanpichardo Really disappointing to hear the PageFreezer API moves so slow but, as you've described, it seems like we have plenty of of other options (and, of course, we always new we had git as a backup). Your 1-3 above sound really great, and I think it makes perfect sense, as you said in point 3, to only parse the diff in the visualizer when it's called back up by an analyst.

Regarding you last comment - I think it sounds great to have the option in the visualizer, maybe in some simple dropdown form, of marking a diff as insignificant. That'll mean everything lives just in that simple visualizer, and everything else will run in the CLI.

The only other thing to add, then, would be to perhaps have a couple of different visualization options, perhaps a "side-by-side" or "in-line" page view for changes, but then also a "changes only" view (very useful for huge pages with only a few changes. I'll write something about that in issue #19, the visualization issue, as well.

dcwalk · 2017-03-09T03:48:16Z

From @titaniumbones on February 10, 2017 17:0

I think this is great. In yr opinion are there pieces of this I should ask folks to work on in SF tomorrow?

On February 10, 2017 10:29:33 AM EST, Allan Pichardo [email protected] wrote:

@titaniumbones Yeah, I suspect that it will. If we do this with
Node.js, then we have the option of using jQuery to parse the HTML
archives. We can determine that certain HTML nodes are insignificant,
such as <head meta> for example or <rel type="stylesheet"> etc
etc... Therefore, at the time that an archive is loaded from disk, then
the CLI uses jQuery to delete such nodes from the text blob and then
execute the diff.

The diff would look like a regular git diff, but then the visualizer
would have some logic that could convert the ++++ ----- syntax into
~~syntax and we'll output that on screen.~~

When viewing the page in the visualizer, an analyst would have the
option of selecting a DOM element and saying that it's insignificant,
thus adding it to an ongoing list that would be fed back to the CLI on
the next cycle.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
edgi-govdata-archiving/web-monitoring-ui#17 (comment)

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

dcwalk · 2017-03-09T03:48:17Z

From @allanpichardo on February 10, 2017 17:15

@ambergman Yes, here's what I see overall at a high level,

I understand that the 30,000 URLs are kept in the spreadsheet. Those are live URLs. Where are the previous versions archived? Is it held remotely on another server?

I ask because if it's possible to have a structure such that:

There is a directory which holds the last known HTML of each page and each filename, for simplicity, could be the URL itself. (plaintext)
Start node script that can open the spreadsheet and go line by line, comparing the stored HTML in the directory with the live version of the page, and update the spreadsheet accordingly for each HTML. (I say Node because JavaScript is so good for DOM traversal, but if there's a better idea for this, I'm all ears)
(this part needs to be worked out) By some heuristic, determine if a diff is significant enough to keep, and store that diff as a text file in another directory.
If the diff text file is readable from the web, that would be OK (preferable), otherwise, we would have to insert the path and some kind of ID into a database table. The visualization link can be something with the ID of the diff text.
Insert that URL in the spreadsheet
Have the visualizer recognize the IDs and pull the corresponding diff file and display it.

@titaniumbones If this architecture is something we can work with, then maybe the SF group can set up the directory structure, and put some test files in it, and start a cli utility that would read the files and use git to diff them with the live URLs and save the diffs into another directory.

dcwalk · 2017-03-09T03:48:17Z

From @titaniumbones on February 10, 2017 17:40

Allan Pichardo [email protected] writes:

@ambergman Yes, here's what I see overall at a high level,

I understand that the 30,000 URLs are kept in the spreadsheet. Those
are live URLs. Where are the previous versions archived? Is it held
remotely on another server?

We don't know where the long-term storage will be. We have talked to a
bunch of differnt sponsors about this and have not been able to get a
firm commitment from anyone yet. For now they are stored on a cluster
that's not entirely easy to access from the web. I've exposed the
zipfile for the 30,000 (actually for some reason there are 2 zipfiles,
one large and one small, fro mthe same day) & a couple of the domains here, over http:

http://edgistorage.hackinghistory.ca/

You can dl it yourself there, but the large zipfile is about 7g.

I've also unzipped the zipfile in
http://edgistorage.hackinghistory.ca/storage, and you can see the funky
directory structure there.

I think there's something about this in the docs in pagefreezer-cli,
but I'm on a low-bandwidth connection in the airport and browsing is a little
hard.

I ask because if it's possible to have a structure such that:

There is a directory which holds the last known HTML of each page and each filename, for simplicity, could be the URL itself. (plaintext)

take a look at the zipfile structure. Probably we could do that but it
might be a little frustrating.

Start node script that can open the spreadsheet and go line by line, comparing the stored HTML in the directory with the live version of the page, and update the spreadsheet accordingly for each HTML. (I say Node because JavaScript is so good for DOM traversal, but if there's a better idea for this, I'm all ears)

yeah sounds great. I think ruby & python also have dom-arware diff
programs, again see docs.

(this part needs to be worked out)
^^ yup!!

By some heuristic, determine if a diff is significant enough to keep, and store that diff as a text file in another directory.

If the diff text file is readable from the web, that would be OK (preferable), otherwise, we would have to insert the path and some kind of ID into a database table. The visualization link can be something with the ID of the diff text.

Whatever solution we come up with, we will need to make all this stuff
accessible to the web. So, let's take that as a given when we're
building a test case.

Insert that URL in the spreadsheet

Have the visualizer recognize the IDs and pull the corresponding diff file and display it.

yup, sounds great.

--

dcwalk · 2017-03-09T03:48:17Z

From @allanpichardo on February 10, 2017 20:30

@titaniumbones the directory structure from the zip files will work because the url is preserved in the file structure. So, it seems that the directories per domain mirror the exact directory structure that is remote. So this is good for knowing what compares to what. The only issue is that the archives come with a lot of other files that we don't need, but that's OK, for this purpose, we can just traverse the tree and take html files and ignore the others.

So I suppose, wherever this service runs, it could download an archive zip, extract it, and run through it creating the diffs and updating the spreadsheet. When the process is done, it can delete the downloaded archive. Then rinse and repeat.

dcwalk · 2017-03-09T03:48:18Z

From @lh00000000 on February 10, 2017 23:41

concerns

how will this address sites with dynamically loaded (e.g. react js) content?
i suspect the pagefreezer isn't really "slow". my guess is that the main causes for latency are network (nontrivial payloads sent to them), and that they might be running the pages in a headless browser and using a few seconds to allow clientside stuff to happen (in my experience, a lot of sites have low-priority / below the fold stuff won't finish loading until 3 or more seconds later) and that they're equiped to handle a bit (polite) multithreading. the issue of 10K daily limit remains though.
i've had bad experiences using jquery selectors to clean up html at scale. things like throwing out and <script> nodes is straightforward, but a lot of dynamic content (even if it's somewhat consistent serverside rendered stuff and not clientside run js) can have branching logic that is unique to that site. personally, i've considered handcrafted jquery/dom/css selectors as a brittle last resort, but i'm definitely no jquery master
re: ui that allows click-to-ignore-dom-node. this would be awesome! i'm worried it would really hard though. i've tried click-to-get-css-selector tools before (chrome extensions) but they never seem to work right. layers of nested divs without strictly correlated visual embedding seemed to be the tough part.

i've been kind of kind of playing around with the idea of an architecture that relies on s3 and aws lambdas (to avoid running-server costs). i wanted to get more details on the current situation to avoid proposing solutions to problems already solved but maybe it's better to just spit it out:

two pieces:
A. archiving / diff emitter system
B. the diff alerting/pre-filtering/exploration system

A. archiving / diff emitter system

using cron-ish timer (e.g. cloudwatch cron events, or something like airflow), an aws lambda function to fill a sqs-toPullRaw SQS queue with all the urls to be scraped. this allows the list of watched urls to be scraped and diffed everyday to be editable in one place. the order of tasks in the queue can be shifted around to avoid hitting the same sites too aggresively. using SQS queue also provides a nice interface to watch the jobs complete in realtime and track failed jobs.
sqs-toPullRaw is consumed by lambda that will hit the urls, and get the RAW html (i.e. dynamic JS not loaded). raw html is dumped to s3 bucket s3-raw-snapshots. (name of the object is reversedata+url e.g. 30-01-2017-poop.gov)
there are three lambdas that are triggered by new-object events in the s3-raw-snapshots bucket:
a lambda to diff raw to raw. given the name of the new snapshot (e.g. 30-01-2017-poop.gov), look for the snapshot of the url at the previous day (29-01-2017-poop.gov) and run it through the diffing service (page-freezer or inhouse). diffs will be written to another s3 bucket s3-raw-diffs for posterity
another lambda just like above, but runs raw html through a text extraction preprocessor (gets all the visible text WITHOUT dynamic content). diffs put into s3-plaintext-diffs
a third lambda just like the above, but runs page through headless browser to evaluate dynamically load content (javascript). both the raw post-evaluated html is diffed (and put into s3-loaded-raw-diffs), and a text-extraction preprocessed version is diffed (and put into s3-loaded-plaintext-diffs). a similar pattern would be used for other ideas of processing the raw html e.g. using an article-text extraction method such as the python package newspaper3k
for each of the s3 diff buckets above, lambdas are triggered to load each of the diff documents (i.e. one json object per diff of a single page) into the B. diff alerting/pre-filtering/exploration system
to cut down on s3 costs, an automated process could move older snapshots from s3 to s3-cold storage or glacier (see https://aws.amazon.com/blogs/aws/archive-s3-to-glacier/)

B. the diff alerting/pre-filtering/exploration system
my suggestion is that this system (which could include another webapp but also entails the generation of the daily csv people are already using) would rely on elasticsearch. reasons why:

hardcoded policies for filtering out junk could be implemented and changed easily (as filter queries), but all the diffs would still be there if needed. figuring out that filtering logic could be done interactively and the daily csv could be regenerated rapidly.
the opposite (i.e. watching for the deletion of the phrase "climate change") could be implemented using the same DSL and take advantage of common elasticsearch alert techniques (elasticsearch perculator API) and send out email alerts in high-priority situations. we could also have the email create a versionista view for the particular diff.
elasticsearch already has an ecosystem of exploratory tools (kibana / sense) that could aid in exploring and finding larger trends
similar to the s3 archiving strategy above, a retiring strategy could be uses to save on cost (see https://www.elastic.co/guide/en/elasticsearch/guide/current/retiring-data.html)

(if you're not familar with aws lambda, it's an aws service that allows you to upload code for single node or python functions, where they are invoked according to "triggers" you define (such as a new object being put in s3 or an http request to some static url). they're kind of a pain in the butt but the payoff is that you only pay $0.000000208 - $0.000002501 per call for month and aws will take care of handling scale-outs (for unexpected bursts of load))

dcwalk · 2017-03-09T03:48:18Z

From @leepro on February 13, 2017 18:23

FYI. I am from Pagefreezer. To make clear on the slowness of our Diff API, I would like to give some idea on it. As @allanpichardo mentioned, one API call takes around avg 5 sec. Actually, it is due to the network latency of AWS Lambda / API Gateway. For the purpose of this project, we took the diff service from our production and made a AWS Lambda version.

Our internal benchmark is as follows:

Native/two local files: 0.3 sec.
API/two URLs: 5 sec.
API/two files uploaded (url1, url2): 4.6 sec

So, to use the diff API with a large number of files, I recommend to use multi-threaded(or event-driven) client codes to make the API calls. AWS Lambda will serve them at scale. To compare 30k files with 100 threads, it simply needs to take 41 minutes. The multithreaded client doesn't do anything except of making a connection to API and waiting its result.

As a note, whatever you use a tool/algorithm to make diff, the time taking to make diff will be somehow in proportion to the source's HTML structure (#DOM nodes and text size).

EOF

dcwalk · 2017-03-09T03:48:18Z

From @mriedijk on February 14, 2017 0:50

In addition: consider bulk-uploading the HTML pages to AWS first before you use the PageFreezer Diff service, it may decrease the network latency.

dcwalk · 2017-03-09T03:48:19Z

From @mekarpeles on February 17, 2017 23:34

Have you folks talked to @markjohngraham about how some of these html diff problems are (or could be addressed) in the Wayback Machine -- web.archive.org?

dcwalk · 2017-03-09T03:48:19Z

From @mekarpeles on February 17, 2017 23:37

Also, in terms of long-term storage, has Internet Archive's S3 API been considered? https://github.com/vmbrasseur/IAS3API

dcwalk · 2017-03-09T03:48:19Z

From @titaniumbones on February 18, 2017 1:58

@mekarpeles have not talked to markjohngraham, but we've been talking to Jefferson a little about IA as the end-game for this effort. Haven't got into the nitty-gritty, but clearly IA seems like the best home for this effort in the long run!

sorry to have missed you in SF! was looking forward to meeting you but missed the connection somehow.

dcwalk · 2017-03-09T03:48:20Z

From @mekarpeles on February 18, 2017 2:5

As long as everything gets backed up and the community is able to find a way to produce a registry so other institutions can cross-check and participate, I'm a super happy camper! Very thankful for your + team's efforts. Sorry to have missed you in SF as well! @flyingzumwalt had great things to say about you :)

dcwalk · 2017-03-12T14:34:42Z

This is such a great conversation of decisions in the project!

I'm having a hard time knowing what the next step from here is with our current architecture/implementation. I'm seeing a few TODOs which I have moved into different issues:

Investigate IA storage (referenced in Need efficient diff-based storage for archives #8)
Investigate IA's html diff implementation (referenced in Research similar projects #18)

But what else from this is left as an 'issue' to address? @titaniumbones or @danielballan your guidance here would be appreciated

danielballan · 2017-03-12T16:35:14Z

I think this can be closed.

dcwalk · 2017-03-13T14:56:06Z

Closing per @danielballan.

This was referenced Mar 9, 2017

Build PageFreezer-Outputter that fits into current Versionista workflow edgi-govdata-archiving/web-monitoring-ui#17

Closed

Need efficient diff-based storage for archives #8

Closed

Research similar projects #18

Open

dcwalk closed this as completed Mar 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build PageFreezer-Outputter that fits into current Versionista workflow #9

Build PageFreezer-Outputter that fits into current Versionista workflow #9

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 12, 2017

danielballan commented Mar 12, 2017

dcwalk commented Mar 13, 2017

Build PageFreezer-Outputter that fits into current Versionista workflow #9

Build PageFreezer-Outputter that fits into current Versionista workflow #9

Comments

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 9, 2017

dcwalk commented Mar 12, 2017

danielballan commented Mar 12, 2017

dcwalk commented Mar 13, 2017