Reads flat/classic files with (mostly) non-bibliographic data and sends nonbib and metrics protobufs to master pipeline.
There are ~30 input files. Each row in every file begins with a bibcode. It is followed by a tab character and then data from that bibcode.
The nonbib protobuf holds a variety of information for a specific bibcode. It is computed by reading all the data for a bibcode from every file and then massaging the data a bit. Computing the metrics protobuf is not as straightforward. Computing the metrics values requires the citation network, the reference network and the list of refereed bibcodes. All three of these are cached in memory in simple python containers.
Master pipeline uses a postgres database to store results from the various pipelines (nonbib, bib, fulltext, etc.). Pipelines avoid sending data to master for bibcodes that have not changed. Nonbib computes which bibcodes have changed data by comparing yesterday's data files to today's data files. It computes a list of changed canonical bibcodes with a series of unix shell commands. The command to compute changed bibcodes is
python3 run.py COMPUTE_DIFF
Changes to Classic bibcodes are stored in the file at logs/input/current/changedBibcodes.txt. Changes to CitationCapture bibcodes are stored in logs/input/current/changedBibcodes_CC.txt.
This pipeline can send nonbib and metrics protobufs for a given list of bibcodes. To update master with the changed bibcodes from a file use the command
#Process only Classic Records
python3 run.py PROCESS_FILE logs/input/current/changedBibcodes.txt
#Process Classic and CitationCapture Records
python3 run.py PROCESS_FILE logs/input/current/changedBibcodes.txt --include-CitationCapture-file logs/input/current/changedBibcodes_CC.txt
#Process only CitationCapture Records
python3 run.py PROCESS_FILE logs/input/current/changedBibcodes_CC.txt --only-CitationCapture
To initialize master, simply process all of the canonical bibcodes:
python3 run.py logs/input/current/bibcodes.list.can --include-CitationCapture logs/input/current/bibcodes_CC.list.can
ADSDataPipeline can also process a space separated list of bibcodes using the command
#Only Classic Records
python3 PROCESS_BIBCODES --bibcodes BIBCODE1 BIBCODE2 ... BIBCODEN
#Only generated nonbib protobufs for Classic Records
python3 PROCESS_BIBCODES --bibcodes BIBCODE1 BIBCODE2 ... BIBCODEN --no-metrics
#Only CitationCapture Records
python3 PROCESS_BIBCODES --bibcodes BIBCODE1 BIBCODE2 ... BIBCODEN --only-CitationCapture
-
Because of the way bibcodes are handled, the pipeline cannot handle mixed groups of Classic and CitationCapture bibcodes specified with the
--bibcodesflag. -
For CitationCapture records, ADSDataPipeline only calculates the metrics protobuf so the
--no-metricsflag cannot be called with any process that handles CitationCapture records.
Computing metrics requires the entire reference and citation networks as well a list of all the refereed bibcodes. The reference and citation networks are cached in memory into python defaultdicts where the key is the bibcode and the default value is []. Refereed status is cached in a python set of bibcodes. The cache code reads these files directly, it does not use the reader.py code.
The COMPUTE_DIFFS command produces merged network files that contain both CitationCapture and Classic records. These can be identified by the .merged extension.
Conceptually, the code reads one line from each of the ~30 input files. This yields all the nonbib data for the next bibcode. It is transformed into a nonbib probobuf and sent to master.
Some implementation details complicate this picture. Every bibcode does not appear in every file so one can't just read the next line from every file and naively merge. In some files the data for a bibcode is spread acroos multiple lines (requiring merging of data). A few values values are computed (e.g., property) from the data read in. Some files contain arrays of data (reads, downloads) for each bibcode, some hold a couple values, some only one. Some detail data is not sent to master (e.g., author list read in but only number of authors is used).
If we have initialized the cache, as we read the nonbib data one bibcode at a time we can also compute the metrics record. Essensically the same core metrics code from the old pipeline is used the compute a metrics record. It is converted to a metrics protobuf and sent to master.
About 30 files need to be consumed. As noted above, the different data files have different customs and quirks. This complicates reading all the files. The code to deal with the complications in a single reader class, NonbibFileReader in adsdata/reader.py. The goal of the reader is generate "ready to use" data that needs little further massaging. It supports a small "description language" where the idiosyncrasies and customs of each file are explicitly declared.
What idiosycrasies and properties are supported?
As noted above, the various files encode data of different type
(boolean, array, hashtables, etc.). So for each file we encode the
default value to use for bibcodes that aren't in the file. We use the
type of the default value to convert the string read from the file.
Some files have a single line for each bibcode and that contains an
list of similar values (e.g., the number of reads per year). We read
lists of similar data into an array. In other files, the multiple
values in a single line contains very different values (e.g.,
relevance holds 4 values: boost, citation_count, read_count and
norm_cites). Data like this is read into a python dict where the
keys are defined in the "description language".
Some data links values are always stored in
arrays of dicts even when there is a single value (for example, url and
title). To indicate this, the keys in the file description is enclosed in
a list.
There are cases where we want to associate some data with all bibcodes in a file. For example, all the data links info read from the file spires/all.links must have a link_type of 'INSPIRE' and a link_sub_type of NA. To accomidate these constants each file property dict can include 'extra_values' that are merged with the data other for a bibcode. This 'extra_values' field also holds additional elements for the 'property' list.
The nonbib protobuf sent to master contains only a subset and a summary of the data read in. There is code in process.py that converts the full nonbib dict to a nonbib protobuf.
A single thread can send nonbib and metrics protobufs to the master pipeline faster than workers on master can process them.