Added JORS scraper and tests

.coveralls.yml

@@ -0,0 +1,2 @@
+repo_token: vHHvsw3QKvjK7zPb9DcFgt6ivLVS8r5uP
+service_name: travis

.gitignore

@@ -0,0 +1,2 @@
+results.json
+coverage.json

.travis.yml

@@ -0,0 +1,14 @@
+language: ruby
+rvm:
+  - 2.1.0
+before_install:
+  - sudo apt-get update -qq
+  - sudo apt-get install -y software-properties-common
+  - sudo apt-get install -y build-essential libfontconfig
+  - nvm install 0.10.32
+  - nvm use 0.10.32
+  - npm update npm -g
+install:
+  - npm install --global [email protected]
+script:
+  - ruby test/test_all.rb

README.md

@@ -1,37 +1,64 @@
 journal-scrapers
 ================
 
-Journal scraper definitions for the ContentMine framework
+[travis]: http://travis-ci.org/ContentMine/journal-scrapers
+[license]: https://creativecommons.org/publicdomain/zero/1.0/
+[coverage]: https://coveralls.io/r/ContentMine/journal-scrapers
 
-### Definition
+[![Build Status](http://img.shields.io/travis/ContentMine/journal-scrapers.svg)][travis]
+[![Coverage](http://img.shields.io/coveralls/ContentMine/journal-scrapers.svg)][coverage]
+[![License](http://img.shields.io/badge/license-CC0-blue.svg)][license]
 
-Scrapers are defined in JSON, using a schema that is currently evolving:
+Journal scraper definitions for the ContentMine framework.
 
-There can be two keys in the root object:
+### Table of Contents
 
-- ***url*** - a string-form regular expression specifying which URL(s) this scraper targets
-- ***elements*** - a dictionary of elements to scrape
+- [Summary](#summary)
+- [Scraper collection status](#scraper-collection-status)
+- [ScraperJSON definitions](#scraperjson-definitions)
+- [Contributing scrapers](#contributing-scrapers)
+- [Usage](#usage)
+- [License](#license)
 
-Elements are defined as key-value pairs, where the key is a description of the element, and the value is a dictionary of specifiers defining the element and its processing. Allowed keys in the specifier dictionary are:
+### Summary
 
-- ***selector*** - an XPath or CSS selector targetting the element to be selected
-- ***attribute*** - a string specifying the attribute to extract from the selected element
-- ***download*** - a boolean flag: true if the element is a URL to a resource that must be downloaded
+This repo is a collection of scraperJSON definitions targeting academic journals. They can be used to extract and download data from URLs of journal articles, such as:
 
-Example:
-```json
-{
-  "url": "plosgenetics.org",
-  "elements": {
-    "fulltext_pdf": {
-      "selector": "//meta[@name='citation_pdf_url']",
-      "attribute": "content",
-      "download": true
-    }
-  }
-}
-```
+- Title, author list, date
+- Figures and their captions
+- Fulltext PDF, HTML, XML, RDF
+- Supplementary materials
+- Reference lists
+
+### Scraper collection status
+
+All the scrapers in the collection are automatically tested daily as well as every time any scraper is changed. The tests work by having the expected results for a set of URLs stored, and randomly selecting one of those URLs to re-scrape. If the results match those expected the test passes. If the badge is green and says `build|passing`, all the scrapers are OK. If the badge is red and says `build|failing`, one or more of the scrapers has stopped working. You can click on the badge to see the test report, to see which scrapers are failing and how.
+
+[![Build Status](http://img.shields.io/travis/ContentMine/journal-scrapers.svg)][travis]
+
+How well the scrapers are covered by the tests is also checked. Coverage should be 100% - this means every element of every scraper is checked at least once in the testing. If coverage is below 100%, you can see exactly which parts of which scrapers are not covered by clicking the `coverage` badge below.
+
+[![Coverage](http://img.shields.io/coveralls/ContentMine/journal-scrapers.svg)][coverage]
+
+### ScraperJSON definitions
+
+Scrapers are defined in JSON, using a schema called scraperJSON which is currently evolving. The current schema is described at [the scraperJSON repo](https://github.com/ContentMine/scraperJSON).
+
+### Contributing scrapers
+
+If your favourite publisher or journal is not covered by a scraper in our collection, we'd love you to submit a new scraper.
+
+We ask that all contributions follow some simple rules that help us maintain a high-quality collection.
+
+1. The scraper covers all [the data elements used in the ContentMine](https://github.com/ContentMine/journal-scrapers/wiki/data_collected_for_ContentMine).
+2. You must submit a set of 5-10 test URLs.
+3. It comes with a regression test ([which can be auto-generated](https://github.com/ContentMine/journal-scrapers/wiki/Generating%20tests%20for%20your%20scrapers)).
+4. You agree to release the scraper definition and tests under the [Creative Commons Zero license](https://creativecommons.org/publicdomain/zero/1.0/).
 
 ### Usage
 
-Currently these definitions can be used with the [quickscrape](http://github.com/ContentMine/quickscrape) tool.
+Currently these definitions can be used with the [quickscrape](http://github.com/ContentMine/quickscrape) tool.
+
+### License
+
+All scrapers are released under the [Creative Commons 0 (CC0)](https://creativecommons.org/publicdomain/zero/1.0/) license.

scrapers/acs.json

@@ -0,0 +1,79 @@
+{
+  "url": "pubs\\.acs\\.org",
+  "headless": true,
+  "elements": {
+    "publisher": {
+      "selector": "//meta[@name='dc.Publisher']",
+      "attribute": "content"
+    },
+    "title": {
+      "selector": "//meta[@name='dc.Title']",
+      "attribute": "content"
+    },
+    "authors": {
+      "selector": "//meta[@name='dc.Creator']",
+      "attribute": "content"
+    },
+    "date": {
+      "selector": "//div[@id='pubDate']",
+      "attribute": "text"
+    },
+    "volume": {
+      "selector": "//span[@class='citation_volume']",
+      "attribute": "text"
+    },
+    "doi": {
+      "selector": "//meta[@scheme='doi']",
+      "attribute": "content"
+    },
+    "description": {
+      "selector": "//meta[@name='dc.Description']",
+      "attribute": "content"
+    },
+    "journal": {
+      "selector": "//cite",
+      "attribute": "text"
+    },
+    "abstract": {
+      "selector": "//p[@class='articleBody_abstractText']",
+      "attribute": "text"
+    },
+    "abstract2": {
+      "selector": "//a[@title='View the Abstract']",
+      "attribute": "html",
+      "download": true
+    },
+    "language": {
+      "selector": "//meta[@name='dc.Language']",
+      "attribute": "content"
+    },
+    "fulltext_html": {
+      "selector": "//a[@title='View the Full Text HTML']",
+      "attribute": "href",
+      "download": {
+        "rename": "fulltext.html"
+      }
+    },
+    "fulltext_pdf": {
+      "selector": "//a[@title='Download the PDF Full Text']",
+      "attribute": "href",
+      "download": {
+        "rename": "fulltext.pdf"
+      }
+    },
+    "supplementary_material": {
+      "selector": "//a[title='View Supporting Information']",
+      "attribute": "href",
+      "download": true
+    },
+    "figure": {
+      "selector": "//img[@alt='Abstract Image']",
+      "attribute": "src",
+      "download": true
+    },
+    "copyright": {
+      "selector": "//div[contains(@id, 'artCopyright')]",
+      "attribute": "text"
+    }
+  }
+}

scrapers/bmc.json

@@ -0,0 +1,90 @@
+{
+"url": "www\\.biomedcentral\\.com",
+"elements": {
+"publisher": {
+"selector": "//meta[@name='citation_publisher']",
+"attribute": "content"
+},
+"journal": {
+"selector": "//meta[@name='citation_journal_title']",
+"attribute": "content"
+},
+"title": {
+"selector": "//meta[@name='citation_title']",
+"attribute": "content"
+},
+"authors": {
+"selector": "//meta[@name='citation_author']",
+"attribute": "content"
+},
+"date": {
+"selector": "//meta[@name='citation_date']",
+"attribute": "content"
+},
+"doi": {
+"selector": "//meta[@name='citation_doi']",
+"attribute": "content"
+},
+"volume": {
+"selector": "//meta[@name='citation_volume']",
+"attribute": "content"
+},
+"issue": {
+"selector": "//meta[@name='citation_issue']",
+"attribute": "content"
+},
+"firstpage": {
+"selector": "//meta[@name='citation_firstpage']",
+"attribute": "content"
+},
+"description": {
+"selector": "//meta[@name='description']",
+"attribute": "content"
+},
+"abstract": {
+"selector": "//meta[@name='description']",
+"attribute": "content"
+},
+"fulltext_html": {
+"selector": "//meta[@name='citation_fulltext_html_url']",
+"attribute": "content",
+"download": {
+"rename": "fulltext.html"
+}
+},
+"fulltext_pdf": {
+"selector": "//meta[@name='citation_pdf_url']",
+"attribute": "content",
+"download": {
+"rename": "fulltext.pdf"
+}
+},
+"fulltext_xml": {
+"selector": "//a[.='Download XML']",
+"attribute": "href",
+"download": {
+"rename": "fulltext.xml"
+}
+},
+"supplementary_material": {
+"selector": "//link[starts-with(@title,'Additional file')]",
+"attribute": "href",
+"download": true
+},
+"figure": {
+"selector": "//div[@class='fig']/p/a/img",
+"attribute": "src",
+"download": true
+},
+"figure_caption": {
+"selector": "//div[@class='fig']//strong"
+},
+"license": {
+"selector": "//p[a/@href='http://creativecommons.org/licenses/by/4.0']"
+},
+"copyright": {
+"selector": "//p[contains(.,'licensee')]"
+}
+}
+}
+

scrapers/elife.json

@@ -0,0 +1,92 @@
+{
+  "url": "elifesciences\\.org",
+  "elements": {
+    "publisher": {
+      "selector": "//meta[@name='citation_publisher']",
+      "attribute": "content"
+    },
+    "journal": {
+      "selector": "//meta[@name='citation_journal_title']",
+      "attribute": "content"
+    },
+    "title": {
+      "selector": "//meta[@name='citation_title']",
+      "attribute": "content"
+    },
+    "authors": {
+      "selector": "//meta[@name='citation_author']",
+      "attribute": "content"
+    },
+    "date": {
+      "selector": "//meta[@name='citation_date']",
+      "attribute": "content"
+    },
+    "doi": {
+      "selector": "//meta[@name='citation_doi']",
+      "attribute": "content"
+    },
+    "volume": {
+      "selector": "//meta[@name='citation_volume']",
+      "attribute": "content"
+    },
+    "issue": {
+      "selector": "//meta[@name='citation_issue']",
+      "attribute": "content"
+    },
+    "firstpage": {
+      "selector": "//meta[@name='citation_firstpage']",
+      "attribute": "content"
+    },
+    "description": {
+      "selector": "//meta[@name='description']",
+      "attribute": "content"
+    },
+    "abstract": {
+      "selector": "//div[contains(class, 'abstract')]//p[1]",
+      "attribute": "content"
+    },
+    "fulltext_html": {
+      "selector": "//meta[@name='citation_fulltext_html_url']",
+      "attribute": "content",
+      "download": {
+        "rename": "fulltext.html"
+      }
+    },
+    "fulltext_pdf": {
+      "selector": "//meta[@name='citation_pdf_url']",
+      "attribute": "content",
+      "download": {
+        "rename": "fulltext.pdf"
+      }
+    },
+    "fulltext_xml": {
+      "selector": "//meta[@name='citation_xml_url']",
+      "attribute": "content",
+      "download": {
+        "rename": "fulltext.xml"
+      }
+    },
+    "supplementary_material": {
+      "selector": "//a[contains(concat(' ', normalize-space(@class), ' '), ' article-supporting-download ')]",
+      "attribute": "href",
+      "download": true
+    },
+    "figure": {
+      "selector": "//div[contains(concat(' ', normalize-space(@class), ' '), ' elife-figure-link-download ')]/a",
+      "attribute": "href",
+      "download": true
+    },
+    "figure_caption": {
+      "selector": "//div[contains(class, 'fig-caption')]",
+      "attribute": "text"
+    },
+    "license": {
+      "selector": "//meta[@name='DC.Rights']",
+      "attribute": "text"
+    },
+    "copyright": {
+      "selector": "//meta[@name='DC.Rights']",
+      "attribute": "text"
+    }
+  }
+}