Merge branch 'effRelease'

.build_exclusions

@@ -16,3 +16,4 @@ webextension/chrome-resources/update-from-chrome-svn.sh
 .eslintrc.json
 webextension/.eslintrc.json
 .eslintignore
+node_modules

.editorconfig

@@ -6,6 +6,7 @@ root = true
 end_of_line = lf
 insert_final_newline = true
 charset = utf-8
+trim_trailing_whitespace = true
 
 [src/chrome/content/rules/*.xml]
 indent_style = tab

.gitignore

@@ -5,6 +5,8 @@
 .idea
 .vagrant
 .venv/
+.venv2/
+.venv3/
 chromium.pem
 from-preloads/
 pkg/

.travis.yml

@@ -2,37 +2,38 @@ sudo: required
 os: linux
 dist: trusty
 language: python
-python: 2.7
+python: 3.6
+group: deprecated-2017Q4
 services:
   - docker
 matrix:
   fast_finish: true
   include:
-    - env: INFO="lint"
+    - env: TEST="lint"
       language: node_js
       node_js:
         - "node"
-    - env: INFO="unittests"
+    - env: TEST="unittests"
       language: node_js
       node_js: node
         - "node"
-    - env: INFO="rules"
-    - env: INFO="fetch"
-    - env: INFO="preloaded"
+    - env: TEST="validations"
+    - env: TEST="fetch"
+    - env: TEST="preloaded"
     - addons:
         chrome: beta
-      env: INFO="chrome beta" BROWSER=google-chrome-beta
+      env: TEST="chrome beta" BROWSER=google-chrome-beta
     - addons:
         chrome: stable
-      env: INFO="chrome stable" BROWSER=google-chrome-stable
+      env: TEST="chrome stable" BROWSER=google-chrome-stable
     - addons:
         firefox: latest
-      env: INFO="firefox" BROWSER=firefox
+      env: TEST="firefox" BROWSER=firefox
     - addons:
         firefox: latest-esr
-      env: INFO="firefox esr" BROWSER=firefox
+      env: TEST="firefox esr" BROWSER=firefox
 before_install:
   - export DISPLAY=:99.0
   - sh -e /etc/init.d/xvfb start
-before_script: travis_retry ./setup_travis.sh
-script: . ./run_travis.sh
+before_script: travis_retry test/setup_travis.sh
+script: . test/run_travis.sh

CONTRIBUTING.md

@@ -30,6 +30,7 @@
         * [Regular Rules](#regular-rules)
         * [HSTS Preloaded Rules](#hsts-preloaded-rules)
 * [Contributing Code](#contributing-code)
+* [Contributing Documentation](#contributing-documentation)
 * [Contributing Translations](#contributing-translations)
 
 * * *
@@ -55,7 +56,7 @@ There are several main areas of development on HTTPS Everywhere: the rulesets, t
 
 The rulesets can be found in the [`rules`](rules) top-level path and include all the rules for redirecting individual sites to HTTPS.  These are written in XML. If you want to get started contributing to HTTPS Everywhere, we recommend starting here.
 
-The core codebase consists of the code that performs the redirects, the UI, logging code, and ruleset loading.  This encompasses all code delivered with the extension itself that is *not* a ruleset.  It is written in JavaScript, using the `WebExtensions` API (located in [`chromium`](chromium)) on all supported browsers.  In Firefox, this extension is wrapped in a thin `XPCOM` layer for the purposes of migrating settings, located in [`src`](src), which will soon be deprecated.
+The core codebase consists of the code that performs the redirects, the UI, logging code, and ruleset loading.  This encompasses all code delivered with the extension itself that is *not* a ruleset.  It is written in JavaScript, using the `WebExtensions` API (located in [`chromium`](chromium)).
 
 The utilities ([`utils`](utils) top-level path) include scripts that build the extension, sanitize and perform normalization on rulesets, simplify rules, and help label GitHub issues.  Historically, these utilities have been written in Python.  Many of the newer utilities are written in JavaScript, and are meant to be run in node.  Some of the wrappers for these utilities are in shell scripts.
 
@@ -103,7 +104,7 @@ Some `rulesets` have the attribute `platform="mixedcontent"`.  These `rulesets`
 
 ## New Rulesets
 
-If you want to create new `rulesets` to submit to us, we expect them to be in the `src/chrome/content/rules` directory. That directory also contains a useful script, `make-trivial-rule`, to create a simple `ruleset` for a specified domain. There is also a script called `utils/trivial-validate.py`, to check all the pending `rulesets` for several common errors and oversights. For example, if you wanted to make a `ruleset` for the `example.com` domain, you could run:
+If you want to create new `rulesets` to submit to us, we expect them to be in the `src/chrome/content/rules` directory. That directory also contains a useful script, `make-trivial-rule`, to create a simple `ruleset` for a specified domain. There is also a script in `test/validations/special/run.py`, to check all the pending `rulesets` for several common errors and oversights. For example, if you wanted to make a `ruleset` for the `example.com` domain, you could run:
 ```
 cd src/chrome/content/rules
 bash ./make-trivial-rule example.com
@@ -148,7 +149,7 @@ Avoid using the left-wildcard (`<target host="*.example.com" />`) unless you int
 
 Instead, prefer listing explicit target hosts and a single rewrite from `"^http:"` to `"^https:"`. This saves you time as a ruleset author because each explicit target host automatically creates an implicit test URL, reducing the need to add your own test URLs. These also make it easier for someone reading the ruleset to figure out which subdomains are covered.
 
-If you know all subdomains of a given domain support HTTPS, go ahead and use a left-wildcard, along with a plain rewrite from `"^http:"` to `"^https:"`. Make sure to add a bunch of test URLs for the more important subdomains. 
+If you know all subdomains of a given domain support HTTPS, go ahead and use a left-wildcard, along with a plain rewrite from `"^http:"` to `"^https:"`. Make sure to add a bunch of test URLs for the more important subdomains.
 
 #### Edge-Case: Right-Wildcards
 
@@ -334,7 +335,7 @@ In `utils` we have a tool called `hsts-prune` which removes `targets` from rules
 Every new pull request automatically has the `hsts-prune` utility applied to it as part of the continual integration process.  If a new PR introduces a `target` which is preloaded, it will fail the CI test suite.  See:
 
 - `.travis.yml`
-- `test/travis.sh`
+- `test/run_travis.sh`
 
 * * *
 
@@ -356,6 +357,20 @@ Several of our utilities and our full test suite is written in Python.  Eventual
 
 * * *
 
+# Contributing Documentation
+
+Standalone documentation should be written in [Markdown](https://en.wikipedia.org/wiki/Markdown) that follows the [Google style guide](https://github.com/google/styleguide/blob/gh-pages/docguide/style.md). If you are updating existing documentation that does not follow the Google style guide, then you should follow the style of the file you are updating.
+
+* * *
+
+# Pull Requests from Deleted Accounts
+
+Sometimes a contributor will [delete their GitHub account](https://help.github.com/articles/deleting-your-user-account/) after submitting a pull request, resulting in the pull request being associated with the [Ghost user (@ghost)](https://github.com/ghost).  These @ghost pull requests can cause problems for HTTPS Everywhere maintainers, leaving questions unanswered and closing off the possibility of receiving maintainer feedback to solicit clarification or request changes.
+
+We ask that if you want to delete your GitHub account, you either close your HTTPS Everywhere pull requests before you delete your account, or wait to delete your account until we merge your pull requests. Otherwise, maintainers are free to close @ghost pull requests without any comment.
+
+* * *
+
 # Contributing Translations
 
 HTTPS Everywhere translations are handled through Transifex.  The easiest way to help with translations is to [create a Transifex account](https://www.transifex.com/signup/) if you don't already have one.  Then log into your account and click "Explore", then search for "Tor Project", and click on The Tor Project.  Then choose the language you plan to translate into, click on the name of that language, and then click "Join team" and "Go" to accept joining the translation team for your language.

Dockerfile

@@ -1,4 +1,4 @@
-FROM ubuntu:16.04
+FROM ubuntu:17.10
 
 # general build deps
 RUN apt-get update && apt-get install -y \
@@ -39,7 +39,8 @@ RUN apt-get install -y \
   xvfb \
   libssl-dev \
   git \
-  chromium-chromedriver
+  chromium-chromedriver \
+  rsync
 
 RUN pip install \
   lxml>=3.3.3 \

Jenkinsfile

@@ -1,4 +1,4 @@
-node {
+node("master") {
 
   stage 'checkout'
 

LICENSE.txt

@@ -8,6 +8,5 @@ To the extent copyright applies to the rulesets, they can be used according to G
 Issue Format Bot (utils/issue-format-bot/*):
 Copyright © 2017 AJ Jordan, AGPLv3+
 
-The build system incorporates code from Python 2.6,
-Copyright © 2001-2006 Python Software Foundation
-Python Software Foundation License Version 2
+The build system incorporates code from Python 3.6
+Copyright © 2001-2018 Python Software Foundation; All Rights Reserved

README.md

@@ -9,7 +9,7 @@ Get the packages you need and install a git hook to run tests before push:
 
     bash install-dev-dependencies.sh
 
-Run all the tests:
+Run the ruleset validations and browser tests:
 
     bash test.sh
 
@@ -50,20 +50,18 @@ This is the source tree for HTTPS Everywhere for Firefox and Chrome.
 
 Important directories you might want to know about
 
-    src/                      The Firefox source
 
-    chromium/                 The Chromium/Chrome source
-                              (not to be confused with Firefox browser "chrome" or UI)
+    chromium/                 WebExtension source code (for Firefox & Chromium/chrome)
+    chromium/external         External dependencies
+    chromium/test             Unit tests
 
-    src/components            |
-    src/chrome/content        | Firefox JavaScript and XUL code
-    src/chrome/content/code   |
+    rules/                    Symbolic link to src/chrome/content/rules
 
-    src/chrome/content/rules  The rulesets live here
+    src/chrome/content/rules  Ruleset files live here
 
-    test/                     The tests live here
+    test/                     Travis unit test source code live here
 
-    utils/                    Various utilities
+    utils/                    Various utilities (includes some Travis test source)
 
 Hacking on the Source Code
 --------------------------