feed.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>Recollection</title><link href="https://recollection.saaj.me/" rel="alternate"></link><link href="https://recollection.saaj.me/feed.xml" rel="self"></link><id>https://recollection.saaj.me/</id><updated>2018-04-15T00:00:00+02:00</updated><entry><title>Developer-friendly job search. The tooling</title><link href="https://recollection.saaj.me/article/developer-friendly-job-search-the-tooling.html" rel="alternate"></link><published>2018-04-15T00:00:00+02:00</published><updated>2018-04-15T00:00:00+02:00</updated><author><name>saaj</name></author><id>tag:recollection.saaj.me,2018-04-15:/article/developer-friendly-job-search-the-tooling.html</id><summary type="html">&lt;p&gt;It is quite interesting to me to see how software job market works in the Netherlands
from an active candidate&amp;#8217;s point of view. Unfortunately one great journey has almost
ended and there is a firm incentive to find something at least as good. Having my
current job through StackOverflow Jobs, I intended to keep using what proved to work,
like marking the profile active, entering &lt;tt class="docutils literal"&gt;python &lt;span class="pre"&gt;-django&lt;/span&gt; &lt;span class="pre"&gt;-sysadmin&lt;/span&gt;&lt;/tt&gt; into the search
field, adding relevant filters and subscribing to the search. That, as such, of course
works, but it looks it is not the most popular software talent recruitment service
that Dutch companies use. And many companies are still quite happy to pay 20% of the
gross year salary per new hire to the volume-oriented keyword-matching salesmen,
colloquially referred as&amp;nbsp;recruiters.&lt;/p&gt;
</summary><content type="html">&lt;p&gt;It is quite interesting to me to see how software job market works in the Netherlands
from an active candidate&amp;#8217;s point of view. Unfortunately one great journey has almost
ended and there is a firm incentive to find something at least as good. Having my
current job through StackOverflow Jobs, I intended to keep using what proved to work,
like marking the profile active, entering &lt;tt class="docutils literal"&gt;python &lt;span class="pre"&gt;-django&lt;/span&gt; &lt;span class="pre"&gt;-sysadmin&lt;/span&gt;&lt;/tt&gt; into the search
field, adding relevant filters and subscribing to the search. That, as such, of course
works, but it looks it is not the most popular software talent recruitment service
that Dutch companies use. And many companies are still quite happy to pay 20% of the
gross year salary per new hire to the volume-oriented keyword-matching salesmen,
colloquially referred as&amp;nbsp;recruiters.&lt;/p&gt;

&lt;p&gt;On the other side of price spectrum there are free (as a beer&amp;#8230; on a degustation)
services like Indeed and Glassdoor &lt;a class="footnote-reference" href="#id23" id="id1"&gt;[1]&lt;/a&gt; where employers can place their vacancies.
The two seem to me most relevant and complete, but because the latter also provides
valuable insight into company state, salary range and interview process I will
focus on&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;StackOverflow Jobs is in between. It&amp;#8217;s definitely not free for employers. For our
small company it costed around €2k for half-year access to active candidate database
and one job posting. Most probably it&amp;#8217;ll require a company to have a full-time
&lt;span class="caps"&gt;HR&lt;/span&gt; employee to manage the interactions with candidates. But hiring 4-5 developers a
year already justifies the investment, so Stack Exchange has growth potential&amp;nbsp;here.&lt;/p&gt;
&lt;p&gt;Glassdoor has poor search features, term-inclusion-only search and location, which
makes it a chore to examine the proposals&amp;nbsp;manually.&lt;/p&gt;
&lt;div class="contents topic" id="contents"&gt;
&lt;p class="topic-title first"&gt;Contents&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference internal" href="#problem" id="id44"&gt;Problem&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#storage" id="id45"&gt;Storage&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a class="reference internal" href="#fts3-and-fts4" id="id46"&gt;&lt;span class="caps"&gt;FTS3&lt;/span&gt; and &lt;span class="caps"&gt;FTS4&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#json1" id="id47"&gt;&lt;span class="caps"&gt;JSON1&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#implementation" id="id48"&gt;Implementation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#retrieval" id="id49"&gt;Retrieval&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a class="reference internal" href="#pyppeteer" id="id50"&gt;Pyppeteer&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#iteration" id="id51"&gt;Iteration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#parsing" id="id52"&gt;Parsing&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#glue" id="id53"&gt;Glue&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#listing-relevant-jobs" id="id54"&gt;Listing relevant&amp;nbsp;jobs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#wrap-up" id="id55"&gt;Wrap-up&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div class="section" id="problem"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id44"&gt;Problem&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;StackOverflow Jobs search features include &lt;a class="footnote-reference" href="#id24" id="id2"&gt;[2]&lt;/a&gt;:&lt;/p&gt;
&lt;ol class="arabic simple"&gt;
&lt;li&gt;tag inclusion and&amp;nbsp;exclusion&lt;/li&gt;
&lt;li&gt;full-text search on body, title and&amp;nbsp;company&lt;/li&gt;
&lt;li&gt;enumeration filters by contract type, industry and&amp;nbsp;seniority&lt;/li&gt;
&lt;li&gt;salary&amp;nbsp;range&lt;/li&gt;
&lt;li&gt;flag filters by remote option, visa and relocation&amp;nbsp;sponsorship&lt;/li&gt;
&lt;li&gt;posted time&amp;nbsp;filtering&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;My job search criteria were the&amp;nbsp;following:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Python (backend software development) job preferably in&amp;nbsp;Amsterdam&lt;/li&gt;
&lt;li&gt;The company develops own product, i.e. it&amp;#8217;s not a&amp;nbsp;consultancy&lt;/li&gt;
&lt;li&gt;The job is available directly, i.e. it&amp;#8217;s not an agency or any
other kind of&amp;nbsp;intermediary&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;At first glance it looks simple. What subset of the above list of feature
is sufficient to cover&amp;nbsp;these?&lt;/p&gt;
&lt;ol class="arabic simple"&gt;
&lt;li&gt;Location data should be good enough for equality comparison, but generally
the country is small, very connected and geospatial search is not&amp;nbsp;needed.&lt;/li&gt;
&lt;li&gt;&lt;em&gt;A Python job&lt;/em&gt; is a match against &lt;tt class="docutils literal"&gt;[python]&lt;/tt&gt; minus some other tags (though only
if carefully tagged), but Glassdoor doesn&amp;#8217;t have tags and implementing &lt;span class="caps"&gt;NLP&lt;/span&gt; to
recognise them was not feasible. Doing just &lt;tt class="docutils literal"&gt;Python&lt;/tt&gt; will not only match all
sorts of jobs with it in nice-to-have, but also match jobs that extensively use
Python but are not about software development as such, e.g. devops, (data)
science, test automation, screen scraping and probably others. Also don&amp;#8217;t forget
full-stack and junior/intern positions (no dedicated seniority field on
Glassdoor&amp;nbsp;either).&lt;/li&gt;
&lt;li&gt;Solving consultancies and intermediaries should be possible with a blacklist,
but needs full-text index on company name to mitigate dealing with optionally
present suffixes like &amp;#8220;B.V.&amp;#8221; and &amp;#8220;&lt;span class="caps"&gt;BV&lt;/span&gt;&amp;#8221;, geographical suffixes and the like.
Using &lt;span class="caps"&gt;NLP&lt;/span&gt; to break down jobs into direct and indirect would be an interesting
research, but again it was not&amp;nbsp;feasible.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Items 1 and 3 look clear. 2 needs more attention. It should also be possible to
solve with smarter blacklisting. But let&amp;#8217;s look at a sample of job titles, which
match &lt;tt class="docutils literal"&gt;Python&lt;/tt&gt; query:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Lead Data&amp;nbsp;Scientist&lt;/li&gt;
&lt;li&gt;DevOps Engineer (Linux, &lt;span class="caps"&gt;AWS&lt;/span&gt;)&lt;/li&gt;
&lt;li&gt;Software Engineer Embedded Smart Systems (C/C++; C#;&amp;nbsp;Python)&lt;/li&gt;
&lt;li&gt;Agile&amp;nbsp;Tester&lt;/li&gt;
&lt;li&gt;Backend Software&amp;nbsp;Engineer&lt;/li&gt;
&lt;li&gt;Experienced Python/Django Developer in Zaandam (&lt;span class="caps"&gt;NL&lt;/span&gt;/&lt;span class="caps"&gt;ENG&lt;/span&gt;)&lt;/li&gt;
&lt;li&gt;Java&amp;nbsp;Developer&lt;/li&gt;
&lt;li&gt;Junior Full Stack&amp;nbsp;Engineer&lt;/li&gt;
&lt;li&gt;Medior Front-end&amp;nbsp;Developer&lt;/li&gt;
&lt;li&gt;Bioinformatic Postdoctoral&amp;nbsp;Fellow&lt;/li&gt;
&lt;li&gt;C++ Software engineer Artificial intelligence &lt;span class="caps"&gt;DELFT&lt;/span&gt;&amp;nbsp;40k-70k&lt;/li&gt;
&lt;li&gt;Portfolio Manager Quant&amp;nbsp;Allocation&lt;/li&gt;
&lt;li&gt;PhD Student “Modelling causal interactions between marine&amp;nbsp;microbes”&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Yes, &lt;em&gt;all sorts of jobs&lt;/em&gt; as promised. The good thing is that indexing title and
body separately can help a lot with blacklisting. Say, if the title includes
&lt;tt class="docutils literal"&gt;Django&lt;/tt&gt; it&amp;#8217;s definitely a no-go, but if &lt;tt class="docutils literal"&gt;Django&lt;/tt&gt; is mentioned in the body
it depends and may be tolerable. The same way other terms, like &lt;tt class="docutils literal"&gt;Java&lt;/tt&gt; or
&lt;tt class="docutils literal"&gt;Embedded&lt;/tt&gt; can be&amp;nbsp;handled.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="storage"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id45"&gt;Storage&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Thus on the level of storage/search it&amp;#8217;s necessary to&amp;nbsp;have:&lt;/p&gt;
&lt;ol class="arabic simple"&gt;
&lt;li&gt;primary key, original, if exposed, or hash of the whole&amp;nbsp;record&lt;/li&gt;
&lt;li&gt;title, full-text&amp;nbsp;indexed&lt;/li&gt;
&lt;li&gt;employer, full-text&amp;nbsp;indexed&lt;/li&gt;
&lt;li&gt;body, full-text&amp;nbsp;indexed&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;JSON&lt;/span&gt; field or just blob for other&amp;nbsp;properties&lt;/li&gt;
&lt;li&gt;status, just integer for own&amp;nbsp;convenience&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;What embedded database has full-text search support with document storage
features and easy Python interface? The latter is about &lt;span class="caps"&gt;JSON&lt;/span&gt;-field support
that PostgeSQL and MySQL have acquired.&amp;nbsp;SQLite!&lt;/p&gt;
&lt;p&gt;The version that I have from Ubuntu Xenial&amp;#8217;s repository, at the time of writing
&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;libsqlite3-0&lt;/span&gt; &lt;span class="pre"&gt;3.11.0-1ubuntu1&lt;/span&gt;&lt;/tt&gt;, supports&amp;nbsp;both:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$ &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;PRAGMA compile_options;&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt; sqlite3 &lt;span class="p"&gt;|&lt;/span&gt; grep -E &lt;span class="s2"&gt;&amp;quot;FTS|JSON&amp;quot;&lt;/span&gt;
ENABLE_FTS3
ENABLE_FTS3_PARENTHESIS
ENABLE_FTS4
ENABLE_JSON1
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;And Python standard library &lt;tt class="docutils literal"&gt;sqlite3&lt;/tt&gt; &lt;a class="footnote-reference" href="#id25" id="id3"&gt;[3]&lt;/a&gt; uses it! Now briefly about
the&amp;nbsp;extensions.&lt;/p&gt;
&lt;div class="section" id="fts3-and-fts4"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id46"&gt;&lt;span class="caps"&gt;FTS3&lt;/span&gt; and &lt;span class="caps"&gt;FTS4&lt;/span&gt;&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;The documentation says &lt;a class="footnote-reference" href="#id26" id="id4"&gt;[4]&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;
&lt;span class="caps"&gt;FTS3&lt;/span&gt; and &lt;span class="caps"&gt;FTS4&lt;/span&gt; are SQLite virtual table modules that allow users to perform
full-text searches on a set of documents&amp;#8230; &lt;span class="caps"&gt;FTS3&lt;/span&gt; and &lt;span class="caps"&gt;FTS4&lt;/span&gt; are nearly identical&amp;#8230;
&lt;span class="caps"&gt;FTS4&lt;/span&gt; is an enhancement to &lt;span class="caps"&gt;FTS3&lt;/span&gt;. &lt;span class="caps"&gt;FTS3&lt;/span&gt; has been available since SQLite version
3.5.0 (2007-09-04). The enhancements for &lt;span class="caps"&gt;FTS4&lt;/span&gt; were added with SQLite version
3.7.4 (2010-12-07).&lt;/blockquote&gt;
&lt;p&gt;A virtual table &lt;a class="footnote-reference" href="#id27" id="id5"&gt;[5]&lt;/a&gt; in SQLite&amp;nbsp;is:&lt;/p&gt;
&lt;blockquote&gt;
&amp;#8230;an object that is registered with an open SQLite database connection. From
the perspective of an &lt;span class="caps"&gt;SQL&lt;/span&gt; statement, the virtual table object looks like any
other table or view. But behind the scenes, queries and updates on a virtual
table invoke callback methods of the virtual table object instead of reading
and writing on the database file.&lt;/blockquote&gt;
&lt;p&gt;In case of &lt;span class="caps"&gt;FTS4&lt;/span&gt; it looks like a view that behind the scenes operates 5
&lt;em&gt;shadow tables&lt;/em&gt; &lt;a class="footnote-reference" href="#id28" id="id6"&gt;[6]&lt;/a&gt; which are stored within the database file. And a note
about &lt;tt class="docutils literal"&gt;ENABLE_FTS3_PARENTHESIS&lt;/tt&gt; and query&amp;nbsp;syntax:&lt;/p&gt;
&lt;blockquote&gt;
New applications should also define the SQLITE_ENABLE_FTS3_PARENTHESIS macro
to enable the enhanced query syntax &lt;a class="footnote-reference" href="#id29" id="id7"&gt;[7]&lt;/a&gt;.&lt;/blockquote&gt;
&lt;p&gt;It means that &lt;tt class="docutils literal"&gt;python &lt;span class="pre"&gt;-django&lt;/span&gt;&lt;/tt&gt; won&amp;#8217;t work. &lt;tt class="docutils literal"&gt;python &lt;span class="caps"&gt;AND&lt;/span&gt; &lt;span class="caps"&gt;NOT&lt;/span&gt; django&lt;/tt&gt;
(or &lt;tt class="docutils literal"&gt;python &lt;span class="caps"&gt;NOT&lt;/span&gt; django&lt;/tt&gt; because &lt;tt class="docutils literal"&gt;&lt;span class="caps"&gt;AND&lt;/span&gt;&lt;/tt&gt; is the implicit operator) should be
used&amp;nbsp;instead.&lt;/p&gt;
&lt;p&gt;To better understand how full-text search is going to contribute to solving the
problem, here is simplified code that represents the data structure &lt;a class="footnote-reference" href="#id28" id="id8"&gt;[6]&lt;/a&gt;, inverted
index, that is used by the SQLite&amp;nbsp;extension.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;collections&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;


&lt;span class="n"&gt;docs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;alpha beta beta gamma gamma gamma&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;gamma delta&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;invertedIndex&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="n"&gt;docIndex&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;term&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;()):&lt;/span&gt;
    &lt;span class="n"&gt;docIndex&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;term&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;offset&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;term&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;offsetList&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;docIndex&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;invertedIndex&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;term&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;docId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;offsetList&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Result &lt;tt class="docutils literal"&gt;invertedIndex&lt;/tt&gt; looks&amp;nbsp;like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="s1"&gt;&amp;#39;alpha&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])],&lt;/span&gt;
  &lt;span class="s1"&gt;&amp;#39;beta&amp;#39;&lt;/span&gt;  &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])],&lt;/span&gt;
  &lt;span class="s1"&gt;&amp;#39;delta&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])],&lt;/span&gt;
  &lt;span class="s1"&gt;&amp;#39;gamma&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Using this structure &lt;span class="caps"&gt;FTS3&lt;/span&gt; and &lt;span class="caps"&gt;FTS4&lt;/span&gt; serve the queries. &lt;tt class="docutils literal"&gt;str.lower&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;str.split&lt;/tt&gt;
in the snippet represent tokenisation of the text. Here no stemming is needed
because most of technical terms are present intact, so I&amp;#8217;ll use &lt;tt class="docutils literal"&gt;unicode61&lt;/tt&gt; for
the virtual&amp;nbsp;table.&lt;/p&gt;
&lt;blockquote&gt;
The &amp;#8220;unicode61&amp;#8221; tokeniser is available beginning with SQLite version 3.7.13
(2012-06-11). Unicode61 works very much like &amp;#8220;simple&amp;#8221; except that it does
simple unicode case folding according to rules in Unicode Version 6.1 and
it recognises unicode space and punctuation characters and uses those to
separate tokens. The simple tokeniser only does case folding of &lt;span class="caps"&gt;ASCII&lt;/span&gt;
characters and only recognises &lt;span class="caps"&gt;ASCII&lt;/span&gt; space and punctuation characters as
token separators&amp;#8230; By default, &amp;#8220;unicode61&amp;#8221; also removes all diacritics
from Latin script characters.&lt;/blockquote&gt;
&lt;p&gt;One thing to point out is that &amp;#8220;C++&amp;#8221; and &amp;#8220;C#&amp;#8221; will be indistinguishable from &amp;#8220;C&amp;#8221;
because non-alphanumeric characters will be treated as separators. SQLite
doesn&amp;#8217;t limit minimal term length or have built-in stop-word list. So to solve
this &lt;tt class="docutils literal"&gt;tokenchars&lt;/tt&gt; is&amp;nbsp;enough:&lt;/p&gt;
&lt;blockquote&gt;
&amp;#8230;&amp;#8221;tokenchars=&amp;#8221; option may be used to specify one or more extra characters
that should be treated as part of tokens instead of as separator characters.&lt;/blockquote&gt;
&lt;p&gt;I also want to exclude &amp;#8220;.&lt;span class="caps"&gt;NET&lt;/span&gt;&amp;#8221; and adding &amp;#8220;.&amp;#8221; to the &lt;tt class="docutils literal"&gt;tokenchars&lt;/tt&gt; will cause
issue with words on the end of a sentence, but it&amp;#8217;s negligible for the use case.
Alternatively &lt;tt class="docutils literal"&gt;&lt;span class="caps"&gt;LIKE&lt;/span&gt;&lt;/tt&gt; can be safely used as a post-filter condition, or
generally it&amp;#8217;s possible to implement tokenisers in Python with &lt;tt class="docutils literal"&gt;sqlitefts&lt;/tt&gt; &lt;a class="footnote-reference" href="#id34" id="id9"&gt;[12]&lt;/a&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="json1"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id47"&gt;&lt;span class="caps"&gt;JSON1&lt;/span&gt;&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;The documentation says &lt;a class="footnote-reference" href="#id30" id="id10"&gt;[8]&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;
The json1 extension is a loadable extension that implements fifteen
application-defined &lt;span class="caps"&gt;SQL&lt;/span&gt; functions and two table-valued functions that are useful
for managing &lt;span class="caps"&gt;JSON&lt;/span&gt; content stored in an SQLite database&amp;#8230; The json1 extension
(currently) stores &lt;span class="caps"&gt;JSON&lt;/span&gt; as ordinary text&amp;#8230; The json1 extension uses the
sqlite3_value_subtype() and sqlite3_result_subtype() interfaces that were
introduced with SQLite version 3.9.0 (2015-10-14).&lt;/blockquote&gt;
&lt;p&gt;Out of the 15 function I&amp;#8217;ll use only &lt;tt class="docutils literal"&gt;json_extract&lt;/tt&gt; &lt;a class="footnote-reference" href="#id31" id="id11"&gt;[9]&lt;/a&gt;. Even though on storage
level of &lt;tt class="docutils literal"&gt;&lt;span class="caps"&gt;JSON1&lt;/span&gt;&lt;/tt&gt; &lt;span class="caps"&gt;JSON&lt;/span&gt; values are just string blobs, on logical level given set of
function extends relation model &lt;a class="footnote-reference" href="#id32" id="id12"&gt;[10]&lt;/a&gt; of SQLite with document storage model &lt;a class="footnote-reference" href="#id33" id="id13"&gt;[11]&lt;/a&gt;.
This provides great flexibility in data modelling, where one can take the best out
of the two models, and still use declarative query language, &lt;span class="caps"&gt;SQL&lt;/span&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="implementation"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id48"&gt;Implementation&lt;/a&gt;&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;sqlite3&lt;/span&gt;


&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;JobStorage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

  &lt;span class="n"&gt;_conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;


  &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sqlite3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;db.sqlite&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;isolation_level&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_setup&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

  &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_setup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;sql&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;&amp;#39;&amp;#39;&lt;/span&gt;
&lt;span class="s1"&gt;      CREATE VIRTUAL TABLE IF NOT EXISTS job USING fts4(&lt;/span&gt;
&lt;span class="s1"&gt;        properties, status, employer, title, body,&lt;/span&gt;
&lt;span class="s1"&gt;        notindexed=properties, notindexed=status,&lt;/span&gt;
&lt;span class="s1"&gt;        tokenize=unicode61 &amp;quot;tokenchars=.+#&amp;quot;)&lt;/span&gt;
&lt;span class="s1"&gt;    &amp;#39;&amp;#39;&amp;#39;&lt;/span&gt;
    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

  &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;sql&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;&amp;#39;&amp;#39;&lt;/span&gt;
&lt;span class="s1"&gt;      INSERT OR IGNORE INTO job(docid, properties, status, employer, title, body)&lt;/span&gt;
&lt;span class="s1"&gt;      VALUES(:docid, :properties, :status, :employer, :title, :body)&lt;/span&gt;
&lt;span class="s1"&gt;    &amp;#39;&amp;#39;&amp;#39;&lt;/span&gt;

    &lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;title&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;body&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;docid&amp;#39;&lt;/span&gt;      &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dataset&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;id&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;employer&amp;#39;&lt;/span&gt;   &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;employer&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;properties&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;status&amp;#39;&lt;/span&gt;     &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;title&amp;#39;&lt;/span&gt;      &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;body&amp;#39;&lt;/span&gt;       &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;isolation_level = None&lt;/tt&gt; re-enables auto-commit (for simplicity, and generally
to let transactions be an explicit measure, because explicit is better than implicit).
&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;job['dataset']&lt;/span&gt;&lt;/tt&gt; will be described in the next&amp;nbsp;section.&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="retrieval"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id49"&gt;Retrieval&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Data retrieval in context of the web or how it&amp;#8217;s called in less neutral terms &amp;#8212; screen
scraping &amp;#8212; is usually implemented with &lt;span class="caps"&gt;HTTP&lt;/span&gt; client using a &lt;span class="caps"&gt;HTML&lt;/span&gt; parser that is capable of
either XPath queries or &lt;span class="caps"&gt;CSS&lt;/span&gt; selectors. This time I wanted to try something new and the
time was right. These days we see a new wave of headless browsers &lt;a class="footnote-reference" href="#id35" id="id14"&gt;[13]&lt;/a&gt;. I&amp;#8217;m not talking
about PhantomJS or the like, but rather ubiquitous desktop browsers that have gained
official automation APIs. For instance, Chromium since version 59 supports DevTools
Protocol (a.k.a. Remote Debugging Protocol) &lt;a class="footnote-reference" href="#id36" id="id15"&gt;[14]&lt;/a&gt; in&amp;nbsp;Linux.&lt;/p&gt;
&lt;p&gt;DevTools Protocol is a WebSocket-based protocol and Google maintains an official client
for it, &lt;tt class="docutils literal"&gt;puppeteer&lt;/tt&gt; &lt;a class="footnote-reference" href="#id37" id="id16"&gt;[15]&lt;/a&gt;, written in JavaScript. What&amp;#8217;s better that there&amp;#8217;s a port to
Python 3.6+. It&amp;#8217;s called &lt;tt class="docutils literal"&gt;pyppeteer&lt;/tt&gt; &lt;a class="footnote-reference" href="#id38" id="id17"&gt;[16]&lt;/a&gt;.&lt;/p&gt;
&lt;div class="section" id="pyppeteer"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id50"&gt;Pyppeteer&lt;/a&gt;&lt;/h3&gt;
&lt;blockquote&gt;
Most things that you can do manually in the browser can be done using Puppeteer!&lt;/blockquote&gt;
&lt;p&gt;As it should be with Pyppeteer. Basically the protocol provides WebSocket-based &lt;span class="caps"&gt;RPC&lt;/span&gt; to
&lt;span class="caps"&gt;BOM&lt;/span&gt;, &lt;span class="caps"&gt;DOM&lt;/span&gt; and user objects, and the implementation provides more-or-less transparent proxy
object &lt;span class="caps"&gt;API&lt;/span&gt;. In case of Pyppeteer it&amp;#8217;s &lt;tt class="docutils literal"&gt;asyncio&lt;/tt&gt; &lt;span class="caps"&gt;API&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;Here it a demonstration of the proxy behaviour. It&amp;#8217;ll use
&lt;tt class="docutils literal"&gt;pyppeteer.element_handle.ElementHandle&lt;/tt&gt; &lt;a class="footnote-reference" href="#id40" id="id18"&gt;[18]&lt;/a&gt; that represents &lt;span class="caps"&gt;DOM&lt;/span&gt; element. Here&amp;#8217;s
a JavaScript that one can run in Chromium&amp;#8217;s Developer&amp;nbsp;Tools.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;div div.flexbox div a&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;hrefProperty&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;href&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;href&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;hrefProperty&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;valueOf&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;titleProperty&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;textContent&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;titleProperty&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;valueOf&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;href&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;And the same thing with&amp;nbsp;Pyppeteer:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;div div.flexbox div a&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;hrefProperty&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;href&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;href&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;hrefProperty&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;titleProperty&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;textContent&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;titleProperty&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;There&amp;#8217;re two must-know things about Pyppeteer. The first is that every attribute
and value access needs to be &lt;tt class="docutils literal"&gt;await&lt;/tt&gt;&amp;#8216;ed. The second is that &lt;tt class="docutils literal"&gt;pyppeteer.page.Page.click&lt;/tt&gt;
and &lt;tt class="docutils literal"&gt;pyppeteer.page.Page.waitForNavigation&lt;/tt&gt; have a race condition and should be
awaited simultaneously &lt;a class="footnote-reference" href="#id41" id="id19"&gt;[19]&lt;/a&gt;,&amp;nbsp;like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gather&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;click&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;#HeroSearchButton&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;waitForNavigation&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Towards the implementation, the idea for job retrieval &lt;span class="caps"&gt;API&lt;/span&gt; is to have an asynchronous
iterator object implemented as asynchronous generator (one that mixes &lt;tt class="docutils literal"&gt;yield&lt;/tt&gt;
and &lt;tt class="docutils literal"&gt;await&lt;/tt&gt;) as defined in &lt;span class="caps"&gt;PEP&lt;/span&gt; 525 &lt;a class="footnote-reference" href="#id39" id="id20"&gt;[17]&lt;/a&gt;. It will encapsulate browser preparation,
pagination, addtional page interactions, throttling sleeps and at the same time in
the calling code there will be just an &lt;tt class="docutils literal"&gt;async for&lt;/tt&gt; loop. I&amp;#8217;ll be&amp;nbsp;using:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Python&amp;nbsp;3.6.5&lt;/li&gt;
&lt;li&gt;Pyppeteer&amp;nbsp;0.0.16&lt;/li&gt;
&lt;li&gt;Chromium&amp;nbsp;64.0.3282.167&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div class="section" id="iteration"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id51"&gt;Iteration&lt;/a&gt;&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;asyncio&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;logging&lt;/span&gt;


&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;JobIter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

  &lt;span class="n"&gt;_page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;

  &lt;span class="n"&gt;_query&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
  &lt;span class="n"&gt;_location&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
  &lt;span class="n"&gt;_days&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;

  &lt;span class="n"&gt;_prepared&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;


  &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;location&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;

    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_query&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;
    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_location&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;location&lt;/span&gt;
    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_days&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;days&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_prepare&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Opening main page&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;goto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;https://www.glassdoor.nl/index.htm&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Entering query&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;#KeywordSearch&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;function(id, loc) {document.getElementById(id).value = loc}&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;LocationSearch&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_location&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Opening result page&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gather&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;click&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;#HeroSearchButton&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
      &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;waitForNavigation&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Selecting posted date range&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;datePosted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;#DKFilters &amp;gt; div &amp;gt; div &amp;gt; div:nth-child(2)&amp;#39;&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;click&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;datePosted&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;daysAgo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;#DKFilters .filter.expanded ul li[value=&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s1"&gt;&amp;quot;]&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_days&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;waitForSelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;daysAgo&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;click&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;daysAgo&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Waiting for list to update&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;waitForSelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.jlGrid.updating&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;waitForSelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.jlGrid.updating&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hidden&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_dismiss&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;button&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;button.mfp-close&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;click&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__aiter__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_prepared&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_prepare&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
      &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_prepared&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;

    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_dismiss&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

      &lt;span class="n"&gt;button&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;#FooterPageNav .page.current&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;textContent&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
      &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Processing page &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

      &lt;span class="n"&gt;jobList&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelectorAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ul.jlGrid &amp;gt; li&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;jobList&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;parser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;JobParser&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Processed job &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;title&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

      &lt;span class="n"&gt;button&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;#FooterPageNav .next&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="n"&gt;disabled&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.disabled&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;disabled&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Last page has been reached&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;

      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gather&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;click&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;waitForNavigation&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="parsing"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id52"&gt;Parsing&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Root element of a job listing is &lt;span class="caps"&gt;HTML&lt;/span&gt; list item tag which looks&amp;nbsp;like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;li&lt;/span&gt; &lt;span class="n"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;jl selected&amp;quot;&lt;/span&gt;
  &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2618878823&amp;quot;&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;emp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;262109&amp;quot;&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="ow"&gt;is&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;organic&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;false&amp;quot;&lt;/span&gt;
  &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;sgoc&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-1&amp;quot;&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;purchase&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;ad&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0&amp;quot;&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="ow"&gt;is&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;easy&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;apply&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;false&amp;quot;&lt;/span&gt;
  &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;njslv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;false&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;/&amp;gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;There&amp;#8217;re several data attributes, notably &lt;tt class="docutils literal"&gt;id&lt;/tt&gt;:&lt;/p&gt;
&lt;ol class="arabic simple"&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;id&lt;/tt&gt; is internal Glassdoor identifier of a job which will
help deduplicate jobs on subsequent&amp;nbsp;runs,&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;empId&lt;/tt&gt; is an employer identifier, it&amp;#8217;s is somewhat&amp;nbsp;useful,&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;isOrganicJob&lt;/tt&gt;, initially I thought it the flag to filter out
intermediaries, but unfortunately I didn&amp;#8217;t grasp how exactly it is&amp;nbsp;set.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The attributes can be accessed as a dictionary under &lt;tt class="docutils literal"&gt;dataset&lt;/tt&gt; property, see
&lt;tt class="docutils literal"&gt;_getDataset&lt;/tt&gt; below.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;datetime&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;decimal&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Decimal&lt;/span&gt;


&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;JobParser&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

  &lt;span class="n"&gt;_elementHandle&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
  &lt;span class="n"&gt;_page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;


  &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;eh&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;eh&lt;/span&gt;
    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_getDataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dataset&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperties&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;dataset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;jsh&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;jsh&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;dataset&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_getRating&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;span&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.compactStars&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;textContent&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;,&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;.&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_getTitleLink&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;div div.flexbox div a&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;href&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;href&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;textContent&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_getEmployer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;div&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.empLoc div:nth-child(1)&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;firstNode&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;div&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;firstChild&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;firstNode&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;nodeValue&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;–&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_getCity&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;span&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.subtle.loc&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;textContent&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_getDate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;div&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.hotListing&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;div&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ts&lt;/span&gt;

    &lt;span class="n"&gt;span&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.showHH span&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;textContent&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;unit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;maxsplit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;unit&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;d&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="n"&gt;ts&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;unit&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;u&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="n"&gt;ts&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hours&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;unit&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;min&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="n"&gt;ts&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;minutes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="ne"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Unsupported unit &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;unit&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ts&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_getBody&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;classList&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_elementHandle&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;click&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;spinner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.jobDetails .ajaxSpinner&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;spinner&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="n"&gt;spinner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;waitForSelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.jobDetails .ajaxSpinner&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hidden&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;div&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_page&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.jobDetails .jobDescriptionContent&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;div&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;textContent&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jsonValue&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_getTitleLink&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rating&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;employer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;city&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gather&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_getDataset&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_getRating&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_getEmployer&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_getCity&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_getDate&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_getBody&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;dataset&amp;#39;&lt;/span&gt;  &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;rating&amp;#39;&lt;/span&gt;   &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rating&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;url&amp;#39;&lt;/span&gt;      &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;employer&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;employer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;city&amp;#39;&lt;/span&gt;     &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;city&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;date&amp;#39;&lt;/span&gt;     &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;title&amp;#39;&lt;/span&gt;    &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s1"&gt;&amp;#39;body&amp;#39;&lt;/span&gt;     &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="glue"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id53"&gt;Glue&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;By default on first run Pyppeteer downloads Chromium build that is guaranteed to work.
But because I already had compatible version pre-installed I explicitly provided
&lt;tt class="docutils literal"&gt;executablePath&lt;/tt&gt; which disables the behaviour. &lt;tt class="docutils literal"&gt;headless = False&lt;/tt&gt; is useful for&amp;nbsp;debugging.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;pyppeteer&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;launch&lt;/span&gt;


&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;location&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
  &lt;span class="n"&gt;browser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;launch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;executablePath&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;/usr/bin/chromium-browser&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headless&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;browser&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;newPage&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
  &lt;span class="n"&gt;storage&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;JobStorage&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;JobIter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;location&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;storage&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;browser&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;close&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;


&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vm"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;__main__&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;basicConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;level&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DEBUG&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;format&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;%(asctime)s&lt;/span&gt;&lt;span class="s1"&gt; &lt;/span&gt;&lt;span class="si"&gt;%(levelname)s&lt;/span&gt;&lt;span class="s1"&gt; &lt;/span&gt;&lt;span class="si"&gt;%(name)s&lt;/span&gt;&lt;span class="s1"&gt; &lt;/span&gt;&lt;span class="si"&gt;%(message)s&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;websockets&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;pyppeteer&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getLogger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;setLevel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WARNING&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

  &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get_event_loop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;run_until_complete&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Python&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;Nederland&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="listing-relevant-jobs"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id54"&gt;Listing relevant&amp;nbsp;jobs&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;And finally after executing the above code, in several minutes there&amp;#8217;s &lt;tt class="docutils literal"&gt;db.sqlite&lt;/tt&gt;.
I used &lt;tt class="docutils literal"&gt;sqliteman&lt;/tt&gt; &lt;a class="footnote-reference" href="#id42" id="id21"&gt;[20]&lt;/a&gt; (available from Ubuntu Xenial&amp;#8217;s repository) to query and
explore it, because it correctly works with &lt;span class="caps"&gt;JSON&lt;/span&gt; attributes and clipboard (surprisingly
SQLiteStudio, which I usually open SQLite databases with, has issues with&amp;nbsp;both).&lt;/p&gt;
&lt;p&gt;A sample record looks&amp;nbsp;like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="s2"&gt;&amp;quot;dataset&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;                &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2622677334&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;&amp;quot;empId&amp;quot;&lt;/span&gt;             &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;823453&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;&amp;quot;isOrganicJob&amp;quot;&lt;/span&gt;      &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;&amp;quot;sgocId&amp;quot;&lt;/span&gt;            &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;&amp;quot;purchaseAdOrderId&amp;quot;&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;&amp;quot;isEasyApply&amp;quot;&lt;/span&gt;       &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;&amp;quot;njslv&amp;quot;&lt;/span&gt;             &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="s2"&gt;&amp;quot;rating&amp;quot;&lt;/span&gt;   &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;2.9&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;&amp;quot;url&amp;quot;&lt;/span&gt;      &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;https://www.glassdoor.nl/partner/jobListing.htm?...&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;&amp;quot;employer&amp;quot;&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Takeaway.com&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;&amp;quot;city&amp;quot;&lt;/span&gt;     &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Amsterdam&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;&amp;quot;date&amp;quot;&lt;/span&gt;     &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;2018-03-23 16:10:59.413456&amp;quot;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Here follows the query I used. Yours most probably will be&amp;nbsp;different.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;
  &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;julianday&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;now&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;julianday&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json_extract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;properties&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;$.date&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt; &lt;span class="n"&gt;ago&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;json_extract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;properties&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;$.employer&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;employer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;json_extract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;properties&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;$.dataset.empId&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;eid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;json_extract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;properties&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;$.rating&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;rating&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;json_extract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;properties&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;$.city&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;city&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;json_extract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;properties&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;$.url&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="k"&gt;MATCH&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
&lt;span class="s1"&gt;  python&lt;/span&gt;
&lt;span class="s1"&gt;  NOT (django OR php OR java OR ruby OR perl OR C# OR .NET)&lt;/span&gt;
&lt;span class="s1"&gt;  NOT (title:devops OR title:analyst OR title:analist OR title:support OR title:graduation OR&lt;/span&gt;
&lt;span class="s1"&gt;    title:director OR title:phd OR title:researcher OR title:biologist OR&lt;/span&gt;
&lt;span class="s1"&gt;    title:test OR title:tester OR title: &amp;quot;quality assurance&amp;quot; OR title:hacker OR&lt;/span&gt;
&lt;span class="s1"&gt;    title:intern OR title:internship OR title:junior OR title:young OR&lt;/span&gt;
&lt;span class="s1"&gt;    title: &amp;quot;full stack&amp;quot; OR title:fullstack OR title: &amp;quot;front end&amp;quot; OR&lt;/span&gt;
&lt;span class="s1"&gt;    title:embedded OR title:firmware OR title:fpga OR title:C++ OR&lt;/span&gt;
&lt;span class="s1"&gt;    title:science OR title:scientist OR title:consultant OR title:coach OR&lt;/span&gt;
&lt;span class="s1"&gt;    title:marketing OR title:network OR title:operations OR title:student OR title:linux OR&lt;/span&gt;
&lt;span class="s1"&gt;    title:architect OR title:android OR title:automation OR title: &amp;quot;ops engineer&amp;quot; OR&lt;/span&gt;
&lt;span class="s1"&gt;    title:verification OR title:validation OR title:manager OR title:expert OR&lt;/span&gt;
&lt;span class="s1"&gt;    title:azure OR title:vmware OR title: &amp;quot;net developer&amp;quot; OR title:trader)&lt;/span&gt;
&lt;span class="s1"&gt;  NOT (employer:CareerValue OR employer: &amp;quot;Bright Cubes&amp;quot; OR employer:Deloitte OR&lt;/span&gt;
&lt;span class="s1"&gt;    employer:Cegeka OR employer: &amp;quot;Star Apple&amp;quot; OR employer:HUMAN-CAPITAL OR employer:Bonque OR&lt;/span&gt;
&lt;span class="s1"&gt;    employer:Sogeti OR employer:Yacht OR employer:iSense OR employer: &amp;quot;Talent Relations&amp;quot; OR&lt;/span&gt;
&lt;span class="s1"&gt;    employer:Place-IT OR employer:Professionals OR employer:ISAAC OR&lt;/span&gt;
&lt;span class="s1"&gt;    employer:Trinamics OR employer:CINQ OR employer:Teqoia OR employer:JouwICTvacature OR&lt;/span&gt;
&lt;span class="s1"&gt;    employer:Gazelle OR employer:Consulting OR employer:Topicus OR employer: &amp;quot;Blue Lynx&amp;quot; OR&lt;/span&gt;
&lt;span class="s1"&gt;    employer:Nobru OR employer:Qualogy OR employer:YER OR employer: &amp;quot;IT Human Resources&amp;quot; OR&lt;/span&gt;
&lt;span class="s1"&gt;    employer:Montash OR employer:CodeGuild OR employer:recruitment OR employer: &amp;quot;hot item&amp;quot; OR&lt;/span&gt;
&lt;span class="s1"&gt;    employer:MobGen OR employer:Mobiquity OR employer: &amp;quot;Orange Quarter&amp;quot; OR&lt;/span&gt;
&lt;span class="s1"&gt;    employer: &amp;quot;You Get&amp;quot; OR employer:ITHR)&lt;/span&gt;
&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;employer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json_extract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;properties&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;$.date&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Yes, it&amp;#8217;s pretty restrictive and thus gives relevant results, although I&amp;#8217;m not very
happy with constant manual blacklisting (there&amp;#8217;s really whole lot of intermediaries).
Maybe next time I&amp;#8217;ll be looking for a job I employ some &lt;span class="caps"&gt;NLP&lt;/span&gt; or other machine learning.
But now now this is&amp;nbsp;it.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="wrap-up"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id55"&gt;Wrap-up&lt;/a&gt;&lt;/h2&gt;
&lt;ol class="arabic simple"&gt;
&lt;li&gt;SQLite a Swiss army knife of an ad-hoc storage and&amp;nbsp;search,&lt;/li&gt;
&lt;li&gt;Ubiquitous browsers have become automatable out of the&amp;nbsp;box,&lt;/li&gt;
&lt;li&gt;Even though some vacancy search platforms provide decent search features and other
try to automate job collection, like Indeed with its &lt;span class="caps"&gt;XML&lt;/span&gt; job feed format &lt;a class="footnote-reference" href="#id43" id="id22"&gt;[21]&lt;/a&gt;, it
looks like the effort is not enough and penetration of developer-friendly automated
job search tools can be significantly&amp;nbsp;improved,&lt;/li&gt;
&lt;li&gt;Python for the&amp;nbsp;win!&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;span class="caps"&gt;P.S.&lt;/span&gt; The new journey has already been scheduled&amp;nbsp;:-)&lt;/p&gt;
&lt;hr class="docutils" /&gt;
&lt;table class="docutils footnote" frame="void" id="id23" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.glassdoor.nl/"&gt;https://www.glassdoor.nl/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id24" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://stackoverflow.com/help/advanced-search-parameters-jobs"&gt;https://stackoverflow.com/help/advanced-search-parameters-jobs&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id25" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://docs.python.org/3/library/sqlite3.html"&gt;https://docs.python.org/3/library/sqlite3.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id26" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id4"&gt;[4]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://sqlite.org/fts3.html"&gt;https://sqlite.org/fts3.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id27" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id5"&gt;[5]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://sqlite.org/vtab.html"&gt;https://sqlite.org/vtab.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id28" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[6]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id6"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id8"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="https://sqlite.org/fts3.html#data_structures"&gt;https://sqlite.org/fts3.html#data_structures&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id29" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id7"&gt;[7]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://sqlite.org/fts3.html#_set_operations_using_the_enhanced_query_syntax"&gt;https://sqlite.org/fts3.html#_set_operations_using_the_enhanced_query_syntax&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id30" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id10"&gt;[8]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://sqlite.org/json1.html"&gt;https://sqlite.org/json1.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id31" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id11"&gt;[9]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://sqlite.org/json1.html#jex"&gt;https://sqlite.org/json1.html#jex&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id32" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id12"&gt;[10]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://en.wikipedia.org/wiki/Relational_database"&gt;https://en.wikipedia.org/wiki/Relational_database&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id33" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id13"&gt;[11]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://en.wikipedia.org/wiki/Document-oriented_database"&gt;https://en.wikipedia.org/wiki/Document-oriented_database&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id34" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id9"&gt;[12]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://pypi.org/project/sqlitefts/"&gt;https://pypi.org/project/sqlitefts/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id35" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id14"&gt;[13]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://en.wikipedia.org/wiki/Headless_browser"&gt;https://en.wikipedia.org/wiki/Headless_browser&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id36" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id15"&gt;[14]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://github.com/ChromeDevTools/devtools-protocol"&gt;https://github.com/ChromeDevTools/devtools-protocol&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id37" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id16"&gt;[15]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://github.com/GoogleChrome/puppeteer"&gt;https://github.com/GoogleChrome/puppeteer&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id38" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id17"&gt;[16]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://pypi.org/project/pyppeteer/"&gt;https://pypi.org/project/pyppeteer/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id39" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id20"&gt;[17]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.python.org/dev/peps/pep-0525/"&gt;https://www.python.org/dev/peps/pep-0525/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id40" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id18"&gt;[18]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://miyakogi.github.io/pyppeteer/reference.html#elementhandle-class"&gt;https://miyakogi.github.io/pyppeteer/reference.html#elementhandle-class&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id41" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id19"&gt;[19]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.click"&gt;https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.click&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id42" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id21"&gt;[20]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://github.com/pvanek/sqliteman"&gt;https://github.com/pvanek/sqliteman&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id43" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id22"&gt;[21]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.indeed.com/intl/en/xmlinfo.html"&gt;https://www.indeed.com/intl/en/xmlinfo.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
</content><category term="python"></category><category term="asyncio"></category><category term="browser-automation"></category><category term="sqlite"></category><category term="full-text-search"></category><category term="document-storage"></category></entry><entry><title>CherryPy questions: testing, SSL and Docker</title><link href="https://recollection.saaj.me/article/cherrypy-questions-testing-ssl-and-docker.html" rel="alternate"></link><published>2015-04-23T00:00:00+02:00</published><updated>2015-04-23T00:00:00+02:00</updated><author><name>saaj</name></author><id>tag:recollection.saaj.me,2015-04-23:/article/cherrypy-questions-testing-ssl-and-docker.html</id><summary type="html">&lt;p&gt;Basically this is a CherryPy work-in-progress article. However, it aims to draw the line
with testing infrastructure I was working on, as it seems to be completed. The article
covers what has been done and what issues I met along the way of reviving the test suite
to serve its purpose. The way is not traversed fully of course and there&amp;#8217;s a lot left to
do. It also seeks to justify concerns about CherryPy &lt;span class="caps"&gt;SSL&lt;/span&gt; functionality&amp;#8217;s correctness and
viability. As well, it tries to answer appeared questions about official Docker image and
its range of&amp;nbsp;application.&lt;/p&gt;
</summary><content type="html">&lt;p&gt;Basically this is a CherryPy work-in-progress article. However, it aims to draw the line
with testing infrastructure I was working on, as it seems to be completed. The article
covers what has been done and what issues I met along the way of reviving the test suite
to serve its purpose. The way is not traversed fully of course and there&amp;#8217;s a lot left to
do. It also seeks to justify concerns about CherryPy &lt;span class="caps"&gt;SSL&lt;/span&gt; functionality&amp;#8217;s correctness and
viability. As well, it tries to answer appeared questions about official Docker image and
its range of&amp;nbsp;application.&lt;/p&gt;

&lt;div class="contents topic" id="contents"&gt;
&lt;p class="topic-title first"&gt;Contents&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference internal" href="#testing" id="id123"&gt;Testing&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a class="reference internal" href="#test-determinism" id="id124"&gt;Test&amp;nbsp;determinism&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#test-dependency" id="id125"&gt;Test&amp;nbsp;dependency&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#test-isolation" id="id126"&gt;Test&amp;nbsp;isolation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#testing-progress" id="id127"&gt;Testing&amp;nbsp;progress&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#tagged-tox-configuration" id="id128"&gt;Tagged Tox&amp;nbsp;configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#tackling-entropy-of-threading-and-in-process-server-juggling" id="id129"&gt;Tackling entropy of threading and in-process server&amp;nbsp;juggling&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#status-indication" id="id130"&gt;Status&amp;nbsp;indication&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#drawing-the-line" id="id131"&gt;Drawing the&amp;nbsp;line&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#ssl" id="id132"&gt;&lt;span class="caps"&gt;SSL&lt;/span&gt;&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a class="reference internal" href="#experiment" id="id133"&gt;Experiment&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#problem" id="id134"&gt;Problem&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#plan" id="id135"&gt;Plan&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#docker" id="id136"&gt;Docker&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a class="reference internal" href="#nature" id="id137"&gt;Nature&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#application" id="id138"&gt;Application&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a class="reference internal" href="#questions" id="id139"&gt;Questions&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div class="section" id="testing"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id123"&gt;Testing&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;As was discovered in &lt;a class="reference external" href="https://recollection.saaj.me/article/future-of-cherrypy-bright-and-shiny.html"&gt;Future of CherryPy&lt;/a&gt; quality
assurance was a glaring problem. After I volunteered and started working on putting
tests back in the right way, I&amp;#8217;ve discovered several testing methodological errors that were
made along in years of CherryPy existence. Also, back in the day CherryPy had its own test
runner and later survived migration to Nose, but lost &lt;span class="caps"&gt;SSL&lt;/span&gt; tests. Not that the tests were purged
away from the codebase, but werern&amp;#8217;t possible to be run automatically and required a tester
to change code to run tests with &lt;span class="caps"&gt;SSL&lt;/span&gt;. Obviously no one did it, and it led to several issues
that rendered &lt;span class="caps"&gt;SSL&lt;/span&gt; functionality broken. Actually it still is, as at the moment of writing
in latest CherryPy release, 3.6, &lt;span class="caps"&gt;SSL&lt;/span&gt; doesn&amp;#8217;t&amp;nbsp;work.&lt;/p&gt;
&lt;p&gt;Besides testing issues, there were issues with CherryPy functionality as such. For instance,
session file locking didn&amp;#8217;t seem to be tested on machines with more than one execution unit.
This way session locks didn&amp;#8217;t work all the time on machines with a multi-core &lt;span class="caps"&gt;CPU&lt;/span&gt;.&lt;/p&gt;
&lt;blockquote&gt;
You may be confused to think that as long as there is &lt;span class="caps"&gt;GIL&lt;/span&gt; &lt;a class="footnote-reference" href="#id67" id="id1"&gt;[1]&lt;/a&gt;, you shouldn&amp;#8217;t care about number
of CPUs. Unfortunately, serial execution doesn&amp;#8217;t mean execution in one certain order. Thus on
different number of execution units and on different &lt;span class="caps"&gt;GIL&lt;/span&gt; implementations (Python 3.2 has new
&lt;span class="caps"&gt;GIL&lt;/span&gt; &lt;a class="footnote-reference" href="#id68" id="id2"&gt;[2]&lt;/a&gt;) a threaded Python program will likely behave differently.&lt;/blockquote&gt;
&lt;p&gt;I&amp;#8217;ve fixed file locking by postponing lock file removal to a cleanup process. Briefly covering
issues with other session backends is to say Postgres session backend was decided to be removed
because it was broken, untested and mostly undocumented. Memcached session backend was mostly
fine, except some minor &lt;tt class="docutils literal"&gt;str&lt;/tt&gt;/&lt;tt class="docutils literal"&gt;bytes&lt;/tt&gt; issue in &lt;em&gt;py3&lt;/em&gt; and the fact it isn&amp;#8217;t very useful in
its current condition. The problem is that it doesn&amp;#8217;t implement distributed locking, so once you
have two or more CherryPy instances that use a Memcached server as a session backend and your
users are not distributed consistently across the instances, all sorts of concurrent update
issues may&amp;nbsp;raise.&lt;/p&gt;
&lt;p&gt;Substantial methodological errors in the test suite&amp;nbsp;were:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;test&amp;nbsp;determinism&lt;/li&gt;
&lt;li&gt;test&amp;nbsp;dependency&lt;/li&gt;
&lt;li&gt;test&amp;nbsp;isolation&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="section" id="test-determinism"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id124"&gt;Test&amp;nbsp;determinism&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;There were garbage collection tests that were automatically added to every subclass of
&lt;tt class="docutils literal"&gt;test.helper.CPWebCase&lt;/tt&gt;. They are non-deterministic and fail sporadically (there are
other normal tests that have non-deterministic behaviour, see below). Because CPython&amp;#8217;s reference
counting garbage collector is deterministic, it&amp;#8217;s a concurrency issue. What was really surprising
for me hear was explanation like, okay, if it has failed run it once again until you&amp;#8217;re sure that
the failure persists. This &lt;tt class="docutils literal"&gt;test_gc&lt;/tt&gt; supplementing is disabled now until everything else is
resolved, and it is clear how to handle it&amp;nbsp;properly.&lt;/p&gt;
&lt;p&gt;Another thing that prevented certain classification to &lt;em&gt;pass&lt;/em&gt; and &lt;em&gt;fail&lt;/em&gt; and led to frequent
freezes were request retries and absence of &lt;span class="caps"&gt;HTTP&lt;/span&gt; client&amp;#8217;s socket timeout. I can only guess about
the original reasons of such &amp;#8220;carefulness&amp;#8221; and not letting a test just fail. For instance, you can
take some of concurrency tests that throw out up to a hundred client threads and imagine stability
and predictability of the undertaking with 10 retires and no client timeout. These two were
also fixed &amp;#8212; &lt;tt class="docutils literal"&gt;test.webtest.openURL&lt;/tt&gt; doesn&amp;#8217;t retry on socket errors and sets 10-second timeout
to its &lt;span class="caps"&gt;HTTP&lt;/span&gt; connections by&amp;nbsp;default.&lt;/p&gt;
&lt;blockquote&gt;
&lt;tt class="docutils literal"&gt;test.helper.CPWebCase&lt;/tt&gt; starts full in-process CherryPy server and then uses real
&lt;tt class="docutils literal"&gt;httplib.HTTPConnection&lt;/tt&gt; to connect to it. Most of CherryPy tests are in fact either
integration &lt;a class="footnote-reference" href="#id69" id="id3"&gt;[3]&lt;/a&gt; or system tests &lt;a class="footnote-reference" href="#id70" id="id4"&gt;[4]&lt;/a&gt;.&lt;/blockquote&gt;
&lt;/div&gt;
&lt;div class="section" id="test-dependency"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id125"&gt;Test&amp;nbsp;dependency&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;How do you like tests named like &lt;tt class="docutils literal"&gt;test_0_check_something&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;test_1_check_something_else&lt;/tt&gt;, and
so on? And what if they were named this way intentionally, so they run in order, and fail
otherwise because latter depend on former. Specifically, you can not run test individually. All
occurrence I&amp;#8217;ve found were renamed and freed from sibling&amp;nbsp;shackles.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="test-isolation"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id126"&gt;Test&amp;nbsp;isolation&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;test.helper.CPWebCase&lt;/tt&gt;&amp;#8216;s server startup routine was made in equivalent of &lt;tt class="docutils literal"&gt;unittest&lt;/tt&gt;&amp;#8216;s
&lt;tt class="docutils literal"&gt;setUpClass&lt;/tt&gt;, so tests in a test case were not isolated from each other, which made it
problematic to deal with the case of tests that interfere with internals as it was really
hard to say whether such test has failed on its own or as a consequence. I&amp;#8217;ve added
&lt;tt class="docutils literal"&gt;per_test_setup&lt;/tt&gt; attribute to &lt;tt class="docutils literal"&gt;CPWebCase&lt;/tt&gt; to conditionally do the routine in &lt;tt class="docutils literal"&gt;setUp&lt;/tt&gt;
and accommodated tests&amp;#8217; code. I apprehended significant performance degradation, but it has just
become around 20% slower, which was the reason for me to make it true by default. Not sure,
maybe it should&amp;#8217;t exist at all&amp;nbsp;now.&lt;/p&gt;
&lt;p&gt;Unfortunately, this doesn&amp;#8217;t resolve all isolation issues because having to do a complete reset
of in-process threaded server is a juggling whatsoever. When a tricky test fails badly, it still
has a side-effect (see below), like&amp;nbsp;this:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
./cherrypy/process/wspbus.py:233: RuntimeWarning: The main thread is exiting, but the Bus is
in the states.STARTING state; shutting it down automatically now. You must either call
bus.block() after start(), or call bus.exit() before the main thread exits.
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="testing-progress"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id127"&gt;Testing&amp;nbsp;progress&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;What has just been said about fixed and improved stuff is happening in my fork &lt;a class="footnote-reference" href="#id71" id="id5"&gt;[5]&lt;/a&gt;, it is not yet
pushed upstream. More detailed description and discussion about current and above sections is in
the thread in CherryPy user group &lt;a class="footnote-reference" href="#id72" id="id6"&gt;[6]&lt;/a&gt;. Because it&amp;#8217;s there until the moment Google decides
to shutdown Google Groups because of low traffic, abuse or any other reason they use for such
an announcement, I will summarise the achievements in improving CherryPy testing&amp;nbsp;infrastructure.&lt;/p&gt;
&lt;ol class="arabic simple"&gt;
&lt;li&gt;Migrated tests to stdlib &lt;tt class="docutils literal"&gt;unittest&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;unitest2&lt;/tt&gt; as fallback for &lt;em&gt;py26&lt;/em&gt;. Nose is unfriendly
to &lt;em&gt;py3&lt;/em&gt; and probably was related to some locking&amp;nbsp;issues.&lt;/li&gt;
&lt;li&gt;Overall improvement in avoiding freezes in various test cases and testing environments.
&lt;span class="caps"&gt;TTY&lt;/span&gt; in Docker, disabled interactive mode, various internals&amp;#8217; tests wait in a thread, et&amp;nbsp;cetera.&lt;/li&gt;
&lt;li&gt;Implemented &amp;#8220;tagged&amp;#8221; Tox 1.8+ configuration for matrix of various environments (see&amp;nbsp;below).&lt;/li&gt;
&lt;li&gt;Integrated with Drone.io &lt;a class="footnote-reference" href="#id73" id="id7"&gt;[7]&lt;/a&gt; &lt;span class="caps"&gt;CI&lt;/span&gt; service and Codecov.io &lt;a class="footnote-reference" href="#id74" id="id8"&gt;[8]&lt;/a&gt; code coverage tracking&amp;nbsp;service.&lt;/li&gt;
&lt;li&gt;Made various changes to allow parallel environment run with Detox to fit in 15 minutes of
Drone.io free tier. It includes running tests in install directory &lt;a class="footnote-reference" href="#id75" id="id9"&gt;[9]&lt;/a&gt;, starting server each
time on free port provided by &lt;span class="caps"&gt;OS&lt;/span&gt;, locking Memcached cases with an atomic operation and&amp;nbsp;other.&lt;/li&gt;
&lt;li&gt;Removed global and persistent configuration that prevented mixing &lt;span class="caps"&gt;HTTP&lt;/span&gt; and &lt;span class="caps"&gt;HTTPS&lt;/span&gt;&amp;nbsp;cases.&lt;/li&gt;
&lt;li&gt;Made it possible to work on the test suite in PyDev. When running tests in PyDev test runner,
it adds another non-daemonic thread that tests don&amp;#8217;t expect which leads to deadlocks or fails
for 3 tests. They are now just skipped under PyDev&amp;nbsp;runner.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Here&amp;#8217;s Drone.io commands required to run the test suite. It uses Deadsnakes &lt;a class="footnote-reference" href="#id76" id="id10"&gt;[10]&lt;/a&gt; to install old
or not yet stable version of Python. Development versions are needed to build pyOpenSSL. The rest
comes with Drone.io container out of the&amp;nbsp;box.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;debconf debconf/frontend select noninteractive&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt; sudo debconf-set-selections
sudo add-apt-repository ppa:fkrull/deadsnakes &lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&amp;gt; /dev/null
sudo apt-get update &lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&amp;gt; /dev/null
sudo apt-get -y install python2.6 python3.4 &lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&amp;gt; /dev/null
sudo apt-get -y install python2.6-dev python3.4-dev &lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&amp;gt; /dev/null

sudo pip install --quiet detox
detox
tox -e post
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;As you can see &lt;tt class="docutils literal"&gt;post&lt;/tt&gt; environment closes the build. Unless the build was successful it won&amp;#8217;t run
and no coverage will be submitted or build artifacts become available. The build artifacts, what
also needs to be listed in Drone.io, are several quality assurance reports that can be helpful for
further improving the&amp;nbsp;codebase:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
coverage_report.txt
coverage_report.html.tgz
maintenance_index.txt
code_complexity.txt
&lt;/pre&gt;
&lt;p&gt;For Codecov.io integration to work, environment variable &lt;tt class="docutils literal"&gt;CODECOV_TOKEN&lt;/tt&gt; should be&amp;nbsp;assigned.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="tagged-tox-configuration"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id128"&gt;Tagged Tox&amp;nbsp;configuration&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Final Tox configuration looks like the following. It emerged through series of rewrites that were
led by tradeoffs between test duration, combined coverage report and need to test dependencies.
I came to the last design after I realised importance of support of pyOpenSSL and thus the need to
test it (see about &lt;span class="caps"&gt;SSL&lt;/span&gt; issues below). Then test sampling facility was easy to integrate to&amp;nbsp;it.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;[tox]&lt;/span&gt;
&lt;span class="na"&gt;minversion&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;1.8&lt;/span&gt;
&lt;span class="na"&gt;envlist&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;pre,docs,py{26-co,27-qa,33-nt,34-qa}{,-ssl,-ossl}&lt;/span&gt;

&lt;span class="c1"&gt;# run tests from install dir, not checkout dir&lt;/span&gt;
&lt;span class="c1"&gt;# http://tox.rtfd.org/en/latest/example/pytest.html#known-issues-and-limitations&lt;/span&gt;
&lt;span class="k"&gt;[testenv]&lt;/span&gt;
&lt;span class="na"&gt;changedir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;{envsitepackagesdir}/cherrypy&lt;/span&gt;
&lt;span class="na"&gt;setenv&lt;/span&gt;    &lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&lt;/span&gt;
&lt;span class="s"&gt;    qa:   COVERAGE_FILE = {toxinidir}/.coverage.{envname}&lt;/span&gt;
&lt;span class="s"&gt;    ssl:  CHERRYPY_TEST_SSL_MODULE = builtin&lt;/span&gt;
&lt;span class="s"&gt;    ossl: CHERRYPY_TEST_SSL_MODULE = pyopenssl&lt;/span&gt;
&lt;span class="s"&gt;    sa:   CHERRYPY_TEST_XML_REPORT_DIR = {toxinidir}/test-xml-data&lt;/span&gt;
&lt;span class="na"&gt;deps&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&lt;/span&gt;
&lt;span class="s"&gt;    routes&lt;/span&gt;
&lt;span class="s"&gt;    py26:      unittest2&lt;/span&gt;
&lt;span class="s"&gt;    py{26,27}: python-memcached&lt;/span&gt;
&lt;span class="s"&gt;    py{33,34}: python3-memcached&lt;/span&gt;
&lt;span class="s"&gt;    qa:        coverage&lt;/span&gt;
&lt;span class="s"&gt;    ossl:      pyopenssl&lt;/span&gt;
&lt;span class="s"&gt;    sa:        unittest-xml-reporting&lt;/span&gt;
&lt;span class="na"&gt;commands&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&lt;/span&gt;
&lt;span class="s"&gt;    python --version&lt;/span&gt;
&lt;span class="s"&gt;    nt: python -m unittest {posargs: discover -v cherrypy.test}&lt;/span&gt;
&lt;span class="s"&gt;    co: unit2 {posargs:discover -v cherrypy.test}&lt;/span&gt;
&lt;span class="s"&gt;    qa: coverage run --branch --source=&amp;quot;.&amp;quot; --omit=&amp;quot;t*/*&amp;quot; \&lt;/span&gt;
&lt;span class="s"&gt;    qa: --module unittest {posargs: discover -v cherrypy.test}&lt;/span&gt;
&lt;span class="s"&gt;    sa: python test/xmltestreport.py&lt;/span&gt;

&lt;span class="k"&gt;[testenv:docs]&lt;/span&gt;
&lt;span class="na"&gt;basepython&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;python2.7&lt;/span&gt;
&lt;span class="na"&gt;changedir&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;docs&lt;/span&gt;
&lt;span class="na"&gt;commands&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;sphinx-build -q -E -n -b html . build&lt;/span&gt;
&lt;span class="na"&gt;deps&lt;/span&gt;       &lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&lt;/span&gt;
&lt;span class="s"&gt;    sphinx &amp;lt; 1.3&lt;/span&gt;
&lt;span class="s"&gt;    sphinx_rtd_theme&lt;/span&gt;

&lt;span class="k"&gt;[testenv:pre]&lt;/span&gt;
&lt;span class="na"&gt;changedir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;{toxinidir}&lt;/span&gt;
&lt;span class="na"&gt;deps&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;coverage&lt;/span&gt;
&lt;span class="na"&gt;commands&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;coverage erase&lt;/span&gt;

&lt;span class="c1"&gt;# must be run separately after main envlist, because of detox&lt;/span&gt;
&lt;span class="k"&gt;[testenv:post]&lt;/span&gt;
&lt;span class="na"&gt;changedir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;{toxinidir}&lt;/span&gt;
&lt;span class="na"&gt;deps&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&lt;/span&gt;
&lt;span class="s"&gt;    coverage&lt;/span&gt;
&lt;span class="s"&gt;    codecov&lt;/span&gt;
&lt;span class="s"&gt;    radon&lt;/span&gt;
&lt;span class="na"&gt;commands&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&lt;/span&gt;
&lt;span class="s"&gt;    bash -c &amp;#39;echo -e &amp;quot;[paths]\nsource = \n  cherrypy&amp;quot; &amp;gt; .coveragerc&amp;#39;&lt;/span&gt;
&lt;span class="s"&gt;    bash -c &amp;#39;echo &amp;quot;  .tox/*/lib/*/site-packages/cherrypy&amp;quot; &amp;gt;&amp;gt; .coveragerc&amp;#39;&lt;/span&gt;
&lt;span class="s"&gt;    coverage combine&lt;/span&gt;
&lt;span class="s"&gt;    bash -c &amp;#39;coverage report &amp;gt; coverage_report.txt&amp;#39;&lt;/span&gt;
&lt;span class="s"&gt;    coverage html&lt;/span&gt;
&lt;span class="s"&gt;    tar -czf coverage_report.html.tgz htmlcov&lt;/span&gt;
&lt;span class="s"&gt;    - codecov&lt;/span&gt;
&lt;span class="s"&gt;    bash -c &amp;#39;radon mi -s -e &amp;quot;cherrypy/t*/*&amp;quot; cherrypy &amp;gt; maintenance_index.txt&amp;#39;&lt;/span&gt;
&lt;span class="s"&gt;    bash -c &amp;#39;radon cc -s -e &amp;quot;cherrypy/t*/*&amp;quot; cherrypy &amp;gt; code_complexity.txt&amp;#39;&lt;/span&gt;
&lt;span class="na"&gt;whitelist_externals&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&lt;/span&gt;
&lt;span class="s"&gt;    /bin/tar&lt;/span&gt;
&lt;span class="s"&gt;    /bin/bash&lt;/span&gt;

&lt;span class="c1"&gt;# parse and collect XML files of XML test runner&lt;/span&gt;
&lt;span class="k"&gt;[testenv:report]&lt;/span&gt;
&lt;span class="na"&gt;deps&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt;
&lt;span class="na"&gt;changedir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;{toxinidir}&lt;/span&gt;
&lt;span class="na"&gt;commands&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;python -m cherrypy.test.xmltestreport --parse&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;This design heavily relies on great multi-dimensional configuration that appeared in Tox 1.8 &lt;a class="footnote-reference" href="#id77" id="id11"&gt;[11]&lt;/a&gt;.
Given an environment, say &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;py27-nt-ossl&lt;/span&gt;&lt;/tt&gt;, Tox treats it like a set of tags,
&lt;tt class="docutils literal"&gt;{'py27', 'nt', 'ossl'}&lt;/tt&gt;, where each entry is matched individually. This way any environment,
that makes sense though, with desired qualities can be constructed even if it isn&amp;#8217;t listed in
&lt;tt class="docutils literal"&gt;envlist&lt;/tt&gt;. Here&amp;#8217;s a table that explains the purpose of the&amp;nbsp;tags.&lt;/p&gt;
&lt;table border="1" class="docutils"&gt;
&lt;colgroup&gt;
&lt;col width="8%" /&gt;
&lt;col width="92%" /&gt;
&lt;/colgroup&gt;
&lt;thead valign="bottom"&gt;
&lt;tr&gt;&lt;th class="head"&gt;tag&lt;/th&gt;
&lt;th class="head"&gt;meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td&gt;nt&lt;/td&gt;
&lt;td&gt;normal test run via &lt;tt class="docutils literal"&gt;unittest&lt;/tt&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;co&lt;/td&gt;
&lt;td&gt;compatibility run via &lt;tt class="docutils literal"&gt;unittest2&lt;/tt&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;qa&lt;/td&gt;
&lt;td&gt;quality assurance run via &lt;tt class="docutils literal"&gt;coverage&lt;/tt&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;sa&lt;/td&gt;
&lt;td&gt;test sampling run via &lt;tt class="docutils literal"&gt;xmlrunner&lt;/tt&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;ssl&lt;/td&gt;
&lt;td&gt;run using Python bulitin &lt;tt class="docutils literal"&gt;ssl&lt;/tt&gt; module&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;ossl&lt;/td&gt;
&lt;td&gt;run using PyOpenSSL&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Note &lt;tt class="docutils literal"&gt;{posargs: discover &lt;span class="pre"&gt;-v&lt;/span&gt; cherrypy.test}&lt;/tt&gt; part. It makes possible to run a subset of the
test suite in desired environments,&amp;nbsp;e.g.:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;tox -e &lt;span class="s2"&gt;&amp;quot;py{26-co-ssl,27-nt,33-qa,34-nt-ssl}&amp;quot;&lt;/span&gt; -- -v cherrypy.test.test_http &lt;span class="se"&gt;\&lt;/span&gt;
  cherrypy.test.test_conn.TestLimitedRequestQueue &lt;span class="se"&gt;\&lt;/span&gt;
  cherrypy.test.test_caching.TestCache.test_antistampede
&lt;/pre&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="tackling-entropy-of-threading-and-in-process-server-juggling"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id129"&gt;Tackling entropy of threading and in-process server&amp;nbsp;juggling&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Even though there was a progress on making test suite more predictable, and actually runnable,
it is still far from desired, and I was scratching my head trying to figure out how to gather
test stats from all the environments. Solution was in the lighted place, right where I was
looking for it. There&amp;#8217;s a package &lt;em&gt;unittest-xml-reporting&lt;/em&gt;, whose name is pretty descriptive.
It&amp;#8217;s a &lt;tt class="docutils literal"&gt;unittest&lt;/tt&gt; runner that writes &lt;span class="caps"&gt;XML&lt;/span&gt; files likes
&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;&lt;span class="caps"&gt;TEST&lt;/span&gt;-test.test_auth_basic.TestBasicAuth-20150410005927.xml&lt;/span&gt;&lt;/tt&gt; with easy-to-guess content.
Then it was just a need to write some glue code for overnight runs and implement parsing and&amp;nbsp;aggregation.&lt;/p&gt;
&lt;p&gt;I&amp;#8217;ve implemented and incorporated module named &lt;tt class="docutils literal"&gt;xmltestreport&lt;/tt&gt; &lt;a class="footnote-reference" href="#id78" id="id12"&gt;[12]&lt;/a&gt; into the &lt;tt class="docutils literal"&gt;test&lt;/tt&gt;
package. &lt;tt class="docutils literal"&gt;sa&lt;/tt&gt; Tox tag corresponds to it. Running the following in terminal at evening,
at morning can give you the following table (except N/A evironments, see&amp;nbsp;below).&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; i in &lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;..64&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; tox -e &lt;span class="s2"&gt;&amp;quot;py{26,27,33,34}-sa{,-ssl,-ossl}&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
tox -e report
&lt;/pre&gt;&lt;/div&gt;
&lt;table border="1" class="docutils"&gt;
&lt;colgroup&gt;
&lt;col width="84%" /&gt;
&lt;col width="5%" /&gt;
&lt;col width="5%" /&gt;
&lt;col width="5%" /&gt;
&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py26-sa&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td colspan="4"&gt;&lt;span class="caps"&gt;OK&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py26-sa-ossl&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_conn.TestLimitedRequestQueue.test_queue_full&lt;/td&gt;
&lt;td&gt;27&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_request_obj.TestRequestObject.testParamErrors&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_static.TestStatic.test_file_stream&lt;/td&gt;
&lt;td&gt;27&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py26-sa-ssl&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_caching.TestCache.test_antistampede&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_config_server.TestServerConfig.testMaxRequestSize&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_conn.TestLimitedRequestQueue.test_queue_full&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_static.TestStatic.test_file_stream&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py27-sa&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td colspan="4"&gt;&lt;span class="caps"&gt;OK&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py27-sa-ossl&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_conn.TestLimitedRequestQueue.test_queue_full&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_session.TestSession.testFileConcurrency&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_states.TestWait.test_wait_for_occupied_port_INADDR_ANY&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_static.TestStatic.test_file_stream&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_tools.TestTool.testCombinedTools&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;31&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py27-sa-ssl&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_caching.TestCache.test_antistampede&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_config_server.TestServerConfig.testMaxRequestSize&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_conn.TestLimitedRequestQueue.test_queue_full&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_static.TestStatic.test_file_stream&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_tools.TestTool.testCombinedTools&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;31&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py33-sa&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_conn.TestPipeline.test_HTTP11_pipelining&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py33-sa-ossl&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td colspan="4"&gt;N/A&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py33-sa-ssl&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_caching.TestCache.test_antistampede&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_static.TestStatic.test_file_stream&lt;/td&gt;
&lt;td&gt;33&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_tools.TestTool.testCombinedTools&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;30&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py34-sa&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_conn.TestLimitedRequestQueue.test_queue_full&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_conn.TestPipeline.test_HTTP11_pipelining&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;14&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py34-sa-ossl&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td colspan="4"&gt;N/A&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;py34-sa-ssl&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;F&lt;/td&gt;
&lt;td&gt;E&lt;/td&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_caching.TestCache.test_antistampede&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_config_server.TestServerConfig.testMaxRequestSize&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;31&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_static.TestStatic.test_file_stream&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;test_tools.TestTool.testCombinedTools&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Actually, as you can see, it only had time for half of the iterations, and at the morning I
interrupted it. Sylvain, you see, I mentioned those three tests in the group correctly.
But there&amp;#8217;re other candidates and this new arisen fragility of &lt;span class="caps"&gt;SSL&lt;/span&gt; sockets, if I understand it
correctly. I mean all the tests with one fail or error. For example,
&lt;tt class="docutils literal"&gt;test_session.TestSession.testFileConcurrency&lt;/tt&gt; which is fine lock-wise but because it stresses
networking layer with several dozen client threads, there&amp;#8217;re two socket&amp;nbsp;errors:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
Exception in thread Thread-1098:
Traceback (most recent call last):
  File &amp;quot;/usr/lib64/python2.7/threading.py&amp;quot;, line 810, in __bootstrap_inner
    self.run()
  File &amp;quot;/usr/lib64/python2.7/threading.py&amp;quot;, line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File &amp;quot;py27-sa-ossl/lib/python2.7/site-packages/cherrypy/test/test_session.py&amp;quot;, line 282, in request
    c.endheaders()
  File &amp;quot;/usr/lib64/python2.7/httplib.py&amp;quot;, line 969, in endheaders
    self._send_output(message_body)
  File &amp;quot;/usr/lib64/python2.7/httplib.py&amp;quot;, line 829, in _send_output
    self.send(msg)
  File &amp;quot;/usr/lib64/python2.7/httplib.py&amp;quot;, line 805, in send
    self.sock.sendall(data)
  File &amp;quot;/usr/lib64/python2.7/ssl.py&amp;quot;, line 293, in sendall
    v = self.send(data[count:])
  File &amp;quot;/usr/lib64/python2.7/ssl.py&amp;quot;, line 262, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe

Exception in thread Thread-1099:
Traceback (most recent call last):
  File &amp;quot;/usr/lib64/python2.7/threading.py&amp;quot;, line 810, in __bootstrap_inner
    self.run()
  File &amp;quot;/usr/lib64/python2.7/threading.py&amp;quot;, line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File &amp;quot;py27-sa-ossl/lib/python2.7/site-packages/cherrypy/test/test_session.py&amp;quot;, line 282, in request
    c.endheaders()
  File &amp;quot;/usr/lib64/python2.7/httplib.py&amp;quot;, line 969, in endheaders
    self._send_output(message_body)
  File &amp;quot;/usr/lib64/python2.7/httplib.py&amp;quot;, line 829, in _send_output
    self.send(msg)
  File &amp;quot;/usr/lib64/python2.7/httplib.py&amp;quot;, line 805, in send
    self.sock.sendall(data)
  File &amp;quot;/usr/lib64/python2.7/ssl.py&amp;quot;, line 293, in sendall
    v = self.send(data[count:])
  File &amp;quot;/usr/lib64/python2.7/ssl.py&amp;quot;, line 262, in send
    v = self._sslobj.write(data)
error: [Errno 104] Connection reset by peer
&lt;/pre&gt;
&lt;p&gt;Other error details are available in &lt;span class="caps"&gt;XML&lt;/span&gt; reports which retain such details. You may think that
I have found the reason for &amp;#8220;carefulness&amp;#8221; I questioned above, but I woulnd&amp;#8217;t agree for several
reasons. First, it looks like an &lt;span class="caps"&gt;SSL&lt;/span&gt;-only and also needs investigation &amp;#8212; someone knows practical
reasons for the &lt;span class="caps"&gt;SSL&lt;/span&gt; server and client to be more fragile? Second, if a test stresses server with a
few dozens of client threads, regardless of &lt;span class="caps"&gt;SSL&lt;/span&gt;, it should be written in a way that tolerates
failures for some of them rather than relying on magic recovery behind the scenes. And after all,
all this integration testing is quite useless if we assume we&amp;#8217;re fine having 9/10 of the client to
fail in any&amp;nbsp;test.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="status-indication"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id130"&gt;Status&amp;nbsp;indication&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Long story told short, I have changed my mind on status badges. After thinking a lot about
organising project&amp;#8217;s status indication and notification, where most important is regression
notification when we have contributors who used to commit without running tests, I came to
conclusion that status badges can do the&amp;nbsp;job.&lt;/p&gt;
&lt;p&gt;This way every contributor will know that an introduced regression will become a public knowledge
in a matter of minutes. And that it won&amp;#8217;t make other members of the community happy about it.
Hopefully, this as a form of enforcement of standards and new testing tools will help us go&amp;nbsp;smoother.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Another thing that made me thinking about status indication was March Google Code shutdown
announcement &lt;a class="footnote-reference" href="#id79" id="id13"&gt;[13]&lt;/a&gt;. For my Open Source code I was mostly using Google Code&amp;#8217;s Mercurial.
It was stable and minimal environment without distraction. Although it wasn&amp;#8217;t a substantially
maintained and over time it had become lacking several features, however it still has several
exclusive ones. Essentially it was a project hosting in contrast to Bitbucket and Github
which are a repository hosting. One could have as many repositories inside a project as needed.
I know it is also feasible with Mercurial named branches, but it still feels a little denser
than it should be. Another handy thing was that Google served raw content with appropriate &lt;span class="caps"&gt;MIME&lt;/span&gt;,
e.g. JavaScript with &lt;tt class="docutils literal"&gt;text/javascript&lt;/tt&gt;, thus allowed right-off-the-repo deployment for various
testing&amp;nbsp;purposes.&lt;/p&gt;
&lt;p&gt;For status indication I used &lt;em&gt;external links&lt;/em&gt;, the block in left side bar on main project page.
You could say here is PyPi page, here&amp;#8217;s &lt;span class="caps"&gt;CI&lt;/span&gt;, and here&amp;#8217;s test coverage tracking. Because it was
visible, in 1-click distance and parallel to project description it was&amp;nbsp;sufficient.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;/div&gt;
&lt;div class="section" id="drawing-the-line"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id131"&gt;Drawing the&amp;nbsp;line&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;As far as I see, testing infrastructure is ready. I also made several related changes here and
there. Now I think it is a good time to put it into upstream. First, of course for other
contributors to make use of the new tools. Second, for the sake of granularity because there&amp;#8217;s
already a bunch of changes since January in my fork. And third, in coming months it is likely that
I won&amp;#8217;t have much time for contribution, thus it&amp;#8217;s good time to discuss the changes if we need&amp;nbsp;to.&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="ssl"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id132"&gt;&lt;span class="caps"&gt;SSL&lt;/span&gt;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;As a part of discussion of design decisions made in the test suite &lt;a class="footnote-reference" href="#id72" id="id14"&gt;[6]&lt;/a&gt;, Joseph Tate, another
CherryPy contributor expressed opinion that &lt;span class="caps"&gt;SSL&lt;/span&gt; support in CherryPy should be deprecated.
Basically the message is about how can we guarantee security of the implementation if we can&amp;#8217;t
even guarantee its&amp;nbsp;functionality.&lt;/p&gt;
&lt;p&gt;Later in discussion with Joseph when I stood on fact that CherryPy&amp;#8217;s &lt;span class="caps"&gt;SSL&lt;/span&gt; adapters are a wrapper on
wrapper, Python &lt;tt class="docutils literal"&gt;ssl&lt;/tt&gt; or pyOpenSSL on OpenSSL, and we don&amp;#8217;t need a professional security expert
in the community or special maintenance, he relaxed his&amp;nbsp;opinion.&lt;/p&gt;
&lt;blockquote&gt;
Distros have patched OpenSSL to remove insecure stuff, so maybe this is not as big a concern
as I&amp;#8217;m making it out to be, but the lack of an expert on this code is hindering its viability.&lt;/blockquote&gt;
&lt;p&gt;Also I want to inform Joseph, that his knowledge of CherryPy support for intermediate certificates
isn&amp;#8217;t up to date. &lt;tt class="docutils literal"&gt;cherrypy.server.ssl_certificate_chain&lt;/tt&gt; is documented &lt;a class="footnote-reference" href="#id80" id="id15"&gt;[14]&lt;/a&gt; and was there at
least since 2011. I&amp;#8217;ve verified it works in default&amp;nbsp;branch.&lt;/p&gt;
&lt;p&gt;Anyway, I took Joseph&amp;#8217;s questions seriously and I&amp;#8217;m sure &lt;strong&gt;we should not in any way give our users
false sense of security&lt;/strong&gt;. Security here is in theoretical sense as the state of art. I&amp;#8217;m neither
about zero-day breaches in algorithms and implementations, nor about fundamental problem of &lt;span class="caps"&gt;CA&lt;/span&gt;
trust, no-such-agencies and generally post-Snowden world we live&amp;nbsp;in.&lt;/p&gt;
&lt;p&gt;On the other hand I believe that having pure Python functional (possibly insecure in theoretical
sense) &lt;span class="caps"&gt;HTTPS&lt;/span&gt; server is beneficial for many practical use cases. There&amp;#8217;re various development tools
that are &lt;span class="caps"&gt;HTTPS&lt;/span&gt;-only unless you fight them. There&amp;#8217;re environments where you have Python but can&amp;#8217;t
install other software. There&amp;#8217;re use cases then any &lt;span class="caps"&gt;SSL&lt;/span&gt;, as an obfuscation, is fairly enough&amp;nbsp;secure.&lt;/p&gt;
&lt;p&gt;CherryPy is described and known as a complete web server and such reduction would be a big
loss. And as I will show you below, although there&amp;#8217;re several issues it is possible to
workaround them and configure CherryPy to serve an application over &lt;span class="caps"&gt;HTTPS&lt;/span&gt; securely even&amp;nbsp;today.&lt;/p&gt;
&lt;div class="section" id="experiment"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id133"&gt;Experiment&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Recently there was a CherryPy StackOverflow question right about the topic &lt;a class="footnote-reference" href="#id81" id="id16"&gt;[15]&lt;/a&gt;. The guy asked
&lt;em&gt;how to block &lt;span class="caps"&gt;SSL&lt;/span&gt; protocols in favour of &lt;span class="caps"&gt;TLS&lt;/span&gt;&lt;/em&gt;. It led me to coming to the following
quick-and-dirty&amp;nbsp;solution.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="ch"&gt;#!/usr/bin/env python&lt;/span&gt;
&lt;span class="c1"&gt;# -*- coding: utf-8 -*-&lt;/span&gt;


&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;sys&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;ssl&lt;/span&gt;

&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;cherrypy&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;cherrypy.wsgiserver.ssl_builtin&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BuiltinSSLAdapter&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;cherrypy.wsgiserver.ssl_pyopenssl&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pyOpenSSLAdapter&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;cherrypy&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;wsgiserver&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;version_info&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
  &lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;cherrypy.wsgiserver.wsgiserver2&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ssl_adapters&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;cherrypy.wsgiserver.wsgiserver3&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ssl_adapters&lt;/span&gt;

&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;OpenSSL&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SSL&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="ne"&gt;ImportError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="k"&gt;pass&lt;/span&gt;


&lt;span class="n"&gt;ciphers&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="s1"&gt;&amp;#39;ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:&amp;#39;&lt;/span&gt;
  &lt;span class="s1"&gt;&amp;#39;DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:&amp;#39;&lt;/span&gt;
  &lt;span class="s1"&gt;&amp;#39;!eNULL:!MD5:!DSS:!RC4:!SSLv2&amp;#39;&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;bundle&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dirname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cherrypy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="vm"&gt;__file__&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;test&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;test.pem&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="s1"&gt;&amp;#39;global&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s1"&gt;&amp;#39;server.socket_host&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;127.0.0.1&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;&amp;#39;server.socket_port&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8443&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;&amp;#39;server.thread_pool&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="s1"&gt;&amp;#39;server.ssl_module&amp;#39;&lt;/span&gt;      &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;custom-pyopenssl&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;&amp;#39;server.ssl_certificate&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;bundle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;&amp;#39;server.ssl_private_key&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;bundle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;


&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;BuiltinSsl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BuiltinSSLAdapter&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
  &lt;span class="sd"&gt;&amp;#39;&amp;#39;&amp;#39;Vulnerable, on py2 &amp;lt; 2.7.9, py3 &amp;lt; 3.3:&lt;/span&gt;
&lt;span class="sd"&gt;    * POODLE (SSLv3), adding ``!SSLv3`` to cipher list makes it very incompatible&lt;/span&gt;
&lt;span class="sd"&gt;    * can&amp;#39;t disable TLS compression (CRIME)&lt;/span&gt;
&lt;span class="sd"&gt;    * supports Secure Client-Initiated Renegotiation (DOS)&lt;/span&gt;
&lt;span class="sd"&gt;    * no Forward Secrecy&lt;/span&gt;
&lt;span class="sd"&gt;  Also session caching doesn&amp;#39;t work. Some tweaks are posslbe, but don&amp;#39;t really&lt;/span&gt;
&lt;span class="sd"&gt;  change much. For example, it&amp;#39;s possible to use ssl.PROTOCOL_TLSv1 instead of&lt;/span&gt;
&lt;span class="sd"&gt;  ssl.PROTOCOL_SSLv23 with little worse compatiblity.&lt;/span&gt;
&lt;span class="sd"&gt;  &amp;#39;&amp;#39;&amp;#39;&lt;/span&gt;

  &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;wrap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sock&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;Wrap and return the given socket, plus WSGI environ entries.&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ssl&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;wrap_socket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;sock&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;ciphers&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ciphers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# the override is for this line&lt;/span&gt;
        &lt;span class="n"&gt;do_handshake_on_connect&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;server_side&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;certfile&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;certificate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;keyfile&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;private_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;ssl_version&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ssl&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;PROTOCOL_SSLv23&lt;/span&gt;
      &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;ssl&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SSLError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exc_info&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;ssl&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SSL_ERROR_EOF&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# This is almost certainly due to the cherrypy engine&lt;/span&gt;
        &lt;span class="c1"&gt;# &amp;#39;pinging&amp;#39; the socket to assert it&amp;#39;s connectable;&lt;/span&gt;
        &lt;span class="c1"&gt;# the &amp;#39;ping&amp;#39; isn&amp;#39;t SSL.&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
      &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;errno&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;ssl&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SSL_ERROR_SSL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;http request&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
          &lt;span class="c1"&gt;# The client is speaking HTTP to an HTTPS server.&lt;/span&gt;
          &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="n"&gt;wsgiserver&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NoSSLError&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;unknown protocol&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
          &lt;span class="c1"&gt;# The client is speaking some non-HTTP protocol.&lt;/span&gt;
          &lt;span class="c1"&gt;# Drop the conn.&lt;/span&gt;
          &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
      &lt;span class="k"&gt;raise&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get_environ&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;ssl_adapters&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;custom-ssl&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;BuiltinSsl&lt;/span&gt;


&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Pyopenssl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pyOpenSSLAdapter&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
  &lt;span class="sd"&gt;&amp;#39;&amp;#39;&amp;#39;Mostly fine, except:&lt;/span&gt;
&lt;span class="sd"&gt;    * Secure Client-Initiated Renegotiation&lt;/span&gt;
&lt;span class="sd"&gt;    * no Forward Secrecy, SSL.OP_SINGLE_DH_USE could have helped but it didn&amp;#39;t&lt;/span&gt;
&lt;span class="sd"&gt;  &amp;#39;&amp;#39;&amp;#39;&lt;/span&gt;

  &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get_context&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;Return an SSL.Context from self attributes.&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SSL&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SSL&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SSLv23_METHOD&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# override:&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_options&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SSL&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;OP_NO_COMPRESSION&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;SSL&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;OP_SINGLE_DH_USE&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;SSL&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;OP_NO_SSLv2&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;SSL&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;OP_NO_SSLv3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_cipher_list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ciphers&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;use_privatekey_file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;private_key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;certificate_chain&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;load_verify_locations&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;certificate_chain&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;use_certificate_file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;certificate&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;

&lt;span class="n"&gt;ssl_adapters&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;custom-pyopenssl&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Pyopenssl&lt;/span&gt;


&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;App&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

  &lt;span class="nd"&gt;@cherrypy.expose&lt;/span&gt;
  &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;index&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;&amp;lt;em&amp;gt;Is this secure?&amp;lt;/em&amp;gt;&amp;#39;&lt;/span&gt;


&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vm"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;__main__&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="n"&gt;cherrypy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;quickstart&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;App&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;/&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;After I regenerated our test certificate&amp;nbsp;to:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;use longer than 1024-bit&amp;nbsp;key&lt;/li&gt;
&lt;li&gt;not use &lt;span class="caps"&gt;MD5&lt;/span&gt; and &lt;span class="caps"&gt;SHA1&lt;/span&gt; (also deprecated &lt;a class="footnote-reference" href="#id92" id="id17"&gt;[26]&lt;/a&gt;) for&amp;nbsp;signature&lt;/li&gt;
&lt;li&gt;use valid domain name for &lt;em&gt;common&amp;nbsp;name&lt;/em&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Using the following versions, we have &lt;strong&gt;A-&lt;/strong&gt; for the customised pyOpenSSL adapter
on Qualys &lt;span class="caps"&gt;SSL&lt;/span&gt; Server Test &lt;a class="footnote-reference" href="#id100" id="id18"&gt;[34]&lt;/a&gt;:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Debian Wheezy&amp;nbsp;(stable)&lt;/li&gt;
&lt;li&gt;Python 2.7.3-4+deb7u1, &lt;tt class="docutils literal"&gt;ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1e 11 Feb 2013'&lt;/tt&gt;&lt;/li&gt;
&lt;li&gt;OpenSSL&amp;nbsp;1.0.1e-2+deb7u16&lt;/li&gt;
&lt;li&gt;pyOpenSSL&amp;nbsp;0.14&lt;/li&gt;
&lt;li&gt;CherryPy from default branch (pre&amp;nbsp;3.7)&lt;/li&gt;
&lt;/ul&gt;
&lt;img alt="Qualys gives CherryPy on pyOpenSSL 0.14 A-" src="https://recollection.saaj.me/figure/cherrypy_questions_1/py273-pyopenssl014-qualys.png" /&gt;
&lt;p&gt;There&amp;#8217;s no error, but the&amp;nbsp;warnings:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Secure Client-Initiated Renegotiation &amp;#8212; Supported, DoS &lt;span class="caps"&gt;DANGER&lt;/span&gt; &lt;a class="footnote-reference" href="#id82" id="id19"&gt;[16]&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Downgrade attack prevention &amp;#8212; No, TLS_FALLBACK_SCSV not supported &lt;a class="footnote-reference" href="#id83" id="id20"&gt;[17]&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Forward Secrecy &amp;#8212; No, &lt;span class="caps"&gt;WEAK&lt;/span&gt; &lt;a class="footnote-reference" href="#id84" id="id21"&gt;[18]&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here&amp;#8217;s what Wikipedia says &lt;a class="footnote-reference" href="#id91" id="id22"&gt;[25]&lt;/a&gt; about &lt;em&gt;forward secrecy&lt;/em&gt;, to inform you that it is not
something critical or required for immediate&amp;nbsp;operation.&lt;/p&gt;
&lt;blockquote&gt;
In cryptography, forward secrecy is a property of key-agreement protocols ensuring that a
session key derived from a set of long-term keys cannot be compromised if one of the
long-term keys is compromised in the future.&lt;/blockquote&gt;
&lt;p&gt;And here&amp;#8217;s what can be squeezed out of built-in &lt;tt class="docutils literal"&gt;ssl&lt;/tt&gt;, which is&amp;nbsp;grievous.&lt;/p&gt;
&lt;img alt="Qualys gives CherryPy on Python 2.7.3's built-in SSL C" src="https://recollection.saaj.me/figure/cherrypy_questions_1/py273-ssl-qualys.png" /&gt;
&lt;p&gt;Here goes the what-is-wrong&amp;nbsp;list:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;span class="caps"&gt;POODLE&lt;/span&gt; (SSLv3) &amp;#8212; Vulnerable, &lt;span class="caps"&gt;INSECURE&lt;/span&gt; &lt;a class="footnote-reference" href="#id85" id="id23"&gt;[19]&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;TLS&lt;/span&gt; compression &amp;#8212; Yes, &lt;span class="caps"&gt;INSECURE&lt;/span&gt; &lt;a class="footnote-reference" href="#id86" id="id24"&gt;[20]&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There&amp;#8217;s also half-implemented &lt;em&gt;session resumption&lt;/em&gt; &amp;#8212; &lt;em&gt;No (IDs assigned but not accepted)&lt;/em&gt;. And
the same three warnings as for PyOpenSSL apply. &lt;tt class="docutils literal"&gt;ssl_builtin&lt;/tt&gt; doesn&amp;#8217;t handle underlying &lt;tt class="docutils literal"&gt;ssl&lt;/tt&gt;
module exceptions correctly, as the log was flooded by exceptions originating mostly from
&lt;tt class="docutils literal"&gt;do_handshake()&lt;/tt&gt;. PyOpenSSL&amp;#8217;s log was the way more quiet with just a couple of page&amp;nbsp;accesses.&lt;/p&gt;
&lt;pre class="literal-block"&gt;
[27/Mar/2015:14:07:49] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File &amp;quot;venv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py&amp;quot;, line 1968, in start
    self.tick()
  File &amp;quot;venv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py&amp;quot;, line 2035, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File &amp;quot;./test.py&amp;quot;, line 53, in wrap
    ssl_version = ssl.PROTOCOL_SSLv23
  File &amp;quot;/usr/lib/python2.7/ssl.py&amp;quot;, line 381, in wrap_socket
    ciphers=ciphers)
  File &amp;quot;/usr/lib/python2.7/ssl.py&amp;quot;, line 143, in __init__
    self.do_handshake()
  File &amp;quot;/usr/lib/python2.7/ssl.py&amp;quot;, line 305, in do_handshake
    self._sslobj.do_handshake()

SSLError: [Errno 1] _ssl.c:504: error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher
SSLError: [Errno 1] _ssl.c:504: error:14076102:SSL routines:SSL23_GET_CLIENT_HELLO:unsupported protocol
SSLError: [Errno 1] _ssl.c:504: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:504: error:14094085:SSL routines:SSL3_READ_BYTES:ccs received early
&lt;/pre&gt;
&lt;p&gt;But nothing is really that bad even with &lt;tt class="docutils literal"&gt;ssl_builtin&lt;/tt&gt;. Let&amp;#8217;s relax stability policy, make sure
you understand the consequences if you&amp;#8217;re going to do it on a production server, and add
Debain Jessie&amp;#8217;s (testing) source to &lt;tt class="docutils literal"&gt;/etc/apt/sources.list&lt;/tt&gt; of the server (Debian Wheezy) and
see what next Debain has in its&amp;nbsp;briefcase:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Python 2.7.9-1, &lt;tt class="docutils literal"&gt;ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1k 8 Jan 2015'&lt;/tt&gt;&lt;/li&gt;
&lt;li&gt;Python 3.4.2-2, &lt;tt class="docutils literal"&gt;ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1k 8 Jan 2015'&lt;/tt&gt;&lt;/li&gt;
&lt;li&gt;OpenSSL&amp;nbsp;1.0.1k-3&lt;/li&gt;
&lt;li&gt;pyOpenSSL 0.15.1 (installed via &lt;tt class="docutils literal"&gt;pip&lt;/tt&gt;, probably has been updated&amp;nbsp;since)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For you information, related excerpt from Python 2.7.9 release change log &lt;a class="footnote-reference" href="#id87" id="id25"&gt;[21]&lt;/a&gt;. It was in fact
a huge security update, not very well represented by the change of a patch&amp;nbsp;version.&lt;/p&gt;
&lt;blockquote&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;The entirety of Python 3.4&amp;#8217;s ssl module has been backported for Python 2.7.9. See
&lt;span class="caps"&gt;PEP&lt;/span&gt; 466 &lt;a class="footnote-reference" href="#id88" id="id26"&gt;[22]&lt;/a&gt; for&amp;nbsp;justification.&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;HTTPS&lt;/span&gt; certificate validation using the system&amp;#8217;s certificate store is now enabled by
default. See &lt;span class="caps"&gt;PEP&lt;/span&gt; 476 &lt;a class="footnote-reference" href="#id89" id="id27"&gt;[23]&lt;/a&gt; for&amp;nbsp;details.&lt;/li&gt;
&lt;li&gt;SSLv3 has been disabled by default in httplib and its reverse dependencies due to the
&lt;span class="caps"&gt;POODLE&lt;/span&gt;&amp;nbsp;attack.&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;
&lt;img alt="Qualys gives CherryPy on Python 2.7.9's built-in SSL A-" src="https://recollection.saaj.me/figure/cherrypy_questions_1/py279-ssl-qualys.png" /&gt;
&lt;p&gt;All errors have gone, and in spite of the warning it supports &lt;em&gt;forward secrecy&lt;/em&gt; for most
clients. Here are just couple of warnings&amp;nbsp;left:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Secure Client-Initiated Renegotiation &amp;#8212; Supported, DoS &lt;span class="caps"&gt;DANGER&lt;/span&gt; &lt;a class="footnote-reference" href="#id82" id="id28"&gt;[16]&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Session resumption (caching) &amp;#8212; No (IDs assigned but not&amp;nbsp;accepted)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The result is pretty good. If you run it on plain &lt;tt class="docutils literal"&gt;ssl_builtin&lt;/tt&gt;, is has &lt;strong&gt;B&lt;/strong&gt;, because &lt;span class="caps"&gt;RC4&lt;/span&gt;
ciphers &lt;a class="footnote-reference" href="#id90" id="id29"&gt;[24]&lt;/a&gt; aren&amp;#8217;t excluded by default. Python 3.4.2 test on the customised &lt;tt class="docutils literal"&gt;ssl_builtin&lt;/tt&gt;
is obviously the same &lt;strong&gt;A-&lt;/strong&gt; because both underlying libraries are the same. Customised
&lt;tt class="docutils literal"&gt;ssl_pyopenssl&lt;/tt&gt; on Python 2.7.9 gives same &lt;strong&gt;A-&lt;/strong&gt; &amp;#8212; again it has functional &lt;em&gt;session
resumption&lt;/em&gt; but doesn&amp;#8217;t have &lt;em&gt;forward secrecy&lt;/em&gt; for any&amp;nbsp;client.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="problem"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id134"&gt;Problem&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;After the experiment has taken place, it is time to generalise the results. Here&amp;#8217;s what is
wrong with CherryPy&amp;#8217;s &lt;span class="caps"&gt;SSL&lt;/span&gt; support&amp;nbsp;now:&lt;/p&gt;
&lt;ol class="arabic simple"&gt;
&lt;li&gt;giving false sense of&amp;nbsp;security&lt;/li&gt;
&lt;li&gt;not flexible configuration: protocol, ciphers,&amp;nbsp;options&lt;/li&gt;
&lt;li&gt;not up to date with builit-in &lt;tt class="docutils literal"&gt;ssl&lt;/tt&gt; (no &lt;span class="caps"&gt;SSL&lt;/span&gt; Context &lt;a class="footnote-reference" href="#id93" id="id30"&gt;[27]&lt;/a&gt;&amp;nbsp;support)&lt;/li&gt;
&lt;li&gt;no support for pyOpenSSL for &lt;em&gt;py3&lt;/em&gt;&lt;/li&gt;
&lt;li&gt;no security&amp;nbsp;assessment&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
&lt;div class="section" id="plan"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id135"&gt;Plan&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;I think that the problem is solvable with a reasonable effort. What has been said above may
already have convinced you that no special cryptography knowledge is required. There&amp;#8217;s a scalar
letter grade for A to F and several design decisions aimed at flexibility and&amp;nbsp;compatibility.&lt;/p&gt;
&lt;div class="section" id="outright-state-of-affairs"&gt;
&lt;h4&gt;Outright state of&amp;nbsp;affairs&lt;/h4&gt;
&lt;p&gt;Now we have the following note in &lt;em&gt;&lt;span class="caps"&gt;SSL&lt;/span&gt; support&lt;/em&gt; section &lt;a class="footnote-reference" href="#id80" id="id31"&gt;[14]&lt;/a&gt; in &lt;em&gt;Deploy&lt;/em&gt; documentation&amp;nbsp;page.&lt;/p&gt;
&lt;blockquote&gt;
You may want to test your server for &lt;span class="caps"&gt;SSL&lt;/span&gt; using the services from Qualys, Inc.&lt;/blockquote&gt;
&lt;p&gt;To me after doing this research it doesn&amp;#8217;t even look like a half-true thing expressed modestly.
It looks we lie to users, giving them false sense of security. It may look like we&amp;#8217;re so sure
that it&amp;#8217;s okay that we let them do a voluntary task of checking it once again, just in case.
But in fact it is to deploy their &lt;em&gt;py2&lt;/em&gt; application and see it&amp;#8217;s vulnerable (I assume
Python 2.7.9 isn&amp;#8217;t going to updated to soon for various reasons). This is absolutely
redundant and we can tell that Python &amp;lt; 2.7.9 is a insecure platform right away. There should be
a warning in documentation, like &amp;#8220;Please update to Python 2.7.9, install pyOpenSSL or start off
with Python 3.4+. Otherwise it will only obfuscate your&amp;nbsp;channel&amp;#8221;.&lt;/p&gt;
&lt;p&gt;Python &amp;lt; 2.7.9 should be considered an insecure platform with appropriate Python warning, see&amp;nbsp;below.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="configuration"&gt;
&lt;h4&gt;Configuration&lt;/h4&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;cherrypy.server&lt;/tt&gt; in addition to &lt;tt class="docutils literal"&gt;ssl_certificate&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;ssl_certificate_chain&lt;/tt&gt;,
&lt;tt class="docutils literal"&gt;ssl_private_key&lt;/tt&gt; should&amp;nbsp;support:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;ssl_cipher_list&lt;/tt&gt; available for all Python versions and both adapters. Mostly needed for
Python &amp;lt;&amp;nbsp;2.7.9.&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;ssl_context&lt;/tt&gt; available for Python 2.7.9+ and 3.3+. pyOpenSSL is almost there. &lt;span class="caps"&gt;SSL&lt;/span&gt;
Context &lt;a class="footnote-reference" href="#id93" id="id32"&gt;[27]&lt;/a&gt; allows user to set: protocol, ciphers,&amp;nbsp;options.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span class="caps"&gt;SSL&lt;/span&gt; Context &lt;a class="footnote-reference" href="#id93" id="id33"&gt;[27]&lt;/a&gt; when not provided from user should be created with
&lt;tt class="docutils literal"&gt;ssl.create_default_context&lt;/tt&gt; which takes care of security defaults. It is available for
Python 2.7.9+ and 3.4+. For Python 3.3 context should be configured at CherryPy side
(the function is two dozen of lines &lt;a class="footnote-reference" href="#id95" id="id34"&gt;[29]&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;ssl_adapters&lt;/tt&gt; should be available at &lt;tt class="docutils literal"&gt;cherrypy.wsgiserver&lt;/tt&gt; for user be able to set
custom adapter (see the code above). Already implemented in my&amp;nbsp;fork.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="built-in-ssl-adapter-update"&gt;
&lt;h4&gt;Built-in &lt;span class="caps"&gt;SSL&lt;/span&gt; adapter&amp;nbsp;update&lt;/h4&gt;
&lt;p&gt;The adapter should make use of &lt;span class="caps"&gt;PEP&lt;/span&gt; 466 &lt;a class="footnote-reference" href="#id88" id="id35"&gt;[22]&lt;/a&gt;. This way, inside it may be two adapters. One for
Python 2.7.9+ with &lt;span class="caps"&gt;SSL&lt;/span&gt; Context &lt;a class="footnote-reference" href="#id93" id="id36"&gt;[27]&lt;/a&gt; support and one for Python &amp;lt; 2.7.9. The latter should be
explicitly documented as vulnerable and emit warning when used. Documentation and warning text
can be taken from &lt;tt class="docutils literal"&gt;urllib3&lt;/tt&gt;&amp;#8216;s &lt;tt class="docutils literal"&gt;InsecurePlatformWarning&lt;/tt&gt; &lt;a class="footnote-reference" href="#id96" id="id37"&gt;[30]&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The adapter should handle &lt;span class="caps"&gt;SSL&lt;/span&gt; exceptions according to the protocol, not flooding the log with&amp;nbsp;them.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="python-3-support-for-pyopenssl-adapter"&gt;
&lt;h4&gt;Python 3 support for pyOpenSSL&amp;nbsp;adapter&lt;/h4&gt;
&lt;p&gt;Purpose of pyOpenSSL adapter should be rethought as Python &amp;lt; 2.6 is no longer supported
and Python 2.6+ has &lt;tt class="docutils literal"&gt;ssl&lt;/tt&gt; module built in. Thus the purpose is not to provide a fallback.
The purpose is to provide more flexibility for users to manage their own security. Python
binary that comes from a disto&amp;#8217;s repository have their own version of OpenSSL. When an OpenSSL
breach is discovered it is reasonable to expect patched OpenSSL package to be shipped first.
This gives an user ability to rebuild pyOpenSSL with latest OpenSSL as soon as possible in matter
of&amp;nbsp;doing:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;pip install -I pyOpenSSL
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Embracing this purpose Python 3 support should be provided. Currently
&lt;tt class="docutils literal"&gt;pyOpenSSLAdapter.makefile&lt;/tt&gt; uses &lt;tt class="docutils literal"&gt;wsgiserver.CP_fileobject&lt;/tt&gt; directly and uses
&lt;tt class="docutils literal"&gt;wsgiserver.ssl_pyopenssl.SSL_fileobject&lt;/tt&gt; which inherits from &lt;tt class="docutils literal"&gt;wsgiserver.CP_fileobject&lt;/tt&gt;.
&lt;tt class="docutils literal"&gt;wsgiserver.CP_fileobject&lt;/tt&gt; is available only in &lt;tt class="docutils literal"&gt;wsgiserver.wsgiserver2&lt;/tt&gt;. Therefore
&lt;tt class="docutils literal"&gt;pyOpenSSLAdapter.makefile&lt;/tt&gt; should use &lt;tt class="docutils literal"&gt;wsgiserver.CP_makefile&lt;/tt&gt; in &lt;em&gt;py3&lt;/em&gt; just like
&lt;tt class="docutils literal"&gt;BuiltinSSLAdapter&lt;/tt&gt; does. Here an advice or involvement of original authors is likely to
be needed because it&amp;#8217;s unclear in what circumstances &lt;tt class="docutils literal"&gt;pyOpenSSLAdapter.makefile&lt;/tt&gt; should return
non-&lt;span class="caps"&gt;SSL&lt;/span&gt;&amp;nbsp;socket.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="automated-security-assessment"&gt;
&lt;h4&gt;Automated security&amp;nbsp;assessment&lt;/h4&gt;
&lt;p&gt;Debian package of OpenSSL is described&amp;nbsp;as:&lt;/p&gt;
&lt;blockquote&gt;
[When] you need it to perform certain cryptographic actions like&amp;#8230;
&lt;span class="caps"&gt;SSL&lt;/span&gt;/&lt;span class="caps"&gt;TLS&lt;/span&gt; client and server tests.&lt;/blockquote&gt;
&lt;p&gt;There are also things like sslyze &lt;a class="footnote-reference" href="#id97" id="id38"&gt;[31]&lt;/a&gt;, which is described as &lt;em&gt;fast and full-featured &lt;span class="caps"&gt;SSL&lt;/span&gt;
scanner&lt;/em&gt;. I think at some extent this approach will give automated security test, but it&amp;#8217;s an
rough road. Basically, the quality of results will depend on our proficiency and maintenance of
the chosen tool. Much easier way is to you a&amp;nbsp;service.&lt;/p&gt;
&lt;p&gt;Luckily, in January Qualys announced &lt;span class="caps"&gt;SSL&lt;/span&gt; Labs &lt;span class="caps"&gt;API&lt;/span&gt; beta &lt;a class="footnote-reference" href="#id98" id="id39"&gt;[32]&lt;/a&gt;. At the dedicated page &lt;a class="footnote-reference" href="#id99" id="id40"&gt;[33]&lt;/a&gt;
there&amp;#8217;re links to official command like client and protocol manual. It makes sense to expect
the same behaviour from the &lt;span class="caps"&gt;API&lt;/span&gt; as the normal tester &lt;a class="footnote-reference" href="#id100" id="id41"&gt;[34]&lt;/a&gt; has. If so it won&amp;#8217;t work on bare &lt;span class="caps"&gt;IP&lt;/span&gt;
address and on non 443&amp;nbsp;port.&lt;/p&gt;
&lt;p&gt;To externalise this idea we have some prerequisite. If there&amp;#8217;s real CherryPy instance running
behind Nginx on cherrypy.org at WebFaction, then do we have access to it? Can we run several
additional instances there? If not, can we run them somewhere&amp;nbsp;else?&lt;/p&gt;
&lt;p&gt;If think of instances for the following&amp;nbsp;environments:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;em&gt;py2-ssl-insecure.ssl.cherrypy.org&lt;/em&gt; with Python &amp;lt; 2.7.9 that will probably be stable version
in distos&amp;#8217; repositories for quite some&amp;nbsp;time&lt;/li&gt;
&lt;li&gt;&lt;em&gt;py2-ssl-secure.ssl.cherrypy.org&lt;/em&gt; with Python&amp;nbsp;2.7.9+&lt;/li&gt;
&lt;li&gt;&lt;em&gt;py2-pyopenssl-secure.ssl.cherrypy.org&lt;/em&gt; with any &lt;em&gt;py2&lt;/em&gt; with&amp;nbsp;pyOpenSSL&lt;/li&gt;
&lt;li&gt;&lt;em&gt;py3-ssl-secure.ssl.cherrypy.org&lt;/em&gt; with Python&amp;nbsp;3.4+&lt;/li&gt;
&lt;li&gt;&lt;em&gt;py3-pyopenssl-secure.ssl.cherrypy.org&lt;/em&gt; with any &lt;em&gt;py3&lt;/em&gt; with pyOpenSSL once it&amp;#8217;s&amp;nbsp;available&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Then we need to multiplex them to outer port 443 because likely it&amp;#8217;ll be running on single
server. Because Nginx doesn&amp;#8217;t seem to be able passthrough upstream &lt;span class="caps"&gt;SSL&lt;/span&gt; connections &lt;a class="footnote-reference" href="#id101" id="id42"&gt;[35]&lt;/a&gt;
we need to employ something else, maybe HAProxy &lt;a class="footnote-reference" href="#id102" id="id43"&gt;[36]&lt;/a&gt; &lt;a class="footnote-reference" href="#id103" id="id44"&gt;[37]&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;What&amp;#8217;s left to do thereafter is to implement &lt;span class="caps"&gt;SSL&lt;/span&gt; Labs &lt;span class="caps"&gt;API&lt;/span&gt;, run in some schedule and we&amp;#8217;re done.
It is able to give us detailed and actual reports on every environment that can be exposed to
users to make their&amp;nbsp;decisions.&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="docker"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id136"&gt;Docker&lt;/a&gt;&lt;/h2&gt;
&lt;blockquote&gt;
People are running in circles, screaming &amp;#8220;DevOps&amp;#8221;. Besides over half of them also
yell &amp;#8220;Docker&amp;#8221; at odd turns.&lt;/blockquote&gt;
&lt;p&gt;Scene of the day; Baron &amp;#8220;Xaprb&amp;#8221; Schwartz authored an article about DevOps identity crisis &lt;a class="footnote-reference" href="#id104" id="id45"&gt;[38]&lt;/a&gt;
which makes it all feel this disordered. Or from the other end, at LinuxCon 2014 talk founder
and now &lt;span class="caps"&gt;CTO&lt;/span&gt; of Docker, Inc. Solomon Hykes presented &lt;a class="footnote-reference" href="#id105" id="id46"&gt;[39]&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;
I know 2 things about Docker: it uses Linux containers, and the Internet won&amp;#8217;t
shut up about it.&lt;/blockquote&gt;
&lt;p&gt;And all you want in the end of the day to all the these people really shut up and stop touting
you their magic medicine. But as long as Sylvain started dedicated topic for collecting feedback
about CherryPy&amp;#8217;s official Docker image &lt;a class="footnote-reference" href="#id106" id="id47"&gt;[40]&lt;/a&gt;, I was one who unintentionally started this
discussion (we talked about Drone.io test environment which is a Docker), and wanted to reply
anyway, I started investigating what the thing that causes itch not only to all the cool kids is
really about, except it eliminates &lt;em&gt;ops&lt;/em&gt; as a class and cures indigestion by the&amp;nbsp;way.&lt;/p&gt;
&lt;p&gt;I have to admit that as soon as I have gotten to the original source of information, have listened
to a couple of Solomon&amp;#8217;s interviews and talks and seen how he was involved into Docker reputation
management by replying to various critical articles, I became more well-disposed toward the
project. Solomon himself looks like a positive and nice kind of guy who knows what he is doing.
Not as nice, actually, as if he hadn&amp;#8217;t rewritten DotCloud&amp;#8217;s Python code to next Google&amp;#8217;s attempt
at language engineering, though reasonably nice anyway. You may often see him saying something
like this &lt;a class="footnote-reference" href="#id107" id="id48"&gt;[41]&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;
I know that Docker is pretty hyped right now &amp;#8212; I personally think it&amp;#8217;s a mixed blessing
for exactly this reason: unexperienced people are bound to talk about Docker, and say stupid
things about what Docker can do and how to use it. I wish I could magically prevent this.&lt;/blockquote&gt;
&lt;p&gt;So I ought to limit myself in saying stupid things&amp;nbsp;;-)&lt;/p&gt;
&lt;div class="section" id="nature"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id137"&gt;Nature&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Basically Docker is like Vagrant + OpenVZ on vanilla kernel. The two are available for
quite some time and have established use cases, reproducible development environment and
lightweight, operating-system-level virtualisation &lt;a class="footnote-reference" href="#id108" id="id49"&gt;[42]&lt;/a&gt;, respectively. Although Docker combines
the two qualities, and it&amp;#8217;s tempting to (re)think that developers can build production-ready
container images and deploy them through Docker magic powers, and that it&amp;#8217;s what Docker is
designated for, it would be a mistake to say&amp;nbsp;so.&lt;/p&gt;
&lt;p&gt;As Solomon says, Docker is mere a building block that comes in at the right time and provides
high-level APIs for set of great low-level features modern Linux kernel has. There are several
things I want to summarise from his latest interview given to &lt;span class="caps"&gt;FLOSS&lt;/span&gt; Weekly in episode 330 &lt;a class="footnote-reference" href="#id109" id="id50"&gt;[43]&lt;/a&gt;:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;it is not a silver bullet, it is a building&amp;nbsp;block&lt;/li&gt;
&lt;li&gt;it is used both ways: as &lt;span class="caps"&gt;VM&lt;/span&gt; and as &amp;#8220;binary&amp;#8221; (process-per-container); neither is the &amp;#8220;true&amp;#8221;&amp;nbsp;one&lt;/li&gt;
&lt;li&gt;it is neither superior nor mutually exclusive with configuration&amp;nbsp;management&lt;/li&gt;
&lt;li&gt;it is not a complete production solution; production is hard and is left to the&amp;nbsp;reader&lt;/li&gt;
&lt;li&gt;it doesn&amp;#8217;t primarily target homogenous applications; the platform&amp;#8217;s tools may be&amp;nbsp;sufficient&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I would agree that except first one, one might find &amp;#8220;however&amp;#8221; to any of them. And especially,
considering the pace of development, and promising trio of new experimental tools, that appeared
to come along Docker for cluster orchestration, in Unix-way of loosely coupled components and
responsibility separation: Machine, Swarm and Compose &lt;a class="footnote-reference" href="#id110" id="id51"&gt;[44]&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;And I really like to think about Docker as a building block. No matter what the hype is, even
though it entails imbalance in available information, there&amp;#8217;re certain design decisions made
that have certain implications. Taking them into account you  decide whether Docker fits your
certain task. For instance, there was a good comparison, &lt;span class="caps"&gt;LXC&lt;/span&gt; versus Docker, by Flockport &lt;a class="footnote-reference" href="#id111" id="id52"&gt;[45]&lt;/a&gt;,
which emphasises the difference between the two as long as Docker goes with own &lt;em&gt;libcontainer&lt;/em&gt;:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;performance penalty for a layered filesystem (e.g. &lt;span class="caps"&gt;AUFS&lt;/span&gt;)&lt;/li&gt;
&lt;li&gt;separate persistence management for immutable&amp;nbsp;containers&lt;/li&gt;
&lt;li&gt;process-per-container design has issues with real world &lt;a class="footnote-reference" href="#id112" id="id53"&gt;[46]&lt;/a&gt;:&lt;ul&gt;
&lt;li&gt;most of existing software is written in expectation of &lt;em&gt;init&lt;/em&gt;&amp;nbsp;system&lt;/li&gt;
&lt;li&gt;monitoring, cron jobs, system logging need a dedicated daemon and/or &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;daemon&lt;/li&gt;
&lt;li&gt;system built of subsystems built of components is much harder to&amp;nbsp;manage&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;As I said above both process-per-container and application-per-container both have merit. But
because Docker originally was conceived with the former idea, most images in Docker Hub are
process-per-container and people begin to fight the tool they have chosen by using Bash scripts,
Monit or Supervisor to manage multiple&amp;nbsp;processes.&lt;/p&gt;
&lt;p&gt;Complete configuration management products, like Ansible, or even simpler Fabric-based approaches
in conjunction with abundance of &lt;span class="caps"&gt;API&lt;/span&gt; for every single IaaS and PaaS provider made available
infrastructure as code with any desired quality long before Docker appeared. Big companies were
implementing their own infrastructure-as-code solutions on top of chosen service provider. Smaller
companies mostly weren&amp;#8217;t in the game. Docker, because of its momentum, can standardise service
providers&amp;#8217; supply and popularise infrastructure as code in the industry which I see quite&amp;nbsp;advantageous.&lt;/p&gt;
&lt;p&gt;Another thing that shoundn&amp;#8217;t fall out of your sight is that Docker is a new technology. It has
known and significant risk for new security issues &lt;a class="footnote-reference" href="#id113" id="id54"&gt;[47]&lt;/a&gt;. And even in its own realm it is not
as nearly as perfect. Indeed Docker makes clearer responsibility separation between &lt;em&gt;devs&lt;/em&gt; and
&lt;em&gt;ops&lt;/em&gt;. But there are cross-boundary configuration, like maximum open file handles or system
networking settings which need to be set on host and have corresponding configuration in
container, e.g. MySQL table cache. There are environment-specific configuration like load testing
in &lt;span class="caps"&gt;QA&lt;/span&gt; and rate-limiting in production. There is a lot more of such things to consider in real&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;There is a lot of discussion of microservice architecture &lt;a class="footnote-reference" href="#id114" id="id55"&gt;[48]&lt;/a&gt; today. One of its aspects is that
what was inner-application complexity is now shifted into infrastructure. And the more you follow
this route the more you immerse into distributed computing while distributed computing is still
very hard. Docker guidelines go clearly along the&amp;nbsp;route.&lt;/p&gt;
&lt;blockquote&gt;
Lately I&amp;#8217;ve seen a comprehensive and all around excellent presentation on distributed computing
by Jonas Bonér &lt;a class="footnote-reference" href="#id115" id="id56"&gt;[49]&lt;/a&gt;. Although it&amp;#8217;s named &amp;#8220;The Road to Akka Cluster and Beyond&amp;#8221; the Akka, which
is distributed application toolkit for &lt;span class="caps"&gt;JVM&lt;/span&gt;, part starts around 3/4 of the slides. The rest is
overview, theoretical aspects and practical challenges of distributed computing.&lt;/blockquote&gt;
&lt;p&gt;Last thing to say in this section is about Docker&amp;#8217;s competitors. There are obviously other fish
in the sea. Besides already mentioned &lt;span class="caps"&gt;LXC&lt;/span&gt; &lt;a class="footnote-reference" href="#id116" id="id57"&gt;[50]&lt;/a&gt;, there are also newer container tools built on
experience with existing tools, realising their shortcomings and addressing them. To name a
few: Rocket &lt;a class="footnote-reference" href="#id117" id="id58"&gt;[51]&lt;/a&gt;, Vagga &lt;a class="footnote-reference" href="#id118" id="id59"&gt;[52]&lt;/a&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="application"&gt;
&lt;h3&gt;&lt;a class="toc-backref" href="#id138"&gt;Application&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;With that being said I think thorough discussion of infrastructure and distributed application
design within the scope of CherryPy project is not only a off-topic. It&amp;#8217;s a hugely out of the
context. However the topic itself is actual and interesting, but I&amp;#8217;m sure everyone who is
dealing or is willing to deal with an application at scale knows how to run
&lt;tt class="docutils literal"&gt;pip install cherrypy&lt;/tt&gt;.&lt;/p&gt;
&lt;p&gt;What I already told to Sylvain, is on-topic and CherryPy &lt;span class="caps"&gt;QA&lt;/span&gt; can benefit from is Docker
&lt;span class="caps"&gt;QA&lt;/span&gt; image with complete CherryPy test suite containing all environments and dependencies so every
contributor can take it and run the test suite against introduced changes effortlessly. Here&amp;#8217;s
a &lt;em&gt;dockerfile&lt;/em&gt; for the&amp;nbsp;purpose.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;FROM ubuntu:trusty

MAINTAINER CherryPy QA Team

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update
RUN apt-get --no-install-recommends -qy install python python-dev python3 python3-dev \
  python-pip python-virtualenv
RUN apt-get --no-install-recommends -qy install software-properties-common build-essential \
  libffi-dev mercurial ca-certificates memcached

RUN add-apt-repository ppa:fkrull/deadsnakes
RUN apt-get update
RUN apt-get --no-install-recommends -qy install python2.6 python2.6-dev python3.3 python3.3-dev

RUN pip install &amp;quot;tox &amp;lt; 2&amp;quot;
RUN pip install &amp;quot;detox &amp;lt; 0.10&amp;quot;

WORKDIR /root
RUN hg clone --branch default https://bitbucket.org/saaj/cherrypy

WORKDIR /root/cherrypy
RUN echo &amp;#39;#!/bin/bash\n \
  service memcached start; hg pull --branch default --update; tox &amp;quot;$@&amp;quot;&amp;#39; &amp;gt; serial.sh
RUN echo &amp;#39;#!/bin/bash\n \
  service memcached start; hg pull --branch default --update; detox &amp;quot;$@&amp;quot;&amp;#39; &amp;gt; parallel.sh
RUN chmod u+x serial.sh parallel.sh
ENTRYPOINT [&amp;quot;./serial.sh&amp;quot;]
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;hg clone&lt;/tt&gt; line should be changed to the source of changes. Thus I think we should distribute
it just as a &lt;em&gt;dockerfile&lt;/em&gt;. Because of development dependencies it isn&amp;#8217;t small &amp;#8212; ~620 &lt;span class="caps"&gt;MB&lt;/span&gt;.
I also made parallel Detox run possible, &lt;tt class="docutils literal"&gt;parallel.sh&lt;/tt&gt;, but I think usually running an
environment at a time is better to stress tests with more concurrency. What&amp;#8217;s really nice about it
is that it can be run just like a&amp;nbsp;binary.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;docker run -it cherrypy/qa -e &lt;span class="s2"&gt;&amp;quot;py{26-co-ssl,27-nt,33-qa,34-nt-ssl}&amp;quot;&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  -- -v cherrypy.test.test_http cherrypy.test.test_conn.TestLimitedRequestQueue
&lt;/pre&gt;&lt;/div&gt;
&lt;div class="section" id="side-project"&gt;
&lt;h4&gt;Side-project&lt;/h4&gt;
&lt;p&gt;As I said, I think that certain answers to questions of infrastructure, orchestration or
application design are off-topic inside the project. CherryPy was never meant to answer them.
But it&amp;#8217;s really nice to have all this information available nearby. Like the series of posts
Sylvain wrote in his blog &lt;a class="footnote-reference" href="#id119" id="id60"&gt;[53]&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Sylvain, I can give you a starting point if you wish. I have a CherryPy project,
&lt;em&gt;cherrypy-webapp-skeleton&lt;/em&gt; &lt;a class="footnote-reference" href="#id120" id="id61"&gt;[54]&lt;/a&gt;, which is a complete, traditional CherryPy deployment on Debain.
I had a &lt;em&gt;fabfile&lt;/em&gt; &lt;a class="footnote-reference" href="#id121" id="id62"&gt;[55]&lt;/a&gt; there which I used test the tutorial against a fresh Debain virtual
machine. What I did is I basically translated in into a &lt;em&gt;dockerfile&lt;/em&gt; &lt;a class="footnote-reference" href="#id122" id="id63"&gt;[56]&lt;/a&gt;. It&amp;#8217;s a
virtual machine style, application-per-container thing, and it&amp;#8217;s good for testing. It is also
one of possible ways of deployment, but it obviously doesn&amp;#8217;t provide means to scaling. So if
you&amp;#8217;re interested in splitting it to separate containers and providing configuration for
containers and orchestration for such an example I would surely accept your changes. You&amp;#8217;re
also free to point me if you see something is wrong there &amp;#8212; it&amp;#8217;s all&amp;nbsp;discussable.&lt;/p&gt;
&lt;p&gt;In case you&amp;#8217;re interested I&amp;#8217;ll give an overview of the project and at the same time address
some of your&amp;nbsp;questions:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;components as-is: CherryPy, Nginx, MySQL,&amp;nbsp;Monit&lt;/li&gt;
&lt;li&gt;containers to-be: 2 x CherryPy, Nginx,&amp;nbsp;MySQL&lt;/li&gt;
&lt;li&gt;it runs through &lt;em&gt;init&lt;/em&gt; script which calls &lt;em&gt;cherryd&lt;/em&gt;-like&amp;nbsp;daemon&lt;/li&gt;
&lt;li&gt;it is installed into a &lt;em&gt;virtualenv&lt;/em&gt; but it probably doesn&amp;#8217;t make sense in a&amp;nbsp;container&lt;/li&gt;
&lt;li&gt;we outght to adhere to distro standard directory layout (&lt;tt class="docutils literal"&gt;/var/www&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;/var/logs&lt;/tt&gt;,
etc.), but the extent is in&amp;nbsp;question&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If we will present it as an example we must also address &lt;em&gt;init&lt;/em&gt; system and other issues &lt;a class="footnote-reference" href="#id112" id="id64"&gt;[46]&lt;/a&gt;,
in some way, I wrote&amp;nbsp;above.&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="questions"&gt;
&lt;h2&gt;&lt;a class="toc-backref" href="#id139"&gt;Questions&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;For these I can expect the answers only from Sylvain, but others&amp;#8217; ideas are also&amp;nbsp;welcomed.&lt;/p&gt;
&lt;ol class="arabic simple"&gt;
&lt;li&gt;Will you review my changes in the fork &lt;a class="footnote-reference" href="#id71" id="id65"&gt;[5]&lt;/a&gt;? Or it&amp;#8217;s easier if I&amp;#8217;ve created a pull
request right&amp;nbsp;away?&lt;/li&gt;
&lt;li&gt;Can you update documentation to explicitly express the state of our &lt;span class="caps"&gt;SSL&lt;/span&gt; support? I
think it is needed as soon as&amp;nbsp;possible.&lt;/li&gt;
&lt;li&gt;Do we have any progress on&amp;nbsp;bugtracker?&lt;/li&gt;
&lt;li&gt;As you have already mentioned a CherryPy roadmap, I think, it&amp;#8217;s getting a direction, like:
3.7 for &lt;span class="caps"&gt;QA&lt;/span&gt;, 3.8 for &lt;span class="caps"&gt;SSL&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;I still want to hear about your ideas about &lt;tt class="docutils literal"&gt;test_gc&lt;/tt&gt;.&lt;/li&gt;
&lt;li&gt;We still have this outdated and poitless wiki home as landing page for the&amp;nbsp;repo.&lt;/li&gt;
&lt;li&gt;Recent coverage report &lt;a class="footnote-reference" href="#id74" id="id66"&gt;[8]&lt;/a&gt; uncovered that we have a dozen with zero-coverage modules which
are probably&amp;nbsp;obsolete.&lt;/li&gt;
&lt;/ol&gt;
&lt;hr class="docutils" /&gt;
&lt;table class="docutils footnote" frame="void" id="id67" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://wiki.python.org/moin/GlobalInterpreterLock"&gt;https://wiki.python.org/moin/GlobalInterpreterLock&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id68" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.dabeaz.com/python/NewGIL.pdf"&gt;http://www.dabeaz.com/python/NewGIL.pdf&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id69" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://en.wikipedia.org/wiki/Integration_testing"&gt;http://en.wikipedia.org/wiki/Integration_testing&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id70" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id4"&gt;[4]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://en.wikipedia.org/wiki/System_testing"&gt;http://en.wikipedia.org/wiki/System_testing&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id71" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[5]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id5"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id65"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="https://bitbucket.org/saaj/cherrypy"&gt;https://bitbucket.org/saaj/cherrypy&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id72" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[6]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id6"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id14"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="https://groups.google.com/forum/#!topic/cherrypy-users/Qm-uLscS4z4"&gt;https://groups.google.com/forum/#!topic/cherrypy-users/Qm-uLscS4z4&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id73" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id7"&gt;[7]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://drone.io/bitbucket.org/saaj/cherrypy"&gt;https://drone.io/bitbucket.org/saaj/cherrypy&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id74" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[8]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id8"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id66"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="https://codecov.io/bitbucket/saaj/cherrypy?ref=default"&gt;https://codecov.io/bitbucket/saaj/cherrypy?ref=default&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id75" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id9"&gt;[9]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://tox.rtfd.org/en/latest/example/pytest.html#known-issues-and-limitations"&gt;http://tox.rtfd.org/en/latest/example/pytest.html#known-issues-and-limitations&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id76" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id10"&gt;[10]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes"&gt;https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id77" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id11"&gt;[11]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://tox.readthedocs.org/en/latest/config-v2.html"&gt;http://tox.readthedocs.org/en/latest/config-v2.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id78" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id12"&gt;[12]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/saaj/cherrypy/src/default/cherrypy/test/xmltestreport.py"&gt;https://bitbucket.org/saaj/cherrypy/src/default/cherrypy/test/xmltestreport.py&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id79" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id13"&gt;[13]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://google-opensource.blogspot.com/2015/03/farewell-to-google-code.html"&gt;http://google-opensource.blogspot.com/2015/03/farewell-to-google-code.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id80" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[14]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id15"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id31"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="http://cherrypy.readthedocs.org/en/latest/deploy.html#ssl-support"&gt;http://cherrypy.readthedocs.org/en/latest/deploy.html#ssl-support&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id81" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id16"&gt;[15]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://stackoverflow.com/q/29260947/2072035"&gt;http://stackoverflow.com/q/29260947/2072035&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id82" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[16]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id19"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id28"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="https://community.qualys.com/blogs/securitylabs/2011/10/31/tls-renegotiation-and-denial-of-service-attacks"&gt;https://community.qualys.com/blogs/securitylabs/2011/10/31/tls-renegotiation-and-denial-of-service-attacks&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id83" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id20"&gt;[17]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://datatracker.ietf.org/doc/draft-ietf-tls-downgrade-scsv/"&gt;https://datatracker.ietf.org/doc/draft-ietf-tls-downgrade-scsv/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id84" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id21"&gt;[18]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://community.qualys.com/blogs/securitylabs/2013/06/25/ssl-labs-deploying-forward-secrecy"&gt;https://community.qualys.com/blogs/securitylabs/2013/06/25/ssl-labs-deploying-forward-secrecy&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id85" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id23"&gt;[19]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://community.qualys.com/blogs/securitylabs/2014/10/15/ssl-3-is-dead-killed-by-the-poodle-attack"&gt;https://community.qualys.com/blogs/securitylabs/2014/10/15/ssl-3-is-dead-killed-by-the-poodle-attack&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id86" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id24"&gt;[20]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://community.qualys.com/blogs/securitylabs/2012/09/14/crime-information-leakage-attack-against-ssltls"&gt;https://community.qualys.com/blogs/securitylabs/2012/09/14/crime-information-leakage-attack-against-ssltls&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id87" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id25"&gt;[21]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.python.org/downloads/release/python-279/"&gt;https://www.python.org/downloads/release/python-279/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id88" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[22]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id26"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id35"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="https://www.python.org/dev/peps/pep-0466/"&gt;https://www.python.org/dev/peps/pep-0466/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id89" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id27"&gt;[23]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.python.org/dev/peps/pep-0476/"&gt;https://www.python.org/dev/peps/pep-0476/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id90" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id29"&gt;[24]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://community.qualys.com/blogs/securitylabs/2013/03/19/rc4-in-tls-is-broken-now-what"&gt;https://community.qualys.com/blogs/securitylabs/2013/03/19/rc4-in-tls-is-broken-now-what&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id91" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id22"&gt;[25]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://en.wikipedia.org/wiki/Forward_secrecy"&gt;http://en.wikipedia.org/wiki/Forward_secrecy&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id92" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id17"&gt;[26]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://community.qualys.com/blogs/securitylabs/2014/09/09/sha1-deprecation-what-you-need-to-know"&gt;https://community.qualys.com/blogs/securitylabs/2014/09/09/sha1-deprecation-what-you-need-to-know&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id93" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[27]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id30"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id32"&gt;2&lt;/a&gt;, &lt;a class="fn-backref" href="#id33"&gt;3&lt;/a&gt;, &lt;a class="fn-backref" href="#id36"&gt;4&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="https://docs.python.org/2/library/ssl.html#ssl-contexts"&gt;https://docs.python.org/2/library/ssl.html#ssl-contexts&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id94" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[28]&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://docs.python.org/2/library/ssl.html#ssl.create_default_context"&gt;https://docs.python.org/2/library/ssl.html#ssl.create_default_context&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id95" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id34"&gt;[29]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://hg.python.org/cpython/file/5b5a22b9327b/Lib/ssl.py#l395"&gt;https://hg.python.org/cpython/file/5b5a22b9327b/Lib/ssl.py#l395&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id96" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id37"&gt;[30]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning"&gt;https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id97" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id38"&gt;[31]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://github.com/nabla-c0d3/sslyze"&gt;https://github.com/nabla-c0d3/sslyze&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id98" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id39"&gt;[32]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://community.qualys.com/blogs/securitylabs/2015/01/22/ssl-labs-apis-now-available-in-beta"&gt;https://community.qualys.com/blogs/securitylabs/2015/01/22/ssl-labs-apis-now-available-in-beta&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id99" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id40"&gt;[33]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.ssllabs.com/projects/ssllabs-apis/"&gt;https://www.ssllabs.com/projects/ssllabs-apis/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id100" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[34]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id18"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id41"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="https://www.ssllabs.com/ssltest/"&gt;https://www.ssllabs.com/ssltest/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id101" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id42"&gt;[35]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://forum.nginx.org/read.php?2,234641,234641"&gt;http://forum.nginx.org/read.php?2,234641,234641&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id102" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id43"&gt;[36]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://serversforhackers.com/using-ssl-certificates-with-haproxy"&gt;https://serversforhackers.com/using-ssl-certificates-with-haproxy&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id103" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id44"&gt;[37]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://scriptthe.net/2015/02/08/pass-through-ssl-with-haproxy/"&gt;https://scriptthe.net/2015/02/08/pass-through-ssl-with-haproxy/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id104" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id45"&gt;[38]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.xaprb.com/blog/2015/02/07/devops-identity-crisis/"&gt;http://www.xaprb.com/blog/2015/02/07/devops-identity-crisis/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id105" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id46"&gt;[39]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.youtube.com/watch?v=UP6HxoC66nw"&gt;http://www.youtube.com/watch?v=UP6HxoC66nw&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id106" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id47"&gt;[40]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://groups.google.com/forum/#!topic/cherrypy-users/WywUwX8dGOM"&gt;https://groups.google.com/forum/#!topic/cherrypy-users/WywUwX8dGOM&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id107" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id48"&gt;[41]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.krisbuytaert.be/blog/docker-vs-reality-0-1#comment-5449"&gt;http://www.krisbuytaert.be/blog/docker-vs-reality-0-1#comment-5449&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id108" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id49"&gt;[42]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://en.wikipedia.org/wiki/Operating-system-level_virtualization"&gt;http://en.wikipedia.org/wiki/Operating-system-level_virtualization&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id109" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id50"&gt;[43]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://twit.tv/show/floss-weekly/330"&gt;http://twit.tv/show/floss-weekly/330&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id110" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id51"&gt;[44]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://blog.docker.com/2015/02/orchestrating-docker-with-machine-swarm-and-compose/"&gt;http://blog.docker.com/2015/02/orchestrating-docker-with-machine-swarm-and-compose/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id111" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id52"&gt;[45]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.flockport.com/lxc-vs-docker/"&gt;http://www.flockport.com/lxc-vs-docker/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id112" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[46]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id53"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id64"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="http://phusion.github.io/baseimage-docker/"&gt;http://phusion.github.io/baseimage-docker/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id113" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id54"&gt;[47]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://zeltser.com/security-risks-and-benefits-of-docker-application/"&gt;https://zeltser.com/security-risks-and-benefits-of-docker-application/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id114" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id55"&gt;[48]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://martinfowler.com/articles/microservices.html"&gt;http://martinfowler.com/articles/microservices.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id115" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id56"&gt;[49]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.slideshare.net/jboner/the-road-to-akka-cluster-and-beyond"&gt;http://www.slideshare.net/jboner/the-road-to-akka-cluster-and-beyond&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id116" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id57"&gt;[50]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://linuxcontainers.org/"&gt;https://linuxcontainers.org/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id117" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id58"&gt;[51]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://coreos.com/blog/rocket/"&gt;https://coreos.com/blog/rocket/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id118" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id59"&gt;[52]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://vagga.readthedocs.org/en/latest/what_is_vagga.html"&gt;http://vagga.readthedocs.org/en/latest/what_is_vagga.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id119" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id60"&gt;[53]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.defuze.org/"&gt;http://www.defuze.org/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id120" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id61"&gt;[54]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/saaj/cherrypy-webapp-skeleton"&gt;https://bitbucket.org/saaj/cherrypy-webapp-skeleton&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id121" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id62"&gt;[55]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/saaj/cherrypy-webapp-skeleton/src/backend/fabfile.py"&gt;https://bitbucket.org/saaj/cherrypy-webapp-skeleton/src/backend/fabfile.py&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id122" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id63"&gt;[56]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/saaj/cherrypy-webapp-skeleton/src/system/docker/Dockerfile"&gt;https://bitbucket.org/saaj/cherrypy-webapp-skeleton/src/system/docker/Dockerfile&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
</content><category term="cherrypy"></category><category term="python"></category><category term="testing"></category><category term="ssl"></category><category term="docker"></category></entry><entry><title>Future of CherryPy: bright and shiny?</title><link href="https://recollection.saaj.me/article/future-of-cherrypy-bright-and-shiny.html" rel="alternate"></link><published>2015-01-25T00:00:00+01:00</published><updated>2015-01-25T00:00:00+01:00</updated><author><name>saaj</name></author><id>tag:recollection.saaj.me,2015-01-25:/article/future-of-cherrypy-bright-and-shiny.html</id><summary type="html">&lt;p&gt;First off, I really hope so, tough at the same time I see the reasons for it. Moreover
I have been trying to contribute back to the community by giving knowledge that could hopefully
cut the rough edges that I’ve met on my way of learning and employing CherryPy [1]. And when
CherryPy is calling, I have something to&amp;nbsp;say.&lt;/p&gt;
&lt;p&gt;This article is my consideration in reply to the holiday post [2] by Sylvain Hellegouarch
in CherryPy user group about its current state and future, which recalls to older big
discussion [3] about the status of development of&amp;nbsp;CherryPy.&lt;/p&gt;
</summary><content type="html">&lt;p&gt;First off, I really hope so, tough at the same time I see the reasons for it. Moreover
I have been trying to contribute back to the community by giving knowledge that could hopefully
cut the rough edges that I’ve met on my way of learning and employing CherryPy &lt;a class="footnote-reference" href="#id32" id="id1"&gt;[1]&lt;/a&gt;. And when
CherryPy is calling, I have something to&amp;nbsp;say.&lt;/p&gt;
&lt;p&gt;This article is my consideration in reply to the holiday post &lt;a class="footnote-reference" href="#id33" id="id2"&gt;[2]&lt;/a&gt; by Sylvain Hellegouarch
in CherryPy user group about its current state and future, which recalls to older big
discussion &lt;a class="footnote-reference" href="#id34" id="id3"&gt;[3]&lt;/a&gt; about the status of development of&amp;nbsp;CherryPy.&lt;/p&gt;

&lt;div class="section" id="acquisition"&gt;
&lt;h2&gt;Acquisition&lt;/h2&gt;
&lt;p&gt;My CherryPy experience started in beginning of 2011. I had a chance to jump ship to Python
web application development and I didn’t lose it. At that moment I had years of web development
experience, so I was looking for the certain piece of software. If I would start this search
today, most of the front page titles are the same. The same &lt;em&gt;you don&amp;#8217;t have to be the best to
be popular&lt;/em&gt; still applies to the very domain of Python&amp;nbsp;application.&lt;/p&gt;
&lt;p&gt;I&amp;#8217;ve spent a couple of weeks reading the web: reviews, comparisons, benchmarks &lt;a class="footnote-reference" href="#id35" id="id4"&gt;[4]&lt;/a&gt; and
StackOverflow &lt;a class="footnote-reference" href="#id36" id="id5"&gt;[5]&lt;/a&gt;. And I have to admit that there are paths for newcomers to CherryPy,
who look for stable, fast, well-designed and pythonic &lt;span class="caps"&gt;HTTP&lt;/span&gt; framework with its own Zen &lt;a class="footnote-reference" href="#id37" id="id6"&gt;[6]&lt;/a&gt;
(like Python itself &lt;a class="footnote-reference" href="#id38" id="id7"&gt;[7]&lt;/a&gt;), that doesn&amp;#8217;t dictate application design, nor what tools to use,
but rather provides structure, testability and primitives that help code&amp;nbsp;scale.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="scope"&gt;
&lt;h2&gt;Scope&lt;/h2&gt;
&lt;p&gt;Where CherryPy stands? I think it perfectly stands as a Python &lt;span class="caps"&gt;HTTP&lt;/span&gt; application server. It
allows a developer to write a genuine Python code, use the best from the Cheese Shop,
design applications in whatever desired way (but with restrictions of threaded execution&amp;nbsp;model).&lt;/p&gt;
&lt;div class="section" id="self-contained-packages"&gt;
&lt;h3&gt;Self-contained&amp;nbsp;packages&lt;/h3&gt;
&lt;p&gt;I agree that stable pure-Python &lt;span class="caps"&gt;HTTP&lt;/span&gt; server is one of major features of CherryPy. Because
basically, in most cases one can just forget about &lt;span class="caps"&gt;WSGI&lt;/span&gt;, and use good old &lt;span class="caps"&gt;HTTP&lt;/span&gt; — bare CherryPy or
behind reverse-proxy like nginx. Less intermediaries means less problems as it simplifies
development and deployment. However, I don&amp;#8217;t think it&amp;#8217;s an additive feature. If you extract the
web-server into a stand-alone package, be it named Cheroot or something else, it won&amp;#8217;t be as
valuable for application development. If there are other frameworks&amp;#8217; users who host their &lt;span class="caps"&gt;WSGI&lt;/span&gt;
applications with CherryPy, it&amp;#8217;s their problem of deployment and of these framework&amp;#8217;s authors.
This usage is ultimately a byproduct of CherryPy&amp;#8217;s &lt;span class="caps"&gt;WSGI&lt;/span&gt; compliance and stability, and let it
stay&amp;nbsp;so.&lt;/p&gt;
&lt;p&gt;On the other hand, as the questions risen in the post are mostly about the cost of CherryPy
maintenance, if the increased modularity will help to reduce it, then it surely makes sense.
But the extraction also has its cost, and if the code of target functionality is tightly coupled
with the rest of the codebase, then benefits of extraction should clearly outweigh its cost.
At least inner-package refactoring towards more encapsulated components is possible with
almost the same&amp;nbsp;benefits.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="implementation-support"&gt;
&lt;h3&gt;Implementation&amp;nbsp;support&lt;/h3&gt;
&lt;p&gt;Regarding supported Python implementation official website says: &lt;em&gt;Python 2.5+, 3.1+, PyPy, Jython
and Android&lt;/em&gt;. Documentation says: &lt;em&gt;CPython, IronPython, Jython and PyPy&lt;/em&gt; and &lt;em&gt;CherryPy supports
Python 2.3 through to 3.4.&lt;/em&gt;. There&amp;#8217;s no need to say about official distribution, CPython, but its
versions are in question. There&amp;#8217;s a lot of compatibility code for old Python versions in CherryPy,
which can be shrunk by lifting the bar closer to actual ones. I don&amp;#8217;t think the move is for 3.x
series, but for CherryPy 4, CPython 2.7 and 3.3+ are fine. This is a real way to make the codebase
cleaner, smaller and fun to deal with&amp;nbsp;again.&lt;/p&gt;
&lt;p&gt;PyPy is constantly taking more ground and is supported by many libraries that also grow in
number. However I can&amp;#8217;t get many real use-cases out of my head (if we don&amp;#8217;t actually include the
case of a long task which should be run in background, but is run synchronously as an effort
tradeoff and assuming it can benefit from &lt;span class="caps"&gt;JIT&lt;/span&gt;), nor do I know if there&amp;#8217;s &lt;span class="caps"&gt;CPU&lt;/span&gt;-bound hot CherryPy
code. Mostly I feel it&amp;#8217;s a cool-factor, but with possible real use-cases. So I think it&amp;#8217;s a good
idea to keep it. The problem with supporting PyPy is its garbage collectors. They aren&amp;#8217;t based on
reference counting, thus files, sockets and everything that was able to close itself passing out
of the execution scope with zero references on CPython, won&amp;#8217;t deterministically keep doing it on
PyPy. It makes more strict demands on code to keep it memory-leak free. Python3&amp;#8217;s
&lt;tt class="docutils literal"&gt;ResourceWarning&lt;/tt&gt; can help with it. I left a bug report &lt;a class="footnote-reference" href="#id39" id="id8"&gt;[8]&lt;/a&gt; about the subject once when I was
porting a CherryPy-based library to&amp;nbsp;Python3.&lt;/p&gt;
&lt;p&gt;The rest is more questionable. CherryPy running on Android is clearly a cool-factor.
&lt;span class="caps"&gt;SL4A&lt;/span&gt; with Python isn&amp;#8217;t straightforward to install, and even though there&amp;#8217;s complete
CPython 2.6, it barely has a single real use-case. There&amp;#8217;s file &lt;span class="caps"&gt;API&lt;/span&gt;, SQLite and the rest
important things we use server-side for, for PhoneGap and the like, for ones who write &lt;span class="caps"&gt;HTML5&lt;/span&gt;
application. There&amp;#8217;s Kivy with established deployment. The same way I feel of Jython and
IronPython support. I have no idea if anyone is actually using CherryPy on them, but having them
officially supported complicates things. Testing, debugging, developing the framework. I think
these highly specific environments should be covered by enthusiasts, if there&amp;#8217;re any out there,
who deal with them. If there&amp;#8217;s a bug, say for Jython, it should be treated as enhancement and
fixed only if the fix doesn&amp;#8217;t impede design, performance and maintainability of primary
implementations&amp;#8217; code. I think guarantees and according measures are needed, so it&amp;#8217;s clear
what is officially supported by CherryPy, and what it may run&amp;nbsp;on.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="to-depend-or-not-to"&gt;
&lt;h3&gt;To depend or not&amp;nbsp;to&lt;/h3&gt;
&lt;p&gt;For an application it is perfectly fine to depend on as many libraries to delegate as most
out-of-domain functionality away from its codebase. On the other hand for a library it&amp;#8217;s not as
good idea. There&amp;#8217;s a functional difference between a library and a framework, but for current
form of CherryPy these terms feel&amp;nbsp;interchangeable.&lt;/p&gt;
&lt;p&gt;For a library having no dependencies is a feature when general &lt;em&gt;less moving parts, less issues&lt;/em&gt;
applies. It may relate to broken, abandoned, license-changed 3rd-party dependencies. It may be
incompatible changes, or used C-modules which complicate installation and use in PyPy and other
implementations and platforms. Thus as CherryPy has grown its own pure-Python internals which
are fast and stable, I think there&amp;#8217;s no point in abandoning&amp;nbsp;them.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="hype-fuss-and-choices"&gt;
&lt;h3&gt;Hype, fuss and&amp;nbsp;choices&lt;/h3&gt;
&lt;p&gt;Last, but not least thing I want to note about the scope of CherryPy is its technology choice.
&lt;span class="caps"&gt;WSGI&lt;/span&gt; is good example here. From inception it likely was a cool thing in Python. &lt;span class="caps"&gt;WSGI&lt;/span&gt; here
and &lt;span class="caps"&gt;WSGI&lt;/span&gt; there, standards, uniformity and stuff. And then, a little later&amp;#8230; oops, a synchronous
protocol. No WebSockets, no &lt;span class="caps"&gt;SSE&lt;/span&gt;, no other close to real-time techniques. This choice affects, for
example things like django-sse &lt;a class="footnote-reference" href="#id40" id="id9"&gt;[9]&lt;/a&gt; and its users &lt;a class="footnote-reference" href="#id41" id="id10"&gt;[10]&lt;/a&gt; (though this community is strongest in
seeing everything as a nail). But also PaaS providers like OpenShift &lt;a class="footnote-reference" href="#id42" id="id11"&gt;[11]&lt;/a&gt;, and probably&amp;nbsp;others.&lt;/p&gt;
&lt;p&gt;That is to say it&amp;#8217;s okay to be compliant (e.g. &lt;span class="caps"&gt;WSGI&lt;/span&gt;-compliant), but it&amp;#8217;s not okay to be a pliable
bigot, who praises next cool X at every turn. In that sense I like how CherryPy stands. Simple is
simple in a threaded server. Real-time stuff needed? Sylvain&amp;#8217;s ws4py &lt;a class="footnote-reference" href="#id43" id="id12"&gt;[12]&lt;/a&gt; to the rescue.
Simple background processing? There you go with &lt;tt class="docutils literal"&gt;cherrypy.process.plugins.BackgroundTask&lt;/tt&gt;.
So I want to keep seeing CherryPy making conscious and pragmatic technology&amp;nbsp;choices.&lt;/p&gt;
&lt;p&gt;2014 has seen new breed of asynchronous libraries within Python 3.4+ batteries, &lt;tt class="docutils literal"&gt;asyncio&lt;/tt&gt;. It
is nice to see co-operative execution in Python finds more expressive language constructs and
tooling, but apart from its niche uses I suspect it&amp;#8217;ll be next asynchronicity-everywhere craze,
which will lead to rewrites, new cool stuff and more &amp;#8220;pressure&amp;#8221; on users of &amp;#8220;old&amp;#8221; software. At
least, there&amp;#8217;s no evidence that it can be faster on generic load, as CherryPy is neck and neck
with Tornado &lt;a class="footnote-reference" href="#id35" id="id13"&gt;[4]&lt;/a&gt;. There&amp;#8217;s good socket performance notice &lt;a class="footnote-reference" href="#id44" id="id14"&gt;[13]&lt;/a&gt; in Python documentation which
also applies generally to such design&amp;nbsp;choice.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;There’s no question that the fastest sockets code uses non-blocking sockets and select to
multiplex them. You can put together something that will saturate a &lt;span class="caps"&gt;LAN&lt;/span&gt; connection without
putting any strain on the &lt;span class="caps"&gt;CPU&lt;/span&gt;. The trouble is that an app written this way can’t do much
of anything else - it needs to be ready to shuffle bytes around at all&amp;nbsp;times.&lt;/p&gt;
&lt;p&gt;&amp;#8230;&lt;/p&gt;
&lt;p&gt;Finally, remember that even though blocking sockets are somewhat slower than non-blocking,
in many cases they are the “right” solution. After all, if your app is driven by the data
it receives over a socket, there’s not much sense in complicating the logic just so your
app can wait on &lt;tt class="docutils literal"&gt;select&lt;/tt&gt; instead of &lt;tt class="docutils literal"&gt;recv&lt;/tt&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="audience"&gt;
&lt;h2&gt;Audience&lt;/h2&gt;
&lt;p&gt;Naturally appreciation comes from the target audience. For a newbie in &lt;span class="caps"&gt;HTTP&lt;/span&gt; (which is a
complex protocol by the way) and application development in general, Django may be a blessed
gift because it helps to meet her goals by answering as many development questions as possible,
and at the same time for an experienced developer it may seem a worthless bulk of bloatware
that solves all its tasks equally poor (like all generic solutions, though). The same newbie
won&amp;#8217;t be able to effectively use CherryPy because it won&amp;#8217;t answer the several dozen of questions
for&amp;nbsp;her.&lt;/p&gt;
&lt;p&gt;If CherryPy stays CherryPy there can&amp;#8217;t and shoundn&amp;#8217;t be answers for many of them. How to design
my application? What application directory layout to use? What persistance layer and storage to
employ? How do I render a template in CherryPy? It&amp;#8217;s all up to you, developer. From my
experience, reading others&amp;#8217; opinions on the web, and particularly in the user group, there is
an unanimously opinion that CherryPy&amp;#8217;s barrier to entry is high, but once you wrapped your head
around its concepts you&amp;#8217;re enlightened and&amp;nbsp;empowered.&lt;/p&gt;
&lt;p&gt;These points lead to two measures of increasing CherryPy user base, and hopefully contributors at
some proportional rate. First is targeting more specific audience. Examining the main page &lt;a class="footnote-reference" href="#id32" id="id15"&gt;[1]&lt;/a&gt; it
is at least not bad. It shows that simple is easy, and gives clues that complex is possible. It
refers to &lt;em&gt;any other object-oriented Python program&lt;/em&gt;, which indicates good &lt;span class="caps"&gt;API&lt;/span&gt;. But it&amp;#8217;s biased
opinion, because I look at things in reverse. Is it really that comprehensible for one who is
making decision at the moment? Is there some real-world CherryPy code that is widely
considered worth exposing, that is done &amp;#8220;right&amp;#8221; way and promotes best CherryPy practices? Second
is means of lowering and alleviating barrier to entry, see &lt;a class="reference internal" href="#improvement"&gt;Improvement&lt;/a&gt;&amp;nbsp;section.&lt;/p&gt;
&lt;p&gt;A note about code hosting that is supposed to affect the audience. I agree with Konstantin
Molchanov, that magic pixies&amp;#8230; wait no, the best friend of Ruby and Rails, to the backbone
awesome, the octocat won&amp;#8217;t make CherryPy popular by the wave of its tentacle. There&amp;#8217;s no outreach
program nor there&amp;#8217;s traffic charity for newcomers on Github. It&amp;#8217;s a popular code hosting which
makes it easy to contribute to a Git repository, in case there&amp;#8217;re ones who are willing to. But
if there are no new contributors, I don&amp;#8217;t see the way it can&amp;nbsp;help.&lt;/p&gt;
&lt;p&gt;Though it may not work in opposite direction, because if there&amp;#8217;s a Python developer willing to
contribute to a Python project, the developer may expect a Python tool for it, which Git is not.
I think it&amp;#8217;s a disillusioned and naive idea to turn away from Mercurial to get more Python users.
At the same time, I cannot disagree with Nic Young that there&amp;#8217;re a few developer services (mostly
for quality assurance) which integrate only with Github, which are free for open source projects
and may be useful for CherryPy (e.g. TravisCI is not Git-only, but Github-only because it&amp;#8217;s tied
to its &lt;span class="caps"&gt;API&lt;/span&gt;), albeit none of them lack&amp;nbsp;alternatives.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="part-taken"&gt;
&lt;h2&gt;Part&amp;nbsp;taken&lt;/h2&gt;
&lt;p&gt;A year after I commenced my CherryPy affair one web application project was in production and I
possessed certain portion of experience. Specifically saying, deployment struggle experience.
Having &lt;span class="caps"&gt;HTTP&lt;/span&gt;-only deployment was very helpful, because I could reuse the knowledge I had, but Linux
part was at most scarce, poorly covered and I was groping for valuable pieces of knowledge all
over the web. Having mint Debian box how do I make my application continuously deployable? How do
I make it first-class daemon and run it as &lt;em&gt;www-data&lt;/em&gt;? How do I put it behind nginx? How do I
monitor its memory and &lt;span class="caps"&gt;CPU&lt;/span&gt; usage? How to restart it if it has crashed? How do I rotate its&amp;nbsp;logs?&lt;/p&gt;
&lt;p&gt;That time I already had the answers, and they were put together and published as tutorial/skeleton
project &lt;a class="footnote-reference" href="#id45" id="id16"&gt;[14]&lt;/a&gt;. I felt it was necessary contribution to make. Obviously, it hasn&amp;#8217;t received high
volume traffic, but qualitatively there&amp;#8217;re constant visits from all over the world (and least from
the countries where people do &lt;span class="caps"&gt;IT&lt;/span&gt;), which indicates that CherryPy still attracts new&amp;nbsp;users.&lt;/p&gt;
&lt;p&gt;In mid-2013 I&amp;#8217;ve written another tutorial/skeleton project &lt;a class="footnote-reference" href="#id46" id="id17"&gt;[15]&lt;/a&gt;, and even though its goal
is maintainable JavaScript website-like application design, its server-side is CherryPy
application with real-world template layout for Jina2 and according CherryPy tool. I&amp;#8217;ve found
time to write documentation for it only in the end of 2014, so it is also an on topic&amp;nbsp;thing.&lt;/p&gt;
&lt;p&gt;In the spring of 2014 I started to participate on StackOverflow&amp;#8217;s CherryPy tag &lt;a class="footnote-reference" href="#id36" id="id18"&gt;[5]&lt;/a&gt;. It was
another pay-off kind of thing, because the site was helpful to me, and its CherryPy part
specifically. But the latter sadly had a lot of unanswered question making it feel a little
abandoned. In fact many questions are quite interesting dealing with performance, design,
execution model of CherryPy, so it requires an answerer to have a good grasp of documentation and
codebase. Besides making other users&amp;#8217; CherryPy experience better, it is fun and cognitive
activity. It sheds the light on how people really use CherryPy, how they start, where are
remaining rough edges and what is the room for&amp;nbsp;improvement.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="improvement"&gt;
&lt;h2&gt;Improvement&lt;/h2&gt;
&lt;p&gt;Inherently I&amp;#8217;m an application guy. I pick out cog-wheels, tune them and make spin together. I
don&amp;#8217;t work on a library oftentimes, nor do I have much time for it, but because I really value
CherryPy I want to have a way to do it, in case I have time and idea for improvement or fix. For
instance, I want to add privilege drop plugin to &lt;tt class="docutils literal"&gt;cherryd&lt;/tt&gt; and make other minor improvement to&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;When you file a bug report, you generally expect someone, a core contributor or so, to tell you
whether it is generally valid or not. Then it may be re-prioritized, assigned to someone or put
on a shelf to wait its day. Without this initial input it&amp;#8217;s a little confusing. &amp;#8220;Okay, no one
replied. It should be working other way&amp;#8221;, — you may think, at&amp;nbsp;best.&lt;/p&gt;
&lt;p&gt;Understanding the flow of one&amp;#8217;s patch to CherryPy codebase is also important. Official site says
&lt;em&gt;fork CherryPy on BitBucket here and submit pull-request with your modifications&lt;/em&gt;. When it&amp;#8217;s a bug
either when it&amp;#8217;s an enhancement? Who and where to ask a flow question, say about branching, or
about eligibility of a change. There&amp;#8217;s &lt;tt class="docutils literal"&gt;&lt;span class="caps"&gt;CONTRIBUTING&lt;/span&gt;.txt&lt;/tt&gt; which didn&amp;#8217;t appear long ago and it
points to Jason Coombs&amp;#8217; post &lt;a class="footnote-reference" href="#id48" id="id19"&gt;[17]&lt;/a&gt; about writing a perfect pull request. It is at least something
but I think it can be more clear and CherryPy-specific. Also I think it should in the
documentation, linked from the website&amp;#8217;s main&amp;nbsp;page.&lt;/p&gt;
&lt;div class="section" id="quality-assurance"&gt;
&lt;h3&gt;Quality&amp;nbsp;assurance&lt;/h3&gt;
&lt;p&gt;I think things like bug #1298 &lt;a class="footnote-reference" href="#id47" id="id20"&gt;[16]&lt;/a&gt; should never happen to a project which is maintained in
a responsible way and puts effort in quality assurance. Okay, I see &lt;tt class="docutils literal"&gt;tox.ini&lt;/tt&gt; in root of the
project. I clone it, and run &lt;tt class="docutils literal"&gt;tox&lt;/tt&gt;. Duh, all environments have failed. Besides, not all the
claimed environments are in the file. Moreover, it&amp;#8217;s some kind of back in Soviet Russia story.
Why the hell unit tests ask me questions!? Are they examining me instead? More surprises?
ShiningPanda is dead &lt;a class="footnote-reference" href="#id49" id="id21"&gt;[18]&lt;/a&gt;, the same way as the link to it from the CherryPy website under
&lt;em&gt;online tests&lt;/em&gt; section (&lt;em&gt;continuous integration&lt;/em&gt; is kind of standard term for the thing, as
online tests refer more to &lt;span class="caps"&gt;IQ&lt;/span&gt; tests, psychology, et&amp;nbsp;cetera).&lt;/p&gt;
&lt;p&gt;To name it honestly, this is a reckless way of maintaining a project. Reckless to users whose
applications will break with next release published this way. And &lt;em&gt;interpretation result&lt;/em&gt; is
as much time-consuming as to see whether the build page is green or red. No, I&amp;#8217;m not promoting
these cool badges one can see all around. I&amp;#8217;m talking about the responsive flow where every
contributor runs tests on all supported Python implementations (with Tox) before pushing her
changes. If one is lazy to care and to set up all implementations, but has itching fingers to
push some changes anyway, having the notification like: &lt;em&gt;Hey, you&amp;#8217;ve just broken CherryPy. Please
fix &lt;span class="caps"&gt;ASAP&lt;/span&gt;&lt;/em&gt; would be helpful. I think for everyone who has write permission to the repository it
makes sense to receive such notification. At very least the one who publishes CherryPy to the
Cheese Shop should have rigid requirement to check &lt;span class="caps"&gt;CI&lt;/span&gt;&amp;nbsp;page.&lt;/p&gt;
&lt;p&gt;The same concern, with lesser strain, applies to the ReadTheDocs builds, which all have failed
for last 3 months &lt;a class="footnote-reference" href="#id50" id="id22"&gt;[19]&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;This is the problem which needs to be solved as soon as possible. In 2014 I tried the &lt;span class="caps"&gt;CI&lt;/span&gt; service
called Drone.io, which supports the three major code hosing services. Well, it turned out to be
usable (build page example &lt;a class="footnote-reference" href="#id51" id="id23"&gt;[20]&lt;/a&gt;). I can help assist or set it up for CherryPy, once flow and
role questions are&amp;nbsp;answered.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="barrier-to-entry"&gt;
&lt;h3&gt;Barrier to&amp;nbsp;entry&lt;/h3&gt;
&lt;p&gt;Hopefully, one of biggest barriers I was facing myself seems to have passed. I mean that
documentation link erosion where almost every single CherryPy-related link on the web leaded
to 404 page. I think ReadTheDocs adoption is huge improvement, especially considering that new
documentation, being clearly a better one, lacks some details that were covered in version 3.3.
Having persistent, versioned and indexable documentation is very&amp;nbsp;helpful.&lt;/p&gt;
&lt;p&gt;I think, it should be stated more or less directly that if one wants a framework to
make application design decisions it&amp;#8217;s better off using something else, not CherryPy. Okay,
here&amp;#8217;s Joe who knows enough about Python. He&amp;#8217;s conscious and is willing to make application design
decisions and take responsibility for them. But he also has a deadline. Joe doesn&amp;#8217;t expect his
first real-world CherryPy application to be shiny from experienced CherryPy developer point of
view, albeit functional from end-user&amp;#8217;s. What&amp;#8217;s the best way to help Joe create functional
prototype quickly, and let him gradually improve the design and codebase of his application
thereafter, along with CherryPy&amp;nbsp;knowledge?&lt;/p&gt;
&lt;p&gt;Let&amp;#8217;s look at CherryPy knowledge as a tree. Each branch represents a functionality. Say a CherryPy
tool branch. As it goes from the root, there&amp;#8217;re nodes like &lt;em&gt;Copy-paste this decorator to turn your
handler result into &lt;span class="caps"&gt;JSON&lt;/span&gt;&lt;/em&gt;, &lt;em&gt;What other tools are capable of?&lt;/em&gt;, &lt;em&gt;What is a CherryPy tool, and how
to write one?&lt;/em&gt;, &lt;em&gt;Tool hook-points, types, priorities, etc.&lt;/em&gt;. The same is for other branches —
from basic to advanced. And what will Joe benefit from the most is breadth-first traversing on the
tree. Better if some of a few first levels are illustrated with runnable snippets. This way it&amp;#8217;s
possible to give overview, tunable blocks of code and directions for improvement. For the later is
important to spur creativity, but not enforcing design decisions, in the way: &lt;em&gt;start with this,
improve by yourself&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;I&amp;#8217;ve just glanced over the latest documentation over again, and surprise, it narrates in very
similar fashion! Sylvain has done a great job. But what is catching my eye is that there&amp;#8217;s two
separate tutorial series now: one in the source code &lt;a class="footnote-reference" href="#id52" id="id24"&gt;[21]&lt;/a&gt;, another in the documentation &lt;a class="footnote-reference" href="#id53" id="id25"&gt;[22]&lt;/a&gt;.
Even though both serve the same purpose, the former is much easier to maintain, covering with
tests which leads to the bonus of better regression test suite. If the later&amp;#8217;s textual flexibility
can be sacrificed for less maintenance cost, I think, it&amp;#8217;s possible to generate tutorial section
from module docstrings and code, and merge the&amp;nbsp;two.&lt;/p&gt;
&lt;p&gt;However, both series deal with simplest cases and there&amp;#8217;s a big gap between them and real-world
application requirements. Here I also a snippet to base on directly or get ideas from to
get a prototype faster, but with real-world tools. Like safe database access in threaded
environment, &lt;span class="caps"&gt;URL&lt;/span&gt; routing and generation, multi-allplication design and so on. Sylvain&amp;#8217;s CherryPy
recipes &lt;a class="footnote-reference" href="#id54" id="id26"&gt;[23]&lt;/a&gt; referred in the documentation several times may answer this to some extent, but in
general I think it&amp;#8217;s still in question. But also whether it should maintained in centralized
fashion within CherryPy. Or if it&amp;#8217;s more suitable for external source, like post on somebody&amp;#8217;s
blog or&amp;nbsp;StackOverflow.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="contribution-packages"&gt;
&lt;h3&gt;Contribution&amp;nbsp;packages&lt;/h3&gt;
&lt;p&gt;I think Eric Larson&amp;#8217;s suggestion about &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;cherrypy-*&lt;/span&gt;&lt;/tt&gt; pageages makes sense. For example there&amp;#8217;s
a package called cherrys &lt;a class="footnote-reference" href="#id55" id="id27"&gt;[24]&lt;/a&gt; which is the good example. It&amp;#8217;s a Redis adapter for
CherryPy session handling. What&amp;#8217;s important it has clearly defined scope and doesn&amp;#8217;t affect
an application design. This way it might have been named &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;cherrypy-redis-session&lt;/span&gt;&lt;/tt&gt;.&lt;/p&gt;
&lt;p&gt;A package that is an implementaion of an CherryPy interface must be appropriate for packaging and
sharing. Sharing a CherryPy application with clearly defined scope, like Dowser &lt;a class="footnote-reference" href="#id56" id="id28"&gt;[25]&lt;/a&gt;, is
also&amp;nbsp;beneficial.&lt;/p&gt;
&lt;p&gt;On the contrary specific tools or plugins that affect application design shouldn&amp;#8217;t be packaged.
For instance, looking at Sylvain&amp;#8217;s recipes they are all better to say recipes. Let&amp;#8217;s look
at Jinja2 recipe &lt;a class="footnote-reference" href="#id57" id="id29"&gt;[26]&lt;/a&gt;. Besides having a class per file isn&amp;#8217;t pythonic, and splitting a thing
into plugin and tool isn&amp;#8217;t always rewarding, it makes certian assumtion on how template file
names are defined for the &lt;span class="caps"&gt;URL&lt;/span&gt;. Jina2 is a flexible template engine with notion of template
inheritance. Real-world projects have dozens of templates and tend to orgianize them by role
and use. Having some convention-over-configuration rule for template name is very handy.
So I want to say it&amp;#8217;s usually better to have a small and effiective application-specific
CherryPy tool or plugin rather than trying to invent something overly broad and generic&amp;nbsp;one.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="other-changes"&gt;
&lt;h3&gt;Other&amp;nbsp;changes&lt;/h3&gt;
&lt;p&gt;Bitbucket pages should have either actual supplementing information or no supplementing
information. Menu community link &lt;a class="footnote-reference" href="#id58" id="id30"&gt;[27]&lt;/a&gt; leads to wiki home page where version section has latest
version of 3.2.5. Menu development link &lt;a class="footnote-reference" href="#id59" id="id31"&gt;[28]&lt;/a&gt;, which points to the repository overview page refers
to Python 2.3, where the site&amp;#8217;s version starts with 2.5+. Referring &lt;tt class="docutils literal"&gt;python setup.py install&lt;/tt&gt; as
installation command, which requires a manual download, ought to be changed to normal
&lt;tt class="docutils literal"&gt;pip install cherrypy&lt;/tt&gt;.&lt;/p&gt;
&lt;p&gt;I think, minor changes on the website are&amp;nbsp;needed:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;no need in download link in the&amp;nbsp;menu,&lt;/li&gt;
&lt;li&gt;add &lt;em&gt;install&lt;/em&gt; section after &lt;em&gt;features&lt;/em&gt; which says &lt;tt class="docutils literal"&gt;pip install cherrypy&lt;/tt&gt;,&lt;/li&gt;
&lt;li&gt;community link is better to refer to the user&amp;nbsp;group,&lt;/li&gt;
&lt;li&gt;rename &lt;em&gt;online tests&lt;/em&gt;, change the service link, use better sentence which will emphasize&amp;nbsp;reliability.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;About &lt;tt class="docutils literal"&gt;_cp&lt;/tt&gt; module prefix. There&amp;#8217;re some public classes like &lt;tt class="docutils literal"&gt;cherrypy._cptools.HandlerTool&lt;/tt&gt;,
which may be used in user code directly or as super classes. This isn&amp;#8217;t correct naming, according
to the underscore convention. So it&amp;#8217;s a good idea to change it some day. Though as a PyDev user I
can&amp;#8217;t recall to any issues with&amp;nbsp;it.&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="wrap-up"&gt;
&lt;h2&gt;Wrap-up&lt;/h2&gt;
&lt;p&gt;I can conclude that there&amp;#8217;re two things that I&amp;#8217;m seeing problematic: quality assurance for various
claimed enviroments CherryPy to support and lack of contribution guiadance, specifically in the
form of bug tracker supervisor. The rest feels normal, if that makes sense and as it is usually
the case needs small constant improment and&amp;nbsp;polishing.&lt;/p&gt;
&lt;hr class="docutils" /&gt;
&lt;table class="docutils footnote" frame="void" id="id32" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[1]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id1"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id15"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="http://cherrypy.org/"&gt;http://cherrypy.org/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id33" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://groups.google.com/forum/#!topic/cherrypy-users/lT1cxovGyy8"&gt;https://groups.google.com/forum/#!topic/cherrypy-users/lT1cxovGyy8&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id34" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://groups.google.com/forum/#!topic/cherrypy-users/MjrIZ_jljRQ"&gt;https://groups.google.com/forum/#!topic/cherrypy-users/MjrIZ_jljRQ&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id35" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[4]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id4"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id13"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="http://nichol.as/benchmark-of-python-web-servers"&gt;http://nichol.as/benchmark-of-python-web-servers&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id36" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[5]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id5"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id18"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="http://stackoverflow.com/questions/tagged/cherrypy"&gt;http://stackoverflow.com/questions/tagged/cherrypy&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id37" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id6"&gt;[6]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/cherrypy/cherrypy/wiki/ZenOfCherryPy"&gt;https://bitbucket.org/cherrypy/cherrypy/wiki/ZenOfCherryPy&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id38" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id7"&gt;[7]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.python.org/dev/peps/pep-0020/"&gt;https://www.python.org/dev/peps/pep-0020/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id39" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id8"&gt;[8]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/cherrypy/cherrypy/issue/1331"&gt;https://bitbucket.org/cherrypy/cherrypy/issue/1331&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id40" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id9"&gt;[9]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://github.com/niwibe/django-sse"&gt;https://github.com/niwibe/django-sse&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id41" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id10"&gt;[10]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://lincolnloop.com/blog/architecting-realtime-applications/"&gt;https://lincolnloop.com/blog/architecting-realtime-applications/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id42" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id11"&gt;[11]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://stackoverflow.com/q/25845161/2072035"&gt;http://stackoverflow.com/q/25845161/2072035&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id43" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id12"&gt;[12]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://ws4py.readthedocs.org/"&gt;https://ws4py.readthedocs.org/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id44" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id14"&gt;[13]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://docs.python.org/2/howto/sockets.html#performance"&gt;https://docs.python.org/2/howto/sockets.html#performance&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id45" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id16"&gt;[14]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/saaj/cherrypy-webapp-skeleton"&gt;https://bitbucket.org/saaj/cherrypy-webapp-skeleton&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id46" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id17"&gt;[15]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/saaj/qooxdoo-website-skeleton"&gt;https://bitbucket.org/saaj/qooxdoo-website-skeleton&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id47" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id20"&gt;[16]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/cherrypy/cherrypy/issue/1298"&gt;https://bitbucket.org/cherrypy/cherrypy/issue/1298&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id48" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id19"&gt;[17]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://blog.jaraco.com/2014/04/how-to-write-perfect-pull-request.html"&gt;http://blog.jaraco.com/2014/04/how-to-write-perfect-pull-request.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id49" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id21"&gt;[18]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://shiningpanda.com/shiningpanda-ci-clap-de-fin.html"&gt;http://shiningpanda.com/shiningpanda-ci-clap-de-fin.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id50" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id22"&gt;[19]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://readthedocs.org/builds/cherrypy"&gt;https://readthedocs.org/builds/cherrypy&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id51" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id23"&gt;[20]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://drone.io/saaj/qooxdoo-cherrypy-json-rpc"&gt;https://drone.io/saaj/qooxdoo-cherrypy-json-rpc&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id52" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id24"&gt;[21]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/cherrypy/cherrypy/src/tip/cherrypy/tutorial/?at=default"&gt;https://bitbucket.org/cherrypy/cherrypy/src/tip/cherrypy/tutorial/?at=default&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id53" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id25"&gt;[22]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://cherrypy.readthedocs.org/en/latest/tutorials.html"&gt;http://cherrypy.readthedocs.org/en/latest/tutorials.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id54" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id26"&gt;[23]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/Lawouach/cherrypy-recipes"&gt;https://bitbucket.org/Lawouach/cherrypy-recipes&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id55" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id27"&gt;[24]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://pypi.python.org/pypi/cherrys"&gt;https://pypi.python.org/pypi/cherrys&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id56" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id28"&gt;[25]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.aminus.net/wiki/Dowser"&gt;http://www.aminus.net/wiki/Dowser&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id57" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id29"&gt;[26]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/Lawouach/cherrypy-recipes/src/tip/web/templating/jinja2_templating/?at=default"&gt;https://bitbucket.org/Lawouach/cherrypy-recipes/src/tip/web/templating/jinja2_templating/?at=default&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id58" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id30"&gt;[27]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/cherrypy/cherrypy/wiki/Home"&gt;https://bitbucket.org/cherrypy/cherrypy/wiki/Home&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id59" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id31"&gt;[28]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://bitbucket.org/cherrypy/cherrypy/overview"&gt;https://bitbucket.org/cherrypy/cherrypy/overview&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
</content><category term="cherrypy"></category><category term="python"></category><category term="future"></category></entry><entry><title>Qooxdoo “next” and targeting website development</title><link href="https://recollection.saaj.me/article/qooxdoo-next-and-targeting-website-development.html" rel="alternate"></link><published>2014-12-08T00:00:00+01:00</published><updated>2014-12-08T00:00:00+01:00</updated><author><name>saaj</name></author><id>tag:recollection.saaj.me,2014-12-08:/article/qooxdoo-next-and-targeting-website-development.html</id><summary type="html">&lt;p&gt;Initially, this post was meant to be an open letter to Martin Wittemann, an urge to do not break the only
robust and consistent JavaScript development environment, in sense of bug #8618 [1]. How robust and
consistent? Most importantly it is&amp;nbsp;about:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Code structure: modern feature-rich object model with properties, events, interfaces, mixins and more,
namespace tree,&amp;nbsp;data-binding,&lt;/li&gt;
&lt;li&gt;Dependency management: automatic and transparent dependency resolution, component&amp;nbsp;packaging,&lt;/li&gt;
&lt;li&gt;Deployment: code optimisation, single- and multi-part&amp;nbsp;builds,&lt;/li&gt;
&lt;li&gt;Standard library: i18n, &lt;span class="caps"&gt;CLDR&lt;/span&gt;, high level &lt;span class="caps"&gt;BOM&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt;, normalised events, normalised ECMAScript, test
infrastructure, &lt;span class="caps"&gt;API&lt;/span&gt; documentation, feature&amp;nbsp;detection.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All this comes out-of-the-box, builds layer on layer, is seamlessly integrated and handed to an end developer
on a silver platter. The codebase is one of the best quality I have ever seen. It is readable,
comprehensible, possible to extend, and is a huge a knowledge resource per se. And note, I am talking about
tools and environment, not &lt;span class="caps"&gt;UI&lt;/span&gt;. Like former Qooxdoo developer, Fabian Jakobs, stated in one of his
presentations [2]:&lt;/p&gt;
&lt;blockquote&gt;
Don’t just look at the widgets, look at the development tools as well.&lt;/blockquote&gt;
&lt;p&gt;Luckily, as was recently stated in &lt;em&gt;Website fundamentals: next challenges&lt;/em&gt; [3], the “next” (branch and then a
separate repo [4]; later referenced Next) turned out to be a experimental website-oriented fork of Qooxdoo,
which doesn’t mean to be merged back to Qooxdoo. At least any time soon. Panic is soothed, impending doom
doesn’t overhang the good times. It’s time to dissect and&amp;nbsp;discuss.&lt;/p&gt;
</summary><content type="html">&lt;p&gt;Initially, this post was meant to be an open letter to Martin Wittemann, an urge to do not break the only
robust and consistent JavaScript development environment, in sense of bug #8618 &lt;a class="footnote-reference" href="#id22" id="id1"&gt;[1]&lt;/a&gt;. How robust and
consistent? Most importantly it is&amp;nbsp;about:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Code structure: modern feature-rich object model with properties, events, interfaces, mixins and more,
namespace tree,&amp;nbsp;data-binding,&lt;/li&gt;
&lt;li&gt;Dependency management: automatic and transparent dependency resolution, component&amp;nbsp;packaging,&lt;/li&gt;
&lt;li&gt;Deployment: code optimisation, single- and multi-part&amp;nbsp;builds,&lt;/li&gt;
&lt;li&gt;Standard library: i18n, &lt;span class="caps"&gt;CLDR&lt;/span&gt;, high level &lt;span class="caps"&gt;BOM&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt;, normalised events, normalised ECMAScript, test
infrastructure, &lt;span class="caps"&gt;API&lt;/span&gt; documentation, feature&amp;nbsp;detection.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All this comes out-of-the-box, builds layer on layer, is seamlessly integrated and handed to an end developer
on a silver platter. The codebase is one of the best quality I have ever seen. It is readable,
comprehensible, possible to extend, and is a huge a knowledge resource per se. And note, I am talking about
tools and environment, not &lt;span class="caps"&gt;UI&lt;/span&gt;. Like former Qooxdoo developer, Fabian Jakobs, stated in one of his
presentations &lt;a class="footnote-reference" href="#id23" id="id2"&gt;[2]&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;
Don’t just look at the widgets, look at the development tools as well.&lt;/blockquote&gt;
&lt;p&gt;Luckily, as was recently stated in &lt;em&gt;Website fundamentals: next challenges&lt;/em&gt; &lt;a class="footnote-reference" href="#id24" id="id3"&gt;[3]&lt;/a&gt;, the “next” (branch and then a
separate repo &lt;a class="footnote-reference" href="#id25" id="id4"&gt;[4]&lt;/a&gt;; later referenced Next) turned out to be a experimental website-oriented fork of Qooxdoo,
which doesn’t mean to be merged back to Qooxdoo. At least any time soon. Panic is soothed, impending doom
doesn’t overhang the good times. It’s time to dissect and&amp;nbsp;discuss.&lt;/p&gt;

&lt;div class="section" id="the-bug-report"&gt;
&lt;h2&gt;The bug&amp;nbsp;report&lt;/h2&gt;
&lt;p&gt;What was meant to break? The bug report mandated to rewrite Qooxdoo test infrastructure with callback
spaghetti of mocha.js. Callback spaghetti?! When most code structure is a callback in callback in callback
and so on, with the state expressed in free closure variables spanning across nesting levels, it is a
callback spaghetti. And for average JavaScript code out there this structure practice is very common, which
is nothing good though, but for Qooxdoo it might have been an awful decision, replacing clean,
object-oriented code with that. But since Next is not Qooxdoo it may depend on some&amp;nbsp;factors.&lt;/p&gt;
&lt;p&gt;In the report Martin&amp;nbsp;told:&lt;/p&gt;
&lt;blockquote&gt;
There has been some hard decisions for me to take lately but I&amp;#8217;m still convinced that not trying to be
the expert for everything gives us the opportunity to focus on our strengths.&lt;/blockquote&gt;
&lt;p&gt;And listed some features that supposedly impressed&amp;nbsp;him:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;pretty&amp;nbsp;printing,&lt;/li&gt;
&lt;li&gt;scaffolding over to &lt;span class="caps"&gt;XML&lt;/span&gt;&amp;nbsp;handling,&lt;/li&gt;
&lt;li&gt;Levenshtein distance&amp;nbsp;calculation,&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;GUI&lt;/span&gt;&amp;nbsp;toolkits.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;First is simple, second is needed only for recorded tests (e.g. Selenium), third is already there with
&lt;tt class="docutils literal"&gt;qx.util.EditDistance&lt;/tt&gt;. And forth is quite a thing Qooxdoo was made for. So these features are kind of
false&amp;nbsp;argument.&lt;/p&gt;
&lt;p&gt;Moreover, &lt;tt class="docutils literal"&gt;mocha.js&lt;/tt&gt; is 5k &lt;span class="caps"&gt;SLOC&lt;/span&gt;, &lt;tt class="docutils literal"&gt;qx.dev.unit&lt;/tt&gt; is 4.4k &lt;span class="caps"&gt;SLOC&lt;/span&gt;, &lt;tt class="docutils literal"&gt;testrunner&lt;/tt&gt; is 3.9k &lt;span class="caps"&gt;SLOC&lt;/span&gt;. Including these
numbers and taking into account more expressive Qooxdoo code, either Martin is trading more for less, or it
is just an expensive&amp;nbsp;experiment.&lt;/p&gt;
&lt;p&gt;Anyway framework consistency is absolutely a strength. Test code is a canonical source of thruth about an &lt;span class="caps"&gt;API&lt;/span&gt;.
Test code is required for maintaining a complex application (which most websites are not though). And test
infrastructure should not be compared to a disposal widget like &lt;tt class="docutils literal"&gt;qx.ui.embed.HtmlArea&lt;/tt&gt; which wasn’t quality
in first place, required special knowledge to maintain, and most importantly it isn’t hard create a wrapper
for TinyMCE implementing &lt;tt class="docutils literal"&gt;qx.ui.form.IStringForm&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;qx.ui.form.IForm&lt;/tt&gt;. Since that moment it is
transparent and seamlessly integrated. Making test code an alien in a framework, makes it seamful,
inconsistent, and what is worst can discourage developers from writing tests. Well, of course unless all the
rest code is the&amp;nbsp;spaghetti.&lt;/p&gt;
&lt;p&gt;Last note here is how framework consistency from language perspective changes in the course of recent time.
Qooxdoo &lt;tt class="docutils literal"&gt;branch_3_0_x&lt;/tt&gt; roughly has nothing but Qooxdoo JavaScript code for a browser and Python code for
generator. Next has Qooxdoo JavaScript code and mocha.js callback spaghetti test code for a browser, CommonJS
code for generator that varies being object-oriented prototypal code, procedural code, and Qooxdoo
object-oriented code, which also doesn’t exclude some Python code still necessary for new generator to
function. So what’s clear, having hard time maintaining this is one of likely&amp;nbsp;outcomes.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="goals"&gt;
&lt;h2&gt;Goals&lt;/h2&gt;
&lt;p&gt;Andreas Ecker wrote in the post &lt;a class="footnote-reference" href="#id24" id="id5"&gt;[3]&lt;/a&gt;, that they
&lt;em&gt;have been looking into the challenges of a more lightweight,  website-oriented approach to web applications&lt;/em&gt;.
Sounds quite ambiguous and curious at the same time. Lightweight in byte size? Website-oriented in a
non-widget &lt;span class="caps"&gt;UI&lt;/span&gt;? At the same time, it sounds as odd as arithmetic approach to differential equations. Then he&amp;nbsp;expanded:&lt;/p&gt;
&lt;blockquote&gt;
Reasons are two-fold: primarily, we are involved in company-internal projects in that very domain.
Secondly, by touching core components of qooxdoo we can experiment with benefits of web technologies and
concepts available today.&lt;/blockquote&gt;
&lt;p&gt;First thing, considering 1&amp;amp;1’s “very domain”, is probably something like Amazon &lt;span class="caps"&gt;AWS&lt;/span&gt; management &lt;span class="caps"&gt;UI&lt;/span&gt;. A
web-application which spans multiple pages, where each page has some &lt;span class="caps"&gt;HTML&lt;/span&gt; from server and complementing
JavaScript code. So it may be looked at as a “website-oriented approach”. Second thing again points to an&amp;nbsp;experiment.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="qxweb"&gt;
&lt;h2&gt;qxWeb&lt;/h2&gt;
&lt;p&gt;Qooxdoo targets websites since 2.0 which was released in June 2012 &lt;a class="footnote-reference" href="#id26" id="id6"&gt;[5]&lt;/a&gt;. Initially I was enthusiastic about
qxWeb, I thought it’ll be another virtue of the framework, and I still think the idea to target websites is
right as even at mid-scale they benefit from most of Qooxdoo non-&lt;span class="caps"&gt;UI&lt;/span&gt; features. But the design and development
of qxWeb seems to me misguided and mostly pointless&amp;nbsp;now.&lt;/p&gt;
&lt;p&gt;A year after qxWeb release, the time I was looking at it from various angles, trying to find its strength and
way to integrate it in my flow to make it handle infrastructure task I used to handle with Qooxdoo in case of
web application, I submitted the bug report &lt;a class="footnote-reference" href="#id27" id="id7"&gt;[6]&lt;/a&gt; that proposes design that enables website development with
Qooxdoo (native application type &lt;a class="footnote-reference" href="#id35" id="id8"&gt;[14]&lt;/a&gt;), points qxWeb shortcomings and its possible shortcut-purpose, and the
problems to be solved for normal operation. The biggest problem with qxWeb is that it wasn’t designed to be a
Qooxdoo thing, but rather be jQuery with &lt;span class="caps"&gt;API&lt;/span&gt; done right. Long story told short: it’s not for writing an
application, it’s for scribbling ad hoc per page scripts. And I would prefer qxWeb over jQuery for writing
one-off script, but in general jQuery is de facto standard in &lt;span class="caps"&gt;DOM&lt;/span&gt; patching and I don’t see any significant
niche for qxWeb as yet another &lt;span class="caps"&gt;DOM&lt;/span&gt; patcher plus this and that. Moreover even for shortcut-purpose in a
Qooxdoo application, qxWeb is still an alien, because it needs manual dependency declaration for its
“modules”. Qooxdoo way of &lt;span class="caps"&gt;DOM&lt;/span&gt; patching was &lt;tt class="docutils literal"&gt;qx.bom.Collection&lt;/tt&gt; &lt;a class="footnote-reference" href="#id28" id="id9"&gt;[7]&lt;/a&gt;, removed in version 3.0. I think it was
a&amp;nbsp;mistake.&lt;/p&gt;
&lt;p&gt;Starting with Qooxdoo 3.5 &lt;a class="footnote-reference" href="#id29" id="id10"&gt;[8]&lt;/a&gt;, there was an addition to qxWeb. Since then it supported low-level widgets.
Such widget expects certain &lt;span class="caps"&gt;HTML&lt;/span&gt; structure and classes, which later is animated on widget instance
construction. Basically, qxWeb started to mimic jQueryUI as well, not just jQuery. Not that I want to say
it’s useless, but rather it’s reinvention of the wheel, reimplementation of a task that was already
implemented long ago, and all who wanted such functionality already have it at hand. What’s worse the
reimplementation is being done with outdated technology. &lt;span class="caps"&gt;DOM&lt;/span&gt; binding, &lt;span class="caps"&gt;MVVM&lt;/span&gt; &lt;a class="footnote-reference" href="#id30" id="id11"&gt;[9]&lt;/a&gt;, libraries emerged in
2009-2010, and at the moment of Qooxdoo 3.5 release, late 2013, were a commodity, as well as understanding
that a lot of manual &lt;span class="caps"&gt;DOM&lt;/span&gt; patching is a chore, that can be automated with a view model. Bug #8101 &lt;a class="footnote-reference" href="#id31" id="id12"&gt;[10]&lt;/a&gt; lead
me to working KnockoutJS &lt;a class="footnote-reference" href="#id32" id="id13"&gt;[11]&lt;/a&gt; adapter for Qooxdoo, but at the framework side it didn’t gain traction, except
that Martin confirmed importance of the topic, and that also it’s the topic of his master thesis &lt;a class="footnote-reference" href="#id33" id="id14"&gt;[12]&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;I want to summarise this section quoting Martin’s &lt;em&gt;focus on our strengths&lt;/em&gt;. How the whole thing looks to me
now, is that in area of website development Qooxdoo team tries to avoid its strengths at all costs. No build
tools, no view model, no application design having great tools, data layer and application design experience.
Instead Qooxdoo team is toying a&amp;nbsp;jQuery/jQueryUI.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="a-qooxdoo-website-today"&gt;
&lt;h2&gt;A Qooxdoo website&amp;nbsp;today&lt;/h2&gt;
&lt;p&gt;Even though initial changes in Next doesn’t seem anything promising, I’m interested what the experiment may
yield. But to address the need of a robust development environment for a website today I want to point again
to the bug #7481 &lt;a class="footnote-reference" href="#id27" id="id15"&gt;[6]&lt;/a&gt; and its proof of concept project &lt;a class="footnote-reference" href="#id34" id="id16"&gt;[13]&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Eventually all blocking bugs for #7481 were resolved and a Qooxdoo native application &lt;a class="footnote-reference" href="#id35" id="id17"&gt;[14]&lt;/a&gt; since 3.5.1 was
already fine as a basis for a website codebase. From my side I’ve been experimenting with the approach on a
few real websites and result are good. When all website’s JavaScript code is centralised, it really starts to
feel like an application, thusly significantly improving maintainability. Late autumn 2014 I’ve written
documentation for the project, backported minor improvements and made it&amp;nbsp;standalone.&lt;/p&gt;
&lt;p&gt;qooxdoo-website-skeleton &lt;a class="footnote-reference" href="#id34" id="id18"&gt;[13]&lt;/a&gt; is a mixture of a tutorial and skeleton. Because initial configuration is
tricky, when starting from a scratch with native application, it’s easier to fork and remove a few unused
files. Here I would like to tell about two aspects that Andreas pointed to (see above), and for the rest
you can follow the link. First, a website approach to &lt;span class="caps"&gt;UI&lt;/span&gt; here is to provide a choice. One can use qxWeb
&lt;span class="caps"&gt;DOM&lt;/span&gt; patching facilities or qxWeb widgets. One can use KnockoutJS adapter and use Qooxdoo data models as
view models (&lt;span class="caps"&gt;MVVM&lt;/span&gt;). There’s also no restriction for using jQuery or KnockoutJS as is or whatever it makes
sense to&amp;nbsp;use.&lt;/p&gt;
&lt;p&gt;Second, is being lightweight. Without external libraries Qooxdoo native application build with basic qxWeb
modules is over 100 kB gzipped. For example boot part for the project is 361/110 kB (optimised/&lt;tt class="docutils literal"&gt;gzip &lt;span class="pre"&gt;-6&lt;/span&gt;&lt;/tt&gt;).
I would agree it is a little too much for a website yesterday, but today I don’t see this concern reasonable.
Here are three hello pages for popular social websites requested from desktop&amp;nbsp;Firefox.&lt;/p&gt;
&lt;table border="1" class="docutils"&gt;
&lt;colgroup&gt;
&lt;col width="43%" /&gt;
&lt;col width="29%" /&gt;
&lt;col width="29%" /&gt;
&lt;/colgroup&gt;
&lt;thead valign="bottom"&gt;
&lt;tr&gt;&lt;th class="head"&gt;website&lt;/th&gt;
&lt;th class="head"&gt;size, kB&lt;/th&gt;
&lt;th class="head"&gt;requests&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td&gt;twitter.com&lt;/td&gt;
&lt;td&gt;156&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;linkedin.com&lt;/td&gt;
&lt;td&gt;180&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;facebook.com&lt;/td&gt;
&lt;td&gt;615&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;And here are home pages for big e-commerce&amp;nbsp;websites.&lt;/p&gt;
&lt;table border="1" class="docutils"&gt;
&lt;colgroup&gt;
&lt;col width="41%" /&gt;
&lt;col width="30%" /&gt;
&lt;col width="30%" /&gt;
&lt;/colgroup&gt;
&lt;thead valign="bottom"&gt;
&lt;tr&gt;&lt;th class="head"&gt;website&lt;/th&gt;
&lt;th class="head"&gt;size, kB&lt;/th&gt;
&lt;th class="head"&gt;requests&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td&gt;ebay.com&lt;/td&gt;
&lt;td&gt;114&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;alibaba.com&lt;/td&gt;
&lt;td&gt;143&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;amazon.com&lt;/td&gt;
&lt;td&gt;560&lt;/td&gt;
&lt;td&gt;30&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;These show that having roughly over 100 kB is a norm today. And if further optimisation of Next will go as
feature-size trade-off, then it wouldn’t probably make&amp;nbsp;sense.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="epilogue"&gt;
&lt;h2&gt;Epilogue&lt;/h2&gt;
&lt;p&gt;Before official announcement Next’s bug #8618 was in line with other dubious decisions taken since Qooxdoo
3.5. Before 3.5 I could agree to virtually every decision taken by the team, as there was clear reason,
argument and pragmatic approach&amp;nbsp;overall.&lt;/p&gt;
&lt;p&gt;Now it’s mixed. There’s still fundamental work like migration to pointer events, but also several other topics
that look like a waste of time. Like what? Like rewriting the generator to NodeJS and GruntJS &lt;a class="footnote-reference" href="#id36" id="id19"&gt;[15]&lt;/a&gt;.
Surprise! There’s ongoing shift to gulp.js and GruntJS is no longer cool JavaScript thiggie. And I’m curious
to see how Qooxdoo team will handle such situations in general. Fancy toys change in the public sandbox fast
&lt;a class="footnote-reference" href="#id38" id="id20"&gt;[17]&lt;/a&gt;. Will you try to catch up every new cool replacement or will maintain an unpopular&amp;nbsp;one?&lt;/p&gt;
&lt;p&gt;Like, unconditional &lt;span class="caps"&gt;RESOLVED&lt;/span&gt; &lt;span class="caps"&gt;NEVER&lt;/span&gt; &lt;a class="footnote-reference" href="#id37" id="id21"&gt;[16]&lt;/a&gt; set to all old bug. No prioritization, there is no past, the future&amp;#8217;s
so bright. The team briefly state something about turning to an agile development, and happily turned to
developing new features. And of course, no doubt developing features is fun, &lt;span class="caps"&gt;PR&lt;/span&gt; is easier with less open
bugs, and resources are always limited. But when new developers stumble the same bugs again and re-report
them, it is a failure. I don’t really understand it, so since that change I’m just discouraged to file new
bug and I don’t. If no one fixes them, and it takes time to analyse, make it reproducible and propose a
solution, I probably shouldn&amp;#8217;t,&amp;nbsp;right?&lt;/p&gt;
&lt;p&gt;I can whine more, but I want to point to two origins of priorities. It’s understanding the scope and
understanding the users. First, is &lt;em&gt;a universal JavaScript framework&lt;/em&gt;. How universal? And what is JavaScript
here? JavaScript like in Netscape? Or like ECMAScript in NodeJS? Should we expect Qooxdoo approach to &lt;span class="caps"&gt;URL&lt;/span&gt;
routing or database connection pooling? In second, I see opposite sides. One commenters in various places
complain that Qooxdoo &lt;em&gt;doesn’t play well with JavaScript ecosystem&lt;/em&gt;. I don’t think they are Qooxdoo users at
all, but ones who couldn&amp;#8217;t handle it. You know, you can’t do your Qooxdoo work copy-pasting from
StackOverflow, you can do jQuery work this way. On the other side are users, including me, who&amp;nbsp;say:&lt;/p&gt;
&lt;blockquote&gt;
Dear Qooxdoo, we don’t really give a damn about JavaScript ecosystem, you are our only concern. We come
from various established application development platforms, in sense “Swing/qt/Cocoa for the web”.  We
value design and maintainability. Please, keep on avoiding us from it.&lt;/blockquote&gt;
&lt;p&gt;That is to say. I’m an optimist, and there’s nothing parallel to Qooxdoo. So I hope the team won’t forget its
strengths and its users, and will keep on the path of solving real&amp;nbsp;problems.&lt;/p&gt;
&lt;hr class="docutils" /&gt;
&lt;table class="docutils footnote" frame="void" id="id22" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://bugzilla.qooxdoo.org/show_bug.cgi?id=8618"&gt;http://bugzilla.qooxdoo.org/show_bug.cgi?id=8618&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id23" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.slideshare.net/fjakobs/masterin-large-scale-java-script-applications"&gt;http://www.slideshare.net/fjakobs/masterin-large-scale-java-script-applications&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id24" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[3]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id3"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id5"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="http://news.qooxdoo.org/website-fundamentals-next-challenges"&gt;http://news.qooxdoo.org/website-fundamentals-next-challenges&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id25" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id4"&gt;[4]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://github.com/qooxdoo/next"&gt;http://github.com/qooxdoo/next&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id26" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id6"&gt;[5]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://news.qooxdoo.org/qooxdoo-2-0-released"&gt;http://news.qooxdoo.org/qooxdoo-2-0-released&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id27" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[6]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id7"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id15"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="http://bugzilla.qooxdoo.org/show_bug.cgi?id=7481"&gt;http://bugzilla.qooxdoo.org/show_bug.cgi?id=7481&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id28" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id9"&gt;[7]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://demo.qooxdoo.org/2.0.4/apiviewer/#qx.bom.Collection"&gt;http://demo.qooxdoo.org/2.0.4/apiviewer/#qx.bom.Collection&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id29" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id10"&gt;[8]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://news.qooxdoo.org/qooxdoo-3-5-released"&gt;http://news.qooxdoo.org/qooxdoo-3-5-released&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id30" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id11"&gt;[9]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://en.wikipedia.org/wiki/Model_View_ViewModel"&gt;http://en.wikipedia.org/wiki/Model_View_ViewModel&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id31" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id12"&gt;[10]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://bugzilla.qooxdoo.org/show_bug.cgi?id=8101"&gt;http://bugzilla.qooxdoo.org/show_bug.cgi?id=8101&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id32" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id13"&gt;[11]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://knockoutjs.com/"&gt;http://knockoutjs.com/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id33" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id14"&gt;[12]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.slideshare.net/wittemann/data-binding-in-qooxdoo"&gt;http://www.slideshare.net/wittemann/data-binding-in-qooxdoo&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id34" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[13]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id16"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id18"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="https://bitbucket.org/saaj/qooxdoo-website-skeleton"&gt;https://bitbucket.org/saaj/qooxdoo-website-skeleton&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id35" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;[14]&lt;/td&gt;&lt;td&gt;&lt;em&gt;(&lt;a class="fn-backref" href="#id8"&gt;1&lt;/a&gt;, &lt;a class="fn-backref" href="#id17"&gt;2&lt;/a&gt;)&lt;/em&gt; &lt;a class="reference external" href="http://manual.qooxdoo.org/3.5.1/pages/development/skeletons.html#native"&gt;http://manual.qooxdoo.org/3.5.1/pages/development/skeletons.html#native&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id36" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id19"&gt;[15]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://news.qooxdoo.org/tool-chain-the-next-step"&gt;http://news.qooxdoo.org/tool-chain-the-next-step&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id37" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id21"&gt;[16]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://www.eclipsezone.com/eclipse/forums/t83053.html"&gt;http://www.eclipsezone.com/eclipse/forums/t83053.html&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;table class="docutils footnote" frame="void" id="id38" rules="none"&gt;
&lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
&lt;tbody valign="top"&gt;
&lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#id20"&gt;[17]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="http://readwrite.com/2014/12/04/node-js-fork-io-js"&gt;http://readwrite.com/2014/12/04/node-js-fork-io-js&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
</content><category term="qooxdoo"></category><category term="javascript"></category><category term="future"></category></entry></feed>