From 26e8ab84eaf77dc483ac5fb08c2ccf13d9cd49f4 Mon Sep 17 00:00:00 2001
From: Maxim Moinat <maxim@thehyve.nl>
Date: Wed, 20 May 2020 20:42:55 +0200
Subject: [PATCH] add detailed installation instructions

---
 docs/RabbitInAHat.html        |  8 +++----
 docs/RabbitInAHat.md          |  6 ++++--
 docs/ReadMe.html              |  8 +++----
 docs/ReadMe.md                |  5 ++++-
 docs/WhiteRabbit.html         | 40 +++++++++++++++++++++++++++++------
 docs/WhiteRabbit.md           | 18 ++++++++++++++--
 docs/best_practices.html      |  2 +-
 docs/images/favicon.html      |  2 +-
 docs/index.html               |  2 +-
 docs/riah_test_framework.html | 10 ++++-----
 10 files changed, 74 insertions(+), 27 deletions(-)

diff --git a/docs/RabbitInAHat.html b/docs/RabbitInAHat.html
index 544c9f35..f38cb4a6 100644
--- a/docs/RabbitInAHat.html
+++ b/docs/RabbitInAHat.html
@@ -11,7 +11,7 @@
 
 
 
-<title>RabbitInAHat.utf8.md</title>
+<title>Rabbit in a Hat</title>
 
 <script src="site_libs/jquery-1.11.3/jquery.min.js"></script>
 <meta name="viewport" content="width=device-width, initial-scale=1" />
@@ -25,7 +25,7 @@
 <script src="site_libs/navigation-1.1/tabsets.js"></script>
 <link href="site_libs/highlightjs-9.12.0/default.css" rel="stylesheet" />
 <script src="site_libs/highlightjs-9.12.0/highlight.js"></script>
-<link rel='shortcut icon' type='image/x-icon' href='favicon.ico' />
+<link rel='shortcut icon' type='image/x-icon' href='images/favicon.ico' />
 
 <style type="text/css">code{white-space: pre;}</style>
 <style type="text/css">
@@ -417,7 +417,7 @@ <h2>Process Overview</h2>
 </div>
 <div id="installation-and-support" class="section level2">
 <h2>Installation and support</h2>
-<p>Rabbit-In-a-Hat comes with WhiteRabbit, refer to <a href="WhiteRabbit.md">WhiteRabbit’s installation section</a>.</p>
+<p>Rabbit-In-a-Hat comes with WhiteRabbit, refer to step 1 and 2 of <a href="WhiteRabbit.html#installation">WhiteRabbit’s installation section</a>.</p>
 </div>
 </div>
 <div id="getting-started" class="section level1">
@@ -437,7 +437,7 @@ <h2>Selecting Desired CDM Version</h2>
 <p>Rabbit-In-a-Hat allows you to select which CDM version (v4, v5 or v6) you’d like to built your ETL specification against.</p>
 <p>See the graphic below for how to select your desired CDM: <img src="http://i.imgur.com/LOqhp7H.gif" alt="Switching between CDMv4 and CDMv5" /></p>
 <p>The CDM version can be changed at any time, but beware that some of your existing mappings may be lost in the process. By default, Rabbit-In-a-Hat will attempt to pereserve as many mappings between the source data and the newly selected CDM as possible. When a new CDM is selected, Rabbit-In-a-Hat will drop any mappings if the mapping’s CDM table or CDM column name no longer exist</p>
-<p>For instance, switching from CDMv4 to CDMv5, a mapping from source to CDM person.person_source_value will be kept because the person table has person_source_value in both CDMv4 and CDMv5. However, person.assocaited_provider_id exists only in CDMv4 and has been renamed to <a href="http://www.ohdsi.org/web/wiki/doku.php?id=documentation:cdm:person">person.provider_id in CDMv5</a> and so that mapping will not be kept when switching between these two CDMs.</p>
+<p>For instance, switching from CDMv4 to CDMv5, a mapping from source to CDM person.person_source_value will be kept because the person table has person_source_value in both CDMv4 and CDMv5. However, person.associated_provider_id exists only in CDMv4 and has been renamed to <a href="https://github.com/OHDSI/CommonDataModel/wiki/PERSON">person.provider_id in CDMv5</a> and so that mapping will not be kept when switching between these two CDMs.</p>
 </div>
 <div id="loading-in-a-custom-cdm" class="section level2">
 <h2>Loading in a Custom CDM</h2>
diff --git a/docs/RabbitInAHat.md b/docs/RabbitInAHat.md
index e48f6cb2..5e3e960d 100644
--- a/docs/RabbitInAHat.md
+++ b/docs/RabbitInAHat.md
@@ -1,4 +1,6 @@
-
+---
+pagetitle: "Rabbit in a Hat"
+---
 ![](images/rabbitinahatlogo.png)
 
 # Introduction
@@ -16,7 +18,7 @@ The typical sequence for using this software to generate documentation of an ETL
 5. Save Rabbit-In-a-Hat work and export to a MS Word document.
 
 ## Installation and support
-Rabbit-In-a-Hat comes with WhiteRabbit, refer to [WhiteRabbit's installation section](WhiteRabbit.md).
+Rabbit-In-a-Hat comes with WhiteRabbit, refer to step 1 and 2 of [WhiteRabbit's installation section](WhiteRabbit.html#installation).
 
 # Getting Started
 ## Creating a New Document
diff --git a/docs/ReadMe.html b/docs/ReadMe.html
index 7f6b39dd..b5cacf27 100644
--- a/docs/ReadMe.html
+++ b/docs/ReadMe.html
@@ -11,7 +11,7 @@
 
 
 
-<title>ReadMe.utf8.md</title>
+<title>ReadMe</title>
 
 <script src="site_libs/jquery-1.11.3/jquery.min.js"></script>
 <meta name="viewport" content="width=device-width, initial-scale=1" />
@@ -25,7 +25,7 @@
 <script src="site_libs/navigation-1.1/tabsets.js"></script>
 <link href="site_libs/highlightjs-9.12.0/default.css" rel="stylesheet" />
 <script src="site_libs/highlightjs-9.12.0/highlight.js"></script>
-<link rel='shortcut icon' type='image/x-icon' href='favicon.ico' />
+<link rel='shortcut icon' type='image/x-icon' href='images/favicon.ico' />
 
 <style type="text/css">code{white-space: pre;}</style>
 <style type="text/css">
@@ -404,8 +404,8 @@
 </div>
 
 
-<div id="documentation" class="section level1">
-<h1>Documentation</h1>
+<div id="white-rabbit-documentation-readme" class="section level1">
+<h1>White Rabbit Documentation Readme</h1>
 <p>This folder contains the raw and rendered documentation of WhiteRabbit.</p>
 <div id="contribute" class="section level2">
 <h2>Contribute</h2>
diff --git a/docs/ReadMe.md b/docs/ReadMe.md
index bcfe266c..04496170 100644
--- a/docs/ReadMe.md
+++ b/docs/ReadMe.md
@@ -1,4 +1,7 @@
-# Documentation
+---
+pagetitle: "ReadMe"
+---
+# White Rabbit Documentation Readme
 This folder contains the raw and rendered documentation of WhiteRabbit.
 
 ## Contribute
diff --git a/docs/WhiteRabbit.html b/docs/WhiteRabbit.html
index b0432216..427da63d 100644
--- a/docs/WhiteRabbit.html
+++ b/docs/WhiteRabbit.html
@@ -11,7 +11,7 @@
 
 
 
-<title>WhiteRabbit.utf8.md</title>
+<title>White Rabbit</title>
 
 <script src="site_libs/jquery-1.11.3/jquery.min.js"></script>
 <meta name="viewport" content="width=device-width, initial-scale=1" />
@@ -25,7 +25,7 @@
 <script src="site_libs/navigation-1.1/tabsets.js"></script>
 <link href="site_libs/highlightjs-9.12.0/default.css" rel="stylesheet" />
 <script src="site_libs/highlightjs-9.12.0/highlight.js"></script>
-<link rel='shortcut icon' type='image/x-icon' href='favicon.ico' />
+<link rel='shortcut icon' type='image/x-icon' href='images/favicon.ico' />
 
 <style type="text/css">code{white-space: pre;}</style>
 <style type="text/css">
@@ -409,7 +409,7 @@
 <h1>Introduction</h1>
 <div id="scope-and-purpose" class="section level2">
 <h2>Scope and purpose</h2>
-<p>WhiteRabbit is a software tool to help prepare for ETLs (Extraction, Transformation, Loading) of longitudinal healthcare databases into the <a href="www.github.com/OHDSI/CommonDataModel/wiki">Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM)</a>. The source data can be in comma-separated text files, SAS files, or in a database (MySQL, SQL Server, Oracle, PostgreSQL, Microsoft APS, Microsoft Access, Amazon RedShift, PDW, Teradata, Google BigQuery). Note that for support of the OHDSI analytical tooling, the OMOP CDM will need to be in one of a limited set of database platforms (SQL Server, Oracle, PostgreSQL, Microsoft APS, Amazon RedShift, Google BigQuery, Impala).</p>
+<p>WhiteRabbit is a software tool to help prepare for ETLs (Extraction, Transformation, Loading) of longitudinal health care databases into the <a href="https://github.com/OHDSI/CommonDataModel">Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM)</a>. The source data can be in comma-separated text files, SAS files, or in a database (MySQL, SQL Server, Oracle, PostgreSQL, Microsoft APS, Microsoft Access, Amazon RedShift, PDW, Teradata, Google BigQuery). Note that for support of the OHDSI analytical tooling, the OMOP CDM will need to be in one of a limited set of database platforms (SQL Server, Oracle, PostgreSQL, Microsoft APS, Amazon RedShift, Google BigQuery, Impala).</p>
 <p>WhiteRabbit’s main function is to perform a scan of the source data, providing detailed information on the tables, fields, and values that appear in a field. This scan will generate a report that can be used as a reference when designing the ETL, for instance by using the Rabbit-In-a-Hat tool. White Rabbit differs from standard data profiling tools in that it attempts to prevent the display of personally identifiable information (PII) data values in the generated output data file.</p>
 </div>
 <div id="process-overview" class="section level2">
@@ -420,10 +420,25 @@ <h2>Process Overview</h2>
 </div>
 <div id="installation-and-support" class="section level1">
 <h1>Installation and support</h1>
-<p>All source code and installation instructions available on GitHub: <a href="https://github.com/OHDSI/WhiteRabbit" class="uri">https://github.com/OHDSI/WhiteRabbit</a></p>
+<div id="installation" class="section level2">
+<h2>Installation</h2>
+<ol style="list-style-type: decimal">
+<li>Download the latest version of WhiteRabbit from Github: <a href="https://github.com/OHDSI/WhiteRabbit/releases/latest" class="uri">https://github.com/OHDSI/WhiteRabbit/releases/latest</a>.<br />
+The packaged application can be found at the bottom of the page under assets, in a file called <em>WhiteRabbit_vX.X.X.zip</em> (where <code>X.X.X</code> is the latest version).</li>
+<li>Unzip the download</li>
+<li>Double-click on bin/whiteRabbit.bat on Windows to start WhiteRabbit, and bin/whiteRabbit on macOS and Linux.<br />
+See <a href="#running_from_the_command_line">Running from the command line</a> for details on how to run from the command line instead.</li>
+<li>Go to <a href="#using_the_application_functions">Using the Application Functions</a> for detailed instructions on how to make a scan of your data.</li>
+</ol>
+<p>Note: on releases earlier than version 0.8.0, open the respective WhiteRabbit.jar or RabbitInAHat.jar files instead.</p>
+</div>
+<div id="support" class="section level2">
+<h2>Support</h2>
+<p>All source code, descriptions and input/output examples are available on GitHub: <a href="https://github.com/OHDSI/WhiteRabbit" class="uri">https://github.com/OHDSI/WhiteRabbit</a></p>
 <p>Any bugs/issues/enhancements should be posted to the GitHub repository: <a href="https://github.com/OHDSI/WhiteRabbit/issues" class="uri">https://github.com/OHDSI/WhiteRabbit/issues</a></p>
 <p>Any questions/comments/feedback/discussion can be posted on the OHDSI Developer Forum: <a href="http://forums.ohdsi.org/c/developers" class="uri">http://forums.ohdsi.org/c/developers</a></p>
 </div>
+</div>
 <div id="using-the-application-functions" class="section level1">
 <h1>Using the Application Functions</h1>
 <div id="specifying-the-location-of-source-data" class="section level2">
@@ -466,14 +481,14 @@ <h4>Oracle</h4>
 <li><em><strong>Server location:</strong></em> this field contains the SID, service name, and optionally the port: <code>&lt;host&gt;/&lt;sid&gt;</code>, <code>&lt;host&gt;:&lt;port&gt;/&lt;sid&gt;</code>, <code>&lt;host&gt;/&lt; service name &gt;</code>, or <code>&lt;host&gt;:&lt;port&gt;/&lt;service name&gt;</code></li>
 <li><em><strong>User name:</strong></em> name of the user used to log into the server</li>
 <li><em><strong>Password:</strong></em> password for the supplied user name</li>
-<li><em><strong>Database name:</strong></em> this field contains the schema (i.e. ‘user’ in Oracle terms) containing the tables</li>
+<li><em><strong>Database name:</strong></em> this field contains the schema (i.e. ‘user’ in Oracle terms) containing the tables</li>
 </ul>
 </div>
 <div id="sql-server" class="section level4">
 <h4>SQL Server</h4>
 <ul>
 <li><em><strong>Server location:</strong></em> the name or IP address of the server running SQL Server. You can also specify the port (ex: <code>&lt;host&gt;:&lt;port&gt;</code>), which defaults to 1433.</li>
-<li><em><strong>User name:</strong></em> name of the user used to log into the server. Optionally, the domain can be specified as <code>&lt;domain&gt;/&lt;user&gt;</code> (e.g. ‘MyDomain/Joe’)</li>
+<li><em><strong>User name:</strong></em> name of the user used to log into the server. Optionally, the domain can be specified as <code>&lt;domain&gt;/&lt;user&gt;</code> (e.g. ‘MyDomain/Joe’)</li>
 <li><em><strong>Password:</strong></em> password for the supplied user name</li>
 <li><em><strong>Database name:</strong></em> name of the database containing the tables</li>
 </ul>
@@ -523,6 +538,7 @@ <h3>Performing the Scan</h3>
 <li>“Rows per table” is an option when scanning field values. By default, WhiteRabbit will random 1 million rows in the table. There are other options to review 100,000 or all rows within the table.</li>
 </ul></li>
 <li>Unchecking the “Scan field values” tells WhiteRabbit to not review or report on any of the raw data items.</li>
+<li>Checking the “Numeric Stats” box will include numeric statistics. See the section on <a href="#numeric-statistics-(develop)">‘Numerical Statistics’</a>.</li>
 </ul>
 <p>Once all settings are completed, press the “Scan tables” button. After the scan is completed the report will be written to the working folder.</p>
 </div>
@@ -539,6 +555,18 @@ <h3>Reading the Scan</h3>
 <p>For a tab that describes a single table, the columns names from the source table (or delimited text file) will be across the columns of the Excel tab. Each source table column will generate two columns in the Excel. One column will list all distinct values that have a “Min cell count” greater than what was set at time of the scan (Section - Performing the Scan). If a list of unique values was truncated, the last value in the list will be “List truncated”; this indicates that there are one or more additional unique source values that appear less than the number entered in the “Min cell count” described in Section - Performing the Scan. Next to each distinct value will be a second column that contains the frequency, or the number of times that value occurs in the data. These two columns (distinct values and frequency) will repeat for all the source columns in the table profiled in the workbook.</p>
 <p><img src="images/whiterabbitscreen-readingthescanex.png" /></p>
 <p>The report is powerful in understanding your source data by highlighting what exists. For example, the above results were given back on the “SEX” column within one of the tables scanned, we can see that there were two common values (1 and 2) that appeared 61,491 and 35,401 times respectively. WhiteRabbit will not define 1 as male and 2 as female; the data holder will typically need to define source codes unique to the source system. However these two values (1 &amp; 2) are not the only values present in the data because we see this list was truncated. These other values appear with very low frequency (defined by “Min cell count”) and often represent incorrect or highly suspicious values. When generating an ETL we should not only plan to handle the high-frequency gender concepts 1 and 2 but the other low-frequency values that exist within this column.</p>
+<div id="numerical-statistics-develop" class="section level4">
+<h4>Numerical statistics (develop)</h4>
+<p>If the option for numerical statistics is checked, then a set of statistics is calculated for all integer, real and date data types. The following statistics are added to the Overview sheet:</p>
+<ul>
+<li>Average</li>
+<li>Standard Deviation (sampled)</li>
+<li>Minimum</li>
+<li>Maximum</li>
+<li>Quartiles: q1, median, q3 (sampled)</li>
+</ul>
+<p>If the number of values is smaller than the set reservoir size, then the standard deviation and three quartile boundaries are the exact population statistics. Otherwise, the statistics are approximated based on a representative sample. The average, minimum and maximum are always true population statistics. For dates, the numerical statistics are calculated by using epoch days. The standard deviation of dates is given in days, the other statistics are converted to a date representation.</p>
+</div>
 </div>
 </div>
 <div id="generating-fake-data" class="section level2">
diff --git a/docs/WhiteRabbit.md b/docs/WhiteRabbit.md
index 6430a023..ddaa6a2c 100644
--- a/docs/WhiteRabbit.md
+++ b/docs/WhiteRabbit.md
@@ -1,4 +1,6 @@
-
+---
+pagetitle: "White Rabbit"
+---
 ![](images/whiterabbitlogo.png )
 
 # Introduction
@@ -22,7 +24,19 @@ Once the scan report is created, this report can then be used in the Rabbit-In-a
 
 # Installation and support
 
-All source code and installation instructions available on GitHub: <https://github.com/OHDSI/WhiteRabbit>
+## Installation
+
+1. Download the latest version of WhiteRabbit from Github: <https://github.com/OHDSI/WhiteRabbit/releases/latest>.  
+   The packaged application can be found at the bottom of the page under assets, in a file called _WhiteRabbit_vX.X.X.zip_ (where `X.X.X` is the latest version).
+2. Unzip the download
+3. Double-click on bin/whiteRabbit.bat on Windows to start WhiteRabbit, and bin/whiteRabbit on macOS and Linux.  
+   See [Running from the command line](#running_from_the_command_line) for details on how to run from the command line instead.
+4. Go to [Using the Application Functions](#using_the_application_functions) for detailed instructions on how to make a scan of your data.
+
+Note: on releases earlier than version 0.8.0, open the respective WhiteRabbit.jar or RabbitInAHat.jar files instead.
+
+## Support
+All source code, descriptions and input/output examples are available on GitHub: <https://github.com/OHDSI/WhiteRabbit>
 
 Any bugs/issues/enhancements should be posted to the GitHub repository: <https://github.com/OHDSI/WhiteRabbit/issues>
 
diff --git a/docs/best_practices.html b/docs/best_practices.html
index 06a37a7a..1775d53f 100644
--- a/docs/best_practices.html
+++ b/docs/best_practices.html
@@ -22,7 +22,7 @@
 <script src="site_libs/navigation-1.1/tabsets.js"></script>
 <link href="site_libs/highlightjs-9.12.0/default.css" rel="stylesheet" />
 <script src="site_libs/highlightjs-9.12.0/highlight.js"></script>
-<link rel='shortcut icon' type='image/x-icon' href='favicon.ico' />
+<link rel='shortcut icon' type='image/x-icon' href='images/favicon.ico' />
 
 <style type="text/css">code{white-space: pre;}</style>
 <style type="text/css">
diff --git a/docs/images/favicon.html b/docs/images/favicon.html
index d13aa17a..2c87f2b0 100644
--- a/docs/images/favicon.html
+++ b/docs/images/favicon.html
@@ -1 +1 @@
-<link rel='shortcut icon' type='image/x-icon' href='favicon.ico' />
\ No newline at end of file
+<link rel='shortcut icon' type='image/x-icon' href='images/favicon.ico' />
\ No newline at end of file
diff --git a/docs/index.html b/docs/index.html
index 688bf6ae..355f6190 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -22,7 +22,7 @@
 <script src="site_libs/navigation-1.1/tabsets.js"></script>
 <link href="site_libs/highlightjs-9.12.0/default.css" rel="stylesheet" />
 <script src="site_libs/highlightjs-9.12.0/highlight.js"></script>
-<link rel='shortcut icon' type='image/x-icon' href='favicon.ico' />
+<link rel='shortcut icon' type='image/x-icon' href='images/favicon.ico' />
 
 <style type="text/css">code{white-space: pre;}</style>
 <style type="text/css">
diff --git a/docs/riah_test_framework.html b/docs/riah_test_framework.html
index b80f3768..ca3b3c72 100644
--- a/docs/riah_test_framework.html
+++ b/docs/riah_test_framework.html
@@ -25,7 +25,7 @@
 <script src="site_libs/navigation-1.1/tabsets.js"></script>
 <link href="site_libs/highlightjs-9.12.0/default.css" rel="stylesheet" />
 <script src="site_libs/highlightjs-9.12.0/highlight.js"></script>
-<link rel='shortcut icon' type='image/x-icon' href='favicon.ico' />
+<link rel='shortcut icon' type='image/x-icon' href='images/favicon.ico' />
 
 <style type="text/css">code{white-space: pre;}</style>
 <style type="text/css">
@@ -436,19 +436,19 @@ <h2>Available functions</h2>
 <p>The test framework defines the following functions <strong>for each table in the source schema</strong>:</p>
 <ul>
 <li><code>get_defaults_&lt;table name&gt;</code> shows the default field values that will be used when creating a record in the table. At the start, these default values have been taken from the White-Rabbit scan report, using the most frequent value.</li>
-<li><code>set_defaults_&lt;table name&gt;</code> can be used to change the default values of one or more fields in the table. For example <code>set_defaults_enrollment(enrollment_date = "2000-01-01")</code>.</li>
-<li><code>add_&lt;table name&gt;</code> can be used to specify that a record should be created in the table. The arguments can be used to specify field values. For fields where the user doesn’t specify a value, the default value is used. For example <code>add_enrollment(member_id = "M00000001")</code>.</li>
+<li><code>set_defaults_&lt;table name&gt;</code> can be used to change the default values of one or more fields in the table. For example <code>set_defaults_enrollment(enrollment_date = &quot;2000-01-01&quot;)</code>.</li>
+<li><code>add_&lt;table name&gt;</code> can be used to specify that a record should be created in the table. The arguments can be used to specify field values. For fields where the user doesn’t specify a value, the default value is used. For example <code>add_enrollment(member_id = &quot;M00000001&quot;)</code>.</li>
 </ul>
 <p>The following functions are defined <strong>for each table in the CDM schema</strong>:</p>
 <ul>
-<li><code>expect_&lt;table name&gt;</code> can be used to state the expectation that at least one record with the defined properties should exist in the table. For example <code>expect_person(person_id = 1, person_source_value = "M00000001")</code>.</li>
+<li><code>expect_&lt;table name&gt;</code> can be used to state the expectation that at least one record with the defined properties should exist in the table. For example <code>expect_person(person_id = 1, person_source_value = &quot;M00000001&quot;)</code>.</li>
 <li><code>expect_no_&lt;table name&gt;</code> can be used to state the expectation that no record with the defined properties should exist in the table. For example <code>expect_no_condition_occurrence(person_id = 1)</code>.</li>
 <li><code>expect_count_&lt;table name&gt;</code> can be used to state the expectation that a specific number of records with the defined properties should exist in the table. For example <code>expect_count_condition_occurrence(person_id = 1, rowCount = 3)</code>.</li>
 <li><code>lookup_&lt;table name&gt;</code> can be used to get a specific value from another table. For example to get the <code>person_id</code> by <code>person_source_value</code>.</li>
 </ul>
 <p>One further function is available:</p>
 <ul>
-<li><code>declareTest</code> is used to group multiple statements under a single identifier. For example <code>declareTest(id = 1, description = "Test person ID")</code>.</li>
+<li><code>declareTest</code> is used to group multiple statements under a single identifier. For example <code>declareTest(id = 1, description = &quot;Test person ID&quot;)</code>.</li>
 </ul>
 </div>
 <div id="defining-unit-tests" class="section level2">