feat(merge): update indexing manifest merging and indexing to use com… (#57)

* feat(merge): update indexing manifest merging and indexing to use common field names

* Apply automatic documentation changes

* feat(tests): update func name in test

* Apply automatic documentation changes

* Apply automatic documentation changes

* fix(indexing): correct error, allow contining after error for merge

* Apply automatic documentation changes

* fix(tests): make sure acl column in output of merge

* Apply automatic documentation changes

* Apply automatic documentation changes

* Apply automatic documentation changes

* chore(indexing): consolidate standard names into single file

* feat(merging): keep non-standard columns and merge correctly, update tests

* chore(indexing): use enumerate instead of manually incrementing

* fix(indexing): don't reprint errors on every row if only one row has error

* Apply automatic documentation changes

* feat(merge): only add new values if not already existing

* Apply automatic documentation changes

* feat(merge): support edge-case where there are multiple existing GUIDs for same md5

* chore(docs): update docstring for clarity

* Apply automatic documentation changes

* fix(manifest merge): dont merge for multiple guids (#58)

* fix(manifest merge): dont merge for multiple guids

* test(manifest merge): sort tsv data by (md5, guid)

Co-authored-by: John McCann <[email protected]>

* Apply automatic documentation changes

* feat(merge): determine delimeter based on file ext

* Apply automatic documentation changes

Co-authored-by: Alexander VT <[email protected]>
Co-authored-by: John McCann <[email protected]>

Author: 3 people

README.md

@@ -1,6 +1,6 @@
 # Gen3 SDK for Python
 
-The Gen3 PSDK for Python provides classes and functions for handling common tasks when interacting with a Gen3 commons. The API for a commons can be overwhelming, so this SDK aims
+The Gen3 SDK for Python provides classes and functions for handling common tasks when interacting with a Gen3 commons. The API for a commons can be overwhelming, so this SDK aims
 to simplify communication with various microservices in a clear Python package.
 
 The docs here contain general descriptions of the different pieces of the SDK and example scripts. For detailed API documentation, see the link below:

docs/_build/doctrees/environment.pickle

@@ -77,8 +77,8 @@ <h1>Source code for gen3.auth</h1><div class="highlight"><pre>
         <span class="n">namespace</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;NAMESPACE&quot;</span><span class="p">)</span>
         <span class="k">if</span> <span class="n">namespace</span><span class="p">:</span>
             <span class="c1"># attempt to get a token from the workspace-token-service</span>
-            <span class="bp">self</span><span class="o">.</span><span class="n">_wts_url</span> <span class="o">=</span> <span class="s2">&quot;http://workspace-token-service.</span><span class="si">{}</span><span class="s2">.svc.cluster.local&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
-                <span class="n">namespace</span>
+            <span class="bp">self</span><span class="o">.</span><span class="n">_wts_url</span> <span class="o">=</span> <span class="p">(</span>
+                <span class="s2">&quot;http://workspace-token-service.</span><span class="si">{}</span><span class="s2">.svc.cluster.local&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">namespace</span><span class="p">)</span>
             <span class="p">)</span>
             <span class="n">resp</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;</span><span class="si">{}</span><span class="s2">/token/&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_wts_url</span><span class="p">))</span>
             <span class="k">if</span> <span class="n">resp</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">200</span><span class="p">:</span>

docs/_build/doctrees/tools/indexing.doctree

@@ -47,10 +47,12 @@ <h1>Source code for gen3.jobs</h1><div class="highlight"><pre>
 
 <span class="kn">from</span> <span class="nn">gen3.utils</span> <span class="kn">import</span> <span class="n">append_query_params</span><span class="p">,</span> <span class="n">DEFAULT_BACKOFF_SETTINGS</span>
 
+<span class="c1"># sower&#39;s &quot;action&quot; mapping to the relevant job</span>
 <span class="n">INGEST_METADATA_JOB</span> <span class="o">=</span> <span class="s2">&quot;ingest-metadata-manifest&quot;</span>
 <span class="n">DBGAP_METADATA_JOB</span> <span class="o">=</span> <span class="s2">&quot;get-dbgap-metadata&quot;</span>
 <span class="n">INDEX_MANIFEST_JOB</span> <span class="o">=</span> <span class="s2">&quot;index-object-manifest&quot;</span>
 <span class="n">DOWNLOAD_MANIFEST_JOB</span> <span class="o">=</span> <span class="s2">&quot;download-indexd-manifest&quot;</span>
+<span class="n">MERGE_MANIFEST_JOB</span> <span class="o">=</span> <span class="s2">&quot;merge-manifests&quot;</span>
 
 
 <div class="viewcode-block" id="Gen3Jobs"><a class="viewcode-back" href="../../jobs.html#gen3.jobs.Gen3Jobs">[docs]</a><span class="k">class</span> <span class="nc">Gen3Jobs</span><span class="p">:</span>

docs/_build/doctrees/tools/metadata.doctree

@@ -76,8 +76,7 @@ <h1>Source code for gen3.submission</h1><div class="highlight"><pre>
         <span class="bp">self</span><span class="o">.</span><span class="n">_endpoint</span> <span class="o">=</span> <span class="n">endpoint</span>
 
     <span class="k">def</span> <span class="nf">__export_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">output</span><span class="p">):</span>
-        <span class="sd">&quot;&quot;&quot;Writes an API response to a file.</span>
-<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="sd">&quot;&quot;&quot;Writes an API response to a file.&quot;&quot;&quot;</span>
         <span class="n">outfile</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s2">&quot;w&quot;</span><span class="p">)</span>
         <span class="n">outfile</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">output</span><span class="p">)</span>
         <span class="n">outfile</span><span class="o">.</span><span class="n">close</span>
@@ -86,9 +85,7 @@ <h1>Source code for gen3.submission</h1><div class="highlight"><pre>
     <span class="c1">### Program functions</span>
 
 <div class="viewcode-block" id="Gen3Submission.get_programs"><a class="viewcode-back" href="../../submission.html#gen3.submission.Gen3Submission.get_programs">[docs]</a>    <span class="k">def</span> <span class="nf">get_programs</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
-        <span class="sd">&quot;&quot;&quot;List registered programs</span>
-
-<span class="sd">        &quot;&quot;&quot;</span>
+        <span class="sd">&quot;&quot;&quot;List registered programs&quot;&quot;&quot;</span>
         <span class="n">api_url</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_endpoint</span><span class="si">}</span><span class="s2">/api/v0/submission/&quot;</span>
         <span class="n">output</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">api_url</span><span class="p">,</span> <span class="n">auth</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_auth_provider</span><span class="p">)</span>
         <span class="n">output</span><span class="o">.</span><span class="n">raise_for_status</span><span class="p">()</span>