Object Manifest CLI functionality

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 4.3.0

Notes to tag/commit: 4.4.0

Generated: 2021-12-02

New Features

• objects CLI with support for calling functions to manage file object
  manifests (functions already available in SDK) (#97)
• Add Aggregate metadata service support for discovery read: --agg flag to
  enable (#99)
• Adding a configure command which lets the user add a profile name and
  attach a credentials file (#93)

Improvements

• Print the API call output when unable to parse it to JSON (#102)
• The "Gen3File.get_presigned_url" function does not have a default
  "protocol" anymore. If no protocol is specified, rely on Fence and Indexd's
  default behavior, which is to use the first URL they find (#91)

Discovery Metadata Functions and Improved Manfiest Merging

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 4.2.0

Notes to tag/commit: d82eeea

Generated: 2021-05-10

New Features

• ability to specify columns that should be treated like arrays (so values
  are combined correctly from various sources) during bucket manifest merging
  (#85)
• add discovery read, discovery publish, delete sub commands to CLI (#83)

Breaking Changes

• Commas are no longer allowed as additional delimiters in a space-delimited
  array for indexing manifests (this was causing a bug for valid file names
  containing commas) (#85)

Bug Fixes

• Include retry logic for caching refreshed access tokens. (#88)
• Fix merging edge cases where multiple rows of same hash, diff guids,
  weren't merging data as expected (#85)
• Fixed issue where an indexing manifest was considered invalid if it had
  commas in filenames in the URL column (#85)
• manifest and ingestion tools break with commas in file names, not anymore
  (#84)
• fix bug where certain fields are not dicts, but we're generically trying to
  do a comparison. AttributeError: 'str' object has no attribute 'keys'
  before, now downstream function handles this case by doing a generic ==
  comparison (#82)
• requests.HTTPError not available in utils b/c missing import (#81)

Improvements

• Added notes for Python set up and manual testing in
  devTest.md
  (#88)
• Additional unit tests for Gen3Auth / Gen3Submission (#88)
• refactor merge code to be more readable and handle edge cases (#85)
• Unit tests for indexing manifest functionality better reflect edge cases
  around spaces & commas in URLs, duplicates, etc. (#85)
• breakout utility function for getting urls from manifest string (#84)
• Functions print error details instead of just exiting when a request fails
  (#79)

Dependency Updates

• Include using backoff lib (#88)

support prev_guid in indexing tooling

4.2.0

feat(indexing): new optional prev_guid column support to create new v…

Gen3Query module

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 4.0.0

Notes to tag/commit: 4.1.0

Generated: 2021-01-13

New Features

• Add Gen3Query module with "query", "graphql_query" and "raw_data_download"
  functions (#70)

Gen3 CLI + Gen3WsStorage module

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 3.1.0

Notes to tag/commit: 4.0.0

Generated: 2021-01-13

Improvements

• use GEN3_API_KEY environment variable - default to
  ~/.gen3/credentials.json (#72)
• derive endpoint from api key (#72)
• add access-token cache (#72)
• add access-token expiration check (#72)
• add workspace storage class and cli integration (#72)
• add auth cli integration (#72)
• arrange markdown docs (#72)
• add Jenkins smoke test (#72)

"delete_records", "delete_node" and "delete_nodes" functions

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 3.0.0

Notes to tag/commit: 3.1.0

Generated: 2020-12-14

New Features

• Add Gen3Submission "delete_records", "delete_node" and "delete_nodes"
  functions (#68)

Improvements

• fix auto generation of docs (#64)

New build tool, auto PyPI deploys, skeleton for CLI

Release Notes

For: uc-cdis/gen3sdk-python
Notes since tag: 2.7.0
Notes to tag/commit: 3.0.0
Generated: 2020-12-03

New Features

• initial CLI skeleton (#63)
• CLI automatically supports pypfb and subcommands with "gen3 pfb" (#63)
• Added ingestion tool for bundles given a manifest. (#60)

Improvements

• use poetry (#63)
• automatic PyPI deploys on master, automatic TestPyPI deploys on branches
  (#63)
• "Gen3Submission.submit_file" improvements: do not fail silently in case of
  unexpected status code, add logs to troubleshoot failures (#59)

Dependency Updates

• updated to latest versions of many deps (#63)

Manifest Merging updates

Release Notes

For: uc-cdis/gen3sdk-python
Notes since tag: 2.6.0
Notes to tag/commit: 4a6e50f
Generated: 2020-10-19

New Features

• Add new is_valid_manifest_format function to gen3.tools.indexing module
  which validates the contents of a manifest of file objects and logs all
  errors found. Each logged error message includes a description along with
  the line number and column in which the error occurred. Validation of md5,
  file size, url, and authz values is supported. (#56)

Improvements

• Indexing manifest merging now handles any indexing column name and outputs
  standard column names (#57)
• Refactor so standard column names in more consolidated location (#57)
• even if the "indexing file" doesn't include GUIDs, this script can still be
  used to combine 2 manifests based on a common column (#50)
• Handle updating a blank record's "authz" and "urls" fields (#54)
• more allowed values for hash column (#49)

WTS auth

Release Notes

For: uc-cdis/gen3sdk-python

Notes since tag: 2.5.0

Notes to tag/commit: 2.6.0

Generated: 2020-08-13

New Features

• When working in a Gen3 Workspace, default to getting access tokens from the
  Workspace Token Service instead of requiring creds (#55)

Deployment Changes

• Depends on the hatchery app being deployed with "env": { "NAMESPACE": <namespace> } (#55)

Bucket manifest merging functionality

chore(hooks): add precommit hooks cfg, add security baseline (#53)

* chore(hooks): add precommit hooks cfg, add security baseline

* Apply automatic documentation changes

Co-authored-by: Alexander VT <[email protected]>