Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docs for dts-flavored bulk_specification endpoint #214

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 26 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -819,21 +819,27 @@ Error Connecting to auth service ...
### Parse bulk specifications

This endpoint parses one or more bulk specification files in the staging area
into a data
structure (close to) ready for insertion into the Narrative bulk import or
analysis cell.
into a data structure (close to) ready for insertion into the Narrative bulk
import or analysis cell.

It can parse `.tsv`, `.csv`, and Excel (`.xls` and `.xlsx`) files. Templates for
the currently
supported data types are available in
By default, it can parse `.tsv`, `.csv`, and Excel (`.xls` and `.xlsx`) files.
Templates for the currently supported data types are available in
the [templates](./import_specifications/templates)
directory of this repo. See
the [README.md](./import_specifications/templates/README.md) file
for instructions on template usage.

When given the `dts` flag in the URL, this endpoint can also parse manifest
files that come from the KBase [Data Transfer Service](https://github.com/kbase/dts) (DTS)
See [here](https://kbase.github.io/dts) for further documentation on the service.
This service copies data files from external sources into a KBase user's staging
area, accompanied by a `manifest.json` file. These also contain information on how to
load files into KBase importer apps. However, since they are a very specific format,
this means adding a `dts` flag to the URL. When this flag is present, all files
are expected to be `.json` files and conform to the [DTS schema](./import_specifications/schema/dts_manifest_schema.json).

See the [import specification ADR document](./docs/import_specifications.ADR.md)
for design
details.
for design details.

**URL** : `ci.kbase.us/services/staging_service/bulk_specification`

Expand All @@ -847,15 +853,24 @@ details.

**Code** : `200 OK`

**Content example**
**Content examples**

```
GET bulk_specification/?files=file1.<ext>[,file2.<ext>,...]
```

`<ext>` is one of `csv`, `tsv`, `xls`, or `xlsx`.

Reponse:

```
GET bulk_specification/?files=file1.json[,file2.json,...]&dts
```

When using the `dts` flag, all files must be JSON and have the `.json` extension.

Both versions of the endpoint respond in the same format.

Response:

```
{
Expand Down Expand Up @@ -1041,7 +1056,7 @@ POST write_bulk_specification/
- `data` contains any data to be written to the file as example data, and is analogous to the data structure returned from the parse endpoint. To specify that no data should be written to the template provide an empty list.
- `<value for ID, row N>` is the value for the input for a given `spec.json` ID and import or analysis instance, where an import/analysis instance is effectively a row in the data file. Each data file row is provided in order for each type. Each row is provided in a mapping of `spec.json` ID to the data for the row. Lines > 3 in the templates are user-provided data, and each line corresponds to a single import or analysis.

Reponse:
Response:

```
{
Expand Down
6 changes: 5 additions & 1 deletion import_specifications/readme.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,6 @@
This directory contains templates for bulk import specifications to be parsed by the
import specfication endpoint, and example code for how to generate them automatically.
import specfication endpoint, and example code for how to generate them automatically.

This also contains a subdirectory for storing JSON Schemas that describe JSON-based
import specifications. These should be referenced by deployment and testing configuration
Comment on lines +4 to +6
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm.. now I think about it, I wanted to delete the import_specifications folder as it was no longer needed since the narrative generates them on the fly. They were a stop gap up until that point.

See #178

Does it make sense for the schemas to live here or somewhere with a better name?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the rest of the folder is going away, I don't see a reason not to move schemas to something like "import_specification_schemas" or something like that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might still make sense to have a template or some user-facing (or dev-facing) documentation for using the XSV/Excel importer specifications, though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the rest of the folder is going away, I don't see a reason not to move schemas to something like "import_specification_schemas" or something like that.

I'd probably call it "import_schemas" to disambiguate from the import specification system

It might still make sense to have a template or some user-facing (or dev-facing) documentation for using the XSV/Excel importer specifications, though.

There's this: https://docs.kbase.us/data/upload-download-guide/bulk

as necessary.
Loading