diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 63a0973..9499503 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -20,7 +20,8 @@ before_script: - mkdir -p $GOPATH/src/github.com/ngageoint - ln -s $(pwd) $GOPATH/src/github.com/ngageoint/seed-cli - cd $GOPATH/src/github.com/ngageoint/seed-cli - + - docker pull rochdev/alpine-asciidoctor:mini + build: stage: build artifacts: @@ -30,6 +31,7 @@ build: - output/seed-windows-amd64 script: - ./build-cli.sh $CI_COMMIT_TAG + - docker run -v readme.adoc:/documents/readme.adoc -v seed-cli.adoc:/documents/seed-cli.adoc --name=seed-man rochdev/alpine-asciidoctor:mini asciidoctor -b manpage -D /documents/output seed-cli.adoc only: - tags diff --git a/.travis.yml b/.travis.yml index 38d5236..894b4ba 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,6 +10,7 @@ install: - go get github.com/ngageoint/seed-common/... - go get golang.org/x/sys/unix/... - go get github.com/zyxar/image2ascii/ascii + - docker pull rochdev/alpine-asciidoctor:mini script: - echo "Validating formatting with gofmt..." @@ -17,6 +18,7 @@ script: - ./build-cli.sh ${TRAVIS_TAG} - docker run -d -p 5000:5000 --restart=always --name registry -v `pwd`/auth:/auth -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd registry:2 - go test ./... + - docker run -v $TRAVIS_BUILD_DIR/readme.adoc:/documents/readme.adoc -v $TRAVIS_BUILD_DIR/seed-cli.adoc:/documents/seed-cli.adoc --name=seed-man rochdev/alpine-asciidoctor:mini asciidoctor -b manpage -D /documents/output seed-cli.adoc deploy: # Deployment that pushes GitHub releases of software on tag diff --git a/readme.adoc b/readme.adoc index b1048e6..4fd65c1 100644 --- a/readme.adoc +++ b/readme.adoc @@ -3,15 +3,19 @@ image:https://badges.gitter.im/ngageoint/seed.svg[link="https://gitter.im/ngageoint/seed?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge"] image:https://travis-ci.org/ngageoint/seed-cli.svg?branch=master[link="https://travis-ci.org/ngageoint/seed-cli"] +//# tag::intro[] The Seed team provides a fully featured Command-Line Interface (CLI) for algorithm developers looking to offer Seed compliant packaging of their jobs. Using the CLI will enable you to quickly iterate without getting bogged down learning how to interface directly with the underlying container technologies (Docker). It also simplifies the process of attaching Seed metadata to your algorithm prior to publish. +//# end::intro[] == Usage +//# tag::command-intro[] Seed CLI offers sub-commands that correlate to common operations performed by an algorithm developer. These can be seen from the command-line by launching the CLI without any arguments: +//# end::command-intro[] ---- seed @@ -22,12 +26,15 @@ usage can be found in the following sections. === Build +//# tag::build-usage[] The first step when starting to package an algorithm for Seed compliance is to define the requirements and interface. This is done by a combination of execution environment configuration (`Dockerfile`), resource requirement definition and input / output specification (`seed.manifest.json`). By default, the `seed build` command assumes both a `Dockerfile` `seed.manifest.json` reside in the current working directory. Using these two files in conjunction the build command is able to construct an re-usable Seed compliant image. +//# end::build-usage[] +//# tag::build-example[] A simple example of this would be the addition-job image. ---- @@ -38,25 +45,39 @@ This will result in a new Docker image that contains `com.ngageoint.seed.manifes constraint: addition-job-0.0.1-seed:1.0.0 This image can now be executed via the `seed run` command or pushed to a remote image registry by way of `seed publish`. +//# end::build-example[] + +//# tag::build-publish-usage[] +The seed build command also provides the option to automatically publish the image after building via the `-publish` flag. +All flags specified by the `seed publish` command are available for use. +//# end::build-publish-usage[] === Init -The init command will initalize a directory with a template seed.manifest.json file. The following command will put -a template seed.manifest.json file in the examples/job directory: +//# tag::init-usage[] +The init command will initalize a directory with a template seed.manifest.json file. +//# end::init-usage[] + +//# tag::init-example[] +The following command will put a template seed.manifest.json file in the examples/job directory: ---- seed init -d examples/job ---- +//# end::init-example[] === Run +//# tag::run-usage[] The primary purpose of the CLI is to easily enable algorithm execution. The common stumbling blocks for new developers are being able to feed data into and retrieve out fo the containers as a part of execution. The `seed run` command facilitates this through introspection of the Seed interface indicating exactly what input data is required, allowing for simply specifying the system locations of data and handling all mounting and output validation and capture operations for the developer. +//# end::run-usage[] -For an Seed interface with a single `inputData.files` element with a `name` of `MY_INPUT` the `seed run` command would +//# tag::run-example[] +For a Seed interface with a single `inputData.files` element with a `name` of `MY_INPUT` the `seed run` command would be as follows: ---- @@ -66,12 +87,16 @@ seed run -in process-file:0.1.0-seed:0.1.0 -i MY_INPUT=/tmp/file_input.txt -o /t This will volume mount `/tmp/file_input.txt` as read only and `/tmp/outputs` into the container, replacing these values with the container relative locations and injecting into the defined `args` placeholders for consumption by the algorithm. +//# end::run-example[] === Batch +//# tag::batch-usage[] Related to the run command, the `seed batch` command will run an image multiple times with varying inputs. It will take an image name and either a directory with files to use as input or a csv file specifying keys and files (preferred). +//# end::batch-usage[] +//# tag::batch-example[] For an Seed interface with two `inputData.files` elements with a `name` of `MY_INPUT` and `MY_INPUT2` the `seed batch` command would be as follows: @@ -90,6 +115,7 @@ currentDirectoryInput.txt, input2.txt The image will be run three times and success or failure will be reported for each run along with the location of any output. +//# end::batch-example[] === List @@ -101,35 +127,51 @@ seed list === Search -Allows for discovery of Seed compliant images hosted within a Docker registry. The 'seed search' command will search -a given registry for repositories (images and all their versions) that end in "-seed". Here is an example of searching quay.io: +//# tag::search-usage[] +Allows for discovery of Seed compliant images hosted within a Docker registry (default is docker.io). +//# end::search-usage[] + +//# tag::search-example-1[] +The 'seed search' command will search a given registry for repositories (images and all their versions) that end in +"-seed". Here is an example of searching quay.io: ---- seed search -r https://quay.io ---- +//# end::search-example-1[] +//# tag::search-example-2[] If no registry url is supplied, the search command uses docker hub, which requires a organization (or user) to be specified: ---- seed search -o geoint ---- +//# end::search-example-2[] +//# tag::search-example-3[] If a registry is private, a username and password can be specified with -u and -p options: ---- seed search -r http://localhost:5000 -u testuser -p testpassword ---- +//# end::search-example-3[] === Publish +//# tag::publish-usage[] Provides a convenient way for algorithm developers to push a Seed image to a registry. This command will tag a seed image appropriately and push it to the specified registry. If successful, it will then remove the local image as well. +//# end::publish-usage[] + +//# tag::publish-example-1[] Here is an example of publishing the extractor image to the geoint organization on docker hub: ---- seed publish -in extractor-0.1.0-seed:0.1.0 -r docker.io -o geoint ---- +//# end::publish-example-1[] +//# tag::publish-example-2[] Publishing will check if an image with the same name and tag exists in the registry and will fail if one is found unless either the force flag (-f) is set or a deconflict tag is specified to increase a version number. A common use case for seed algorithm developers is to publish new versions of their image and this can be done by specifying one of the job or @@ -138,38 +180,60 @@ package version flags. Here is an example of updating the extractor image to 1. ---- seed publish -in extractor-0.1.0-seed:0.1.0 -r docker.io -o geoint -A -P ---- +//# end::publish-example-2[] +//# tag::publish-example-3[] This will rebuild the extractor image with the appropriate name & label and publish the image extractor-1.0.0-seed:1.0.0 to docker hub. Finally, if the registry is private, a username and password can be specified with -u and -p options: ---- seed publish -in extractor-0.1.0-seed:0.1.0 -r localhost:5000 -u testuser -p testpassword ---- +//# end::publish-example-3[] + +=== Pull + +The Pull command will pull the a Seed compliant image from the remote Docker registry. + +//# tag::pull-example[] +This will pull the extractor-0.1.0-seed:1.0.0 image from the docker.io/geoint registry: +---- +seed pull -in extractor-0.1.0-seed:0.1.0 -r docker.io -o geoint +---- +//# end::pull-example[] === Validate +//# tag::validate-usage[] The Validate command will validate a Seed json file against the Seed schema. This is also done as part of the build and run commands, but if a user is having problems getting their Seed file to validate this can be useful to debug without -those additional steps. This command will validate the Seed file in the examples/extractor directory: +those additional steps. +//# end::validate-usage[] + +//# tag::validate-example-1[] +This command will validate the Seed file in the examples/extractor directory using the schema built-in to the Seed CLI tool: ---- seed validate -d examples/extractor ---- +//# end::validate-example-1[] -That will use the schema built-in to the Seed CLI tool. To use a different schema, pass it in using the -s flag: +//# tag::validate-example-2[] +To use a different schema, pass it in using the -s flag: ---- seed validate -d examples/extractor -s schema/0.1.0/seed.manifest.schema.json ---- +//# end::validate-example-2[] === Version - +//# tag::version[] The version command will print the version of the Seed CLI tool: ---- seed version ---- - +//# end::version[] == Development diff --git a/seed-cli.adoc b/seed-cli.adoc new file mode 100644 index 0000000..dba78bf --- /dev/null +++ b/seed-cli.adoc @@ -0,0 +1,309 @@ += seed(1) +Jonathan Meyer ; Mike Holt ; Derick Faller ; John Tobe +vSEED_VERSION +:doctype: manpage +:manmanual: SEED +:mansource: SEED +:man-linkstyle: pass:[blue R < >] + +== Name + +seed - A test runner for Seed spec compliant algorithms + +== Synopsis + +*seed* [COMMAND] [OPTIONS] + +*seed* batch -in IMAGE_NAME [-b BATCH_FILE | -d BATCH_DIRECTORY] [-e SETTING=SETTING_VALUE] [-m MOUNT_KEY=HOST_PATH] [-o OUTPUT_DIRECTORY] + +*seed* build [-d JOB_DIRECTORY] [-u USER_NAME -p PASSWORD] [-publish Publish Options] + +*seed* init [-d JOB_DIRECTORY] + +*seed* list + +*seed* publish -in IMAGE_NAME [-r REGISTRY_NAME] [-o ORG_NAME] [-u username] [-p password] [Conflict Options] + +*seed* pull -in IMAGE_NAME [-r REGISTRY_NAME] [-o ORGANIZATION_NAME] [-u USER_NAME] [-p PASSWORD] + +*seed* run -in IMAGE_NAME [-rm] [-q] [-i INPUT_FILE_KEY=INPUT_FILE_VALUE] [-e SETTING_KEY=SETTING_VALUE] [-m MOUNT_KEY=HOST_PATH] [-o OUTPUT_DIRECTORY] [-rep 5] [-s SCHEMA_FILE] + +*seed* search [-r REGISTRY_NAME] [-o ORGANIZATION_NAME] [-f FILTER] [-u Username] [-p password] + +*seed* validate [-d MANIFEST_DIRECTORY] [-s SCHEMA_FILE] + +*seed* version + +== Description + +include::readme.adoc[tag=intro] + +== Commands +include::readme.adoc[tag=command-intro] + +=== batch + +Executes Seed compliant Docker image over multiple iterations + +include::readme.adoc[tag=batch-usage] + +seed batch -in IMAGE_NAME [-b BATCH_FILE | -d BATCH_DIRECTORY] [-e SETTING=SETTING_VALUE] [-m MOUNT_KEY=HOST_PATH] [-o OUTPUT_DIRECTORY] + +*-in, -imageName* :: + Docker image name to run; Required argument. +*-b, -batch* :: + Optional file specifying input keys and file mapping for batch processing. Supersedes directory flag. +*-d, -directory* :: + Alternative to batch file; Specifies a directory of files to batch process (default is current directory). +*-e, -setting* :: + Specifies the key/value setting values of the seed spec in the format SETTING_KEY=VALUE. +*-m, -mount* :: + Specifies the key/value mount values of the seed spec in the format MOUNT_KEY=HOST_PATH. +*-o, -outDir* :: + Specifies the job output directory. +*-rm* :: + Automatically removes the container when the job exits (i.e. docker run --rm) +*-s, -schema* :: + External Seed metadata schema file; Overrides built in schema to validate side-car metadata files + +*EXAMPLE:* + +include::readme.adoc[tag=batch-example] + +=== build + +Builds Seed compliant Docker image + +include::readme.adoc[tag=build-usage] + +seed build [-d JOB_DIRECTORY] [-c] [-D DOCKERFILE_DIRECTORY] [-m MANIFEST_DIRECTORY] [-u USER_NAME -p PASSWORD] + +*-c, -cache-from* :: + Utilizes the --cache-from option when building the docker image +*-d, -directory* :: + Directory containing Seed spec and Dockerfile (default is current directory). +*-D, -dockerfile* :: + Specifies the Dockerfile to use (default is Dockerfile within current directory) +*-m, -manifest* :: + Specifies the seed manifest file to use (default is seed.manifest.json within the current directory) +*-v, -version* :: + Version of built in seed manifest to validate against (default is 1.0.0). +*-u, -user* :: + Username to login if needed to pull images (default anonymous). +*-p, -password* :: + Password to login if needed to pull images (default anonymous). +*-publish* :: + Will publish image after a successful build. May require extra arguments defined in the following sections + +*EXAMPLE*: + +include::readme.adoc[tag=build-example] + +*BUILD AND PUBLISH OPTIONS:* + +include::readme.adoc[tag=build-publish-usage] + +seed build [...] -publish -r REGISTRY -o ORGANIZATION [-f] + +*-r, -registry* :: + Specifies a specific registry to publish the image (default is docker.io). +*-o, -org* :: + Required argument that specifies a specific organization to publish the image under. +*-f* :: + Forces overwrite of the remote image if publish conflict is found. + +*PUBLISH CONFLICT OPTIONS:* + +seed build [...] -publish -r REGISTRY -o ORGANIZATION [-f [-pp | -pm | -P | -jp | -jm | -JM ]] + +If the force flag (-f) is not set, the following options specify how a publish conflict is handled: + + +*-pp* :: + Force Patch version bump of 'packageVersion' in manifest on disk if publish conflict found +*-pm* :: + Force Minor version bump of 'packageVersion' in manifest on disk if publish conflict found +*-P* :: + Force Major version bump of 'packageVersion' in manifest on disk if publish conflict found +*-jp* :: + Force Patch version bump of 'jobVersion' in manifest on disk if publish conflict found +*-jm* :: + Force Minor version bump of 'jobVersion' in manifest on disk if publish conflict found +*-JM* :: + Force Major version bump of 'jobVersion' in manifest on disk if publish conflict found + +=== init + +include::readme.adoc[tag=init-usage] + +seed init [-d JOB_DIRECTORY] + +*-d, -directory* :: + Directory to place the generated seed.manifest.json example (default is current directory). + +*EXAMPLE:* + +include::readme.adoc[tag=init-example] + +=== list + +Allows for listing of all Seed compliant images residing on the local system + +seed list + +*Example Output:* + +REPOSITORY TAG IMAGE ID CREATED SIZE + +extractor-0.1.0-seed 2.0.0 cb4065fe9c8f 7 days ago 4.15MB + +addition-job-0.0.1-seed 1.0.0 15c15cf4a896 7 days ago 684MB + +my-job-1.0.0-seed 1.0.0 dc955d34436f 13 days ago 3.97MB + +test-seed latest 71e4d4addfdd 10 months ago 0B + +=== publish + +Publishes Seed compliant images to remote Docker registry + +include::readme.adoc[tag=publish-usage] + +Publishing will check if an image with the same name and tag exists in the registry and will fail if one is found unless either the force flag (-f) is set or a deconflict tag is specified to increase a version number. A common use case for seed algorithm developers is to publish new versions of their image and this can be done by specifying one of the job or package version flags. + +seed publish -in IMAGE_NAME [-r REGISTRY_NAME] [-o ORG_NAME] [-u username] [-p password] [Conflict Options] + +*-in, -imageName* :: + Specifies the Docker image name to publish; Must be an existing image residing on the local system. +*-r, -registry* :: + Specifies a specific registry to publish the image (default is docker.io) +*-o, -org* :: + Specifies a specific organization to publish the image under. +*-u, -user* :: + Username to login if needed to publish images (default anonymous). +*-p, -password* :: + Password to login if needed to publish images (default anonymous). +*-f* :: + Forces overwrite of the remote image if publish conflict is found. + +*CONFLICT OPTIONS* + +seed publish ... -f [-d SEED_DIRECTORY] [-pp] [-pm] [-P] [-jp] [-J] + +If the force flag (-f) is not set, the following options specify how a publish conflict is handled: + +*-d, -directory* :: + Specifies the directory containing the seed.manifest.json and dockerfile to rebuild the image. +*-pp* :: + Force Patch version bump of 'packageVersion' in manifest on disk if publish conflict found +*-pm* :: + Force Minor version bump of 'packageVersion' in manifest on disk if publish conflict found +*-P* :: + Force Major version bump of 'packageVersion' in manifest on disk if publish conflict found +*-jp* :: + Force Patch version bump of 'jobVersion' in manifest on disk if publish conflict found +*-jm* :: + Force Minor version bump of 'jobVersion' in manifest on disk if publish conflict found +*-J* :: + Force Major version bump of 'jobVersion' in manifest on disk if publish conflict found + +*EXAMPLE:* + +This will build a new image example-0.2.0-seed:1.0.0 and publish it to hub.docker.com/geoint + + + seed publish -in example-0.1.3-seed:0.1.3 -r hub.docker.com -o geoint -jm -P + +Deconfliction is obtained by incrementing the minor version number of the 'jobVersion' and the major version number of the 'packageVersion'. + +*EXAMPLE:* + +This is an example of updating the existing extractor image to 1.0.0 by forcing a major package version bump: + + seed publish -in extractor-0.1.0-seed:0.1.0 -r docker.io -o geoint -P + +*EXAMPLE:* + +include::readme.adoc[tag=publish-example-3] + +=== pull + +Allows for pulling Seed compliant images from remote Docker registry + +seed pull -in IMAGE_NAME [-r REGISTRY_NAME] [-o ORGANIZATION_NAME] [-u USER_NAME] [-p PASSWORD] + +*-in, -imageName* :: + Docker image name to pull +*-r, -registry* :: + Specifies a specific registry (default is index.docker.io). +*-o, -org* :: + Specifies a specific organization (default is no organization). +*-u, -user* :: + Username to login to remote registry (default anonymous). +*-p, -password* :: + Password to login to remote registry (default anonymous). + +*EXAMPLE:* + +include::readme.adoc[tag=pull-example] + +=== run + +Executes Seed compliant Docker docker image + +include::readme.adoc[tag=run-usage] + +seed run -in IMAGE_NAME [-rm] [-q] [-i INPUT_FILE_KEY=INPUT_FILE_VALUE] [-e SETTING_KEY=SETTING_VALUE] [-m MOUNT_KEY=HOST_PATH] [-o OUTPUT_DIRECTORY] [-rep 5] [-s SCHEMA_FILE] + +*-in, -imageName* :: + Docker image name to run + +*-i, -inputs* :: + Specifies the key/value input data values of the seed spec in the format INPUT_FILE_KEY=INPUT_FILE_VALUE + +*-e, -setting* :: + Specifies the key/value setting values of the seed spec in the format SETTING_KEY=VALUE + +*-m, -mount* :: + Specifies the key/value mount values of the seed spec in the format MOUNT_KEY=HOST_PATH + +*-o, -outDir* :: + Job Output Directory Location + +*-rm* :: + Automatically remove the container when it exits (docker run --rm) + +*-q, -quiet* :: + Suppress stdout when running docker image + +*-rep, -repetitions* :: + Run docker image multiple times (i.e. -rep 5 runs the image 5 times) + +*-s, -schema* :: + External Seed metadata schema file; Overrides built in schema to validate side-car metadata files + +*EXAMPLE:* + +include::readme.adoc[tag=run-example] + +=== search + +include::readme.adoc[tag=search-usage] + +seed search [-r REGISTRY_NAME] [-o ORGANIZATION_NAME] [-f FILTER] [-u Username] [-p password] + +*-r, -registry* :: + Specifies a specific registry to search (default is index.docker.io). +*-o, -org* :: + Specifies a specific organization to filter (default is no filter). +*-f, -filter* :: + Specifies a filter to apply (default is no filter). +*-u, -user* :: + Username to login to remote registry (default is anonymous). +*-p, -password* :: + Password to login to remote registry (default is anonymous). + +*Note:* The Docker engine API does not support returning result sets of over 100 images due to no pagination capability. Until the API supports paginating over 100 results, the `seed search` command will only return a maximum of 100 seed image results. Furthermore, the docker engine search capability strips all special characters and will return images that contain any search term. Searching for foo-seed will return images with foo or seed in the name. + +*EXAMPLE:* + +include::readme.adoc[tag=search-example-1] + +*EXAMPLE:* + +include::readme.adoc[tag=search-example-2] + +*EXAMPLE:* + +include::readme.adoc[tag=search-example-3] + +=== validate +Validates a Seed spec + +include::readme.adoc[tag=validate-usage] + +seed validate [-d MANIFEST_DIRECTORY] [-s SCHEMA_FILE] + +*-d, -directory* :: + Specifies the directory in which Seed is located (default is the current directory) +*-s, -schema* :: + External Seed schema file; Overrides built in schema to validate Seed spec against + +*EXAMPLE:* + +include::readme.adoc[tag=validate-example-1] + +*EXAMPLE:* + +include::readme.adoc[tag=validate-example-2] + +=== version +include::readme.adoc[tag=version] \ No newline at end of file