Merge pull request #26 from civitaspo/develop

0.0.5

CHANGELOG.md

@@ -1,3 +1,15 @@
+0.0.5 (2018-11-13)
+==================
+
+* [Experimental] Implement `ecs_task.rb>` operator.
+* [Experimental] Implement `ecs_task.sh>` operator.
+* [Enhancement] Add interface for another storage except S3 used by scripting operators.
+* [Enhancement] Add abstract class for scripting operators.
+* [Enhancement] Request ECS TaskRun with some retry.
+* [Fix] Prevent the influence of prior task registration.
+* [Enhancement] Add Logging for registered TaskDefinition arn. 
+* [Enhancement] Define VERSION var as package object val.
+
 0.0.4 (2018-11-06)
 ==================
 

README.md

@@ -15,7 +15,7 @@ _export:
     repositories:
       - https://jitpack.io
     dependencies:
-      - pro.civitaspo:digdag-operator-ecs_task:0.0.4
+      - pro.civitaspo:digdag-operator-ecs_task:0.0.5
   ecs_task:
     auth_method: profile
 
@@ -197,12 +197,26 @@ In addition, the below configurations exist.
 - **started_by**: An optional tag specified when a task is started. (string, optional)
 - **workspace_s3_uri_prefix**: S3 uri prefix for using as workspace. (string, required)
     - Currently, input params, output params, stdout, stderr, and internal scripts are put on S3, and then they are not removed. So it's insecure unless strict access control to S3.
+    - This option is **deprecated**. Please use **tmp_storage** option instead.
+- **tmp_storage**: Temporary storage for the data and files scripting operator uses. (map, required)
+    - **type**: storage type. Currently, only `"s3"` is valid. (string, required)
+    - **uri**: storage uri. (string, required)
 
 ## Configuration for `ecs_task.py>` operator
 
 - **ecs_task.py>**: Name of a method to run. The format is `[PACKAGE.CLASS.]METHOD`. (string, required)
 - **pip_install**: packages to install before task running. (array of string, optional)
 
+## Configuration for `ecs_task.rb>` operator
+
+- **ecs_task.rb>**: Name of a method to run. The format is `[MODULE::CLASS.]METHOD`. (string, required)
+- **gem_install**: packages to install before task running. (array of string, optional)
+- **require**: Name of a file to require. e.g. `require: task/my_workflow` (string, required)
+
+## Configuration for `ecs_task.sh>` operator
+
+- **ecs_task.sh>**: command to run on shell. (string, required)
+
 ## Configuration for `ecs_task.embulk>` operator
 
 - **ecs_task.embulk>**: Embulk config yaml or file. You can use digdag's template engine like `${...}` in the config yaml or file. (string or map, required)

build.gradle

@@ -6,7 +6,7 @@ plugins {
 }
 
 group = 'pro.civitaspo'
-version = '0.0.4'
+version = '0.0.5'
 
 def digdagVersion = '0.9.31'
 def scalaSemanticVersion = "2.12.6"

example/example.dig

@@ -4,7 +4,7 @@ _export:
       - file://${repos}
       # - https://jitpack.io
     dependencies:
-      - pro.civitaspo:digdag-operator-ecs_task:0.0.4
+      - pro.civitaspo:digdag-operator-ecs_task:0.0.5
   ecs_task:
     auth_method: profile
 

src/main/resources/pro/civitaspo/digdag/plugin/ecs_task/rb/run.sh

@@ -0,0 +1,55 @@
+set -ex
+set -o pipefail
+
+mkdir -p ./digdag-operator-ecs_task
+cd digdag-operator-ecs_task
+
+# Create output files
+touch out.json stdout.log stderr.log
+
+# Download requirements
+aws s3 cp s3://${ECS_TASK_RB_BUCKET}/${ECS_TASK_RB_PREFIX}/ ./ --recursive
+
+# Move workspace
+cd workspace
+
+# Unset e option for returning ruby results to digdag
+set +e
+
+# Run setup command
+${ECS_TASK_RB_SETUP_COMMAND} \
+        2>> ../stderr.log \
+    | tee -a ../stdout.log
+
+# Run
+cat ../runner.rb \
+    | ruby \
+        -I . \
+        -r ${ECS_TASK_RB_REQUIRE} \
+        -- - \
+        "${ECS_TASK_RB_COMMAND}" \
+        ../in.json \
+        ../out.json \
+        2>> ../stderr.log \
+    | tee -a ../stdout.log
+
+# Capture exit code
+EXIT_CODE=$?
+
+# Set e option
+set -e
+
+# Move out workspace
+cd ..
+
+# For logging driver
+cat stderr.log 1>&2
+
+# Upload results
+aws s3 cp ./out.json s3://${ECS_TASK_RB_BUCKET}/${ECS_TASK_RB_PREFIX}/
+aws s3 cp ./stdout.log s3://${ECS_TASK_RB_BUCKET}/${ECS_TASK_RB_PREFIX}/
+aws s3 cp ./stderr.log s3://${ECS_TASK_RB_BUCKET}/${ECS_TASK_RB_PREFIX}/
+
+# Exit with the ruby exit code
+exit $EXIT_CODE
+