Skip to content

Commit

Permalink
Refactor meta tags from nomad-pipeline/ -> nomad-pipeline.
Browse files Browse the repository at this point in the history
  • Loading branch information
@waquidvp
waquidvp committed Jul 25, 2022
1 parent 75de3c5 commit fed3b3c
Show file tree
Hide file tree
Showing 7 changed files with 101 additions and 73 deletions.
30 changes: 17 additions & 13 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ variable "nomad_addr" {
job "example-job" {
meta = {
"nomad-pipeline.enabled" = "true"
}
group "▶️" {
count = 1
Expand All @@ -27,7 +31,7 @@ job "example-job" {
config {
image = "ghcr.io/hyperbadger/nomad-pipeline:main"
args = ["-init"]
args = ["agent", "init"]
}
env {
Expand All @@ -51,8 +55,8 @@ job "example-job" {
count = 0 # <-- Important! nomad-pipeline will control the count
meta = {
"nomad-pipeline/root" = "true" # <-- Indicates the starting task group
"nomad-pipeline/next" = "2-second-task-group"
"nomad-pipeline.root" = "true" # <-- Indicates the starting task group
"nomad-pipeline.next" = "2-second-task-group"
}
...
Expand Down Expand Up @@ -92,7 +96,7 @@ job "example-job" {

***Using dependencies***

To support running tasks in parallel and having a final task that runs at the end of all parallel tasks (eg. fan-out fan-in pattern), you can use the `nomad-pipeline/dependencies` tag.
To support running tasks in parallel and having a final task that runs at the end of all parallel tasks (eg. fan-out fan-in pattern), you can use the `nomad-pipeline.dependencies` tag.

```mermaid
graph TD;
Expand All @@ -110,7 +114,7 @@ group "E" {
count = 0
meta = {
"nomad-pipeline/dependencies" = "C, D"
"nomad-pipeline.dependencies" = "C, D"
}
...
Expand All @@ -121,17 +125,17 @@ See [`dependencies.hcl`](examples/dependencies.hcl) for a more complete example.

***Using count***

Another way to implement the fan-out fan-in pattern is to have multiple instances of a task group that can all pick up some work. Without nomad-pipeline, this is quite easy, you just set the [`count` stanza](https://www.nomadproject.io/docs/job-specification/group#count) on the task group. However, when using nomad-pipeline, the control of count is not in your hands. So if you want to set a count greater than 1, you can set the `nomad-pipeline/count` tag.
Another way to implement the fan-out fan-in pattern is to have multiple instances of a task group that can all pick up some work. Without nomad-pipeline, this is quite easy, you just set the [`count` stanza](https://www.nomadproject.io/docs/job-specification/group#count) on the task group. However, when using nomad-pipeline, the control of count is not in your hands. So if you want to set a count greater than 1, you can set the `nomad-pipeline.count` tag.

> 💡 *Tip: The [`count` stanza](https://www.nomadproject.io/docs/job-specification/group#count) doesn't support variable interpolation since the config value is an integer and not a string - currently Nomad only support variable interpolation for string config values. This means that `count` can't be set from a `NOMAD_META_` variable, which is required for setting the `count` dynamically in a parameterized job. Using the `nomad-pipeline/count` tag allows you work around this. All `nomad-pipeline/*` tags interpolates variables, so you can use something like `"nomad-pipeline/count" = "${NOMAD_META_count}"`*
> 💡 *Tip: The [`count` stanza](https://www.nomadproject.io/docs/job-specification/group#count) doesn't support variable interpolation since the config value is an integer and not a string - currently Nomad only support variable interpolation for string config values. This means that `count` can't be set from a `NOMAD_META_` variable, which is required for setting the `count` dynamically in a parameterized job. Using the `nomad-pipeline.count` tag allows you work around this. All `nomad-pipeline.*` tags interpolates variables, so you can use something like `"nomad-pipeline.count" = "${NOMAD_META_count}"`*
See [`examples/fan-out-fan-in.hcl`](examples/fan-out-fan-in.hcl) for a more complete example.

**Dynamic tasks**

Dynamic tasks allows you to have a task that outputs more tasks 🤯. These tasks are then run as part of the job. This can open up the possibility to create some powerful pipelines. An example use case is for creating periodic splits of a longer task, if you have a task that processes 5 hours of some data, you could split the task into 5x 1 hour tasks and run them in parallel. This can be achieved by having an initial task that outputs the 5 split tasks as an output.

To use dynamic tasks, set the `nomad-pipeline/dynamic-tasks` tag to a path/glob of where the task JSON's will be outputted. This path should be relative to [`NOMAD_ALLOC_DIR`](https://www.nomadproject.io/docs/runtime/environment#alloc).
To use dynamic tasks, set the `nomad-pipeline.dynamic-tasks` tag to a path/glob of where the task JSON's will be outputted. This path should be relative to [`NOMAD_ALLOC_DIR`](https://www.nomadproject.io/docs/runtime/environment#alloc).

In the following example, the 1-generate-tasks first runs and outputs the 2-echo-hey task group which then gets launched after 1-generate-tasks finishes.

Expand All @@ -140,8 +144,8 @@ group "1-generate-tasks" {
count = 0
meta = {
"nomad-pipeline/root" = "true"
"nomad-pipeline/dynamic-tasks" = "tasks/*"
"nomad-pipeline.root" = "true"
"nomad-pipeline.dynamic-tasks" = "tasks/*"
}
task "generate-tasks" {
Expand All @@ -158,7 +162,7 @@ group "1-generate-tasks" {
"Name": "2-echo-hey",
"Count": 0,
"Meta": {
"nomad-pipeline/root": "true"
"nomad-pipeline.root": "true"
},
"Tasks": [{
"Name": "echo",
Expand All @@ -182,14 +186,14 @@ See [`dynamic-job.hcl`](examples/dynamic-job.hcl) for a more complete example.

Nomad currently allows you to set a [`leader`](https://www.nomadproject.io/docs/job-specification/task#leader) at the task level. This allows you to gracefully shutdown all other tasks in the group when the leader task exits.

Using the `nomad-pipeline/leader` tag, you can get the same functionality at the job level. You can set the tag on a task group, and when that task group completes, all other task groups will be gracefully shutdown.
Using the `nomad-pipeline.leader` tag, you can get the same functionality at the job level. You can set the tag on a task group, and when that task group completes, all other task groups will be gracefully shutdown.

```hcl
group "leader" {
count = 0
meta = {
"nomad-pipeline/leader" = "true"
"nomad-pipeline.leader" = "true"
}
...
Expand Down
18 changes: 11 additions & 7 deletions examples/dependencies.hcl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ job "dependencies" {
datacenters = var.datacenters
type = "batch"

meta = {
"nomad-pipeline.enabled" = "true"
}

group "▶️" {
task "init" {
driver = "docker"
Expand All @@ -46,8 +50,8 @@ job "dependencies" {
count = 0

meta = {
"nomad-pipeline/root" = "true"
"nomad-pipeline/next" = "2-dependent"
"nomad-pipeline.root" = "true"
"nomad-pipeline.next" = "2-dependent"
}

task "normal" {
Expand Down Expand Up @@ -76,8 +80,8 @@ job "dependencies" {
count = 0

meta = {
"nomad-pipeline/root" = "true"
"nomad-pipeline/next" = "2-dependent"
"nomad-pipeline.root" = "true"
"nomad-pipeline.next" = "2-dependent"
}

task "normal" {
Expand Down Expand Up @@ -106,8 +110,8 @@ job "dependencies" {
count = 0

meta = {
"nomad-pipeline/root" = "true"
"nomad-pipeline/next" = "2-dependent"
"nomad-pipeline.root" = "true"
"nomad-pipeline.next" = "2-dependent"
}

task "normal" {
Expand Down Expand Up @@ -136,7 +140,7 @@ job "dependencies" {
count = 0

meta = {
"nomad-pipeline/dependencies" = "1a-task, 1b-task, 1c-task"
"nomad-pipeline.dependencies" = "1a-task, 1b-task, 1c-task"
}

task "dependent" {
Expand Down
16 changes: 10 additions & 6 deletions examples/dynamic-job.hcl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ job "dynamic" {
datacenters = var.datacenters
type = "batch"

meta = {
"nomad-pipeline.enabled" = "true"
}

group "▶️" {
task "init" {
driver = "docker"
Expand All @@ -46,8 +50,8 @@ job "dynamic" {
count = 0

meta = {
"nomad-pipeline/root" = "true"
"nomad-pipeline/dynamic-tasks" = "tasks/*"
"nomad-pipeline.root" = "true"
"nomad-pipeline.dynamic-tasks" = "tasks/*"
}

task "generate-tasks" {
Expand All @@ -65,8 +69,8 @@ job "dynamic" {
"Name": "${TASK_NAME}",
"Count": 0,
"Meta": {
"nomad-pipeline/root": "true",
"nomad-pipeline/next": "3-last"
"nomad-pipeline.root": "true",
"nomad-pipeline.next": "3-last"
},
"Tasks": [
{
Expand All @@ -92,8 +96,8 @@ job "dynamic" {
"Name": "3-last",
"Count": 0,
"Meta": {
"nomad-pipeline/root": "true",
"nomad-pipeline/dependencies": "2a-echo,2b-echo"
"nomad-pipeline.root": "true",
"nomad-pipeline.dependencies": "2a-echo,2b-echo"
},
"Tasks": [
{
Expand Down
14 changes: 9 additions & 5 deletions examples/fan-out-fan-in.hcl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ job "fan-out-fan-in" {
datacenters = var.datacenters
type = "batch"

meta = {
"nomad-pipeline.enabled" = "true"
}

group "▶️" {
task "init" {
driver = "docker"
Expand All @@ -46,8 +50,8 @@ job "fan-out-fan-in" {
count = 0

meta = {
"nomad-pipeline/root" = "true"
"nomad-pipeline/next" = "2-do-work"
"nomad-pipeline.root" = "true"
"nomad-pipeline.next" = "2-do-work"
}

task "submit" {
Expand Down Expand Up @@ -76,8 +80,8 @@ job "fan-out-fan-in" {
count = 0

meta = {
"nomad-pipeline/count" = "5"
"nomad-pipeline/next" = "3-process-output"
"nomad-pipeline.count" = "5"
"nomad-pipeline.next" = "3-process-output"
}

scaling {
Expand Down Expand Up @@ -112,7 +116,7 @@ job "fan-out-fan-in" {
count = 0

meta = {
"nomad-pipeline/dependencies" = "2-do-work"
"nomad-pipeline.dependencies" = "2-do-work"
}

task "process" {
Expand Down
18 changes: 11 additions & 7 deletions examples/happy-job.hcl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ job "happy" {
datacenters = var.datacenters
type = "batch"

meta = {
"nomad-pipeline.enabled" = "true"
}

group "▶️" {
count = 1

Expand All @@ -48,8 +52,8 @@ job "happy" {
count = 0

meta = {
"nomad-pipeline/root" = "true"
"nomad-pipeline/next" = "2-multi-task-group"
"nomad-pipeline.root" = "true"
"nomad-pipeline.next" = "2-multi-task-group"
}

task "normal" {
Expand Down Expand Up @@ -78,7 +82,7 @@ job "happy" {
count = 0

meta = {
"nomad-pipeline/next" = "3a-parallel,3b-parallel-i"
"nomad-pipeline.next" = "3a-parallel,3b-parallel-i"
}

task "first_task" {
Expand All @@ -102,7 +106,7 @@ job "happy" {
count = 0

meta = {
"nomad-pipeline/next" = "4-dependent"
"nomad-pipeline.next" = "4-dependent"
}

task "parallel" {
Expand Down Expand Up @@ -130,7 +134,7 @@ job "happy" {
count = 0

meta = {
"nomad-pipeline/next" = "3b-parallel-ii"
"nomad-pipeline.next" = "3b-parallel-ii"
}

task "parallel" {
Expand Down Expand Up @@ -158,7 +162,7 @@ job "happy" {
count = 0

meta = {
"nomad-pipeline/next" = "4-dependent"
"nomad-pipeline.next" = "4-dependent"
}

task "parallel" {
Expand Down Expand Up @@ -188,7 +192,7 @@ job "happy" {
meta = {
# BUG: when whole job is restarted, it will not wait for this task group,
# 4-dependent will run as soon as 3b-parallel-i finishes
"nomad-pipeline/dependencies" = "3b-parallel-ii"
"nomad-pipeline.dependencies" = "3b-parallel-ii"
}

task "dependent" {
Expand Down
10 changes: 7 additions & 3 deletions examples/leader-task-group.hcl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ job "leader-task-group" {
datacenters = var.datacenters
type = "batch"

meta = {
"nomad-pipeline.enabled" = "true"
}

group "▶️" {
task "init" {
driver = "docker"
Expand All @@ -46,8 +50,8 @@ job "leader-task-group" {
count = 0

meta = {
"nomad-pipeline/root" = "true"
"nomad-pipeline/leader" = "true"
"nomad-pipeline.root" = "true"
"nomad-pipeline.leader" = "true"
}

task "some-process" {
Expand Down Expand Up @@ -75,7 +79,7 @@ job "leader-task-group" {
count = 0

meta = {
"nomad-pipeline/root" = "true"
"nomad-pipeline.root" = "true"
}

task "forever-run" {
Expand Down
Loading

0 comments on commit fed3b3c

Please sign in to comment.