Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#768] dag: Support precondition as a shell command #771

Merged
merged 9 commits into from
Jan 3, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 38 additions & 10 deletions docs/source/schema.rst
Original file line number Diff line number Diff line change
Expand Up @@ -152,15 +152,33 @@ These fields apply to the entire DAG. They appear at the root of the YAML file.
- FOO: 1
- BAR: "`echo 2`"

``preconditions``
``precondition``
~~~~~~~~~~~~~~~
A list of conditions that must be satisfied before the DAG can run. Each condition can use shell expansions or command substitutions to validate external states.
The condition(s) that must be satisfied before the DAG can run. Each condition can use shell expansions or command substitutions to validate external states.

**Example**:
**Example**: Condition based on command exit code:

.. code-block:: yaml

precondition:
- "test -f /path/to/file"

# or more simply
precondition: "test -f /path/to/file"

**Example**: Condition based on environment variables:

.. code-block:: yaml

preconditions:
precondition:
- condition: "$ENV_VAR"
expected: "value"

**Example**: Condition based on command output (stdout):

.. code-block:: yaml

precondition:
- condition: "`echo $2`"
expected: "param2"

Expand Down Expand Up @@ -285,21 +303,31 @@ Each element in the top-level ``steps`` list has its own fields for customizatio
repeat: true
intervalSec: 60 # run every minute

``preconditions``
``precondition``
~~~~~~~~~~~~~~
Conditions that must be met for this step to run. Each condition block has:

- **condition** (string): A command or expression to evaluate.
- **expected** (string): The expected output. If the output matches, the step runs; otherwise, it is skipped.
Condition(s) that must be met for this step to run. It works same as the DAG-level ``precondition`` field. See :ref:`DAG-Level Fields <DAG-Level-Fields>` for examples.

.. code-block:: yaml

steps:
# Example 1: based on exit code
- name: daily task
command: daily.sh
precondition: "test -f /path/to/file"

# Example 2: based on command output (stdout)
- name: monthly task
command: monthly.sh
preconditions:
precondition:
- condition: "`date '+%d'`"
expected: "01"

# Example 3: based on environment variables
- name: weekly task
command: weekly.sh
precondition:
- condition: "$WEEKDAY"
expected: "Friday"

``depends``
~~~~~~~~~
Expand Down
38 changes: 35 additions & 3 deletions docs/source/yaml_format.rst
Original file line number Diff line number Diff line change
Expand Up @@ -201,10 +201,42 @@ Send output to files:
Conditional Execution
------------------

Preconditions
Precondition
~~~~~~~~~~~~
Run steps only when conditions are met:

.. code-block:: yaml

steps:
- name: monthly task
command: monthly.sh
preconditions: "test -f file.txt" # Run only if the file exists

Use multiple conditions:

.. code-block:: yaml

steps:
- name: monthly task
command: monthly.sh
preconditions: # Run only if all commands exit with 0
- "test -f file.txt"
- "test -d dir"

Use environment variables in conditions:

.. code-block:: yaml

steps:
- name: monthly task
command: monthly.sh
preconditions:
- condition: "${TODAY}" # Run only if TODAY is set as "01"
expected: "01"


Use command substitution in conditions:

.. code-block:: yaml

steps:
Expand Down Expand Up @@ -431,7 +463,7 @@ Complete list of DAG-level configuration options:
- ``delaySec``: Delay between steps
- ``maxActiveRuns``: Maximum parallel steps
- ``params``: Default parameters
- ``preconditions``: DAG-level conditions
- ``precondition``: DAG-level conditions
- ``mailOn``: Email notification settings
- ``MaxCleanUpTimeSec``: Cleanup timeout
- ``handlerOn``: Lifecycle event handlers
Expand All @@ -456,7 +488,7 @@ Example DAG configuration:
delaySec: 1
maxActiveRuns: 1
params: param1 param2
preconditions:
precondition:
- condition: "`echo $2`"
expected: "param2"
mailOn:
Expand Down
52 changes: 49 additions & 3 deletions internal/cmdutil/cmdutil.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,15 +105,13 @@ func SplitCommandWithEval(cmd string) (string, []string, error) {
}
for i, arg := range command {
command[i] = arg
// escape the command
// Escape the command
command[i] = escapeReplacer.Replace(command[i])
// Substitute command in the command.
command[i], err = SubstituteCommands(command[i])
if err != nil {
return "", nil, fmt.Errorf("failed to substitute command: %w", err)
}
// unescape the command
// command[i] = unescapeReplacer.Replace(command[i])
}
}

Expand Down Expand Up @@ -240,6 +238,54 @@ func WithVariables(vars map[string]string) EvalOption {
}
}

// RunExitCode runs the command and returns the exit code.
// func RunExitCode(cmd string) (int, error) {
// command, args, err := SplitCommandWithEval(cmd)
// if err != nil {
// return 0, err
// }
// shellCommand := GetShellCommand("")
// }

var regEscapedKeyValue = regexp.MustCompile(`^[^\s=]+="[^"]+"$`)

// BuildCommandEscapedString constructs a single shell-ready string from a command and its arguments.
// It assumes that the command and arguments are already escaped.
func BuildCommandEscapedString(command string, args []string) string {
quotedArgs := make([]string, 0, len(args))
for _, arg := range args {
// If already quoted, skip
if strings.HasPrefix(arg, `"`) && strings.HasSuffix(arg, `"`) {
quotedArgs = append(quotedArgs, arg)
continue
}
if strings.HasPrefix(arg, `'`) && strings.HasSuffix(arg, `'`) {
quotedArgs = append(quotedArgs, arg)
continue
}
// If the argument contains spaces, quote it.
if strings.ContainsAny(arg, " ") {
// If it includes '=' and is already quoted, skip
if regEscapedKeyValue.MatchString(arg) {
quotedArgs = append(quotedArgs, arg)
continue
}
// if it contains double quotes, escape them
arg = strings.ReplaceAll(arg, `"`, `\"`)
quotedArgs = append(quotedArgs, fmt.Sprintf(`"%s"`, arg))
} else {
quotedArgs = append(quotedArgs, arg)
}
}

// If we have no arguments, just return the command without trailing space.
if len(quotedArgs) == 0 {
return command
}

return fmt.Sprintf("%s %s", command, strings.Join(quotedArgs, " "))
}

// EvalString substitutes environment variables and commands in the input string
func EvalString(input string, opts ...EvalOption) (string, error) {
options := &EvalOptions{}
Expand Down
59 changes: 59 additions & 0 deletions internal/cmdutil/cmdutil_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -401,3 +401,62 @@ func TestReplaceVars(t *testing.T) {
})
}
}

// TestBuildCommandString demonstrates table-driven tests for BuildCommandString.
func TestBuildEscapedCommandString(t *testing.T) {
type testCase struct {
name string
cmd string
args []string
want string
}

tests := []testCase{
{
name: "piping",
cmd: "echo",
args: []string{"hello", "|", "wc", "-c"},
want: "echo hello | wc -c",
},
{
name: "redirection",
cmd: "echo",
args: []string{"'test content'", ">", "testfile.txt", "&&", "cat", "testfile.txt"},
want: `echo 'test content' > testfile.txt && cat testfile.txt`,
},
{
name: `key="value" argument`,
cmd: "echo",
args: []string{`key="value"`},
want: `echo key="value"`,
},
{
name: "JSON argument",
cmd: "echo",
args: []string{`{"foo":"bar","hello":"world"}`},
want: `echo {"foo":"bar","hello":"world"}`,
},
{
name: "key=value argument",
cmd: "echo",
args: []string{`key="some value"`},
want: `echo key="some value"`,
},
{
name: "double quotes",
cmd: "echo",
args: []string{`a "b" c`},
want: `echo "a \"b\" c"`,
},
}

for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
// Build the final command line that will be passed to `sh -c`.
cmdStr := BuildCommandEscapedString(tc.cmd, tc.args)

// Check if the built command string is as expected.
require.Equal(t, tc.want, cmdStr, "unexpected command string")
})
}
}
Loading
Loading