[incubator-kie-issues#1350] Refactor ForExpressionNode to allow dynamically-generated iteration contexts

[gitgabrio]

Fixes apache/incubator-kie-issues#1350

@bncriju

How to replicate CI configuration locally?

Build Chain tool does "simple" maven build(s), the builds are just Maven commands, but because the repositories relates and depends on each other and any change in API or class method could affect several of those repositories there is a need to use build-chain tool to handle cross repository builds and be sure that we always use latest version of the code for each repository.

build-chain tool is a build tool which can be used on command line locally or in Github Actions workflow(s), in case you need to change multiple repositories and send multiple dependent pull requests related with a change you can easily reproduce the same build by executing it on Github hosted environment or locally in your development environment. See local execution details to get more information about it.

How to retest this PR or trigger a specific build:

• for pull request and downstream checks

  • Push a new commit to the PR. An empty commit would be enough.

• for a full downstream build

  • for github actions job: add the label run_fdb

• for Jenkins PR check only

  • If you are an ASF committer for KIE podling, login to Jenkins (https://ci-builds.apache.org/job/KIE/job/drools/), go to the specific PR job, and click on Build Now button.

[gitgabrio]

Build failures are unrelated - please do not restart the build

Kogito Downstream / kogito-runtimes (ubuntu-latest / Java-17 / Maven-3.9.6) (pull_request) failure is related to a flacky test

1. it worked locally
2. reducing the timeout here, reproduce the error locally

Kogito Downstream / kogito-quarkus-examples (ubuntu-latest / Java-17 / Maven-3.9.6) (pull_request) failure due to apache_incubator-kie-kogito-examples/kogito-quarkus-examples/dmn-drools-quarkus-metrics/src/test/java/org/kie/kogito/examples/quarkus/GrafanaDockerComposeIT.java:1 | Local Docker Compose not found. Is docker-compose on the PATH?

Kogito Downstream / kogito-springboot-examples (ubuntu-latest / Java-17 / Maven-3.9.6) (pull_request) failure due to apache_incubator-kie-kogito-examples/kogito-quarkus-examples/dmn-drools-quarkus-metrics/src/test/java/org/kie/kogito/examples/quarkus/GrafanaDockerComposeIT.java:1 | Local Docker Compose not found. Is docker-compose on the PATH?

Note: for the latter failure, the situation seems more complicated, because the quarkus failure seems to be appended to springboot execution

Going to parse results form **/*-reports/TEST-*.xml
##[debug]followSymbolicLinks 'false'
##[debug]followSymbolicLinks 'false'
##[debug]implicitDescendants 'true'
##[debug]matchDirectories 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]Search path '/home/runner/work/incubator-kie-drools/incubator-kie-drools'
##[debug]Parsing file /home/runner/work/incubator-kie-drools/incubator-kie-drools/apache_incubator-kie-kogito-examples/kogito-springboot-examples/decisiontable-springboot-example/target/surefire-reports/TEST-org.kie.kogito.decisiontable.springboot.RestQueryTest.xml
##[debug]Parsing file /home/runner/work/incubator-kie-drools/incubator-kie-drools/apache_incubator-kie-kogito-examples/kogito-springboot-examples/dmn-drools-springboot-metrics/target/failsafe-reports/TEST-org.kie.kogito.examples.springboot.GrafanaDockerComposeIT.xml
##[debug]Resolving path for GrafanaDockerComposeIT
##[debug]followSymbolicLinks 'false'
##[debug]followSymbolicLinks 'false'
##[debug]implicitDescendants 'true'
##[debug]matchDirectories 'true'
##[debug]omitBrokenSymbolicLinks 'true'
##[debug]Search path '/home/runner/work/incubator-kie-drools/incubator-kie-drools'
##[debug]Matched files: /home/runner/work/incubator-kie-drools/incubator-kie-drools/apache_incubator-kie-kogito-examples/kogito-quarkus-examples/dmn-drools-quarkus-metrics/src/test/java/org/kie/kogito/examples/quarkus/GrafanaDockerComposeIT.java,/home/runner/work/incubator-kie-drools/incubator-kie-drools/apache_incubator-kie-kogito-examples/kogito-springboot-examples/dmn-drools-springboot-metrics/src/test/java/org/kie/kogito/examples/springboot/GrafanaDockerComposeIT.java,/home/runner/work/incubator-kie-drools/incubator-kie-drools/apache_incubator-kie-kogito-examples/kogito-springboot-examples/dmn-drools-springboot-metrics/target/test-classes/org/kie/kogito/examples/springboot/GrafanaDockerComposeIT.class
##[debug]Resolved path: apache_incubator-kie-kogito-examples/kogito-quarkus-examples/dmn-drools-quarkus-metrics/src/test/java/org/kie/kogito/examples/quarkus/GrafanaDockerComposeIT.java
##[debug]Canonical path: apache_incubator-kie-kogito-examples/kogito-quarkus-examples/dmn-drools-quarkus-metrics/src/test/java/org/kie/kogito/examples/quarkus/GrafanaDockerComposeIT.java
apache_incubator-kie-kogito-examples/kogito-quarkus-examples/dmn-drools-quarkus-metrics/src/test/java/org/kie/kogito/examples/quarkus/GrafanaDockerComposeIT.java:1 | Local Docker Compose not found. Is docker-compose on the PATH?

@jstastny-cz @rodrigonull ^^ any hint on that last finding ?

[baldimir]

The docker compose failures are solved here I think https://kie.zulipchat.com/#narrow/stream/386449-kie-dev/topic/Several.20PR.20CI.20failing.20because.20missing.20docker.20compose

[yesamer]

Tested against TCK Test Suite. 1 Test fixed, no regressions. Excellent work!

[yesamer]

Excellent!!

[baldimir]

Maybe it would be good to also do benchmarks on this. I guess it is expected, that it will be a bit slower, as it is doing more things now, however it might be good to know by how much. What do you think please?

[gitgabrio]

@baldimir @yesamer @bncriju
Created specific benchmarks here

Results.

Main branch:

"Benchmark","Mode","Threads","Samples","Score","Score Error (99,9%)","Unit","Param: baseNodeExpression"
"org.drools.benchmarks.dmn.ast.ForExpressionNodeBenchmark.evaluateBaseNodeBenchmark","avgt",1,100,"1,109441","0,021942","us/op","for x in [ 1, 2, 3, 4 ] return x"
"org.drools.benchmarks.dmn.ast.ForExpressionNodeBenchmark.evaluateBaseNodeBenchmark","avgt",1,100,"1,158235","0,020001","us/op","for x in [ [1, 2], [3, 4] ] return x"
"org.drools.benchmarks.dmn.ast.ForExpressionNodeBenchmark.evaluateBaseNodeBenchmark","avgt",1,100,"1,664903","0,029775","us/op","for x in [ 1, 2, 3, 4 ], y in x return y"
"org.drools.benchmarks.dmn.ast.ForExpressionNodeBenchmark.evaluateBaseNodeBenchmark","avgt",1,100,"1,640637","0,028808","us/op","for x in [ [1,2], [3,4] ], y in x return y"

Current PR

"Benchmark","Mode","Threads","Samples","Score","Score Error (99,9%)","Unit","Param: baseNodeExpression"
"org.drools.benchmarks.dmn.ast.ForExpressionNodeBenchmark.evaluateBaseNodeBenchmark","avgt",1,100,"1,083477","0,030394","us/op","for x in [ 1, 2, 3, 4 ] return x"
"org.drools.benchmarks.dmn.ast.ForExpressionNodeBenchmark.evaluateBaseNodeBenchmark","avgt",1,100,"1,182300","0,030478","us/op","for x in [ [1, 2], [3, 4] ] return x"
"org.drools.benchmarks.dmn.ast.ForExpressionNodeBenchmark.evaluateBaseNodeBenchmark","avgt",1,100,"1,634832","0,027725","us/op","for x in [ 1, 2, 3, 4 ], y in x return y"
"org.drools.benchmarks.dmn.ast.ForExpressionNodeBenchmark.evaluateBaseNodeBenchmark","avgt",1,100,"1,662278","0,031083","us/op","for x in [ [1,2], [3,4] ], y in x return y"

It seems results are pretty similar. Probably the current PR increase slightly the time variability.
The most evident change is that, currently, all "iteration contexts" are initialized "statically", at the beginning of execution, thus preventing any kind of "dynamic assignment" over iterations; but that does not completely fulfill specifications.