Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FLPATH-1066: swf-builder: generate manifests using kn-workflow #1751

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

rgolangh
Copy link

@rgolangh rgolangh commented Feb 28, 2024

Generate manifests and save them in the runtime image.

Generating manifests is an opt-in as part of the build ran by /launch/build-app.sh
When setting GEN_MANIFESTS=true during container build the build-app.sh
script will call a new generate-manifests.sh script that will use
kn-workflow cli and will store the manifests under
$KOGITO_HOME/build/manifests

Later in the runtime image user can copy and store those manifests for
later use:

COPY --from=builder --chown=185 /home/kogito/build/manifests /deployments/manifests

The manifests can be extracted for example by echoing them from the image:

podman run builder:latest cat /deployment/manifests/* > manifests-all.yaml

or by podman cp:

podman cp $(podman create builder:latest):/deployment/manifests .

Fixes: #1739
Related-to: https://issues.redhat.com/browse/FLPATH-1066

Signed-off-by: Roy Golan [email protected]

Many thanks for submitting your Pull Request ❤️!

Please make sure your PR meets the following requirements:

  • You have read the contributors guide
  • Pull Request title is properly formatted: [KOGITO|RHPAM-XYZ] Subject
  • Pull Request contains link to the JIRA issue
  • Pull Request contains description of the issue
  • Pull Request does not include fixes for issues other than the main ticket
  • Your feature/bug fix has a testcase that verifies it
  • You've tested the new feature/bug fix in an actual OpenShift cluster
  • You've added a RELEASE_NOTES.md entry regarding this change
How to retest this PR or trigger a specific build:
  • (Re)run Jenkins tests
    Please add comment: Jenkins [test|retest] this
How to backport a pull request to a different branch?

In order to automatically create a backporting pull request please add one or more labels having the following format backport-<branch-name>, where <branch-name> is the name of the branch where the pull request must be backported to (e.g., backport-7.67.x to backport the original PR to the 7.67.x branch).

NOTE: backporting is an action aiming to move a change (usually a commit) from a branch (usually the main one) to another one, which is generally referring to a still maintained release branch. Keeping it simple: it is about to move a specific change or a set of them from one branch to another.

Once the original pull request is successfully merged, the automated action will create one backporting pull request per each label (with the previous format) that has been added.

If something goes wrong, the author will be notified and at this point a manual backporting is needed.

NOTE: this automated backporting is triggered whenever a pull request on main branch is labeled or closed, but both conditions must be satisfied to get the new PR created.

@rgolangh rgolangh marked this pull request as ready for review February 29, 2024 14:42
@rgolangh rgolangh changed the title WIP: swf-builder generate manifests using kn-workflow FLPATH-1066: swf-builder: generate manifests using kn-workflow Feb 29, 2024
Copy link
Member

@ricardozanini ricardozanini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks! A few comments if you don't mind. Also, can you please create one or more bats tests to validate this new module and the builder calling the new script?


SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
INSTALL_DIR=/usr/local/bin
curl -L https://github.com/rgolangh/kie-tools/releases/download/0.0.2/kn-workflow-linux-amd64 -o "${INSTALL_DIR}"/kn-workflow
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the official repo and uname to identify the OS and arch in this URL?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

till now I'm using this private build because the latest fixes with ''--namespace" flag are still not published. @ederign is this still true?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rgolangh not yet, but soon we will have kn workflow cli 1.32 published together with serverless release.

modules/kogito-workflow-manifests/install.sh Outdated Show resolved Hide resolved
modules/kogito-workflow-manifests/module.yaml Show resolved Hide resolved
@rgolangh rgolangh force-pushed the add-kn-workflow-to-builder branch 3 times, most recently from e963f7f to eda46ff Compare February 29, 2024 15:26
@apache apache deleted a comment from kie-ci3 Feb 29, 2024
[ ! -f "${KOGITO_HOME}"/bin/kn-workflow_invocation.txt ]
}

mock_bin() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pretty cool!

@apache apache deleted a comment from kie-ci3 Feb 29, 2024
@apache apache deleted a comment from kie-ci3 Feb 29, 2024
@apache apache deleted a comment from kie-ci3 Feb 29, 2024
@rgolangh rgolangh force-pushed the add-kn-workflow-to-builder branch 3 times, most recently from e419825 to 86fedf8 Compare March 3, 2024 08:21
Generate manifests and save them in the runtime image.

Generating manifests is an opt-in as part of the build ran by /launch/build-app.sh
When setting GEN_MANIFESTS=true during container build the build-app.sh
script will call a new generate-manifests.sh script that will use
_kn-workflow_ cli and will store the manifests under
$KOGITO_HOME/build/manifests

Later in the runtime image user can copy and store those manifests for
later use:
  COPY --from=builder --chown=185 /home/kogito/build/manifests /deployments/manifests

The manifests can be extracted for example by echoing them from the image:

  podman run builder:latest cat /deployment/manifests/* > manifests-all.yaml

or by podman cp:

  podman cp $(podman create builder:latest):/deployment/manifests .

Signed-off-by: Roy Golan <[email protected]>
@kie-ci3
Copy link
Contributor

kie-ci3 commented Mar 3, 2024

PR job #121 was: FAILURE
Possible explanation: Pipeline failure or project build failure

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/121/display/redirect
See console log:

Console Logs [Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-data-index-ephemeral: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-jobs-service-ephemeral: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-runtime-jvm: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-management-console: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-task-console: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-swf-builder: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-s2i-builder: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-jobs-service-allinone: https://github.com//pull/1751 completed: FAILURE
[Pipeline] error
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch kogito-jobs-service-allinone
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ wget --no-check-certificate -qO - 'https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/121/api/json?depth=0'
[Pipeline] readJSON
[Pipeline] sh

@domhanak
Copy link

domhanak commented Mar 4, 2024

@tomasdavidorg fyi: concerns kn cli

@apache apache deleted a comment from kie-ci3 Mar 4, 2024
@apache apache deleted a comment from kie-ci3 Mar 4, 2024
@apache apache deleted a comment from kie-ci3 Mar 4, 2024
@apache apache deleted a comment from kie-ci3 Mar 4, 2024
@apache apache deleted a comment from kie-ci3 Mar 4, 2024
@ricardozanini
Copy link
Member

Errors in the CI are not related. These are tied to the recent renaming work we are doing on runtimes libraries.

@ricardozanini
Copy link
Member

@jakubschwan @tomasdavidorg can you guys verify?

Copy link

@tomasdavidorg tomasdavidorg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ricardozanini Is this still relevant with the latest planned changes? I mean the Unblocking and releasing Apache KIE 10. The kie-tools will be the last in the queue and it does not seems to be ok to include them here. But maybe I am missing something.


artifacts:
- name: kn-workflow-x86_64
url: https://github.com/rgolangh/kie-tools/releases/download/0.0.2/kn-workflow-linux-amd64

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The link does not seems to be correct as it points to the fork.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will review this once we move kn-workflow to the operator.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a release I can use already?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, there's not.

@ricardozanini
Copy link
Member

@tomasdavidorg We will continue with this since we are moving to kie-tools just temporarily. After the release, we will fix the release stream.

Additionally, the workflow CLI will be in the operator repo.

@kie-ci3
Copy link
Contributor

kie-ci3 commented May 6, 2024

PR job #1 was: FAILURE
Possible explanation: Pipeline failure or project build failure

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/1/display/redirect
See console log:

Console Logs From https://github.com/rgolangh/incubator-kie-kogito-images
* branch add-kn-workflow-to-builder -> FETCH_HEAD
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint: git config pull.rebase false # merge (the default strategy)
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] echo

-------------------------------------------------------------
[ERROR] Can't merge source into Target. Please rebase PR branch.
-------------------------------------------------------------
Source: git://github.com/rgolangh/incubator-kie-kogito-images add-kn-workflow-to-builder
Target: a595d8c Upgrade to and align with Quarkus 3.8.4 LTS release. (#1760)
-------------------------------------------------------------

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Validate CeKit Image and Modules descriptors)
Stage "Validate CeKit Image and Modules descriptors" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build & Test Images)
Stage "Build & Test Images" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ wget --no-check-certificate -qO - 'https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/1/api/json?depth=0'
[Pipeline] readJSON
[Pipeline] sh

@kie-ci3
Copy link
Contributor

kie-ci3 commented Jun 4, 2024

PR job #3 was: FAILURE
Possible explanation: Pipeline failure or project build failure

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/3/display/redirect
See console log:

Console Logs From https://github.com/rgolangh/incubator-kie-kogito-images
* branch add-kn-workflow-to-builder -> FETCH_HEAD
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint: git config pull.rebase false # merge (the default strategy)
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] echo

-------------------------------------------------------------
[ERROR] Can't merge source into Target. Please rebase PR branch.
-------------------------------------------------------------
Source: git://github.com/rgolangh/incubator-kie-kogito-images add-kn-workflow-to-builder
Target: 431715c Add @jstastny-cz and @rodrigonull to CODEOWNERS (#1771)
-------------------------------------------------------------

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Validate CeKit Image and Modules descriptors)
Stage "Validate CeKit Image and Modules descriptors" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build & Test Images)
Stage "Build & Test Images" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ wget --no-check-certificate -qO - 'https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/3/api/json?depth=0'
[Pipeline] readJSON
[Pipeline] sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Generate manifests as part of build time
6 participants