chore(ci): make libbpf as the default attacher #1077

vimalk78 · 2023-11-24T04:19:21Z

tag existing 'latest' image as 'latest-bcc'
tag existing 'latest-libbpf' as 'latest'
make libbpf image as default released image
use libbpf as attacher for PR image
make unit_test run with libbpf attacher
build kepler with libbpf attacher for rpm build
rename integration_test to integration_test_bcc
rename integration_test_libbpf to integration_test

vimalk78 · 2023-11-24T04:43:50Z

Cc: @rootfs @SamYuan1990

.github/workflows/image.yml

packaging/rpm/kepler.spec

vimalk78 · 2023-11-24T06:20:31Z

@rootfs @marceloamaral can i pls get rights to request a PR review?

vimalk78 · 2023-11-24T06:26:44Z

.github/workflows/integration_test.yml

this file is renamed
old name: integration_test_libbpf.yml
new name: integration_test.yml
there are no other changes in file

vimalk78 · 2023-11-24T06:27:18Z

.github/workflows/integration_test_bcc.yml

this file is renamed
old name: integration_test.yml
new name: integration_test_bcc.yml
there are no other changes in file

vprashar2929 · 2023-11-24T06:38:50Z

.github/workflows/image.yml

This should be changed to Build and push kepler latest (bcc)

.github/workflows/integration_test.yml

sthaha · 2023-11-24T08:01:14Z

.github/workflows/integration_test_bcc.yml


 on:
  workflow_call:

 env:
  OUTPUT_DIR: "_output/"
-  FILE_NAME: "kepler_libbpf.tar.gz"
+  FILE_NAME: "kepler.tar.gz"


kepe

Suggested change

FILE_NAME: "kepler.tar.gz"

FILE_NAME: "kepler_bcc.tar.gz"

.github/workflows/integration_test_bcc.yml

sthaha · 2023-11-24T08:01:46Z

.github/workflows/integration_test_bcc.yml

@@ -24,30 +24,30 @@ jobs:
      - name: install libbpf


Suggested change

- name: install libbpf

- name: install bcc

.github/workflows/integration_test_bcc.yml

.github/workflows/integration_test.yml

sthaha · 2023-11-24T08:16:36Z

.github/workflows/release.yml

@@ -130,17 +130,17 @@ jobs:
          context: .
          platforms: linux/amd64
          push: true
-          tags: quay.io/sustainable_computing_io/kepler:latest, quay.io/sustainable_computing_io/kepler:${{ github.event.inputs.release }}
+          tags: quay.io/sustainable_computing_io/kepler:latest, quay.io/sustainable_computing_io/kepler:${{ github.event.inputs.release }}-bcc


should we have the latest tag here?

lets take this question in some other PR

The reason I ask is because I am assuming that the previous release tagged bcc as latest and now we want to switch that to libbpf. Would that be the case if we add latest here?

tags field for both images has quay.io/sustainable_computing_io/kepler:latest

vimalk78 · 2023-11-24T15:08:36Z

@rootfs we wanted this PR to merge before we merged xgboost PR #1052
now can kepler compile without xgboost ? kepler needs #include <xgboost/c_api.h> to build.

sthaha · 2023-11-24T23:22:05Z

@rootfs could we please revert #1052 , wait until this PR gets merged so that a release without xgboost can be made?

- tag existing 'latest' image as 'latest-bcc' - tag existing 'latest-libbpf' as 'latest' - make libbpf image as default released image - use libbpf as attacher for PR image - make unit_test run with libbpf attacher - updated unit_test workflow to install libbpf from source - build kepler with libbpf attacher for rpm build - rename integration_test to integration_test_bcc - rename integration_test_libbpf to integration_test Signed-off-by: Vimal Kumar <[email protected]>

Updated following files to not use normalized_cpu_arch.csv, and use cpus.yaml - updated build/Dockerfile - doc/dev/prepare_dev_env.sh Signed-off-by: Vimal Kumar <[email protected]>

rootfs · 2023-11-27T12:08:02Z

Per @SamYuan1990 comment, the bcc code will held in the source and be removed in the next release.
Following this PR, the libbpf ebpf module should be also built on the CI. It is currently updated manually.

SamYuan1990 · 2023-11-27T12:09:12Z

Per @SamYuan1990 comment, the bcc code will held in the source and be removed in the next release. Following this PR, the libbpf ebpf module should be also built on the CI. It is currently updated manually.

so we will remove bcc code after 0.8 release?

rootfs · 2023-11-27T12:12:15Z

@vimalk78 we need a CI step to make genlibbpf

@SamYuan1990 yes, let's remove all the bcc code in 0.8 once we are fully confident on libbpf.

SamYuan1990 · 2023-11-27T12:17:22Z

.github/workflows/unit_test.yml

    - name: Prepare environment 
      run: |
          sudo apt-get install -y cpuid
          cd doc/ && sudo ./dev/prepare_dev_env.sh && cd - 
          git config --global --add safe.directory /kepler
    - name: Run 
      run: |
-          make test-verbose
+          sudo apt remove libbpf-dev


if libbpf-dev is not enough of our libbpf driver, we should make a deb package at https://github.com/sustainable-computing-io/kepler-ci-artifacts and update our kepler-action.

sustainable-computing-io/kepler-action#87 created accordingly.

@rootfs / @vimalk78 / @sthaha could you please help double confirm if we need sustainable-computing-io/kepler-action#87 ?

new issue related to this PR as #1086

if libbpf-dev is not enough of our libbpf driver, we should make a deb package at

yes @SamYuan1990, opening a PR for the same. in this PR i wanted to test what works. will update kepler-action, and then we dont needs this setup in CI

@vimalk78 I am going to sleep, please take a look at #1086 briefly, the libbpf-dev ubuntu distribution has the static link file but it seems the version for ubuntu distribution is too old. or mismatch with the golang code. hence I am worry about if libbpf is ready for use? as if we build it from source each time.... dependence maintain as OpenSSF or SBOM sounds like disaster for us.

@SamYuan1990 pls check this PR sustainable-computing-io/kepler-action#90
Cc: @rootfs

SamYuan1990 · 2023-11-27T12:19:41Z

Makefile

@@ -277,6 +277,7 @@ test: ginkgo-set tidy-vendor

 test-verbose: ginkgo-set tidy-vendor
 	@echo TAGS=$(GO_BUILD_TAGS)
+	@echo GOENV=$(GOENV)


do we need echo this?

the information echoed in this line is helpful in checking errors we get from go build, if any.

SamYuan1990 · 2023-11-27T12:20:38Z

some further comments been added related pr code but not release process.

…-io#1077) * chore(ci): make libbpf image as the default image - tag existing 'latest' image as 'latest-bcc' - tag existing 'latest-libbpf' as 'latest' - make libbpf image as default released image - use libbpf as attacher for PR image - make unit_test run with libbpf attacher - updated unit_test workflow to install libbpf from source - build kepler with libbpf attacher for rpm build - rename integration_test to integration_test_bcc - rename integration_test_libbpf to integration_test Signed-off-by: Vimal Kumar <[email protected]> * fix(build): use correct cpus data file Updated following files to not use normalized_cpu_arch.csv, and use cpus.yaml - updated build/Dockerfile - doc/dev/prepare_dev_env.sh Signed-off-by: Vimal Kumar <[email protected]> --------- Signed-off-by: Vimal Kumar <[email protected]>

vimalk78 force-pushed the libbpf-default-image branch 2 times, most recently from cceecea to 0eef904 Compare November 24, 2023 04:40

vimalk78 changed the title ~~chore(ci): make libbpf image as the default image~~ chore(ci): make libbpf as the default attacher Nov 24, 2023

sthaha reviewed Nov 24, 2023

View reviewed changes

.github/workflows/image.yml Outdated Show resolved Hide resolved

sthaha reviewed Nov 24, 2023

View reviewed changes

.github/workflows/image.yml Outdated Show resolved Hide resolved

sthaha reviewed Nov 24, 2023

View reviewed changes

packaging/rpm/kepler.spec Show resolved Hide resolved

sthaha marked this pull request as draft November 24, 2023 05:34

vimalk78 force-pushed the libbpf-default-image branch 2 times, most recently from 6e77fa4 to 08227c7 Compare November 24, 2023 06:19

vimalk78 marked this pull request as ready for review November 24, 2023 06:19

vimalk78 commented Nov 24, 2023

View reviewed changes

vprashar2929 reviewed Nov 24, 2023

View reviewed changes

vimalk78 force-pushed the libbpf-default-image branch 4 times, most recently from 4aaf4d1 to e6db42a Compare November 24, 2023 07:20

sthaha reviewed Nov 24, 2023

View reviewed changes

.github/workflows/integration_test.yml Show resolved Hide resolved

sthaha reviewed Nov 24, 2023

View reviewed changes

.github/workflows/integration_test.yml Show resolved Hide resolved

sthaha reviewed Nov 24, 2023

View reviewed changes

.github/workflows/integration_test_bcc.yml Outdated Show resolved Hide resolved

sthaha reviewed Nov 24, 2023

View reviewed changes

.github/workflows/integration_test_bcc.yml Show resolved Hide resolved

sthaha reviewed Nov 24, 2023

View reviewed changes

.github/workflows/integration_test.yml Show resolved Hide resolved

sthaha reviewed Nov 24, 2023

View reviewed changes

.github/workflows/integration_test.yml Show resolved Hide resolved

sthaha reviewed Nov 24, 2023

View reviewed changes

vimalk78 force-pushed the libbpf-default-image branch 2 times, most recently from 9fb5b58 to b71c37a Compare November 24, 2023 10:12

rootfs assigned SamYuan1990 and vprashar2929 Nov 24, 2023

This was referenced Nov 24, 2023

Revert "Support local xgboost regressor" #1080

Closed

Revert "Support local xgboost regressor (#1052)" #1081

Merged

vimalk78 force-pushed the libbpf-default-image branch 11 times, most recently from e4d38f5 to 9b991e5 Compare November 27, 2023 11:44

vimalk78 added 2 commits November 27, 2023 17:19

fix(build): use correct cpus data file

e04259c

Updated following files to not use normalized_cpu_arch.csv, and use cpus.yaml - updated build/Dockerfile - doc/dev/prepare_dev_env.sh Signed-off-by: Vimal Kumar <[email protected]>

vimalk78 force-pushed the libbpf-default-image branch from 9b991e5 to e04259c Compare November 27, 2023 11:49

rootfs approved these changes Nov 27, 2023

View reviewed changes

rootfs merged commit e9e1c8b into sustainable-computing-io:main Nov 27, 2023

SamYuan1990 reviewed Nov 27, 2023

View reviewed changes

SamYuan1990 mentioned this pull request Nov 27, 2023

if libbpf-dev is not enough of our libbpf driver, we should make a deb package at https://github.com/sustainable-computing-io/kepler-ci-artifacts and update our kepler-action. sustainable-computing-io/kepler-action#87

Closed

SamYuan1990 reviewed Nov 27, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(ci): make libbpf as the default attacher #1077

chore(ci): make libbpf as the default attacher #1077

vimalk78 commented Nov 24, 2023 •

edited

Loading

vimalk78 commented Nov 24, 2023

vimalk78 commented Nov 24, 2023

vimalk78 Nov 24, 2023 •

edited

Loading

vimalk78 Nov 24, 2023 •

edited

Loading

vprashar2929 Nov 24, 2023

vimalk78 Nov 24, 2023

sthaha Nov 24, 2023

sthaha Nov 24, 2023

sthaha Nov 24, 2023

vimalk78 Nov 24, 2023

sthaha Nov 24, 2023

vimalk78 Nov 24, 2023

vimalk78 commented Nov 24, 2023 •

edited

Loading

sthaha commented Nov 24, 2023

rootfs commented Nov 27, 2023

SamYuan1990 commented Nov 27, 2023

rootfs commented Nov 27, 2023

SamYuan1990 Nov 27, 2023

SamYuan1990 Nov 27, 2023

SamYuan1990 Nov 27, 2023

SamYuan1990 Nov 27, 2023

vimalk78 Nov 27, 2023

SamYuan1990 Nov 27, 2023

vimalk78 Nov 27, 2023

SamYuan1990 Nov 27, 2023

vimalk78 Nov 27, 2023

SamYuan1990 commented Nov 27, 2023

chore(ci): make libbpf as the default attacher #1077

chore(ci): make libbpf as the default attacher #1077

Conversation

vimalk78 commented Nov 24, 2023 • edited Loading

vimalk78 commented Nov 24, 2023

vimalk78 commented Nov 24, 2023

vimalk78 Nov 24, 2023 • edited Loading

Choose a reason for hiding this comment

vimalk78 Nov 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vimalk78 commented Nov 24, 2023 • edited Loading

sthaha commented Nov 24, 2023

rootfs commented Nov 27, 2023

SamYuan1990 commented Nov 27, 2023

rootfs commented Nov 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SamYuan1990 commented Nov 27, 2023

vimalk78 commented Nov 24, 2023 •

edited

Loading

vimalk78 Nov 24, 2023 •

edited

Loading

vimalk78 Nov 24, 2023 •

edited

Loading

vimalk78 commented Nov 24, 2023 •

edited

Loading