Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): make libbpf as the default attacher #1077

Merged
merged 2 commits into from
Nov 27, 2023
Merged

chore(ci): make libbpf as the default attacher #1077

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8fff251
chore(ci): make libbpf image as the default image
vimalk78 Nov 24, 2023
e04259c
fix(build): use correct cpus data file
vimalk78 Nov 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/image.yml
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be changed to Build and push kepler latest (bcc)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,14 @@ jobs:
username: ${{ secrets.username }}
password: ${{ secrets.password }}

- name: Build and push kepler latest
- name: Build and push kepler latest (bcc)
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64
push: true
tags: quay.io/sustainable_computing_io/kepler:latest
labels: latest
tags: quay.io/sustainable_computing_io/kepler:latest-bcc
labels: latest-bcc
file: build/Dockerfile

- name: Build and push kepler latest (libbpf)
Expand All @@ -41,6 +41,6 @@ jobs:
context: .
platforms: linux/amd64
push: true
tags: quay.io/sustainable_computing_io/kepler:latest-libbpf
labels: latest-libbpf
file: build/Dockerfile.libbpf.kepler
tags: quay.io/sustainable_computing_io/kepler:latest
labels: latest
file: build/Dockerfile.libbpf.kepler
2 changes: 1 addition & 1 deletion .github/workflows/image_pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,4 @@ jobs:
push: true
tags: quay.io/sustainable_computing_io/kepler
labels: ${{ github.event.inputs.commitSHA }}
file: build/Dockerfile
file: build/Dockerfile.libbpf.kepler
14 changes: 7 additions & 7 deletions .github/workflows/integration_test.yml
View file
Open in desktop
Copy link
Collaborator Author

@vimalk78 vimalk78 Nov 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file is renamed
old name: integration_test_libbpf.yml
new name: integration_test.yml
there are no other changes in file

Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on:

env:
OUTPUT_DIR: "_output/"
FILE_NAME: "kepler.tar.gz"
FILE_NAME: "kepler_libbpf.tar.gz"
ARTIFACT_DIR: "/tmp/artifacts"

jobs:
Expand All @@ -24,29 +24,29 @@ jobs:
- name: install libbpf
uses: sustainable-computing-io/[email protected]
with:
ebpfprovider: bcc
ebpfprovider: libbpf
# build kepler image
- name: build and export Kepler image
run: |
make build_containerized
sthaha marked this conversation as resolved.
Show resolved Hide resolved
make save-image
env:
IMAGE_REPO: "localhost:5001"
IMAGE_TAG: "devel"
IMAGE_NAME: "kepler"
IMAGE_TAG: "devel"
CTR_CMD: docker
IMAGE_OUTPUT_PATH: ${{env.OUTPUT_DIR}}${{env.FILE_NAME}}
# save kepler image
- name: save Kepler image as artifact
uses: actions/upload-artifact@v3
with:
name: kepler
name: keplerlibbpf
sthaha marked this conversation as resolved.
Show resolved Hide resolved
path: ${{env.OUTPUT_DIR}}${{env.FILE_NAME}}
retention-days: 1
# ref https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
# as PR or Push event, we don't keep artifact in 90 days hence use 1 day here to save resources.

integration_test:
integration_test_with_libbpf:
needs: [build-kepler]
runs-on: ubuntu-latest
strategy:
Expand All @@ -59,7 +59,7 @@ jobs:
- name: download Kepler image artifact
uses: actions/download-artifact@v3
with:
name: kepler
name: keplerlibbpf
sthaha marked this conversation as resolved.
Show resolved Hide resolved

- name: build manifest
run: make build-manifest OPTS="CI_DEPLOY"
sthaha marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -81,7 +81,7 @@ jobs:
- name: use Kepler action to deploy cluster
uses: sustainable-computing-io/[email protected]
with:
ebpfprovider: bcc
ebpfprovider: libbpf
cluster_provider: ${{matrix.cluster_provider}}
local_dev_cluster_version: v0.0.3

Expand Down
22 changes: 11 additions & 11 deletions ...hub/workflows/integration_test_libbpf.yml → .github/workflows/integration_test_bcc.yml
View file
Open in desktop
Copy link
Collaborator Author

@vimalk78 vimalk78 Nov 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file is renamed
old name: integration_test.yml
new name: integration_test_bcc.yml
there are no other changes in file

Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
name: Integration test with libbpf
name: Integration test with bcc

on:
workflow_call:

env:
OUTPUT_DIR: "_output/"
FILE_NAME: "kepler_libbpf.tar.gz"
FILE_NAME: "kepler-bcc.tar.gz"
ARTIFACT_DIR: "/tmp/artifacts"

jobs:
build-kepler_with_libbpf:
build-kepler-with-bcc:
runs-on: ubuntu-latest
steps:
# checkout source code
Expand All @@ -21,33 +21,33 @@ jobs:
with:
go-version-file: go.mod
# set up ebpf
- name: install libbpf
- name: install bcc
uses: sustainable-computing-io/[email protected]
with:
ebpfprovider: libbpf
ebpfprovider: bcc
# build kepler image
- name: build and export Kepler image
run: |
make build_containerized
make save-image
env:
IMAGE_REPO: "localhost:5001"
IMAGE_NAME: "kepler"
IMAGE_TAG: "devel"
IMAGE_NAME: "kepler"
CTR_CMD: docker
IMAGE_OUTPUT_PATH: ${{env.OUTPUT_DIR}}${{env.FILE_NAME}}
# save kepler image
- name: save Kepler image as artifact
uses: actions/upload-artifact@v3
with:
name: keplerlibbpf
name: kepler
sthaha marked this conversation as resolved.
Show resolved Hide resolved
path: ${{env.OUTPUT_DIR}}${{env.FILE_NAME}}
retention-days: 1
# ref https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
# as PR or Push event, we don't keep artifact in 90 days hence use 1 day here to save resources.

integration_test_with_libbpf:
needs: [build-kepler_with_libbpf]
integration_test:
needs: [build-kepler-with-bcc]
runs-on: ubuntu-latest
strategy:
matrix:
Expand All @@ -59,7 +59,7 @@ jobs:
- name: download Kepler image artifact
uses: actions/download-artifact@v3
with:
name: keplerlibbpf
name: kepler

- name: build manifest
run: make build-manifest OPTS="CI_DEPLOY"
Expand All @@ -81,7 +81,7 @@ jobs:
- name: use Kepler action to deploy cluster
uses: sustainable-computing-io/[email protected]
with:
ebpfprovider: libbpf
ebpfprovider: bcc
cluster_provider: ${{matrix.cluster_provider}}
local_dev_cluster_version: v0.0.3

Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/pull_request.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ jobs:
developer_local:
uses: ./.github/workflows/developer_local.yml
# for each PR run integration test
integration_test_bcc:
uses: ./.github/workflows/integration_test_bcc.yml
# for each PR run integration test
integration_test:
uses: ./.github/workflows/integration_test.yml
# for each PR run integration test
integration_test_libbpf:
uses: ./.github/workflows/integration_test_libbpf.yml


6 changes: 3 additions & 3 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,17 +130,17 @@ jobs:
context: .
platforms: linux/amd64
push: true
tags: quay.io/sustainable_computing_io/kepler:latest, quay.io/sustainable_computing_io/kepler:${{ github.event.inputs.release }}
tags: quay.io/sustainable_computing_io/kepler:latest, quay.io/sustainable_computing_io/kepler:${{ github.event.inputs.release }}-bcc
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we have the latest tag here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets take this question in some other PR

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I ask is because I am assuming that the previous release tagged bcc as latest and now we want to switch that to libbpf. Would that be the case if we add latest here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tags field for both images has quay.io/sustainable_computing_io/kepler:latest

labels: latest, ${{ github.event.inputs.release }}, bcc
file: build/Dockerfile

- name: Build and push kepler libbpf latest image
- name: Build and push kepler latest libbpf image
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64
push: true
tags: quay.io/sustainable_computing_io/kepler:latest, quay.io/sustainable_computing_io/kepler:${{ github.event.inputs.release }}-libbpf
tags: quay.io/sustainable_computing_io/kepler:latest, quay.io/sustainable_computing_io/kepler:${{ github.event.inputs.release }}
labels: latest, ${{ github.event.inputs.release }}, libbpf
file: build/Dockerfile.libbpf.kepler

Expand Down
15 changes: 12 additions & 3 deletions .github/workflows/unit_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,27 @@ jobs:
env:
GOPATH: /home/runner/go
GOBIN: /home/runner/go/bin
- name: install bcc
- name: install libbpf
uses: sustainable-computing-io/[email protected]
with:
ebpfprovider: bcc
ebpfprovider: libbpf
- name: Prepare environment
run: |
sudo apt-get install -y cpuid
cd doc/ && sudo ./dev/prepare_dev_env.sh && cd -
git config --global --add safe.directory /kepler
- name: Run
run: |
make test-verbose
sudo apt remove libbpf-dev
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if libbpf-dev is not enough of our libbpf driver, we should make a deb package at https://github.com/sustainable-computing-io/kepler-ci-artifacts and update our kepler-action.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sustainable-computing-io/kepler-action#87 created accordingly.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rootfs / @vimalk78 / @sthaha could you please help double confirm if we need sustainable-computing-io/kepler-action#87 ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new issue related to this PR as #1086

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if libbpf-dev is not enough of our libbpf driver, we should make a deb package at

yes @SamYuan1990, opening a PR for the same. in this PR i wanted to test what works. will update kepler-action, and then we dont needs this setup in CI

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vimalk78 I am going to sleep, please take a look at #1086 briefly, the libbpf-dev ubuntu distribution has the static link file but it seems the version for ubuntu distribution is too old. or mismatch with the golang code. hence I am worry about if libbpf is ready for use? as if we build it from source each time.... dependence maintain as OpenSSF or SBOM sounds like disaster for us.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SamYuan1990 pls check this PR sustainable-computing-io/kepler-action#90
Cc: @rootfs

mkdir temp-libbpf
cd temp-libbpf
git clone https://github.com/libbpf/libbpf
cd libbpf/src
sudo make install_headers
sudo make install_uapi_headers
sudo prefix=/usr BUILD_STATIC_ONLY=y make install
cd ../../../
ATTACHER_TAG=libbpf make test-verbose
go tool cover -func=coverage.out -o=coverage.out
- name: Go Coverage Badge # Pass the `coverage.out` output to this action
uses: tj-actions/coverage-badge-go@v2
Expand Down
3 changes: 2 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ ifdef ATTACHER_TAG
ifeq ($(ATTACHER_TAG),libbpf)
LIBBPF_HEADERS := /usr/include/bpf
KEPLER_OBJ_SRC := $(SRC_ROOT)/bpfassets/libbpf/bpf.o/$(GOARCH)_kepler.bpf.o
LIBBPF_OBJ := /usr/lib64/libbpf.a
LIBBPF_OBJ ?= /usr/lib64/libbpf.a
endif
else
# auto determine
Expand Down Expand Up @@ -277,6 +277,7 @@ test: ginkgo-set tidy-vendor

test-verbose: ginkgo-set tidy-vendor
@echo TAGS=$(GO_BUILD_TAGS)
@echo GOENV=$(GOENV)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need echo this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the information echoed in this line is helpful in checking errors we get from go build, if any.

@$(GOENV) go test -tags $(GO_BUILD_TAGS) -covermode=atomic -coverprofile=coverage.out -v $$(go list ./... | grep pkg | grep -v bpfassets) --race --bench=. -cover --count=1 --vet=all

test-mac-verbose: ginkgo-set
Expand Down
5 changes: 3 additions & 2 deletions build/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ FROM quay.io/sustainable_computing_io/kepler_base:ubi-9-bcc-0.26
COPY --from=builder /opt/app-root/src/github.com/sustainable-computing-io/kepler/_output/bin/kepler /usr/bin/kepler

RUN mkdir -p /var/lib/kepler/data
COPY --from=builder /opt/app-root/src/github.com/sustainable-computing-io/kepler/data/normalized_cpu_arch.csv /var/lib/kepler/data/normalized_cpu_arch.csv
COPY --from=builder /opt/app-root/src/github.com/sustainable-computing-io/kepler/data/cpus.yaml /var/lib/kepler/data/cpus.yaml


# copy model weight
COPY --from=builder /opt/app-root/src/github.com/sustainable-computing-io/kepler/data/model_weight/acpi_AbsPowerModel.json /var/lib/kepler/data/acpi_AbsPowerModel.json
Expand All @@ -41,4 +42,4 @@ RUN mkdir -p /usr/share/kepler/kernel_sources

COPY --from=quay.io/sustainable_computing_io/kepler_kernel_source_images:ubi9 /usr/src/kernels /usr/share/kepler/kernel_sources

ENTRYPOINT ["/usr/bin/kepler"]
ENTRYPOINT ["/usr/bin/kepler"]
4 changes: 2 additions & 2 deletions doc/dev/prepare_dev_env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ echo "copy data files"
DATAPATH="/var/lib/kepler/data/"
mkdir -p ${DATAPATH}

cp ../data/normalized_cpu_arch.csv ${DATAPATH}
cp ../data/cpus.yaml ${DATAPATH}

mkdir -p /var/lib/kepler/data/
cp ../data/model_weight/acpi_AbsPowerModel.json ${DATAPATH}/acpi_AbsPowerModel.json
cp ../data/model_weight/acpi_DynPowerModel.json ${DATAPATH}/acpi_DynPowerModel.json
cp ../data/model_weight/rapl_AbsPowerModel.json ${DATAPATH}/rapl_AbsPowerModel.json
cp ../data/model_weight/rapl_DynPowerModel.json ${DATAPATH}/rapl_DynPowerModel.json
cp ../data/model_weight/rapl_DynPowerModel.json ${DATAPATH}/rapl_DynPowerModel.json
2 changes: 1 addition & 1 deletion packaging/rpm/kepler.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ CROSS_BUILD_BINDIR=_output/bin
GOARCH=amd64
%endif

make _build_local GOOS=${GOOS} GOARCH=${GOARCH}
make _build_local GOOS=${GOOS} GOARCH=${GOARCH} ATTACHER_TAG=libbpf
sthaha marked this conversation as resolved.
Show resolved Hide resolved

cp ./${CROSS_BUILD_BINDIR}/${GOOS}_${GOARCH}/kepler ./_output/kepler

Expand Down