Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: rearrange github actions a bit #2868

Merged
merged 2 commits into from
Sep 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 0 additions & 25 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,31 +26,6 @@ jobs:
- name: Format
run: cargo fmt -- --check

coverage:
runs-on: ubuntu-latest
env:
CARGO_TERM_COLOR: always
steps:
- uses: actions/checkout@v4
- name: Install rust
uses: actions-rs/toolchain@v1
with:
profile: default
toolchain: '1.80'
override: true
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@cargo-llvm-cov
- uses: Swatinem/rust-cache@v2
- name: Generate code coverage
run: cargo llvm-cov --features ${DEFAULT_FEATURES} --workspace --codecov --output-path codecov.json -- --skip read_table_version_hdfs
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
files: codecov.json
fail_ci_if_error: true
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

build:
strategy:
fail-fast: false
Expand Down
36 changes: 36 additions & 0 deletions .github/workflows/codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: coverage

on:
push:
branches: [main, "rust-v*"]
pull_request:
branches: [main, "rust-v*"]

env:
DEFAULT_FEATURES: "azure,datafusion,s3,gcs,glue,hdfs "

jobs:
coverage:
runs-on: ubuntu-latest
env:
CARGO_TERM_COLOR: always
steps:
- uses: actions/checkout@v4
- name: Install rust
uses: actions-rs/toolchain@v1
with:
profile: default
toolchain: '1.80'
override: true
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@cargo-llvm-cov
- uses: Swatinem/rust-cache@v2
- name: Generate code coverage
run: cargo llvm-cov --features ${DEFAULT_FEATURES} --workspace --codecov --output-path codecov.json -- --skip read_table_version_hdfs
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
files: codecov.json
fail_ci_if_error: true
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
54 changes: 54 additions & 0 deletions .github/workflows/python_benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
name: python_benchmark


# This is separate from the python_build so that it doesn't need to run on the merge group
on:
push:
branches: [main]
pull_request:
branches: [main]

defaults:
run:
working-directory: ./python

jobs:
benchmark:
name: Python Benchmark
runs-on: ubuntu-latest
env:
RUSTFLAGS: "-C debuginfo=line-tables-only"
CARGO_INCREMENTAL: 0

steps:
- uses: actions/checkout@v2

- name: Setup Environment
uses: ./.github/actions/setup-env

- name: Build deltalake in release mode
run: |
python -m venv venv
source venv/bin/activate
MATURIN_EXTRA_ARGS=--release make develop

# Download previous benchmark result from cache (if exists)
- name: Download previous benchmark data
uses: actions/cache@v2
with:
path: ./cache
key: ${{ runner.os }}-benchmark

- name: Run benchmark
run: |
source venv/bin/activate
pytest tests/test_benchmark.py -m benchmark --benchmark-json output.json

- name: Store benchmark result
uses: benchmark-action/github-action-benchmark@v1
with:
tool: "pytest"
output-file-path: python/output.json
external-data-json-path: ./cache/benchmark-data.json
fail-on-alert: true

39 changes: 0 additions & 39 deletions .github/workflows/python_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,45 +97,6 @@ jobs:
python -m pytest -m "not pandas and not integration and not benchmark"
pip install pandas

benchmark:
name: Python Benchmark
runs-on: ubuntu-latest
env:
RUSTFLAGS: "-C debuginfo=line-tables-only"
CARGO_INCREMENTAL: 0

steps:
- uses: actions/checkout@v2

- name: Setup Environment
uses: ./.github/actions/setup-env

- name: Build deltalake in release mode
run: |
python -m venv venv
source venv/bin/activate
MATURIN_EXTRA_ARGS=--release make develop

# Download previous benchmark result from cache (if exists)
- name: Download previous benchmark data
uses: actions/cache@v2
with:
path: ./cache
key: ${{ runner.os }}-benchmark

- name: Run benchmark
run: |
source venv/bin/activate
pytest tests/test_benchmark.py -m benchmark --benchmark-json output.json

- name: Store benchmark result
uses: benchmark-action/github-action-benchmark@v1
with:
tool: "pytest"
output-file-path: python/output.json
external-data-json-path: ./cache/benchmark-data.json
fail-on-alert: true

test-pyspark:
name: PySpark Integration Tests
runs-on: ubuntu-latest
Expand Down
Loading