Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Stub out CI for new python package #661

Merged
merged 95 commits into from
Aug 19, 2023
Merged
Show file tree
Hide file tree
Changes from 58 commits
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
4e8b833
ci: Stub out CI for new python package
bjchambers Aug 11, 2023
8920b80
python workflows (no release yet)
bjchambers Aug 15, 2023
5e74d54
fix syntax
bjchambers Aug 15, 2023
d789434
poetry
bjchambers Aug 15, 2023
1d3795e
default working directory
bjchambers Aug 15, 2023
e9e7fb3
install poetry with pip
bjchambers Aug 15, 2023
b27d43d
fix poetry install
bjchambers Aug 15, 2023
40bba15
peotry->poetry
bjchambers Aug 15, 2023
40bd544
pipx
bjchambers Aug 15, 2023
bfccac0
simplify
bjchambers Aug 15, 2023
486b890
maturin working directory
bjchambers Aug 15, 2023
022f99b
lint
bjchambers Aug 15, 2023
70f57b6
pydocstyle
bjchambers Aug 15, 2023
3f20186
maturin working directory
bjchambers Aug 15, 2023
23dc53d
pydocstyle
bjchambers Aug 15, 2023
aec6771
no fail fast
bjchambers Aug 15, 2023
baa3ee6
install proto; disable s390x
bjchambers Aug 15, 2023
038865b
combine build and test; disable failing architectures
bjchambers Aug 16, 2023
51e08c2
multi-line versions
bjchambers Aug 16, 2023
45fcb60
tweak tests
bjchambers Aug 16, 2023
d029ee7
tweak run on arch
bjchambers Aug 16, 2023
91c2336
attempt to fix paths
bjchambers Aug 16, 2023
cbf0f12
debugging
bjchambers Aug 16, 2023
a23289a
attempt
bjchambers Aug 16, 2023
7764526
wip
bjchambers Aug 16, 2023
0832ac6
wip
bjchambers Aug 16, 2023
081a451
lto
bjchambers Aug 16, 2023
5c38959
work on os x build
bjchambers Aug 16, 2023
79f5a30
grouping
bjchambers Aug 16, 2023
51d0e83
install protoc
bjchambers Aug 16, 2023
7e48cef
wip: installing
bjchambers Aug 16, 2023
3dec3d9
new approach based on https://github.com/adriangb/graphlib2/blob/main…
bjchambers Aug 16, 2023
9a2ea60
fix linux working-directory
bjchambers Aug 16, 2023
0faeee2
zig on linux?
bjchambers Aug 16, 2023
e95b007
fix linux protoc version
bjchambers Aug 16, 2023
84eea49
protoc for windows
bjchambers Aug 16, 2023
05531c7
fixes
bjchambers Aug 16, 2023
2ffc5d0
fix lint
bjchambers Aug 16, 2023
fc2c1a0
Merge branch 'py-lint' into python-ci
bjchambers Aug 16, 2023
75046d0
wip
bjchambers Aug 17, 2023
58fd8b4
poetry on linux
bjchambers Aug 17, 2023
355796e
try again
bjchambers Aug 17, 2023
383f6e2
attempt
bjchambers Aug 17, 2023
a1b15f8
attempt
bjchambers Aug 17, 2023
ad366d3
don't mypy docs
bjchambers Aug 17, 2023
656485d
fix
bjchambers Aug 17, 2023
7e5da42
wip: docs
bjchambers Aug 17, 2023
6169eee
specify dependencies
bjchambers Aug 17, 2023
5772ac7
fix pyproject.toml
bjchambers Aug 17, 2023
eaec82d
wip! docs
bjchambers Aug 17, 2023
8bb4ed0
Merge branch 'main' into python-ci
bjchambers Aug 17, 2023
eec5dd6
update lock
bjchambers Aug 17, 2023
6cbc572
diagnostic ls
bjchambers Aug 17, 2023
d1d2583
Merge branch 'main' into py-lint
bjchambers Aug 17, 2023
b8ae418
add more fix
bjchambers Aug 17, 2023
a300b12
Merge branch 'py-lint' into python-ci
bjchambers Aug 17, 2023
10e8259
Merge branch 'main' into python-ci
bjchambers Aug 17, 2023
ba33bed
fixes
bjchambers Aug 17, 2023
584d58c
use matrix to parallelize macos builds
bjchambers Aug 17, 2023
aaa63c5
add a somke run; skip wheels on non-push
bjchambers Aug 17, 2023
38474e8
update versions
bjchambers Aug 17, 2023
01ea376
Merge branch 'main' into python-ci
bjchambers Aug 17, 2023
45b9111
fix linting
bjchambers Aug 17, 2023
eef9f75
don't upload debug wheel
bjchambers Aug 17, 2023
b4467e7
debug
bjchambers Aug 17, 2023
9fe25d1
dev profile
bjchambers Aug 17, 2023
ce325c6
use mimalloc
bjchambers Aug 17, 2023
e6a8e49
local dynamic tls
bjchambers Aug 17, 2023
74600d6
add note
bjchambers Aug 17, 2023
3c4f58f
no zig
bjchambers Aug 18, 2023
15b9eb6
install clang
bjchambers Aug 18, 2023
bbbda99
epel release?
bjchambers Aug 18, 2023
5104f45
make cache
bjchambers Aug 18, 2023
6bcf1a5
scl
bjchambers Aug 18, 2023
ff330f1
update manylinux
bjchambers Aug 18, 2023
a118277
manylinux 2.28
bjchambers Aug 18, 2023
3494229
protobuf
bjchambers Aug 18, 2023
93c4eae
protobuf-compiler
bjchambers Aug 18, 2023
e2de757
target
bjchambers Aug 18, 2023
afc4394
protobuf-devel
bjchambers Aug 18, 2023
e03788c
gcc-toolset
bjchambers Aug 18, 2023
154cdc5
plotly
bjchambers Aug 18, 2023
0a60802
zig?
bjchambers Aug 18, 2023
d79d97c
ld?
bjchambers Aug 18, 2023
4ec02db
zig?
bjchambers Aug 18, 2023
049b167
binutils
bjchambers Aug 18, 2023
e80238f
lld
bjchambers Aug 18, 2023
4b91877
mirror
bjchambers Aug 18, 2023
1b736b8
wip
bjchambers Aug 18, 2023
5d87621
try with find-links
bjchambers Aug 19, 2023
c03e625
quote
bjchambers Aug 19, 2023
fc315ff
fixes
bjchambers Aug 19, 2023
de18521
cleanup
bjchambers Aug 19, 2023
920827c
Merge branch 'main' into python-ci
bjchambers Aug 19, 2023
d620ae2
sdist
bjchambers Aug 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
327 changes: 327 additions & 0 deletions .github/workflows/ci_python.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,327 @@
# This file was initially generated by maturin v1.2.0, using:
#
# maturin generate-ci github --pytest -o ../.github/workflows/ci_python.yml
#
# TODO:
# - `safety`
# - `typeguard`
# - `xdoctest`
name: CI

# Only one job per-ref
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

on:
push:
branches:
- main
tags:
- '*'
pull_request:
branches:
- main
merge_group:
branches:
- main

defaults:
run:
shell: bash
working-directory: python

permissions:
contents: read

jobs:
bjchambers marked this conversation as resolved.
Show resolved Hide resolved
# Run lint separately from the build -- it doesn't need the built Rust code.
#
# Also, only do it on one machine.
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Install poetry
run: |
pipx install poetry
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
- uses: actions/setup-python@v4
with:
python-version: 3.11
cache: poetry
- name: install deps
run: |
poetry install --only=main --only=lint
- name: black
run: |
poetry run black --diff pysrc pytests docs/source
- name: flake8
run: |
poetry run flake8 pysrc pytests docs/source
- name: isort
run: |
poetry run isort --filter-files --diff pysrc pytests docs/source
- name: darglint
run: |
poetry run darglint pysrc
- name: pydocstyle
run: |
poetry run pydocstyle pysrc

build-wheel-macos:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Install poetry
run: |
pipx install poetry
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
- uses: actions/setup-python@v4
with:
python-version: |
3.8
3.9
3.10
3.11
architecture: x64
cache: poetry
- name: Install Protoc
uses: arduino/setup-protoc@v1
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build wheel (Mac x86_64)
uses: messense/maturin-action@v1
with:
target: x86_64
args: --release --out dist --sdist
working-directory: python
- name: pytest and mypy (Mac x86_64)
run: |
for V in 3.8 3.9 3.10 3.11; do
echo "::group::Install for Python $V"
poetry env use $V
source $(poetry env info --path)/bin/activate
poetry install --only=test --only=typecheck
pip install dist/kaskada-*.whl --force-reinstall
echo "::endgroup::"
echo "::group::Test Python $V"
poetry run pytest
echo "::endgroup::"
echo "::group::MyPy Python $V"
poetry run mypy -- --install-types --non-interactive pysrc pytests
echo "::endgroup::"
deactivate
done
- name: Build wheel (Mac universal2)
uses: messense/maturin-action@v1
with:
target: universal2
args: --release --out dist --sdist
working-directory: python
- name: pytest and mypy (Mac universal2)
run: |
echo "::group::Install for Python 3.11"
poetry env use 3.11
source $(poetry env info --path)/bin/activate
poetry install --only=test --only=typecheck
pip install dist/kaskada-*universal2.whl --force-reinstall
echo "::endgroup::"
echo "::group::Test Python 3.11"
poetry run pytest
echo "::endgroup::"
echo "::group::MyPy Python 3.11"
poetry run mypy -- --install-types --non-interactive pysrc pytests
echo "::endgroup::"
deactivate
- name: Upload wheels
uses: actions/upload-artifact@v2
with:
name: wheels
path: ${{ github.workspace }}/python/dist

build-wheel-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@v3
- name: Install poetry
run: |
pipx install poetry
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
- uses: actions/setup-python@v4
with:
python-version: 3.11
architecture: x64
cache: poetry
# - name: Install Rust toolchain
# uses: actions-rs/toolchain@v1
# with:
# toolchain: stable
# profile: minimal
# default: true
- name: Install Protoc
uses: arduino/setup-protoc@v1
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build wheels
uses: messense/maturin-action@v1
with:
target: x64
args: --release --out dist
working-directory: python
- name: pytest and mypy (Windows x64)
shell: bash
run: |
echo "::group::Install for Python 3.11"
source $(poetry env info --path)\\Scripts\\activate
poetry install --only=test --only=typecheck
pip install dist/kaskada-*.whl --force-reinstall
echo "::endgroup::"
echo "::group::Test Python 3.11"
poetry run pytest
echo "::endgroup::"
echo "::group::MyPy Python 3.11"
poetry run mypy -- --install-types --non-interactive pysrc pytests
echo "::endgroup::"
deactivate
- name: Upload wheels
uses: actions/upload-artifact@v2
with:
name: wheels
path: ${{ github.workspace }}/python/dist

build-wheel-linux:
runs-on: ubuntu-latest
strategy:
matrix:
include:
- target: x86_64
protoc_target: x86_64
- target: i686
protoc_target: x86_32
steps:
- uses: actions/checkout@v3
- name: Install poetry
run: |
pipx install poetry
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
- uses: actions/setup-python@v4
with:
python-version: |
3.8
3.9
3.10
3.11
cache: poetry
- name: Build wheels
uses: messense/maturin-action@v1
with:
target: ${{ matrix.target }}
manylinux: auto
args: --release --out dist --zig
working-directory: python
before-script-linux: |
set -e
# We need to add the protobuf-compiler since manylinux builds happen
# in a docker container that doesn't have it.
PB_REL="https://github.com/protocolbuffers/protobuf/releases"
curl -LO $PB_REL/download/v24.0/protoc-24.0-linux-${{ matrix.protoc_target }}.zip
unzip protoc-24.0-linux-${{ matrix.protoc_target }}.zip -d $HOME/.local
export PATH="$PATH:$HOME/.local/bin"
- name: pytest and mypy (Linux x86_64)
if: matrix.target == 'x86_64'
run: |
for V in 3.8 3.9 3.10 3.11; do
echo "::group::Install for Python $V"
poetry env use $V
poetry env info
source $(poetry env info --path)/bin/activate
poetry install --only=test --only=typecheck
pip install dist/kaskada-*.whl --force-reinstall
echo "::endgroup::"
echo "::group::Test Python $V"
poetry run pytest
echo "::endgroup::"
echo "::group::MyPy Python $V"
poetry run mypy -- --install-types --non-interactive pysrc pytests
echo "::endgroup::"
deactivate
done
- name: Upload wheels
uses: actions/upload-artifact@v2
with:
name: wheels
path: ${{ github.workspace }}/python/dist
- name: Build docs
if: matrix.target == 'x86_64'
run: |
poetry env use 3.11
source $(poetry env info --path)/bin/activate
poetry install --with=docs
ls docs/source/_static
sphinx-build docs/source docs/_build -j auto -W
deactivate
- name: Upload artifact
if: matrix.target == 'x86_64'
uses: actions/upload-pages-artifact@v2
with:
# Automatically uploads an artifact from the './_site' directory by default
path: ${{ github.workspace }}/python/docs/_build

# Make the source distribution
sdist:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Build sdist
uses: PyO3/maturin-action@v1
with:
command: sdist
args: --out dist
working-directory: python
- name: Upload sdist
uses: actions/upload-artifact@v3
with:
name: wheels
path: ${{ github.workspace }}/python/dist

docs-deploy:
# Deployment job
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}

# Grant GITHUB_TOKEN the permissions required to make a Pages deployment
permissions:
pages: write # to deploy to Pages
id-token: write # to verify the deployment originates from an appropriate source

if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
needs: [build-wheel-linux]
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v2

# release:
# name: Release
# runs-on: ubuntu-latest
# if: "startsWith(github.ref, 'refs/tags/')"
# needs: [linux, windows, macos, sdist]
# steps:
# - uses: actions/download-artifact@v3
# with:
# name: wheels
# - name: Publish to PyPI
# uses: PyO3/maturin-action@v1
# env:
# MATURIN_PYPI_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
# with:
# command: upload
# args: --skip-existing *
# working-directory: python
Loading
Loading