Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix conda upload #2374

Merged
merged 15 commits into from
Oct 31, 2024
Merged

Fix conda upload #2374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
394a958
wip: fix conda upload
Oct 23, 2024
ed4b047
better conda workflow
Oct 23, 2024
cfbf87a
Suggested change
benoit-cty Oct 31, 2024
ed93f12
Reduce verbosity to avoid "This step has been truncated due to its la…
benoit-cty Oct 31, 2024
b91d572
Ajout d'un commentaire sur l'usage
benoit-cty Oct 31, 2024
09147e4
Conda uniquement sur Windows
benoit-cty Oct 31, 2024
5be45e8
Passe à un chemin windows
benoit-cty Oct 31, 2024
a59a5ac
Utilise ${{ runner.temp }}
benoit-cty Oct 31, 2024
dc9ae44
fix path
benoit-cty Oct 31, 2024
2f645d1
try fix cache
benoit-cty Oct 31, 2024
426a3a1
Remet seulement sur master
benoit-cty Oct 31, 2024
dd809e2
Retire TODO
benoit-cty Oct 31, 2024
b93e95f
CI python version
benoit-cty Oct 31, 2024
c7edbb9
anti-slash
benoit-cty Oct 31, 2024
998e14c
bump
benoit-cty Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .conda/recipe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ source:
build:
noarch: python
number: 0
script: "pip install . -v"
script: "pip install ."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Permet d'éviter un job rouge en sortie de CI pour log trop long.


requirements:
host:
Expand Down
122 changes: 53 additions & 69 deletions .github/workflows/workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,64 +44,53 @@ jobs:
key: release-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}-${{ github.sha }}-${{ matrix.os }}-${{ matrix.openfisca-dependencies }}

build-conda:
runs-on: ubuntu-22.04
runs-on: windows-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Cache build
uses: actions/cache@v4
with:
path: /tmp/conda-bld
key: build-conda-${{ hashFiles('.conda/recipe.yaml') }}-${{ github.sha }}
restore-keys: |
build-conda-${{ hashFiles('.conda/recipe.yaml') }}
build-conda-
path: ${{ runner.temp }}\conda-bld
key: build-conda-${{ hashFiles('pyproject.toml') }}-${{ github.sha }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ Pour la source unique qu'est pyproject.toml maniement pour l'information sur les dépendances.

- name: set version
run: |
python3 .github/pyproject_version.py --replace True
- name: Build conda package
uses: prefix-dev/[email protected]
with:
build-args: --channel openfisca --channel conda-forge --output-dir /tmp/conda-bld
build-args: --channel openfisca --channel conda-forge --output-dir ${{ runner.temp }}\conda-bld
recipe-path: .conda/recipe.yaml
upload-artifact: false
- name: Upload Conda build
sandcha marked this conversation as resolved.
Show resolved Hide resolved
uses: actions/upload-artifact@v4
with:
name: conda-build-${{ env.PACKAGE_VERSION }}-${{ github.sha }}
path: /tmp/conda-bld
retention-days: 30

test-conda:
runs-on: ubuntu-22.04
runs-on: windows-latest
needs: [ build-conda ]
steps:
# Checkout needed to get github.sha
- uses: actions/checkout@v4

- name: Setup conda
uses: conda-incubator/setup-miniconda@v3
with:
activate-environment: openfisca
miniforge-version: latest
python-version: 3.9.12
use-mamba: true

- name: Restore build
uses: actions/cache@v4
with:
path: /tmp/conda-bld
key: build-conda-${{ hashFiles('.conda/recipe.yaml') }}-${{ github.sha }}

path: ${{ runner.temp }}\conda-bld
key: build-conda-${{ hashFiles('pyproject.toml') }}-${{ github.sha }}
fail-on-cache-miss: true
- name: Install package
# `bash -l {0}` démarre un login shell avec les variables d'environnement
# voir https://github.com/conda-incubator/setup-miniconda/issues/128
shell: bash -l {0}
run: mamba install --channel file:///tmp/conda-bld --channel openfisca openfisca-france

run: mamba install --channel file:///${RUNNER_TEMP//\\//}/conda-bld --channel openfisca openfisca-france
- name: Test conda package
shell: bash -l {0}
run: openfisca test tests/formulas/irpp.yaml
sandcha marked this conversation as resolved.
Show resolved Hide resolved



lint-files:
runs-on: ubuntu-20.04
strategy:
Expand Down Expand Up @@ -223,35 +212,32 @@ jobs:
run: "${GITHUB_WORKSPACE}/.github/test-api.sh"

test-on-windows:
runs-on: "windows-latest"
runs-on: windows-latest
if: github.ref == 'refs/heads/master' # Only triggered for the `master` branch
needs: [ build-conda ]
needs: [ test-conda ]
steps:
# Checkout needed to get github.sha
- uses: actions/checkout@v4
- uses: conda-incubator/setup-miniconda@v3
with:
auto-update-conda: true
python-version: "3.9.9"
# Add conda-forge for OpenFisca-Core
channels: openfisca,conda-forge
activate-environment: true
- uses: actions/checkout@v4
activate-environment: openfisca
miniforge-version: latest
python-version: 3.10.6
use-mamba: true
sandcha marked this conversation as resolved.
Show resolved Hide resolved
- name: Restore build
uses: actions/cache@v4
with:
path: ${{ runner.temp }}\conda-bld
key: build-conda-${{ hashFiles('pyproject.toml') }}-${{ github.sha }}
fail-on-cache-miss: true
- name: Test max path length
run: "python3 openfisca_france/scripts/check_path_length.py"
- name: Get version
run: |
# chcp 65001 #set code page to utf-8
echo ("PACKAGE_VERSION=" + (python3 ./setup.py --version) ) >> $env:GITHUB_ENV
echo "Version setup.py: ${{ env.PACKAGE_VERSION }}"
- name: Download conda build
uses: actions/download-artifact@v3
with:
name: conda-build-${{ env.PACKAGE_VERSION }}-${{ github.sha }}
path: conda-build-tmp
- name: Install with conda
run: |
conda install -c ./conda-build-tmp/noarch/openfisca-france-dev-${{ env.PACKAGE_VERSION }}-py_0.tar.bz2 openfisca-france-dev
- name: openfisca test
run: openfisca test --country-package openfisca_france tests
- name: Install package
shell: bash -l {0}
sandcha marked this conversation as resolved.
Show resolved Hide resolved
run: mamba install --channel file:///${RUNNER_TEMP//\\//}/conda-bld --channel openfisca openfisca-france
- name: Test conda package
shell: bash -l {0}
run: openfisca test tests/formulas/irpp.yaml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pourquoi réduisons-nous à un seul test ? Dans le passé, les tests permettaient de relever les erreurs de chemins de paramètres trop longs sous Windows qui pouvaient survenir à n'importe quel endroit modifié par une PR. Considérons-nous que Test max path length suffit ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C'est un problème de temps de CI : tout tester prend 1 heure. On pourrait aussi mettre des tests sous forme de matrice. Cependant vu que nous ne savons pas s'il y a des utilisateurs des paquets conda, il ne semble pas nécessaire de gêner les contributeurs avec des tests supplémentaires.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bilan, pour détecter les erreurs sur Windows de :

  • paramètres : le script Test max path length (du jobtest-on-windows à ne pas confondre avec le job test-path-length exécuté sur ubuntu) vérifie le fait de ne pas dépasser la limite par défaut de longueur de path
  • variables, chargement du TaxBenefitSystem et bonne installation de la commande openfisca : l'exécution d'un seul test YAML suffit et est économe en temps de calcul ; ici on teste déjà tout ce qu'il y a dans irpp.yaml (pas d'erreur attendue/rencontrée spécifique à des syntaxes de règles python)


check-version-and-changelog:
runs-on: ubuntu-20.04
Expand Down Expand Up @@ -323,32 +309,30 @@ jobs:
run: "${GITHUB_WORKSPACE}/.github/publish-git-tag.sh"

publish-to-conda:
runs-on: "ubuntu-20.04"
runs-on: windows-latest
needs: [ deploy, test-on-windows ]
steps:
- uses: conda-incubator/setup-miniconda@v3
- uses: actions/checkout@v4
- name: Restore build
uses: actions/cache@v4
with:
auto-update-conda: true
python-version: "3.9.9"
# Add conda-forge for OpenFisca-Core
channels: conda-forge
activate-environment: true
- name: Get source code
uses: actions/checkout@v4
- name: Get version
run: echo "PACKAGE_VERSION=$(python3 ./pyproject.toml --version)" >> $GITHUB_ENV
# Get the last commit hash on the PR (-2 : before the merge commit)
- uses: actions/github-script@v6
id: last_pr_commit
path: ${{ runner.temp }}\conda-bld
key: build-conda-${{ hashFiles('pyproject.toml') }}-${{ github.sha }}
fail-on-cache-miss: true
- uses: conda-incubator/setup-miniconda@v3
with:
script: |
const commits = ${{ toJSON(github.event.commits) }}
return commits.at(-2).id;
result-encoding: string
- name: Conda build and upload
# This shell is made necessary by https://github.com/conda-incubator/setup-miniconda/issues/128
activate-environment: openfisca
miniforge-version: latest
python-version: 3.10.6
use-mamba: true
- name: Install package
shell: bash -l {0}
run: mamba install --channel file:///${RUNNER_TEMP//\\//}/conda-bld --channel openfisca openfisca-france
- name: Test conda package
shell: bash -l {0}
run: openfisca test tests/formulas/irpp.yaml
- name: Conda upload already built package
shell: bash -l {0}
run: |
conda install --yes conda-build anaconda-client
conda config --set anaconda_upload yes
conda build --channel conda-forge --channel openfisca --token ${{ secrets.ANACONDA_TOKEN }} --user openfisca .conda
conda install --yes anaconda-client
anaconda -t ${{ secrets.ANACONDA_TOKEN }} upload --user openfisca ${{ runner.temp }}\conda-bld\noarch\openfisca-france-*
9 changes: 9 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
# Changelog

### 169.3.1 [2374](https://github.com/openfisca/openfisca-france/pull/2374)

* Correction d'un crash.
* Périodes concernées : aucune.
* Zones impactées : `workflow.yml`.
* Détails :
- La publication conda ne fonctionnait plus suite au passage à `pyproject.toml` : `C:\hostedtoolcache\windows\Python\3.9.13\x64\python3.exe: can't open file 'D:\a\openfisca-france\openfisca-france\setup.py': [Errno 2] No such file or directory`
- Met en place la pipeline conda faite dans Openfisca/country-template

### 169.3.0 [2376](https://github.com/openfisca/openfisca-france/pull/2376)

* Évolution du système socio-fiscal
Expand Down
3 changes: 2 additions & 1 deletion openfisca_france/scripts/check_path_length.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
FR: Ce script sert à estimer la longueur des chemins d'arborescence des paramètres,
afin de ne pas avoir de chemins > 150 caractères (incompatible Windows).
Il est à utiliser avant de contribuer à l'harmonisation
Il est à utiliser avant de contribuer à l'harmonisation.
⚠️ Il est également utilisé par la CI, dans l'étape de test sur Windows.
benoit-cty marked this conversation as resolved.
Show resolved Hide resolved
'''


Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "OpenFisca-France"
version = "169.3.0"
version = "169.3.1"
description = "OpenFisca Rules as Code model for France."
readme = "README.md"
keywords = ["microsimulation", "tax", "benefit", "rac", "rules-as-code", "france"]
Expand Down
Loading