Add Versioning Support from `version.txt` #3140

ahnaf-tahmid-chowdhury · 2024-09-24T06:28:42Z

Overview

This pull request introduces a versioning mechanism that utilizes a version.txt file located in the tools folder. Both pyproject.toml and CMakeLists.txt have been updated to read version information from this file, ensuring consistency across the project.

Changes

Added version.txt: A new file in the tools directory containing the version string in the format X.X.X or X.X.X-any.
Updated pyproject.toml:
- The version field in pyproject.toml is now set to read the version from version.txt. This enables automatic versioning based on the contents of version.txt.
Updated CMakeLists.txt:
- The CMake configuration has been modified to read the version from version.txt and set the OPENMC_VERSION accordingly.
- Implemented logic to handle versions with optional suffixes (e.g., -dev, -any), ensuring that the header files use only the main version number without the suffix.

Benefits

Consistency: Centralizing version information in a single file (version.txt) ensures that both Python and CMake projects remain in sync.
Flexibility: The version can be easily updated by modifying just one file, simplifying version management.

Fixs

Create single source of truth for version identifier #3136

ahnaf-tahmid-chowdhury · 2024-09-24T13:38:40Z

Not sure how to resolve this

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. 
If you have updated the package versions, please update the hashes. Otherwise, 
examine the package contents carefully; someone may have tampered with them.
unknown package:
    Expected sha256 f728bb61f43fce850d622ced3b3d51b3116f767685ca4e4e0076f624e2d2307d
         Got        afe0e1873a0a0858a245ccd771066eddc07d20068859f1dd669a002e5dc68a65

shimwell · 2024-09-26T14:59:47Z

pyproject.toml

@@ -58,6 +58,9 @@ Documentation = "https://docs.openmc.org"
 Repository = "https://github.com/openmc-dev/openmc"
 Issues = "https://github.com/openmc-dev/openmc/issues"

+[tool.setuptools.dynamic]
+version = {file = "tools/version.txt"}


Does this file need locating elsewhere, perhaps in the source code. I'm not sure but perhaps that helps when packaging as I don't think the tools folder is included in the packaging (e.g. conda)

This file is only required at build time. If it is necessary at runtime, we can put it in the root dir.

It appears that openmc-feedstock uses a specific release tar file, which includes all the folders, including the tools folder.

Thanks for answering this

I'm not an expert, but this doesn't feel like a tools sort of thing. To me it feels like a top level file.

Doing it this way means openmc.__version__ still works due to how __version__ is populated.

I have stored this file in the tools directory because it is only required during build time. Python's wheel uses it to create the wheel, and CMake uses it to generate the version.h file. Additionally, we can use this file for other purposes. I placed it in that folder to keep the root directory clean. However, if necessary, we can move it to the root directory.

I tend to agree that this is more of a root-level file.

shimwell · 2024-09-26T16:48:07Z

The CI passed when I rerun a single job just now. So I have re triggered the others and I guess it will pass now. I think the previous failure was unrelated to this CI

shimwell · 2024-09-26T21:00:11Z

CI is all green 🎉

Do you think it is worth adding a version test to the pytests, perhaps something like this
https://github.com/fusion-energy/cad_to_dagmc/blob/main/tests/test_version.py

ahnaf-tahmid-chowdhury · 2024-09-27T01:33:53Z

I have already added something similar to this in the CMakeLists.txt, which will perform a similar task during configuration. I am also using setuptools to get the version number on the Python side, which will cause an error if an improper format is set.

shimwell · 2024-09-27T11:15:21Z

This looks good to me, does anyone else want to take a peak. Perhaps @MicahGale

MicahGale

Design looks good. However this doesn't seem to be propagating to the C++ side properly in my testing.

I followed the build from source method cmake .. && make. However build/bin/openmc -v was still showing openmc version 0.15.0.

MicahGale · 2024-09-27T13:44:59Z

pyproject.toml

@@ -58,6 +58,9 @@ Documentation = "https://docs.openmc.org"
 Repository = "https://github.com/openmc-dev/openmc"
 Issues = "https://github.com/openmc-dev/openmc/issues"

+[tool.setuptools.dynamic]
+version = {file = "tools/version.txt"}


I'm not an expert, but this doesn't feel like a tools sort of thing. To me it feels like a top level file.

MicahGale · 2024-09-27T13:46:56Z

tools/version.txt

@@ -0,0 +1 @@
+1.15.1-dev


This is probably beyond the scope of this PR alone, but this isn't PyPA compliant. It should be: 0.15.1.devN (Probably dev1?). Also you accidentally incremented a major release.

Setuptools will handle the task for us, so we can continue using the legacy format.

This is something setuptools does because it's nice to do; not because it has to. I think changing to PyPA compliant is a small change that would reduce the risk of something breaking in the future. (Honestly I don't think many people would even notice beyond some of the core devs.)

PyPA does allow -dev but probably better to use a dot since it's perferred

Thanks. TIL.

MicahGale · 2024-09-27T13:58:26Z

pyproject.toml

@@ -58,6 +58,9 @@ Documentation = "https://docs.openmc.org"
 Repository = "https://github.com/openmc-dev/openmc"
 Issues = "https://github.com/openmc-dev/openmc/issues"

+[tool.setuptools.dynamic]
+version = {file = "tools/version.txt"}


Doing it this way means openmc.__version__ still works due to how __version__ is populated.

ahnaf-tahmid-chowdhury · 2024-09-27T14:35:54Z

I followed the build from source method cmake .. && make. However build/bin/openmc -v was still showing openmc version 0.15.0.

It seems you have an old version of openmc installed. And you have exported that on your PATH. You can easily review it by calling

which openmc

MicahGale · 2024-09-27T14:46:30Z

I followed the build from source method cmake .. && make. However build/bin/openmc -v was still showing openmc version 0.15.0.

It seems you have an old version of openmc installed. And you have exported that on your PATH. You can easily review it by calling
which openmc

Oh yep that was my issue. I'm used to python where $PATH does not shadow the current directory.

pshriwise

Adding a suggestion for convenience.

tools/version.txt

pshriwise · 2024-09-27T15:26:28Z

pyproject.toml

@@ -58,6 +58,9 @@ Documentation = "https://docs.openmc.org"
 Repository = "https://github.com/openmc-dev/openmc"
 Issues = "https://github.com/openmc-dev/openmc/issues"

+[tool.setuptools.dynamic]
+version = {file = "tools/version.txt"}


I tend to agree that this is more of a root-level file.

paulromano

A few immediate thoughts:

Some people argue that you shouldn't store version information in files in the first place. If there's a way we can do this that automatically pulls information from git, that would probably be better.
If we do go with what you have here, there's one more instance of hardcoding of the version in docs/source/conf.py.

MicahGale · 2024-09-27T15:49:58Z

Some people argue that you shouldn't store version information in files in the first place. If there's a way we can do this that automatically pulls information from git. If that case can be addressed I really prefer using git tag.

@paulromano: @gonuke brought up some good points, that there are use cases for when the source code may be used while detached from git. I'm not sure right now of a work-around for that case using git based methods, which I generally agree would be the ideal solution.

If we do go with what you have here, there's one more instance of hardcoding of the version in docs/source/conf.py.

I'd recommend that @ahnaf-tahmid-chowdhury go with using a importlib.... call in this file no matter what because that will be agnostic to however python retrieves version information.

ahnaf-tahmid-chowdhury · 2024-09-27T16:12:38Z

I have added the get_version_info function to extract the version info, but I am unsure what working directory Read the Docs is using. For now, I am assuming it is using docs/source/ as the working directory.

Recently, I noticed @MicahGale's comment on this. We can use importlib, but again, I need to understand how Read the Docs processes this. If it runs the conf.py file before installing OpenMC, it will fail.

ahnaf-tahmid-chowdhury · 2024-09-27T16:49:52Z

It seems docs buils are passing https://readthedocs.org/projects/openmc/builds/25766475/

We can use importlib.metadata.version, but we still need to use the get_version_info function as there are two var.

gonuke · 2024-10-08T00:53:44Z

Some people argue that you shouldn't store version information in files in the first place. If there's a way we can do this that automatically pulls information from git, that would probably be better.

We definitely came here, @paulromano , to get broader perspective on this issue. I definitely like the elegance of git being a single source of truth, but worry about folks who build from tarballs (e.g. those generated automatically by GitHub upon release) rather than clones. Do we ignore them? Or do we find a way to make that work, too?

MicahGale · 2024-10-08T01:06:13Z

@gonuke one thing I thought about is that setuptools-scm will create a _version.py file. As part of the GH release process a source tar ball with a version file could be included. I think building from a source tar ball for a specific release would be a lot more common than from a specific branch.

ahnaf-tahmid-chowdhury · 2024-10-08T14:28:38Z

An additional note: If we use setuptools-scm, when someone installs OpenMC from a tarball, it will build a version labeled 0.0.0.

MicahGale · 2024-10-08T15:02:12Z

@ahnaf-tahmid-chowdhury Good point. Do you want to try and see if we could make a CI release that would embed a version file in a release tar ball from setuptools-scm?

ahnaf-tahmid-chowdhury · 2024-10-08T15:19:27Z

For this, we may need to use a workflow to build OpenMC that will create the version file. Once done, it will push the updated commit to the develop branch. We can use any existing workflow, like this one, and it will only run if the PR gets merged. Let me know if I should proceed with that approach.

MicahGale · 2024-10-08T16:03:30Z

No this is a bad idea. _version.py should never be in git. What I was thinking was as part of the release process having python -m build . create the version file. Create a complete source tar ball which include _version.py and then upload that to the GH release as an artifact.

ahnaf-tahmid-chowdhury · 2024-10-08T17:17:43Z

We are currently working on PR #3087, which will create wheel (.whl) artifacts after a release. Your approach is good, but GitHub automatically creates its own tarball with each release. So, in the end, we will be releasing multiple tarballs. And, I am unfamiliar with resolving that.

MicahGale

The changes look good, but testing is still needed to ensure all edge cases are handled. See my previous comment on some of these.

MicahGale · 2024-10-24T03:38:45Z

Let me explain how the metadata works. Whenever we install something with pip, it always comes with a package.dist-info folder that contains a METADATA file, which stores the version number. So, instead of relying on _version.py, we can rely on that METADATA file, as they will be the same. We don't need the setuptools_scm, it is a buildtime dependency.

I dug into this a little bit more. One clarification is needed: importlib.metadata will only get the version of OpenMC that is installed. So if it hasn't been installed or a dev version is being used that is different from the installed version this could lead to issues. Overall I don't think that it's too important of an edge case.

ahnaf-tahmid-chowdhury · 2024-10-24T03:56:08Z

ensure that cmake always builds with the same version as setuptools_scm provides

As I mentioned, it is quite hard to achieve here, or we may need to change the OpenMC header so that it can work with strings. After that, we need to implement the setuptools_scm format, as git describe does not follow the setuptools_scm format.

setuptools_scm format
0.15.0.dev144+gddb6c96ef

git describe format
0.15.0-144-gddb6c96ef

For now, if the format is not supported, the build will fail. So, the test is actually implemented during the configuration stage.

ensure that the python api gets a version in all circumstances

I think we can rely on setuptools_scm and let it handle this for us. Once the wheel is created, Python's built-in library importlib.metadata can easily read the version info from the METADATA. We may not need to write a dedicated test script for that, as they may have their own tests in their repository. However, for consistency, we can limit the version of setuptools_scm if it is recommended.

Esnure that a git archive has a version attached to it.

I believe this is an initial-time test that I have already run on my branch while releasing a version. Creating a dedicated test for this may consume resources and increase the time required for processing tests. I think there is no need to test this until we modify the .git_archival.txt. However, any suggestions on this are welcome.

ahnaf-tahmid-chowdhury · 2024-10-24T04:34:29Z

I dug into this a little bit more. One clarification is needed: importlib.metadata will only get the version of OpenMC that is installed. So if it hasn't been installed or a dev version is being used that is different from the installed version this could lead to issues. Overall I don't think that it's too important of an edge case.

It is always recommended to install Python packages in editable mode for developers, and this is also mentioned in the OpenMC developer docs. So, once OpenMC is installed, even in development mode, it comes with the METADATA file.

ahnaf-tahmid-chowdhury · 2024-10-24T05:09:29Z

Yes, developers may need to periodically update the OpenMC installation; otherwise, the METADATA will always contain the last installed version.

I think we can add logic where, if someone tries to use OpenMC directly from the source, it will prompt them with a message like, 'Hey, please install OpenMC and run it outside the source root directory.' Let me know if this approach is feasible.

MicahGale · 2024-10-24T12:12:10Z

I think the issue though is should it just return a version number, or the version number of the code that you are running right now always. The more I think about it needs to always return the version of what is actively running for trace-ability.

ahnaf-tahmid-chowdhury · 2024-10-24T15:40:56Z

Even if we include the _version.py file, it will only be generated after installing OpenMC, and the only way to update the _version.py file requires reinstalling OpenMC. Therefore, if we use setuptools_scm, we could rely on the METADATA. Alternatively, we could revert to the previous approach of using a hard-coded version file.

ahnaf-tahmid-chowdhury · 2024-10-24T15:52:29Z

Please note that OpenMC is already using the METADATA to extract version information, so I haven’t changed anything in that regard. I think @paulromano or @pshriwise might have some insights on how to proceed, or we are good to go.

MicahGale

Pending @paulromano @pshriwise decision on the meta-data question.

ahnaf-tahmid-chowdhury · 2024-11-30T14:27:51Z

We are planning to adopt the OpenMC version format in PyNE through the PR #1550. Currently, we are waiting for this PR to be merged. It would be greatly appreciated if we continue making progress on this.

paulromano

Sorry for the delay on this PR @ahnaf-tahmid-chowdhury and thanks for all your work on it. I think I'm on board for moving forward with this approach. The one thing that appears to need fixing is that the release version should be incremented for the "in development" version. This is what setuptools_scm does internally. Matching that behavior shouldn't be too difficult from CMakeLists.txt.

@pshriwise Do you have any further thoughts on this PR since you had requested changes before?

pshriwise · 2025-01-10T21:19:41Z

I'm on board with this as well. One detail that remains to be addressed is how the output version in the OpenMC header will be managed at runtime. Right now we get a nice -dev suffix in the output if the OpenMC executable isn't versioned. This is pretty helpful in providing context for user questions and it would be nice to maintain it if possible.

Right now the appearance of this suffix is hardcoded in the version.h.in file.

https://github.com/openmc-dev/openmc/blob/51f0e6f35039a61925477033a8eea4b2c871a728/include/openmc/version.h.in#L13C1-L14C1

shimwell · 2025-01-11T07:15:10Z

Just spotted an easy to fix conflict so fixed it up to keep this branch easy to merge. Delighted to hear there is general support for this

ahnaf-tahmid-chowdhury · 2025-01-14T14:28:54Z

@paulromano, @pshriwise,

This is the current configuration:

openmc -v
OpenMC version 0.15.0-dev
Commit hash: 66efc3a8e63a47128a306257cd6699d987fadf3c
Copyright (c) 2011-2025 MIT, UChicago Argonne LLC, and contributors
MIT/X license at <https://docs.openmc.org/en/latest/license.html>
Build type:            Debug
Compiler ID:           GNU 11.4.0
MPI enabled:           no
Parallel HDF5 enabled: no
PNG support:           yes
DAGMC support:         no
libMesh support:       no
MCPL support:          yes
NCrystal support:      no
Coverage testing:      yes
Profiling flags:       no
UWUW support:          no

You will also get commit hash information even if you haven’t used the git repository or don’t have git installed.

paulromano · 2025-01-15T12:45:29Z

@ahnaf-tahmid-chowdhury Thanks for the updates. There is still one outstanding request from above:

The one thing that appears to need fixing is that the release version should be incremented for the "in development" version. This is what setuptools_scm does internally. Matching that behavior shouldn't be too difficult from CMakeLists.txt.

ahnaf-tahmid-chowdhury · 2025-01-15T13:13:24Z

I think you're suggesting adding the number of commits above the stable release, like 0.15.0-dev+123. If that's not the case, could you please elaborate? It would help to see an example of the desired output.

Currently, the version appears like this:

OpenMC version 0.15.0-dev
Commit hash: {commit hash}

I believe you're proposing something like this:

OpenMC version 0.15.0-dev+{number of commits}
Commit hash:  {commit hash}

For reference, setuptools_scm formats it as openmc-0.15.0.dev{number of commits}.

Could you please confirm if this is what you have in mind, or clarify further?

paulromano · 2025-01-15T14:03:23Z

Sorry, to clarify further, what I mean is that if the last tagged release is 0.15.0 and there are commits past that, it should increment the release number so that it shows the 0.15.1-dev. This is what we currently do manually, and it also matches the behavior of setuptools_scm. If you run python -m setuptools_scm on your branch (which already contains a v0.15.1 tag), you'll see that it gives:

$ python -m setuptools_scm
0.15.2.dev42+g2b47355f2

This means that CMakeLists.txt would be responsible for incrementing the release number if there are changes past the last tag.

MicahGale · 2025-01-15T19:09:05Z

Sorry, to clarify further, what I mean is that if the last tagged release is 0.15.0 and there are commits past that, it should increment the release number so that it shows the 0.15.1-dev. This is what we currently do manually, and it also matches the behavior of setuptools_scm. If you run python -m setuptools_scm on your branch (which already contains a v0.15.1 tag), you'll see that it gives:
$ python -m setuptools_scm
0.15.2.dev42+g2b47355f2
This means that CMakeLists.txt would be responsible for incrementing the release number if there are changes past the last tag.

I would like to bring up this discussion: #3140 (comment). I still think it would be easiest to just call setuptools_scm rather than trying to reimplement it. Yes this would add build dependencies, but I feel like the situation in which a user wants to build from scratch but doesn't have the python dependencies is a bit contrived.

ahnaf-tahmid-chowdhury · 2025-01-19T18:26:58Z

Update

Run openmc -v
OpenMC version 0.15.0-dev115
Commit hash: 375dec94e99afd6c382827b4a750e47eddf93b4c

We can shift to setuptools_scm and make it a core build dependency, but we still need to rely on CMake variables since these values need to be written into the version.h file. Let me know if this method is good to go, or if we should fully transition to setuptools_scm.

paulromano · 2025-01-21T22:31:32Z

@ahnaf-tahmid-chowdhury It still doesn't look like there is logic to bump the release number (the second "0" in "0.15.0"). It should show 0.15.1-dev115, not 0.15.0-dev115 to match the behavior of setuptools_scm because 0.15.0 is the most recent release.

ahnaf-tahmid-chowdhury · 2025-01-23T13:32:34Z

Thanks for the explanation. I apologize for the misunderstanding. The release number increment for dev versions is important and makes sense. I have added a logic where in dev mode cmake will increment the release number by 1.

single source of truth for version identifier

5779779

shimwell reviewed Sep 26, 2024

View reviewed changes

shimwell approved these changes Sep 27, 2024

View reviewed changes

MicahGale suggested changes Sep 27, 2024

View reviewed changes

MicahGale mentioned this pull request Sep 27, 2024

Update guidance on build from source to retrieve python dependencies #3146

Open

pshriwise requested changes Sep 27, 2024

View reviewed changes

paulromano requested changes Sep 27, 2024

View reviewed changes

ahnaf-tahmid-chowdhury added 2 commits September 27, 2024 21:31

Follow PyPA compliant

104dfb0

move version file to root-level

cb849ee

add get_version_info function

0ce318c

ahnaf-tahmid-chowdhury requested a review from paulromano September 27, 2024 16:50

Merge remote-tracking branch 'upstream/develop' into version

0eff6a4

ahnaf-tahmid-chowdhury requested a review from pshriwise October 5, 2024 17:45

MicahGale suggested changes Oct 23, 2024

View reviewed changes

MicahGale approved these changes Nov 6, 2024

View reviewed changes

paulromano requested changes Jan 10, 2025

View reviewed changes

Merge branch 'develop' into version

2eb6c93

ahnaf-tahmid-chowdhury added 4 commits January 12, 2025 11:25

add hash

5219406

Add more log

d4b37cf

clang format

160bf43

Find git at the beginning

2b47355

add VERSION_COMMIT_COUNT

923f84a

ahnaf-tahmid-chowdhury requested a review from ebknudsen as a code owner January 19, 2025 17:10

ahnaf-tahmid-chowdhury added 2 commits January 23, 2025 19:04

Increment release number for dev versions

07c65a3

Merge remote-tracking branch 'upstream/develop' into version

940c571

		@@ -0,0 +1 @@
		1.15.1-dev

Add Versioning Support from version.txt #3140

Are you sure you want to change the base?

Add Versioning Support from version.txt #3140

Conversation

ahnaf-tahmid-chowdhury commented Sep 24, 2024

Overview

Changes

Benefits

Fixs

ahnaf-tahmid-chowdhury commented Sep 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shimwell commented Sep 26, 2024

shimwell commented Sep 26, 2024 • edited Loading

ahnaf-tahmid-chowdhury commented Sep 27, 2024

shimwell commented Sep 27, 2024

MicahGale left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahnaf-tahmid-chowdhury commented Sep 27, 2024

MicahGale commented Sep 27, 2024

pshriwise left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulromano left a comment

Choose a reason for hiding this comment

MicahGale commented Sep 27, 2024 • edited Loading

ahnaf-tahmid-chowdhury commented Sep 27, 2024

ahnaf-tahmid-chowdhury commented Sep 27, 2024

gonuke commented Oct 8, 2024

MicahGale commented Oct 8, 2024

ahnaf-tahmid-chowdhury commented Oct 8, 2024

MicahGale commented Oct 8, 2024

ahnaf-tahmid-chowdhury commented Oct 8, 2024

MicahGale commented Oct 8, 2024

ahnaf-tahmid-chowdhury commented Oct 8, 2024

MicahGale left a comment

Choose a reason for hiding this comment

MicahGale commented Oct 24, 2024

ahnaf-tahmid-chowdhury commented Oct 24, 2024 • edited Loading

ahnaf-tahmid-chowdhury commented Oct 24, 2024

ahnaf-tahmid-chowdhury commented Oct 24, 2024

MicahGale commented Oct 24, 2024

ahnaf-tahmid-chowdhury commented Oct 24, 2024

ahnaf-tahmid-chowdhury commented Oct 24, 2024

MicahGale left a comment

Choose a reason for hiding this comment

ahnaf-tahmid-chowdhury commented Nov 30, 2024

paulromano left a comment

Choose a reason for hiding this comment

pshriwise commented Jan 10, 2025 • edited Loading

shimwell commented Jan 11, 2025

ahnaf-tahmid-chowdhury commented Jan 14, 2025

paulromano commented Jan 15, 2025

ahnaf-tahmid-chowdhury commented Jan 15, 2025

paulromano commented Jan 15, 2025

MicahGale commented Jan 15, 2025 • edited Loading

ahnaf-tahmid-chowdhury commented Jan 19, 2025

paulromano commented Jan 21, 2025

ahnaf-tahmid-chowdhury commented Jan 23, 2025

Add Versioning Support from `version.txt` #3140

Add Versioning Support from `version.txt` #3140

shimwell commented Sep 26, 2024 •

edited

Loading

MicahGale commented Sep 27, 2024 •

edited

Loading

ahnaf-tahmid-chowdhury commented Oct 24, 2024 •

edited

Loading

pshriwise commented Jan 10, 2025 •

edited

Loading

MicahGale commented Jan 15, 2025 •

edited

Loading