Finalize staged data tests is timing out when tested with a real AWS S3 storage backend #2078
Labels
flaky test
Tracking tests that fail inconsistently in CI
Comments
G-D-Petrov
added a commit
that referenced
this issue
Jan 14, 2025
#### Reference Issues/PRs #2048 #### What does this implement or fix? There quite a few distinct changes as most of the work was in the CI and testing. I will try to separate the different changes here: - Numpy 2 Support: - [support for new c++ interface](https://github.com/man-group/ArcticDB/pull/2050/files#diff-d23d2e3a46aa783e6c27de2ff06123f6149cda2727b09eb48e0755bc29a3ede2) - [small change for how == and is are evaluated](https://github.com/man-group/ArcticDB/pull/2050/files#diff-8d6759b6a2a11f86f70ee321e943d6c8e7898d4419ed281ad5a7358426a49a9a) - [added some more checks during normalization](https://github.com/man-group/ArcticDB/pull/2050/files#diff-fa52ad937e8dc0daf708927a99994e7264fd9cb62557326919c023696808859eL121) - changes to tests to support new Numpy conventions (all of the changes to the pytests) - Pybind11 2.13 Support (Needed for numpy2): - removed python 3.6 and added 3.12 (relevant changes [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-5c3fa597431eda03ac3339ae6bf7f05e1a50d6fc7333679ec38e21b337cb6721L30), [here ](https://github.com/man-group/ArcticDB/pull/2050/files#diff-ea07d77332dfd2427efe1d2148bec3d99c7dd3c06160f7975d5ad53069b9083f), and [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-ea07d77332dfd2427efe1d2148bec3d99c7dd3c06160f7975d5ad53069b9083f) - Added new version of pybind11 and made changes to support the new interface: - [changed PythonUtils](https://github.com/man-group/ArcticDB/pull/2050/files#diff-57acdaae55064408141a79c555b1fe7c4f2d4a6ce7f7b291e814b36abe4bdbce) to add more info about the Python executable/libs neded for the newer version of pybind11 - Changes to support new pybind11 interfaces (relevant changes [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-9357e7e6906b72682dbae796b1a310682b3f7b334e3bba0cfee71c08f2e8cf57), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-dc9806fb1c292ea747ea6f0d333206c084159dcf6fc94919b98bfce3fd90d8c8), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-02770184046bcf181939ea70f927d1f835687e57fbcc843a71a500684674b8e9), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-87b188a9b8a3a1ab16c10f975faa03e7f33e42347eb3019505b240432a4502e6), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-6caec3c44be32d3a881cccbff1f03bbf8aedcdbf65e6e0c2c963e34bc675a246), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-6c2d76c22148e822d651baf5e8c3c8e23528d7360afe98d3ba085104e46237db), [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-c7f1753eda666a2b8fd8c840f0ed2be21b809132b58ea6adbb14f49afb34039f) and [here](https://github.com/man-group/ArcticDB/pull/2050/files#diff-ed781e679ed39eaca3b7b13549967958654ab47eb42f920bfa5535933a1c097b) - Changes for testing: (also added tickets for these): - [Changed the persistent tests to use http instead of https, because the sts is broken on windows ](https://github.com/man-group/ArcticDB/pull/2050/files#diff-9c13e1c7a3ca0bbad7e24e48944e815de7bfe7ed57584ce31b0549eb7be6c94dL3) - #2080 - [change for persistent tests to use only encoding version 1, so as to not write the encoding_version to the cfg, because older versions cannot read it](https://github.com/man-group/ArcticDB/pull/2050/files#diff-11d84f04de731d90230614eaaeb91eb376a0b00061ce90dacbef195db6125005) - #2077 - [skip for storage lock test because it is flaky](https://github.com/man-group/ArcticDB/pull/2050/files#diff-f0c82624218a7a1ad65345a6e6ae054e689288ded7b0670999c5eabb9a1c89d9) - #2079 - [change finalize stage data to not be run against real s3 because it is too slow and it times out](https://github.com/man-group/ArcticDB/pull/2050/files#diff-67a5a1e0b68a1442a89e31f917eeaf651954e0b389d15c0ce0a6e7fd80706574) - #2078 I've also tested in against the full suite of tests incl the persistent ones (run [here](https://github.com/man-group/ArcticDB/actions/runs/12388677078/job/34581688721)) **N.B.:** We need to continue using the submodule on pybind11 instead of getting it from vcpkg, because the latest version of pybind11 are not available on its vcpkg distribution. Specifically, the versions that introduce the Numpy 2 Support. #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: phoebusm <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
The test is timing out when tested with a real s3 storage because it because too slow (e.g. failing run here)
For that reason the test is now using only lmdb as backend (e.g. changed here).
Steps/Code to Reproduce
Run
test_finalize_monotonic_unique_chunkswithbasic_arctic_libraryExpected Results
The test shouldn't be timing out
OS, Python Version and ArcticDB Version
Any Python Version
Backend storage used
AWS S3
Additional Context
No response
The text was updated successfully, but these errors were encountered: