Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature[next]: ITIR transformation & gtfn codegen caching #1474

Closed
wants to merge 19 commits into from

Conversation

tehrengruber
Copy link
Contributor

Significantly improves time to first computed value by caching the ITIR transformations and GTFN code generation.

This is not meant to be merged. Just an intermediate solution to make debugging more practical. In PMAP-G this improves the time for the first time-step for a larger test-case from ~10 minutes to ~40s.

@tehrengruber tehrengruber marked this pull request as draft February 28, 2024 09:38
@tehrengruber tehrengruber marked this pull request as ready for review June 21, 2024 12:24
@tehrengruber tehrengruber requested a review from egparedes June 21, 2024 12:24
@SF-N
Copy link
Contributor

SF-N commented Oct 24, 2024

Note: We just observed that even in case the compilation fails the cache is populated, which is not so great.

SF-N added a commit that referenced this pull request Nov 7, 2024
…1690)

Add memory and disk-based caching to other workflow steps and,
therefore, removing unnecessary overhead of Program calls and
significantly improving time to first computed value.

Changes:

- setting `cached = True` for `func_to_past_factory`
- wrapping the GTFN code generation into a `CachedStep` (using
`Diskcache`) which is activated when setting
`otf_workflow__cached_translation=True`, similar as in
[PR#1474](#1474) (without
CachedStep)
- Fixing hash function of `ProgramDefinition`
- New tests for added functionality

This leads to a runtime decrease of about 25% for PMAP-G in the
advect-uniform testcase (5 hours) after caches are populated.

TODOs: 
- [x] improving hash functions of `fingerprint_stage`

---------

Co-authored-by: Till Ehrengruber <[email protected]>
Co-authored-by: Enrique G. Paredes <[email protected]>
@tehrengruber
Copy link
Contributor Author

Absorbed in #1690

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants