Pytest with f2s ops #3099
Closed
Pytest with f2s ops #3099
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For more information, see https://pre-commit.ci
…l into parallel-pytest
Additionally, switch integration tests to use tox under the hood.
This will result in a nicer more compact output.
Instead of using single monolith op that loops over all forms, we can use ops factory and XbrlRuntimeSettings resource to simplify how stuff is passed in. This way, single runtime settings exists for all xbrl ops and one op is generated for each form. This should allow for better parallelism, even though it might interfere with the num_workers being set to num cpus by default (i.e. this way we will oversubscribe available cores twice, one for dagster workers, and once for xbrl num workers). This, hovewer, should be an easy fix.
Refactor monolithic dbf2sqlite and xbrl2sqlite methods into per-dataset smaller ops that are invoked within the graphs. This should allow us to better make use of dagster parallelism and speed up ferc_to_sqlite processing. It seems that current unit/integration tests only use FERC1 raw data, so I've modified the fixtures to only run the relevant pieces of processing.
add tuning for xbrl workers, set it to 2 here
To test the speed-up characteristics of different tuning params, and to determine what fits on the worker.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dummy PR that combines pytest refactor and ferc_to_sqlite break-up of monolithic ops; this PR exists so that pytests can be automatically run and evaluated for speedups.