Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-41962: Use storage classes from QG in PreExecInit. #276

Merged
merged 4 commits into from
Dec 2, 2023

Commits on Dec 1, 2023

  1. Use storage classes from QG in PreExecInit.

    PipelineDatasetTypes does preserve storage classes (that's why
    it's being deprecated).
    TallJimbo committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    dd2f147 View commit details
    Browse the repository at this point in the history
  2. Fix unit test that was looking for the wrong behavior.

    First problem was just that the test assumed it could register a
    dataset type after the QG was made and not run into trouble when
    running that QG.
    
    Second problem was that it was just expecting something other than what
    its own code comments suggested, which was also contrary to correct
    behavior.
    TallJimbo committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    ae2ae85 View commit details
    Browse the repository at this point in the history
  3. Clean up storage class conversion tests.

    This includes:
    
     - Using TaskMetadata.from_dict instead of to_dict in comparisons to
       make the intent clearer.
    
     - Documenting what the log-inspection utility method actually tests.
    
     - Making one test (test_from_pipeline_intermediates_differ)
       registering a dataset type before building a QG because that's
       what's needed in general for correctness (even though it didn't
       matter here).
    
     - Renaming and re-documenting another test
       (test_from_pipeline_inconsistent_dataset_types) where we were
       actually testing that building the QG and then changing a
       dataset type out from under it can be a problem.
    TallJimbo committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    1a05997 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2023

  1. Guard against QGs that don't populate registryDatasetTypes.

    We have a custom QG builders in the wild and they're not as
    well-behaved as the main one; guard against them until we can fix them.
    TallJimbo committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    2e33486 View commit details
    Browse the repository at this point in the history