Updating or creating a new workflow_generator

Introduction

This is a guide for developer wishing to update or create a new workflow_generator. The WorkflowGenerators are classes that have create_osw function that generate openstudio osw file based on user input in the project yaml file.

Example use from here

In the example above, residential_hpxml version 2024.07.19 workflow_generator is being used to run the simulation.

The directory structure for the workflow_generator looks like this:

buildstockbatch/
└── workflow_generator/
    ├── residential/
    │   ├── latest/
    │   │   ├── __init__.py  # __version__="2024.07.20"
    │   │   ├── other_workflow_generator_files.py
    │   │   └── tests/
    │   │       ├── test_workflow_generator.py
    │   │       └── testing_data/
    │   │           └── stock_snapshot_data
    │   └── v2024_07_19/
    │       ├── __init__.py  # __version__="2024.07.19"
    │       ├── other_workflow_generator_files.py
    │       └── tests/
    │           ├── test_workflow_generator.py
    │           └── testing_data/
    │               └── stock_snapshot_data
    └── commercial/
    │   └── latest/
    │       ├── __init__.py  # __version__="2024.07.18"
    │       ├── other_workflow_files.py
    │       └── tests/
    │           ├── test_workflow_generator.py
    │           └── testing_data/
    │               └── stock_snapshot_data
    └── __init__.py  # main_init that imports all versions
    ├── base.py  # base class
    └── test_common_workflow_generator.py  # tests for the base class

The latest version is always placed in the folder named "latest" and the other version files are placed inside folder named after the version. Make sure to use underscore instead of a period for the folder names - otherwise it will cause problem during import.

Creating a new version vs updating existing version

You can update the latest workflow generator without creating a new version for minor revisions/extension if all of the following conditions apply, otherwise create a new version.

• The workflow generator will continue to generate exact same osw files for the existing input i.e. existing yaml files. This means, you shouldn't have to rewrite or touch the existing tests to make them pass, or modify existing yamls to run with the new workflow generator.
• The change is small enough that someone can quickly review the changes and have high degree of confidence on the changes doing the intended behavior. Even if the 1st criteria is met, if the refactoring/changes are large, there is a risk of introducing subtle bugs - hence it's better to create a new version to preserve existing functionality.
• The features being introduced are not mandatory for any version of the stock modelling tool (ComStock or ResStock). For example, if a new feature being added to ResStock/ComStock requires corresponding new feature in the workflow generator, create a new version of the workflow generator and specify the minimum required workflow_generator version to the new version in the ResStock/ComStock side. This is required even if the new workflow generator is backward compatible so that we can ensure the necessary feature exists in the workflow generator being used. This allows us to throw an error early when one attempts to use the new ResStock/ComStock feature with out-of-date workflow_generator.

Process for creating a new version of workflow_generator

Here are the step by step process for creating a new version of the residential workflow_generator. We are going use the example file structure above. Say, you plan to release the next version by 2024.08.23

• Checkout a new branch.
• Copy the latest version to a new folder - say latest - copy.
• Rename the latest - copy folder name to v2024_07_20 based on the version info stored at latest - copy\__init__.py
• Go to test_workflow_generator.py inside v2024_07_20\tests and replace all latest module in the import statements with v2024_07_20. We could have use relative import to avoid having to do this step, but pytest doesn't work well with relative import. So, we are stuck with this.
• Edit the main_init file at workflow_generator\__init__.py to import the v2024_07_20 workflow_generator following the pattern of other versions. This step might be unnecessary in the future when refactor this to import all versions automatically.
• We won't be touching this version - this is now frozen for life - except for trivial bug fixes.
• Edit the version in latest\__init__.py to 2024.08.23. This is the latest version now.
• Edit the version_info in latest\__init__.py to describe what's new in this version.
• Replace the testing_data inside latest\tests with most up to date snapshot of the stock modelling tool that this generator is supposed to work with. For ResStock, you would download the xml files from the branch of ResStock this new generator is compatible with.
• Edit other files inside latest as required to support the new behavior of the generator. You don't have to worry about backward compatibility because older stock models will be using older version of the workflow generator.
• Commit your changes to latest and also commit the new 2024_07_20 folder.
• Push your branch, create a PR and pray that all the tests pass!

After this process is completed, the new directory structure will look like:

buildstockbatch/
└── workflow_generator/
    ├── residential/
    │   ├── latest/
    │   │   ├── __init__.py  # __version__="2024.08.23" # updated version
    │   │   ├── other_workflow_generator_files.py  # updated code
    │   │   └── tests/
    │   │       ├── test_workflow_generator.py  # updated tests
    │   │       └── testing_data/
    │   │           └── stock_snapshot_data  # Updated with most up-to-date snapshot
    │   ├── v2024_07_19/
    │   │   ├── __init__.py  # __version__="2024.07.19"
    │   │   ├── other_workflow_generator_files.py
    │   │   └── tests/
    │   │       ├── test_workflow_generator.py
    │   │       └── testing_data/
    │   │           └── stock_snapshot_data
    │   └── v2024_07_20/
    │       ├── __init__.py  # __version__="2024.07.20"
    │       ├── other_workflow_generator_files.py
    │       └── tests/
    │           ├── test_workflow_generator.py  # Updated import statements
    │           └── testing_data/
    │               └── stock_snapshot_data
    └── commercial/
    │   ├── latest/
    │   │   ├── __init__.py  # __version__="2024.07.18"
    │   │   ├── other_workflow_files.py
    │   │   └── tests/
    │   │       ├── test_workflow_generator.py
    │   │       └── testing_data/
    │   │           └── stock_snapshot_data
    └── __init__.py  # main_init that imports all versions, including v2024_07_20
    ├── base.py  # base class
    └── test_common_workflow_generator.py  # tests for the base class