Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Method to cache critical results #3

Open
Chan-Dong-Jun opened this issue Aug 18, 2024 · 3 comments
Open

Method to cache critical results #3

Chan-Dong-Jun opened this issue Aug 18, 2024 · 3 comments
Assignees

Comments

@Chan-Dong-Jun
Copy link
Owner

What's the problem this feature will solve?
There should be a method to cache the critical results.

Describe the solution you'd like
Introduce a method to cache the results at critical steps and write them to a dataframe before caching them.

@Chan-Dong-Jun
Copy link
Owner Author

Main edit

 def save_critical_result(self, condition_function: Callable[[dict], bool]):
        model_vars = self.model.datacollector.model_vars
        self.cache_file_path.mkdir(parents=True, exist_ok=True)

        current_step = self.model._steps
        special_results_file = f"{self.cache_file_path}/special_results.parquet"
        if condition_function(model_vars):
            step_data = {key: [value[-1]] for key, value in model_vars.items()}
            step_data["Step"] = current_step
            special_results_df = pd.DataFrame(step_data)

            # Append the current step data to the Parquet file
            if os.path.exists(special_results_file):
                existing_data = pq.read_table(special_results_file).to_pandas()
                combined_data = pd.concat(
                    [existing_data, special_results_df], ignore_index=True
                )
                special_results_table = pa.Table.from_pandas(combined_data)
            else:
                special_results_table = pa.Table.from_pandas(special_results_df)

            pq.write_table(special_results_table, special_results_file)

condition_function (Callable[[dict], bool]): A function that takes a dictionary of model variables and returns a boolean indicating whether the current step's data should be saved.
save_critical_result takes in a condition function that caches model level results whenever the condition in the function is met.

@Chan-Dong-Jun
Copy link
Owner Author

Example
image

@Chan-Dong-Jun Chan-Dong-Jun self-assigned this Aug 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

1 participant