logs/report-{timestamp}.md

[tristan-f-r]

To make it easier to understand the output structure (originally intended for #369), we may want to include some extra file, report-{timestamp}.md (or .html, etc.., any convenient readable file), which would be a very basic report file specifying the entire output directory for the current run - e.g. algorithm outputs, analysis steps, visualization, etc...

This would include files that were not changed with the particular configuration run, but are still tied to the run (i.e. cached results), while excluding files that were not part of the aforementioned run.

A hypothetical example of this file may begin with:

# SPRAS run at {timestamp}

## Algorithm outputs

- dataset data0:
  - omicsintegrator1, Parameter combination {hash} (parameters-omicsintegrator1-params-{hash}.yaml: data0-omicsintegrator1-params-{hash}/
  - ...

## Downstream analysis

- data0-gs0-eval/
- ...

Essentially, this should be a much lighter, single-file version of the display we have in the spras-benchmarking repository that only links to file paths.

This is not very helpful compared to documentation if the user already cleared output before, or uses separate folders for different runs, but in the case of a user re-using the same output folder, this can clear up some confusion. (Other than improving some user experience, this also seems useful for navigating the information we'll eventually store in OSDF from all the benchmarking runs.)

[ntalluri]

(put this in my pr comment, but adding it here as well)

I’m still not entirely sure how this is supposed to look based on the explanation in #380. Could you provide a pseudocode example of the expected output? To me, it seems the same as what we already generate in the output folder, just reformatted into a single file for clarity that is just a different formatted version of what is already specified in the config file and output folder.

I’m also unclear on what kind of clarity this is meant to provide, should it include only the newly generated files, or all files tied to a specific config? If yes wouldn’t it be simpler to just create a subfolder in output/ or a separate output directory with a new name that we can set in a config file? I was planning on making subfolders anyways for this tutorial when I make the configs that will be used for this tutorial.

[tristan-f-r]

The output would be something as so, only containing output files tied to that run: it would still contain files that didn't need to be re-run for the changes made in the configuration, but it won't contain all the (edit: files) in a folder.

# SPRAS run at {timestamp}

## Algorithm outputs

- dataset data0:
  - omicsintegrator1, Parameter combination {hash} (parameters-omicsintegrator1-params-{hash}.yaml: data0-omicsintegrator1-params-{hash}/
  - ...

## Downstream analysis

(until #320, these folder names are much more obvious than the hashes above, but it would still look something like:)

- data0-gs0-eval/
- data0-gs1-eval/

To me, it seems the same as what we already generate in the output folder, just reformatted into a single file for clarity that is just a different formatted version of what is already specified in the config file and output folder.

I'm specifically worried about multiple configuration files being run while pointing to the same output folder. If configuration files pointed to different output folders, or were cleared before every run, this feature would just be exactly as you describe.

[ntalluri]

I'm still confused by why we would want a user to clear every run? Part of the tutorial is to show the caching aspect of SPRAS.

[tristan-f-r]

Since caching is in the tutorial, I would absolutely recommend having this feature beforehand - at the moment, it's difficult to tell what files are actually associated to the configuration you just ran.

[ntalluri]

My plan is to create subfolders within the output directory so that results are separated by configuration or tutorial section (right now just "basic" and "medium"). I think I understand what you mean now, without subfolders, both the basic and medium runs would just dump everything into the main output folder with no organization. I usually rely on subfolders when I’m running many different SPRAS tests across datasets or testing different things with different config, since it helps keep everything organized which is what will be used here. In the basic case, we will be doing caching for the egfr dataset in its own subfolder. In the medium case, we will be building a new dataset and then running on that in its own subfolder. We also won't be running 100s of runs for this tutorial. Maybe only 15 runs max for the medium and 3 max for the basic.

[ntalluri]

I do think this is a good idea though; it’s just not something we need for the tutorial right now. It also shouldn’t be a top priority to add compared to other tasks that are more pressing. That said, I think this would make an excellent introductory project for a new developer who joins the team in the future. It’s a contained, well scoped feature that would give them a chance to get familiar with the codebase without being overwhelming.

[tristan-f-r]

I'll clarify the original issue then with some more details from this conversation 👍

[agitter]

I also think this would be a good improvement to the code in case someone is using one output directory (without subdirectories) for many runs. I agree it isn't on the critical path for the benchmarking paper so we could save it for a new contributor later.