Having trouble with changing RESULTS_DIR for each job

[honghuikim]

I'm trying to set a unique result directory for each job, but it's not working.

I tried the three ways presented in quacc document, but the only working one is changing ~/.quacc.yaml file directly.
This is off, but scratch and result directory are saved in my $HOME directory, not the current working directory.

For example,

print(SETTINGS.RESULTS_DIR) # it gives the current working directory
with change_settings({"RESULTS_DIR": "/some/new/directory"}):
    print(SETTINGS.RESULTS_DIR) # it gives the "/some/new/directory"
    future = static_job(atoms)
    future.result() # But the result files and directory are stored in my $HOME

os.environ["QUACC_RESULTS_DIR"] = "/some/new/directory" is not working as well.

I suspect myself of doing something wrong, what would cause the issue?
I'm using quacc=0.8.1 and Parsl as a workflow engine. Let me know if you need additional information.

Thank you

[Andrew-S-Rosen]

@honghuikim: Thank you for your question.

First, let me ask: does it work when you do not use a workflow engine? I imagine the answer is yes.

If so, note that when deploying calculations via a workflow engine, changes to in-memory global variables on the local machine (e.g. login node) will not be reflected on the remote machine (e.g. compute node). You are basically shipping off the @job decorated function to the remote machine; it does not know anything about the variables in the current Python process from which it is dispatched.

For the environment variable approach, you are changing the environment variable via os.environ and therefore this is only reflected in the current Python processes' working memory. A more robust mechanism would be to change the environment variable (via the export bash command) in your job submission script itself rather than in the Python process. In Parsl, that means the worker_init keyword argument within your configuration.

For the change_settings context handler, you'd need to ensure the setting re-assignment takes place within the decorated function itself. Currently, you can use from quacc import strip_decorator to strip the @job decorator off, and you can then redefine the @job function with the change to the setting inside. Let me know if you need an example.

To make this process easier, I'd like to propose two things:

1. An update to the documentation to further clarify this. It is mentioned in one of the drop down menus at the bottom, but this is hidden by default. This I can do easily.
2. Introduce a new function that will essentially strip the decorator off a function, apply a setting to it, and re-introduce the decorator. I think it is possible. I'll think about this later today.

[honghuikim]

@Andrew-S-Rosen Thank you for the detailed explanation!

Yes, it works without setting a workflow. With Parsl, adding RESULTS_DIR info in worker_init solved the issue. I forgot that I should transfer variables to compute nodes if needed. Thank you for the explanation.

Either way will be fine, but for now, controlling Parsl configuration seems simpler. Nonetheless, change_settings + strip_decorator looks more flexible.

This is an another question, but does Quacc support defining the name of the result directory? Or is this a role of Parsl? For example, if I run qmof_relax_job, there will be 4~5 steps, and an equal number of directories under RESULTS_DIR will be made (and the directory name is qmof-{time}). I was wondering whether I can set these directories' name more informative (e.g. loose_relax, double_relax1), even though I can know their information from the pickle output or MongoDB.

[Andrew-S-Rosen]

Glad to hear the settings management works now!

This is an another question, but does Quacc support defining the name of the result directory? Or is this a role of Parsl? For example, if I run qmof_relax_job, there will be 4~5 steps, and an equal number of directories under RESULTS_DIR will be made (and the directory name is qmof-{time}). I was wondering whether I can set these directories' name more informative (e.g. loose_relax, double_relax1), even though I can know their information from the pickle output or MongoDB.

This is a good question. Unfortunately, at this time, there is not a mechanism to update the individual subdirectory names, and a standard format is used based on the datetime both for informational purposes but also to ensure that each name is guaranteed to be unique. I wouldn't be opposed to a feature addition to allow more flexibility in the future, however.

[honghuikim]

I see, that should be fine. Since we have almost every data and dir_name in pickle, the current scheme looks good. The question was just from curiosity. Again thank you for the fast and detailed reply, this helps me a lot!

[Andrew-S-Rosen]

No problem!

In f4bcced, I have tried to clarify the documentation a bit. I also removed the suggestion for os.environ because I felt it was doing more harm than good.

I opened #2147 to track progress on improving the user experience related to dynamic settings updates.