Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOI not retrievable with too many processes #263

Open
haukekoehn opened this issue Oct 10, 2023 · 2 comments
Open

DOI not retrievable with too many processes #263

haukekoehn opened this issue Oct 10, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@haukekoehn
Copy link
Contributor

haukekoehn commented Oct 10, 2023

Describe the bug
When starting a multimessenger analysis nmma-analysis with many processes (>100), the function get_latest_zenodo_doi in nmma/utils/models.py is called with every processes. Instead of providing the latest zenodo doi it returns a 429 too many requests error.

To Reproduce
srun -n 128 nmma-analysis --args

Expected behavior
To simply return the latest doi from zenodo. But in principle only one process would be needed for this, as also just one process should download everything. Instead now every process is downloading everything.

Platform information:

  • NMMA version: 0.1.0

Additional context
In my recent pull request, I have put a work around into load_models_list to set the doi to the PERMANENT_DOI if get_latest_zenodo_doi fails to retrieve it.

@haukekoehn haukekoehn added the bug Something isn't working label Oct 10, 2023
@mcoughlin
Copy link
Member

@haukekoehn yeah we saw this early on and our recommendation is to just run the download directly first and then to trigger the MPI script. It's tricky to work around.

@haukekoehn
Copy link
Contributor Author

perhaps the easiest thing would be to assert the initialization of the AnalysisRun instance in nmma-analysis to only one process (which would be totally efficient from how i understand it)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants