Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new command flepimop-push, flepimop-pull #296

Closed
wants to merge 40 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
d85f6e4
add new command flepimop-pull
fang19911030 Jul 25, 2024
edfbb68
bug fix
fang19911030 Jul 25, 2024
1cb9ef1
change format
fang19911030 Jul 25, 2024
d9d59a2
add the test
fang19911030 Jul 25, 2024
c18ecde
change argument type
fang19911030 Jul 25, 2024
ea59026
add file check
fang19911030 Jul 29, 2024
aafc88b
add new file for push command
fang19911030 Aug 5, 2024
d3a4cac
Merge branch 'main' into python_script_resume
fang19911030 Aug 5, 2024
9e838b0
Merge branch 'main' into python_script_resume
fang19911030 Aug 8, 2024
62c0979
add function creating file names for pushing
fang19911030 Aug 8, 2024
5407c71
add body for flepimop-push
fang19911030 Aug 8, 2024
e8c1c42
add command flepimop-push
fang19911030 Aug 8, 2024
cde74d4
change error message
fang19911030 Aug 8, 2024
f1a57fb
fix wrong parameter
fang19911030 Aug 12, 2024
8c6b65f
rename file
fang19911030 Aug 12, 2024
b0d8895
wrong file name
fang19911030 Aug 12, 2024
fc8b4fa
update doc and fix format
fang19911030 Aug 12, 2024
5fce4d4
fix
fang19911030 Aug 12, 2024
ce734c4
black fix format
fang19911030 Aug 12, 2024
6cacb69
print message
fang19911030 Aug 13, 2024
34b18cf
clean
fang19911030 Aug 13, 2024
534d932
correct variable name
fang19911030 Aug 13, 2024
72ef61b
correct tests
fang19911030 Aug 13, 2024
1cbce6c
Merge branch 'main' into python_script_resume
jcblemai Sep 13, 2024
d4ba408
Merge branch 'main' into python_script_resume
jcblemai Sep 13, 2024
cee4259
Merge branch 'main' into python_script_resume
fang19911030 Oct 22, 2024
da0b989
address comments
fang19911030 Oct 23, 2024
3c21a82
address comments 2
fang19911030 Oct 23, 2024
1d8879a
Merge branch 'python_script_resume' of https://github.com/HopkinsIDD/…
fang19911030 Oct 23, 2024
c9a2307
Merge branch 'main' into python_script_resume
fang19911030 Oct 31, 2024
29cf95d
change doc string of file_paths
fang19911030 Nov 14, 2024
62c56f0
remove main
fang19911030 Nov 14, 2024
8194214
remove main and relocate import
fang19911030 Nov 14, 2024
14193ff
add test file
fang19911030 Nov 19, 2024
22f5188
change
fang19911030 Nov 20, 2024
a417e38
new unit test
fang19911030 Nov 20, 2024
f796ce2
add test
fang19911030 Nov 20, 2024
25d79d3
change click type
fang19911030 Nov 20, 2024
fea9449
change click string to path
fang19911030 Dec 5, 2024
1522e8f
format change
fang19911030 Dec 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
add new command flepimop-pull
fang19911030 committed Jul 25, 2024
commit d85f6e4e5186311f70883b2a3b5cdb080e4a2368
1 change: 1 addition & 0 deletions flepimop/gempyor_pkg/setup.cfg
Original file line number Diff line number Diff line change
@@ -53,6 +53,7 @@ console_scripts =
gempyor-seir = gempyor.simulate_seir:simulate
gempyor-simulate = gempyor.simulate:simulate
flepimop-calibrate = gempyor.calibrate:calibrate
flepimop-pull = gempyor.resume_pull:fetching_resume_files

[options.packages.find]
where = src
154 changes: 154 additions & 0 deletions flepimop/gempyor_pkg/src/gempyor/resume_pull.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
#!/usr/bin/env python
"""
Script for fetching resume files based on various input parameters.
Overview:
This script is designed to fetch resume files based on various input parameters.
It uses Click for command-line interface (CLI) options and handles the fetching process either by downloading from an S3 bucket or moving files locally.
Dependencies:
- click: A package for creating command-line interfaces.
- os: A module for interacting with the operating system.
- gempyor.utils: A module containing utility functions create_resume_file_names_map, download_file_from_s3, and move_file_at_local.
CLI Options:
The script uses Click to define the following command-line options:
--resume_location:
- Environment Variables: LAST_JOB_OUTPUT, RESUME_LOCATION
- Type: STRING
- Required: Yes
- Description: The path for the last run's output.
--discard_seeding:
- Environment Variable: RESUME_DISCARD_SEEDING
- Type: BOOL
- Required: Yes
- Description: Boolean value indicating whether to discard seeding or not.
- valid values: true, 1, y, yes, True, False, false, f, 0, no, n
--block_index:
- Environment Variable: FLEPI_BLOCK_INDEX
- Type: STRING
- Required: Yes
- Description: The block index for the FLEPI.
--resume_run_index:
- Environment Variable: RESUME_RUN_INDEX
- Type: STRING
- Required: Yes
- Description: The run index for resuming.
--flepi_run_index:
- Environment Variable: FLEPI_RUN_INDEX
- Type: STRING
- Required: Yes
- Description: The run index for the FLEPI.
--flepi_prefix:
- Environment Variable: FLEPI_PREFIX
- Type: STRING
- Required: Yes
- Description: The prefix for the FLEPI.
Function: fetching_resume_files
Parameters:
- resume_location (str): Path to the last run's output.
- discard_seeding (bool): Whether to discard seeding.
- flepi_block_index (str): Block index for FLEPI.
- resume_run_index (str): Run index for resuming.
- flepi_run_index (str): Run index for FLEPI.
- flepi_prefix (str): Prefix for FLEPI.
Description:
The function fetching_resume_files fetches resume files based on the provided parameters. It checks if the resume_location is an S3 path and decides to download from S3 or move files locally accordingly.
Workflow:
1. Retrieves the environment variable SLURM_ARRAY_TASK_ID for the slot index.
2. Converts the discard_seeding boolean to a string "true" if it is True.
3. Creates a resume file name map using the create_resume_file_names_map function.
4. Checks if resume_location starts with "s3://":
- If yes, downloads the file from S3 using download_file_from_s3.
- If no, moves the file locally using move_file_at_local.
Example Usage:
To use this script, you can run it from the command line with the required options:
```sh
python script_name.py --resume_location "path/to/resume" --discard_seeding True --block_index "block123" --resume_run_index "run456" --flepi_run_index "run789" --flepi_prefix "prefix"
"""
import click
import os
from gempyor.utils import create_resume_file_names_map, download_file_from_s3, move_file_at_local

@click.command()
@click.option(
"--resume_location",
"resume_location",
envvar=["LAST_JOB_OUTPUT", "RESUME_LOCATION"],
type=click.STRING,
required=True,
help="the path for the last run's output",
)
@click.option(
"--discard_seeding",
"discard_seeding",
envvar="RESUME_DISCARD_SEEDING",
type=click.BOOL,
required=True,
help="required bool value for discarding seeding or not"
)
@click.option(
"--block_index",
"flepi_block_index",
envvar="FLEPI_BLOCK_INDEX",
type=click.STRING,
required=True
)
@click.option(
"--resume_run_index",
"resume_run_index",
envvar="RESUME_RUN_INDEX",
type=click.STRING,
required=True,
)
@click.option(
"--flepi_run_index",
"flepi_run_index",
envvar="FLEPI_RUN_INDEX",
type=click.STRING,
required=True
)
@click.option(
"--flepi_prefix",
"flepi_prefix",
envvar="FLEPI_PREFIX",
type=click.STRING,
required=True
)
def fetching_resume_files(resume_location,
discard_seeding,
flepi_block_index,
resume_run_index,
flepi_run_index,
flepi_prefix):
flep_slot_index = os.environ["SLURM_ARRAY_TASK_ID"]
if discard_seeding is True:
discard_seeding = "true"

resume_file_name_map = create_resume_file_names_map(resume_discard_seeding=discard_seeding,
flepi_block_index=flepi_block_index,
resume_run_index=resume_run_index,
flepi_prefix=flepi_prefix,
flepi_slot_index=flep_slot_index,
flepi_run_index=flepi_run_index,
last_job_output=resume_location)
print(resume_file_name_map)
if resume_location.startswith("s3://"):
download_file_from_s3(resume_file_name_map)
else:
move_file_at_local(resume_file_name_map)


if __name__ == "__main__":
fetching_resume_files()