Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates to Docker worker image entrypoints #405

Merged
merged 10 commits into from
Aug 28, 2023
Merged

Updates to Docker worker image entrypoints #405

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
f5a1c61
Move NWM image entrypoint args to explicit flags.
robertbartel Jul 20, 2023
eef9f3a
Updates to ngen Docker image entrypoint script.
robertbartel Jul 20, 2023
a10f96f
Update Launcher funcs for generating Docker args.
robertbartel Jul 24, 2023
17d244f
Have ngen entrypoint copy configs to output.
robertbartel Jul 26, 2023
0fe280b
Fix a few sanity checks in ngen image entrypoint.
robertbartel Jul 26, 2023
5269c9e
Add ngen entrypoint node count sanity check.
robertbartel Jul 26, 2023
4ada56d
Improve ngen entrypoint cleanup logic.
robertbartel Jul 26, 2023
08eabe3
Copy configs to output in ngen-cal Docker image.
robertbartel Jul 27, 2023
387abce
Update ngen-cal Docker image to match ngen.
robertbartel Jul 27, 2023
df67199
Update docker/main/ngen-calibration/entrypoint.sh
robertbartel Aug 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 99 additions & 47 deletions docker/main/ngen-calibration/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,50 +1,94 @@
#!/bin/sh
# Managed by the _generate_docker_cmd_args function in scheduler.py of dmod.scheduler
#
# $1 will have the number of nodes associated with this run
# $2 will have comma-delimited host strings in MPI form; e.g., hostname:N,hostname:M
# $3 will have the unique job id
# $4 is the worker index
# $5 will be the name of the output dataset (which will imply a directory location)
# $6 will be the name of the hydrofabric dataset (which will imply a directory location)
# $7 will be the name of the realization configuration dataset (which will imply a directory location)
# $8 will be the name of the BMI configuration dataset (which will imply a directory location)
# $9 will be the name of the partition configuration dataset (which will imply a directory location)
# TODO: wire up $10
# $10 will be the name of the calibration configuration dataset (which will imply a directory location)

# Not yet supported
# no-op
MPI_NODE_COUNT="${1:?No MPI node count given}"
# no-op
MPI_HOST_STRING="${2:?No MPI host string given}"
# no-op
PARTITION_DATASET_NAME="${9:?}"

JOB_ID=${3:?No Job id given}
WORKER_INDEX=${4:?No worker index given}

OUTPUT_DATASET_NAME="${5:?}"
HYDROFABRIC_DATASET_NAME="${6:?}"
REALIZATION_CONFIG_DATASET_NAME="${7:?}"
BMI_CONFIG_DATASET_NAME="${8:?}"
CALIBRATION_CONFIG_DATASET_NAME="${10:?}"
#!/bin/bash
# Args are managed by the _generate_docker_cmd_args function in scheduler.py of dmod.scheduler

# TODO: Docker secret variable values need to be parameterized
ACCESS_KEY_SECRET="object_store_exec_user_name"
SECRET_KEY_SECRET="object_store_exec_user_passwd"
DOCKER_SECRETS_DIR="/run/secrets"
ACCESS_KEY_FILE="${DOCKER_SECRETS_DIR}/${ACCESS_KEY_SECRET}"
SECRET_KEY_FILE="${DOCKER_SECRETS_DIR}/${SECRET_KEY_SECRET}"

MPI_USER="mpi"
if [ "$(whoami)" = "${MPI_USER}" ]; then
MPI_HOSTS_FILE="${HOME}/.mpi_hosts"
else
MPI_HOSTS_FILE="$(su ${MPI_USER} -c 'echo "${HOME}"')/.mpi_hosts"
fi
RUN_SENTINEL="/home/${MPI_USER}/.run_sentinel"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we put this in /var instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. To go that route, I'd want us to add a dedicated directory under /var as part of the image. It would hold job-execution-items like this and be chown-ed to mpiuser. Otherwise we eventually won't be able to create the sentinel file in the container (see #238).

We also aren't really writing anything to this file, just creating, watching, and removing it. We can discuss further, but I'm inclined to leave it as-is for now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im totally okay with leave it how it is. The only reason I asked was in the situation that an MPI_USER did not have a /home directory.

MPI_RUN="mpirun"

NGEN_SERIAL_EXECUTABLE="/ngen/ngen/cmake_build_serial/ngen"
NGEN_PARALLEL_EXECUTABLE="/ngen/ngen/cmake_build_parallel/ngen"
# This will be symlinked to the parallel one currently
NGEN_EXECUTABLE="/ngen/ngen/cmake_build/ngen"

ALL_DATASET_DIR="/dmod/datasets"
OUTPUT_DATASET_DIR="${ALL_DATASET_DIR}/output/${OUTPUT_DATASET_NAME}"
HYDROFABRIC_DATASET_DIR="${ALL_DATASET_DIR}/hydrofabric/${HYDROFABRIC_DATASET_NAME}"
REALIZATION_CONFIG_DATASET_DIR="${ALL_DATASET_DIR}/config/${REALIZATION_CONFIG_DATASET_NAME}"
BMI_CONFIG_DATASET_DIR="${ALL_DATASET_DIR}/config/${BMI_CONFIG_DATASET_NAME}"
PARTITION_DATASET_DIR="${ALL_DATASET_DIR}/config/${PARTITION_DATASET_NAME}"
CALIBRATION_CONFIG_DATASET_DIR="${ALL_DATASET_DIR}/config/${CALIBRATION_CONFIG_DATASET_NAME}"

while [ ${#} -gt 0 ]; do
case "${1}" in
--config-dataset)
CONFIG_DATASET_NAME="${2:?}"
shift
;;
--host-string)
MPI_HOST_STRING="${2:?}"
shift
;;
--hydrofabric-dataset)
HYDROFABRIC_DATASET_NAME="${2:?}"
shift
;;
--job-id)
JOB_ID="${2:?}"
shift
;;
--node-count)
MPI_NODE_COUNT="${2:?}"
shift
;;
--output-dataset)
OUTPUT_DATASET_NAME="${2:?}"
shift
;;
--partition-dataset)
PARTITION_DATASET_NAME="${2:?}"
shift
;;
--worker-index)
WORKER_INDEX="${2:?}"
shift
;;
esac
shift
done

# Run some sanity checks
# TODO: not sure if this is appropriate or not for calibration exec
if [ -z "${MPI_HOST_STRING:?No MPI hosts string given}" ]; then
echo "Error: MPI host string is empty" > 2>&1
exit 1
fi
# Using complement of valid range to catch non-integer values
if ! [ "${WORKER_INDEX:?No MPI worker index/rank given}" -ge 0 ] 2>/dev/null; then
echo "Error: invalid value '${WORKER_INDEX}' given for MPI worker index/rank" > 2>&1
exit 1
fi
if ! [ "${MPI_NODE_COUNT:?No MPI node count provided}" -gt 0 ] 2>/dev/null; then
echo "Error: invalid value '${MPI_NODE_COUNT}' given for MPI node count" > 2>&1
exit 1
fi

# These serve as both sanity checks and initialization of some derived values
OUTPUT_DATASET_DIR="${ALL_DATASET_DIR:?}/output/${OUTPUT_DATASET_NAME:?No output dataset provided}"
HYDROFABRIC_DATASET_DIR="${ALL_DATASET_DIR}/hydrofabric/${HYDROFABRIC_DATASET_NAME:?No hydrofabric dataset provided}"
CONFIG_DATASET_DIR="${ALL_DATASET_DIR}/config/${CONFIG_DATASET_NAME:?No config dataset provided}"

# Require a partitioning config unless using just a single node for serial exec
# TODO: account for CPU count more appropriately when that gets merged in for ngen image
if [ ${MPI_NODE_COUNT:?No MPI node count provided} -gt 1 ]; then
PARTITION_DATASET_DIR="${ALL_DATASET_DIR:?}/config/${PARTITION_DATASET_NAME:?No partition config dataset name given}"
fi

print_date() {
date "+%Y-%m-%d,%H:%M:%S"
Expand Down Expand Up @@ -83,12 +127,9 @@ load_object_store_keys_from_docker_secrets() {
start_calibration() {
# Start ngen calibration
echo "$(print_date) Starting serial ngen calibration"
# CALIBRATION_CONFIG_FILE=${CALIBRATION_CONFIG_DATASET_DIR}/$(basename $(find ${CALIBRATION_CONFIG_DATASET_DIR} -name "*.yaml" -maxdepth 1 | head -1))

# TODO: move this to CALIBRATION_CONFIG_DATASET_DIR
# NOTE: assumes that calibration dataset will be in realization config dataset AND that it is
# the only yaml file at the top level of that dataset.
CALIBRATION_CONFIG_FILE=${REALIZATION_CONFIG_DATASET_DIR}/$(basename $(find ${REALIZATION_CONFIG_DATASET_DIR} -name "*.yaml" -maxdepth 1 | head -1))
# Find and use copy of config in output dataset
CALIBRATION_CONFIG_FILE=$(find ${OUTPUT_DATASET_DIR} -type f -iname "*.yaml" -o -iname "*.yml" -maxdepth 1 | head -1)

if [ -z "${CALIBRATION_CONFIG_FILE}" ]; then
echo "Error: NGEN calibration yaml file not found" 2>&1
Expand All @@ -106,12 +147,23 @@ start_calibration() {
}

# Sanity check that the output, hydrofabric, and config datasets are available (i.e., their directories are in place)
check_for_dataset_dir "${REALIZATION_CONFIG_DATASET_DIR}"
check_for_dataset_dir "${BMI_CONFIG_DATASET_DIR}"
check_for_dataset_dir "${PARTITION_DATASET_DIR}"
check_for_dataset_dir "${HYDROFABRIC_DATASET_DIR}"
check_for_dataset_dir "${OUTPUT_DATASET_DIR}"
# check_for_dataset_dir "${CALIBRATION_CONFIG_DATASET_DIR}"
check_for_dataset_dir "${CONFIG_DATASET_DIR}"
if [ -n "${PARTITION_DATASET_DIR:-}" ]; then
check_for_dataset_dir "${PARTITION_DATASET_DIR:?No partition dataset directory defined}"
fi
check_for_dataset_dir "${HYDROFABRIC_DATASET_DIR}"

# Copy config files to output dataset for record keeping, but only from the "main" worker node
if [ ${WORKER_INDEX} -eq 0 ]; then
# TODO: perform copy of configs to output dataset outside of image (in service) for better performance
cp -a ${CONFIG_DATASET_DIR}/. ${OUTPUT_DATASET_DIR}
if [ -n "${PARTITION_DATASET_DIR:-}" ]; then
# Also, when partition config present, copy that for record keeping
# TODO: perform copy of configs to output dataset outside of image (in service) for better performance
cp -a ${PARTITION_DATASET_DIR}/. ${OUTPUT_DATASET_DIR}
fi
fi

# Move to the output dataset mounted directory
cd ${OUTPUT_DATASET_DIR}
Expand Down
Loading
Loading