Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[IA-4840] [DO NOT MERGE] Testing running jupyter server from docker container for ToA #4465

Open
wants to merge 60 commits into
base: develop
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
4ee4a5d
run jupyter server from docker container in dsvm
LizBaldo Apr 22, 2024
251443e
point to new azure init script
LizBaldo Apr 22, 2024
6331285
retry command does not exist yet...
LizBaldo Apr 22, 2024
8c99d32
update path url
LizBaldo Apr 22, 2024
341f2f3
try to address command not found
LizBaldo Apr 22, 2024
c48163b
update script url
LizBaldo Apr 22, 2024
37cd343
there are no log functions either...
LizBaldo Apr 22, 2024
df50311
update url
LizBaldo Apr 22, 2024
2a1d3eb
mount persistent disk volume and make run-jupyter.sh command found
LizBaldo Apr 23, 2024
1718499
update path
LizBaldo Apr 23, 2024
178a241
fix path
LizBaldo Apr 23, 2024
39458a3
do not run as jupyter user
LizBaldo Apr 23, 2024
3299b5f
update path
LizBaldo Apr 23, 2024
801cc82
control jupyter-user uid to match the docker container
LizBaldo Apr 23, 2024
5e7e618
update path
LizBaldo Apr 23, 2024
7eb1f19
force the creation of the jupyter user home directory
LizBaldo Apr 23, 2024
07f1003
update path
LizBaldo Apr 23, 2024
ffa7990
no need to add to a user grooup
LizBaldo Apr 23, 2024
132f4e8
update path
LizBaldo Apr 23, 2024
490da5a
revert to original useradd commands
LizBaldo Apr 23, 2024
3944c4d
revert sudo when making pd directory
LizBaldo Apr 23, 2024
1bec438
update path
LizBaldo Apr 23, 2024
e6642ca
fix typo
LizBaldo Apr 23, 2024
7fa0556
update path
LizBaldo Apr 23, 2024
aebdb24
set up environment variables properly
LizBaldo Apr 24, 2024
14d3e02
update path
LizBaldo Apr 24, 2024
1e0d205
Merge branch 'develop' into IA-4840-toa-jupyter-docker
LizBaldo Apr 24, 2024
b434b90
mirror the google logic to mount PD and open permissions
LizBaldo Apr 24, 2024
c890157
update path
LizBaldo Apr 24, 2024
121fd9a
mount PD on to the original location...
LizBaldo Apr 24, 2024
5979b89
update path
LizBaldo Apr 24, 2024
e370bc6
publish the port not just exposing it
LizBaldo Apr 25, 2024
3cc6140
update path
LizBaldo Apr 25, 2024
b8d0dcd
create bridge network between containers
LizBaldo Apr 25, 2024
eb702a6
update path
LizBaldo Apr 25, 2024
6614fa9
tripple regex escape fun!
LizBaldo Apr 25, 2024
c88fa0d
clean up - I need it
LizBaldo Apr 25, 2024
f3e2a2a
update path
LizBaldo Apr 25, 2024
6174b98
revert to using host network
LizBaldo Apr 25, 2024
1ec5eab
update path
LizBaldo Apr 25, 2024
c37b678
mount jupyter user home to welder work folder
LizBaldo Apr 25, 2024
f23fae0
update path
LizBaldo Apr 25, 2024
a9072ab
give access to the working directory not just the pd directory
LizBaldo Apr 25, 2024
1822443
update path
LizBaldo Apr 25, 2024
10324be
clean up my sassy comment
LizBaldo Apr 25, 2024
ab54c67
update paths
LizBaldo Apr 25, 2024
b743f9b
add reboot command and fix shared PD volume between jupyter and welder
LizBaldo May 1, 2024
ddda60e
update paths
LizBaldo May 1, 2024
3275c4a
change notebook dir to correspond to the persistent disk and address …
LizBaldo May 1, 2024
c20c8bb
update paths
LizBaldo May 1, 2024
dfa7cc2
fix crontab command
LizBaldo May 1, 2024
a42b85d
update paths
LizBaldo May 1, 2024
0b8f8e6
triple escaping fun...
LizBaldo May 2, 2024
8947c4d
update paths
LizBaldo May 2, 2024
6a900ce
adding the cloud provider as an environment variable to the jupyter s…
LizBaldo May 31, 2024
edd5ff6
fix merge conflicts
LizBaldo Jun 3, 2024
03fa90d
update paths
LizBaldo Jun 3, 2024
45e370d
enable docker container to access GPUs
LizBaldo Jun 4, 2024
e97dcd7
update paths
LizBaldo Jun 4, 2024
f868925
Merge branch 'develop' into IA-4840-toa-jupyter-docker
LizBaldo Jun 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
revert to original useradd commands
LizBaldo committed Apr 23, 2024
commit 490da5a5ef05d41b8bc4ebe8618ced9dd861bb57
23 changes: 11 additions & 12 deletions http/src/main/resources/init-resources/azure_vm_init_script.sh
Original file line number Diff line number Diff line change
@@ -9,12 +9,11 @@ set -e
export DEBIAN_FRONTEND=noninteractive

#create the jupyter user that will have ownership of the persistent disk
JUPYTER_USER=jupyter-user
JUPYTER_USER_UID=1002
JUPYTER_USER_HOME="/home/$JUPYTER_USER"
VM_JUP_USER=jupyter-user
VM_JUP_USER_UID=1002

sudo useradd -m -d $JUPYTER_USER_HOME -N -u $JUPYTER_USER_UID $JUPYTER_USER
#sudo usermod -a -G $JUPYTER_USER,adm,dialout,cdrom,floppy,audio,dip,video,plugdev,lxd,netdev $JUPYTER_USER
sudo useradd -m -c "Jupyter User" -u $VM_JUP_USER_UID $VM_JUP_USER
sudo usermod -a -G $VM_JUP_USER,adm,dialout,cdrom,floppy,audio,dip,video,plugdev,lxd,netdev $VM_JUP_USER
#
### Change ownership for the new user
#
@@ -30,9 +29,9 @@ sudo useradd -m -d $JUPYTER_USER_HOME -N -u $JUPYTER_USER_UID $JUPYTER_USER


# Formatting and mounting persistent disk
WORK_DIRECTORY="/home/$JUPYTER_USER/persistent_disk"
WORK_DIRECTORY="/home/$VM_JUP_USER/persistent_disk"
## Create the PD working directory
mkdir -p ${WORK_DIRECTORY}
sudo mkdir -p ${WORK_DIRECTORY}

## The PD should be the only `sd` disk that is not mounted yet
AllsdDisks=($(lsblk --nodeps --noheadings --output NAME --paths | grep -i "sd"))
@@ -90,7 +89,7 @@ echo "UUID="$OUTPUT" ${WORK_DIRECTORY} ext4 defaults 0 1" | sudo
echo "successful write of PD UUID to fstab"

## Change ownership of the mounted drive to the user
sudo chown -R $JUPYTER_USER:$JUPYTER_USER ${WORK_DIRECTORY}
sudo chown -R $VM_JUP_USER:$VM_JUP_USER ${WORK_DIRECTORY}


# Read script arguments
@@ -210,7 +209,7 @@ echo "docker run -d --restart always --network host --name jupyter \
--volume ${WORK_DIRECTORY}:${NOTEBOOKS_DIR}/persistent_disk \
-e SERVER_APP_BASE_URL=$SERVER_APP_BASE_URL \
-e SERVER_APP_WEBSOCKET_URL=$SERVER_APP_WEBSOCKET_URL \
-e NOTEBOOKS_DIR=/home/$JUPYTER_USER \
-e NOTEBOOKS_DIR=/home/$VM_JUP_USER \
-e WORKSPACE_ID=$WORKSPACE_ID \
-e WORKSPACE_NAME=$WORKSPACE_NAME \
-e WORKSPACE_STORAGE_CONTAINER_URL=$WORKSPACE_STORAGE_CONTAINER_URL \
@@ -222,7 +221,7 @@ docker run -d --restart always --network host --name jupyter \
--volume ${WORK_DIRECTORY}:${NOTEBOOKS_DIR}/persistent_disk \
--env SERVER_APP_BASE_URL=$SERVER_APP_BASE_URL \
--env SERVER_APP_WEBSOCKET_URL=$SERVER_APP_WEBSOCKET_URL \
--env NOTEBOOKS_DIR=/home/$JUPYTER_USER\
--env NOTEBOOKS_DIR=/home/$VM_JUP_USER
--env WORKSPACE_ID=$WORKSPACE_ID \
--env WORKSPACE_NAME=$WORKSPACE_NAME \
--env WORKSPACE_STORAGE_CONTAINER_URL=$WORKSPACE_STORAGE_CONTAINER_URL \
@@ -285,7 +284,7 @@ echo "------ Welder version: ${WELDER_WELDER_DOCKER_IMAGE} ------"
echo " Starting Welder with command...."

echo "docker run -d --restart always --network host --name welder \
--volume \"/home/${JUPYTER_USER}\":\"/work\" \
--volume \"/home/${VM_JUP_USER}\":\"/work\" \
-e WSM_URL=$WELDER_WSM_URL \
-e PORT=8081 \
-e WORKSPACE_ID=$WORKSPACE_ID \
@@ -299,7 +298,7 @@ echo "docker run -d --restart always --network host --name welder \
$WELDER_WELDER_DOCKER_IMAGE"

docker run -d --restart always --network host --name welder \
--volume "/home/${JUPYTER_USER}":"/work" \
--volume "/home/${VM_JUP_USER}":"/work" \
--env WSM_URL=$WELDER_WSM_URL \
--env PORT=8081 \
--env WORKSPACE_ID=$WORKSPACE_ID \