Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: Test Models Against the Full Docker Stack #3183

Closed

Conversation

allgandalf
Copy link
Collaborator

@allgandalf allgandalf commented Jun 11, 2023

Description

In this PR, we have used the full docker stack to test SIPNET by using Docker Compose, this test will run weekly on Mondays.

Motivation and Context

From Discussions in #3184, It would be wise to have a script file and have our cURL POST requests for different models in it as the Matrix option would have to bring up the whole docker stack for each job of the Matrix.

In this PR, we have added the SIPNET model test, which outputs the sipnet.out (model output) file, Test results can be found in this workflow

In the script file, template has been provided to correctly input data into the cURL request so that in future, People can add more models in the script file against which tests can be performed.

Review Time Estimate

  • Immediately
  • Within one week
  • When possible

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My change requires a change to the documentation.
  • My name is in the list of CITATION.cff
  • I have updated the CHANGELOG.md.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

This PR closes #3184

@@ -300,6 +300,7 @@ services:
api:
image: pecan/api:${PECAN_VERSION:-latest}
user: "${UID:-1001}:${GID:-1001}"
restart: unless-stopped
Copy link
Collaborator Author

@allgandalf allgandalf Jun 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be useful to automatically restart the api container if it ever fails when we bring up the whole docker stack

@allgandalf allgandalf marked this pull request as ready for review June 14, 2023 23:15
sudo apt-get -y install docker-compose
- name: Build and run containers
run: |
docker system prune --all --force
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is a prune needed here? Won't it be running in a fresh VM (presumably with ~empty docker cache) every time?

Comment on lines +38 to +39
container:
image: pecan/depends
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm rusty on the details here -- Does running this test inside the depends container mean these deps are used directly, or do we wind up pulling the whole container again to be run inside itself? Would it work to not specify a container at all let the Docker stack run directly on the host?

Comment on lines +78 to +83
docker-compose ps
sleep 20 # Adjust the duration as needed
docker-compose ps
sleep 20
docker-compose ps
sleep 20
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be a loop over an actual status check; maybe something like while [[ $(docker compose ps --format '{{.State}}'|sort|uniq) != "running" ]]; do sleep 10; done

Comment on lines +86 to +90
- name: Display running containers
run: |
docker-compose ps
docker network ls
docker network inspect bridge
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How useful is this step for unattended CI run? Will these outputs be helpful for debugging or will they just be filling the log up?

-F 'notes=' \
'http://172.17.0.1/pecan/04-runpecan.php'

sleep 200
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, probably want to loop over a status check and sleep rather than assume a set duration

Comment on lines +21 to +47
# -----------------------------------------------
# cURL Request Template
#-----------------------------------------------

# Be sure to use the -H header and also use the bridge IP address of the Docker host (172.17.0.1) and not localhost


# curl -v -L -X POST -H "Host: pecan.localhost" \
# -F 'hostname=' \
# -F 'modelid=' \
# -F 'sitegroupid=' \
# -F 'siteid=' \
# -F 'sitename=' \
# -F 'pft[]=' \
# -F 'start=' \
# -F 'end=' \
# -F 'input_met=' \
# -F 'email=' \
# -F 'notes=' \
# 'http://172.17.0.1/pecan/04-runpecan.php'

# Adjust the sleep time according to the model you are running

# Append the workflowid and ID after `out/` by 1 for every consecutive run

# curl -v -L -H "Host: pecan.localhost" \
# 'http://172.17.0.1/pecan/dataset.php?workflowid=(ID)&type=file&name=out/(ID)/sipnet.out'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines seem redundant with the ones above

Suggested change
# -----------------------------------------------
# cURL Request Template
#-----------------------------------------------
# Be sure to use the -H header and also use the bridge IP address of the Docker host (172.17.0.1) and not localhost
# curl -v -L -X POST -H "Host: pecan.localhost" \
# -F 'hostname=' \
# -F 'modelid=' \
# -F 'sitegroupid=' \
# -F 'siteid=' \
# -F 'sitename=' \
# -F 'pft[]=' \
# -F 'start=' \
# -F 'end=' \
# -F 'input_met=' \
# -F 'email=' \
# -F 'notes=' \
# 'http://172.17.0.1/pecan/04-runpecan.php'
# Adjust the sleep time according to the model you are running
# Append the workflowid and ID after `out/` by 1 for every consecutive run
# curl -v -L -H "Host: pecan.localhost" \
# 'http://172.17.0.1/pecan/dataset.php?workflowid=(ID)&type=file&name=out/(ID)/sipnet.out'


curl -v -L -H "Host: pecan.localhost" \
'http://172.17.0.1/pecan/dataset.php?workflowid=99000000001&type=file&name=out/99000000001/sipnet.out'

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a step here verifying that sipnet.out is free from errors / contains the expected output values?

@infotroph
Copy link
Member

infotroph commented May 19, 2024

Left some comments above focused on technical implementation, but from a broader perspective it's worth making sure we're aligned that "Whole-docker-stack Sipnet test via GitHub Actions every Monday" is a good approach. For my part I agree it's smart to make sure we have tests that exercise the whole docker-compose project, but given we already test Sipnet in the daily CI suite I don't have a good sense how much more information we get from testing it again inside the full Docker stack. Emphasis that I really do mean "I don't know", not "I think the proposed approach is wrong"

Pinging @meetagrawal09 for thoughts on how this aligns/interacts with your planned CI development this summer

@meetagrawal09
Copy link
Collaborator

@infotroph Some points that come to my mind when I check this implementation are :

  1. Should we be more selective as to which containers we are running as part of this full docker stack rather than running all of them at once.

  2. We are restricting ourselves with the features provided by the PEcAn API by using curl queries to trigger model runs. A more extensible approach seems to be by using a settings file and triggering workflows by running the workflow.R script.

  3. We can run multiple models here as part of the script in the same job but I prefer having separate models to be running as separate jobs. E.g. Here I can run SIPNET and ED2 as part of the shell script but they both are part of the same job. In case SIPNET fails to run correctly this entire job fails without even trying if ED2 was able to run or not. Rather if I configure them as 2 separate jobs we can analyse issues with each model separately.
    By creating a new job for each model run we are separating out concerns with their failures and also reducing time on each run. It also gives us the liberty to choose which model we want to trigger a run for as part of the CI.

@allgandalf
Copy link
Collaborator Author

Closing this PR in favour of #3345, please reopen if you disagree

@allgandalf allgandalf closed this Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Should the new SIPNET test run on every PR or should it be scheduled on weekly basis
4 participants