Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

impose memory limit on exercise notebooks #188

Closed
buedaswag opened this issue Nov 1, 2021 · 4 comments
Closed

impose memory limit on exercise notebooks #188

buedaswag opened this issue Nov 1, 2021 · 4 comments
Assignees

Comments

@buedaswag
Copy link
Contributor

  • exercise notebooks need to run on a limited amount of ram
  • if an exercise notebook takes up too much ram, the portal will not be able to grade it
  • we need to define how much ram is the limit
  • we need to enforce this limit in the cicd pipeline
@cimendes
Copy link
Member

TO DO: check how to impose this limit on CircleCI and on Kubernetes

@cimendes
Copy link
Member

Option: impose memory limit on docker command in CircleCI !!!!

@cimendes
Copy link
Member

cimendes commented Sep 1, 2022

The grading of BLU09 exercise notebooks consistently fails due to the notebooks consuming too much memory and crashing the containers that run the grading.
This happens when more than one grading container is running.
On my pc, running the exercise notebook took an extra 4.5gb of ram. The cluster that runs the containers for the portal and grading has 8gb of ram in total.
This means that only one BLU09 grading container can run at a time.

To overcome this problem, several alternatives were mentioned:

Limit grading to one container at a time through changing the code for the portal
Limit grading to one container at a time through a slack channel where students coordinate with each other to guarantee that only one student is grading at a time
Redeploy the cluster on a machine with more ram (this would mean redeploying the portal and i, Miguel anticipate would take several days of full time work to accomplish, therefore in my opinion is not practical)
Ask the students to submit the notebooks as soon as possible
Extend the deadline for submissions
Manually grade and manually set the grade on the portal for each notebook
Next steps

Evaluate option 1’s feasibility. If its not feasible then evaluate what other options are available, discuss with other stakeholders, decide and implement
Coordinate with teaching so that future BLUs have more manageable ram requirements (what would a good amount of ram be?)
More issues

#188
LDSSA/ldsagrader#7

@Gustavo-SF
Copy link
Contributor

They will be addressed by #255 and #256. I'll close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants