Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enlarge the production DB volume #604

Closed
33 tasks done
mfocko opened this issue Oct 9, 2024 · 3 comments · Fixed by #605
Closed
33 tasks done

Enlarge the production DB volume #604

mfocko opened this issue Oct 9, 2024 · 3 comments · Fixed by #605
Assignees
@mfocko
Labels
area/general Related to whole service, not a specific part/integration. complexity/single-task Regular task, should be done within days. gain/high This brings a lot of value to (not strictly a lot of) users. impact/high This issue impacts multiple/lot of users. kind/internal Doesn't affect users directly, may be e.g. infrastructure, DB related.

Comments

@mfocko
Copy link
Member

mfocko commented Oct 9, 2024 •
edited
Loading

We are slowly running out of the space on the production DB volume, therefore we need to enlarge it. Skimmed through the API code, it should be possible to not drop any tasks during the “enlargement procedure”.

  • Try it on stage
    • Scale down workers
      • short-running
      • wait for the queue
      • long-running
    • Redeploy database with enlarged volume
    • Rescale postgres
    • Verify the PVC size
      • OpenShift
      • Within the postgres pod
    • Check integrity of the data
    • Scale the workers back up
    • Trigger packit/hello-world
      • Verify it works
  • Showdown!
    • Scale down workers
      • short-running
      • wait for the queue
      • long-running
    • Redeploy database with enlarged volume
    • Rescale postgres
    • Verify the PVC size
      • OpenShift
      • Within the postgres pod
    • Check integrity of the data
    • Scale the workers back up
    • Trigger packit/hello-world
      • Verify it works
  • Remove banners
    • Matrix
    • Slack
  • Remove the status notice
  • (optional) Rant if needed… Canceled in the end
@mfocko mfocko added area/general Related to whole service, not a specific part/integration. complexity/single-task Regular task, should be done within days. gain/high This brings a lot of value to (not strictly a lot of) users. impact/high This issue impacts multiple/lot of users. kind/internal Doesn't affect users directly, may be e.g. infrastructure, DB related. labels Oct 9, 2024
@mfocko mfocko self-assigned this Oct 9, 2024
@mfocko
Copy link
Member Author

mfocko commented Oct 9, 2024

Rescaling packit-service pod, it appears that even read-only API endpoints fail on postgres redeployment /o\

@mfocko
Copy link
Member Author

mfocko commented Oct 9, 2024

redeployed, :binoculars: Sentry

@mfocko
Copy link
Member Author

mfocko commented Oct 9, 2024

refreshed dashboard few times to make trip, didn't reappear

mfocko added a commit to mfocko/deployment that referenced this issue Oct 9, 2024
@mfocko
chore(postgres): enlarge the postgres volume
4425eec 
Also resized the stage for the testing purposes, it ain't much(, but
it's an honest work), if we reserve a lot of resources on preprod, we
can relax it (we're using barely 300MB on stage).

Turns out that you, indeed, can resize the PVC with the pod running and
then just rescale the deployment (postgres).

Fixes packit#604

Signed-off-by: Matej Focko <[email protected]>
@mfocko mfocko mentioned this issue Oct 9, 2024
Merged
mfocko added a commit that referenced this issue Oct 14, 2024
@mfocko
chore(postgres): enlarge the postgres volume (#605)
56c1bb0 
Also resized the stage for the testing purposes, it ain't much(, but
it's an honest work), if we reserve a lot of resources on preprod, we
can relax it (we're using barely 300MB on stage).

Turns out that you, indeed, can resize the PVC with the pod running and
then just rescale the deployment (postgres).

Fixes #604
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mfocko mfocko

Labels
area/general Related to whole service, not a specific part/integration. complexity/single-task Regular task, should be done within days. gain/high This brings a lot of value to (not strictly a lot of) users. impact/high This issue impacts multiple/lot of users. kind/internal Doesn't affect users directly, may be e.g. infrastructure, DB related.
Projects
Packit Kanban Board
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

chore(postgres): enlarge the postgres volume mfocko/deployment
1 participant
@mfocko