Skip to content

CPU Limit Adjustments #1181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: main
Choose a base branch
from
+5 −3
Open

CPU Limit Adjustments #1181

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
f0d8cf0
wip
mrnicegyu11 Sep 19, 2024
e906b41
Merge remote-tracking branch 'upstream/main' into main
mrnicegyu11 Oct 23, 2024
14c751d
Merge remote-tracking branch 'upstream/main' into main
mrnicegyu11 Oct 23, 2024
293f63c
Add csi-s3 and have portainer use it
mrnicegyu11 Oct 24, 2024
f7f72ec
Change request @hrytsuk 1GB max portainer volume size
mrnicegyu11 Oct 25, 2024
94cfb76
t push
mrnicegyu11 Oct 28, 2024
509c717
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Oct 29, 2024
1a65ecf
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Nov 13, 2024
77ee45e
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Nov 25, 2024
c9c70d6
Arch Linux Certificates Customization
mrnicegyu11 Dec 3, 2024
7b8be53
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Dec 5, 2024
bcd61cd
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Dec 12, 2024
58e1030
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Dec 13, 2024
ed8d479
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Jan 10, 2025
dda6e01
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Feb 4, 2025
f6f4f36
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Feb 25, 2025
5dca5c3
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Mar 13, 2025
4a653ef
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Mar 20, 2025
3a21f0f
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Mar 28, 2025
48fbbca
Fix pgsql exporter failure
mrnicegyu11 Apr 24, 2025
08c57db
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 May 6, 2025
5ecbfec
[Kubernetes] Introduce on-prem persistent Storage (Longhorn) :tada: …
YuryHrytsuk May 6, 2025
3ea41b5
Experimental: Try to add tracing to simcore-traefik on master
mrnicegyu11 May 9, 2025
29f2f2e
Fixes https://github.com/ITISFoundation/osparc-simcore/issues/7363
mrnicegyu11 May 14, 2025
cdef57f
Merge branch 'ITISFoundation:main' into main
mrnicegyu11 May 21, 2025
c0f393e
t push
mrnicegyu11 May 23, 2025
34a86fd
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Jul 2, 2025
df3f5df
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Jul 3, 2025
ac44663
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Jul 8, 2025
4100b87
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Jul 21, 2025
759f657
Merge branch 'ITISFoundation:main' into main
mrnicegyu11 Jul 21, 2025
d60fd0c
t checkout -b 2025/redactoMerge remote-tracking branch 'upstream/main'
mrnicegyu11 Jul 25, 2025
a1e36c7
Merge branch 'ITISFoundation:main' into main
mrnicegyu11 Jul 30, 2025
b856eb0
Arch Linux Certificates Customization - 2
mrnicegyu11 Jul 30, 2025
81ce9fb
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Aug 7, 2025
0e32699
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 Aug 19, 2025
dc2846c
Unthrottle
mrnicegyu11 Aug 19, 2025
0840c61
revert
mrnicegyu11 Aug 19, 2025
b7c0dbc
Merge branch 'main' into 2025/change/CPULimitAdjustment
mrnicegyu11 Aug 19, 2025
1db1257
Dont [place registry on GPU or sidecar machines
mrnicegyu11 Aug 19, 2025
7726cff
Merge branch 'main' into 2025/change/CPULimitAdjustment
mrnicegyu11 Aug 20, 2025
5f95bb7
Merge branch 'main' into 2025/change/CPULimitAdjustment
mrnicegyu11 Aug 20, 2025
b0f962c
⚗️ upgrade distribution registry to v3.0.0, add tracing (#1182)
mrnicegyu11 Aug 20, 2025
dede0a1
Increase registry & dv0 CPU limit
mrnicegyu11 Aug 27, 2025
a7a9af9
FluentD configuration: remove `/` in front of container names (#1188)
sanderegg Aug 26, 2025
4d2e914
ensure graylog source is correct (#1189)
sanderegg Aug 26, 2025
1aa8e2e
merge
mrnicegyu11 Aug 27, 2025
a9d7b35
Merge remote-tracking branch 'upstream/main' into 2025/change/CPULimi…
mrnicegyu11 Aug 27, 2025
0086be0
Merge branch 'main' into 2025/change/CPULimitAdjustment
mrnicegyu11 Sep 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions services/registry/docker-compose.yml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,12 @@ services:
failure_action: rollback
parallelism: 1
placement:
constraints:
constraints: ${OPS_REGISTRY_PLACEMENT_CONSTRAINTS}
- node.labels.ops==true
resources:
limits:
memory: 1G
cpus: '6'
cpus: ${OPS_REGISTRY_CPU_LIMIT}
reservations:
memory: 128M
cpus: '0.1'
Expand Down Expand Up @@ -125,7 +125,7 @@ services:
resources:
limits:
memory: 1G
cpus: '2'
cpus: '6'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this number? What is the reasoning behind?
Do we take into account available number of CPU on machines?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the number of available CPUs are not taken into account, and afaik this is correct, they do not matter (as much). [talk to me for clarification ;)]

The number is a guess based on prometheus observation on osparc-master (e.g. PromQL sum(clamp_max(rate(container_cpu_cfs_throttled_seconds_total{image=~"(registry:.*)|(traefik:.*)|(.*itisfoundation.*)|(.*director:.*)"}[2m]) > 0.2,3)) by (container_label_com_docker_swarm_service_name))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nevertheless @mrnicegyu11 if you set a limit above what is available in reality, the container never starts.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK then I vote for removing the limit alltogether. I can see that it was throttled >7 (additional) CPUs at a time on master, so 8 CPUs would be the right number. But this is too high to put into the section if what @sanderegg says is correct. @YuryHrytsuk @sanderegg

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: Test CPU Limit
Optional: Benchmark with and without registry CPU limit

reservations:
memory: 128M
cpus: '0.1'
Expand Down
2 changes: 2 additions & 0 deletions services/registry/template.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
STACK_NAME=${STACK_NAME}

OPS_REGISTRY_REPLICAS=${OPS_REGISTRY_REPLICAS}
OPS_REGISTRY_PLACEMENT_CONSTRAINTS=${OPS_REGISTRY_PLACEMENT_CONSTRAINTS}
OPS_REGISTRY_CPU_LIMIT=${OPS_REGISTRY_CPU_LIMIT}

REGISTRY_DOMAIN=${REGISTRY_DOMAIN}

Expand Down
Loading