Add info on soft memory limits (#3672)

* add info on soft memory limits Signed-off-by: Filinto Duran <[email protected]> * add mention about the allowed suffixes Signed-off-by: Filinto Duran <[email protected]> * make format similar to other annotations format Signed-off-by: Filinto Duran <[email protected]> * Update daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md Co-authored-by: Mark Fussell <[email protected]> Signed-off-by: Filinto Duran <[email protected]> * Update daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md Co-authored-by: Mark Fussell <[email protected]> Signed-off-by: Filinto Duran <[email protected]> * Update daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md Co-authored-by: Mark Fussell <[email protected]> Signed-off-by: Filinto Duran <[email protected]> * Update daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md Co-authored-by: Mark Fussell <[email protected]> Signed-off-by: Filinto Duran <[email protected]> * Update daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md Co-authored-by: Mark Fussell <[email protected]> Signed-off-by: Filinto Duran <[email protected]> * Update daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md Co-authored-by: Mark Fussell <[email protected]> Signed-off-by: Filinto Duran <[email protected]> * Update daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md Co-authored-by: Mark Fussell <[email protected]> Signed-off-by: Filinto Duran <[email protected]> * Update daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md Signed-off-by: Mark Fussell <[email protected]> --------- Signed-off-by: Filinto Duran <[email protected]> Signed-off-by: Filinto Duran <[email protected]> Signed-off-by: Mark Fussell <[email protected]> Co-authored-by: Mark Fussell <[email protected]>
dapr · Aug 11, 2023 · 518276b · 518276b
1 parent 44c0256
commit 518276b
Showing 1 changed file with 22 additions and 0 deletions.
diff --git a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md
@@ -68,6 +68,28 @@ Since Dapr is intended to do much of the I/O heavy lifting for your app, it's ex
 
 The CPU and memory limits above account for the fact that Dapr is intended to support a high number of I/O bound operations. It is strongly recommended that you use a monitoring tool to get a baseline for the sidecar (and app) containers and tune these settings based on those baselines.
 
+### Setting soft memory limits on Dapr sidecar
+
+It is recommended to set soft memory limits on the Dapr sidecar when you have set up memory limits. 
+This allows the sidecar garbage collector to free up memory when the memory usage is above the limit instead of 
+waiting to be double of the last amount of memory present in the heap when it was run, which is the default behavior 
+of the [garbage collector](https://tip.golang.org/doc/gc-guide#Memory_limit) used in Go, and can lead to OOM Kill events.
+
+For example, for an app with app-id `nodeapp`, if you have set your memory limit to be 1000Mi as mentioned above, you can use the following in your pod annotations:
+
+```yaml
+  annotations:
+    dapr.io/enabled: "true"
+    dapr.io/app-id: "nodeapp"
+    # our daprd memory settings
+    dapr.io/sidecar-memory-limit: "1000Mi"   # your memory limit
+    dapr.io/env: "GOMEMLIMIT=900MiB"         # 90% of your memory limit. Also notice the suffix "MiB" instead of "Mi"
+```
+
+In this example, the soft limit has been set to be 90% as recommended in [garbage collector tips](https://tip.golang.org/doc/gc-guide#Memory_limit) where it is recommend to leave 5-10% for other services.
+
+The `GOMEMLIMIT` environment variable [allows](https://pkg.go.dev/runtime) certain suffixes for the memory size: `B, KiB, MiB, GiB, and TiB.`
+
 ## Highly-available mode
 
 When deploying Dapr in a production-ready configuration, it is recommend to deploy with a highly available (HA) configuration of the control plane, which creates 3 replicas of each control plane pod in the dapr-system namespace. This configuration allows the Dapr control plane to retain 3 running instances and survive individual node failures and other outages.