Optimize avni_server jvm config for prod env

[himeshr]

Optimize avni_server jvm config for prod env.
We have recently made following config changes to optimize memory and cache usage for avni_server:

avni_server_opts: "... -Xmx7168m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/avni_server/ ..."


avni_server_cache_max_entries: "100"
avni_server_cache_ttl_seconds: "300"
avni_server_cache_max_weight: "3000"

There is need to further optimize this keeping in mind that the prod ec2 instance will be a 4gb RAM , 2 CPU units machine with 1 GB of additional swap space.

#Approach
We will retain the current t3.large ec2 instance for prod-avni-server, which has 8gb ram, but apply jvm config that should work for a 4gb machine. After which we'll monitor for performance and threshold limits breach / Repeated GC need. If they remain within bounds, we'll downgrade the system to a t3.medium ec2-instance.

[himeshr]

Explanation for config changes done:

Config	Meaning
avni_server_cache_max_entries: "100"	Cache max of 100 entries like GlificContactPhoneNumber, etc.
avni_server_cache_ttl_seconds: "300"	Cache entries for a max of 5 mins
avni_server_cache_max_weight: "3000"	Cache max of 3000 * 100 locations
-Xms2560m -Xmx2560m	Set start and max heap space to 2.5GB
-XX:MaxMetaspaceSize=512m -XX:CompressedClassSpaceSize=256m	Set max MetaSpace and CompressedClassSpace to these to limit non-heap space
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath={{avni_server_access_log_dir}}/ -XX:OnOutOfMemoryError="shutdown -r"	Configuration for taking heapDUmp and restart on OOM error
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=50M -Xloggc:{{avni_server_access_log_dir}}/gc.log"	Config to capture gcc logs info for later analysis
-XX:+UseParallelGC -XX:+UseGCOverheadLimit -Xss=1m	Defaults for jvm 8, not set but would like to be applied

[himeshr]

As of now(23 Apr 2024, 5th day from deployment of the jvm and server config changes), it looks like the jvm config targeting a 4gb machine is working fine. Total JVM memory has remained at around 3 GB.
This leaves us with enough spare mem for other operations to do on the Prod-server for debug, etc..

Will monitor for the day and change the instanceType back to t3.medium if things look fine.

CC: @1t5j0y, @vinayvenu

[himeshr]

Downgraded Prod server EC2 instanceType back to t3.mediumfew days ago, performance looks good.