WPE 2.38: Out of memory error while streaming any content with WebInspector open

[janani-balasubramanian-infosys]

In RPi, there is a significant increase in memory consumption of WPEThunder when Webinspector is started along with the streaming of any content. But no crash is seen because there is no limit set for WPEThunder and We were able to stream for more than 1.5 hours. But, the memory max usage rapidly increases. Below are the observations in RPi.

root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.limit_in_bytes
9223372036854771712

root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
531464192
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
544538624
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
550207488
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
561790976
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
584716288
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
620036096
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
635936768
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
646520832
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
655519744
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
695853056
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
702447616
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
711151616
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
721469440
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
737206272
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
835309568
root@raspberrypi4-64-rdk-android-mc:/sys/fs/cgroup/memory# cat memory.max_usage_in_bytes
838696960

Every memory usage values are noted within 1-2 mins.

Scenario to reproduce

1. Start any video streaming app
2. Open WebInspector
3. Play any VOD
4. Notice the memory consumption

[janani-balasubramanian-infosys]

Below is an additional observation made by making the JSC_criticalGCMemoryThreshold=0.10 .

With this change, the playback was running for almost 45 mins. In this, for the first 30 mins the memory increase was stable. But after 30 mins, the memory increase is very fast.

Image tested:

root@raspberrypi4-64-rdk-android-mc:~# cat /version.txt 
imagename:rdk-generic-mediaclient-wpe-image_FBT_TDK_nightly-20240503-dunfell_20240503043146
BRANCH=nightly-20240503-dunfell
YOCTO_VERSION=dunfell
VERSION=nightly-20240503-dunfell.05.03.24
SPIN=0
BUILD_TIME="2024-05-03 04:31:46"
WPE_WEBKIT_VERSION=2.38.4+gitAUTOINC+fc1703ed69-r4
WPEFRAMEWORK-VERSION=R4.2
Generated on Fri May 03  04:31:46 UTC 2024
JENKINS_JOB=0
JENKINS_BUILD_NUMBER=0

[calvaris]

Could you please test with latest wpe-2.38 ? I landed several build fixes.

[suresh-khurdiya-infosys]

thanks @calvaris ,
We reviewed the last 4 to 6 months patches and we don't see any patch related memory pressure, web inspector side.
Do you have few patch in your mind which we may fix this issue?

[calvaris]

Yes, all patches merged during merge commit 0cb500e .

[hridhya-narayanan-infosys]

@calvaris ,

I have pulled the latest wpe-2.38 code and tested the scenario on RPI4 device.
CNN App was launched and tested video playback while keeping the web inspector open and monitored memory usage.

Observation
I could see memory is getting increased and after some time about 35 mins, I could see WebKitBrowser is deactivated by reason MemoryExceeded.

2024-10-04T11:23:20.509Z WPEFramework[1696]: [SysLog]:[Fatal]: FORCED Shutdown: WebKitBrowser by reason: MemoryExceeded.

root@raspberrypi4-64-rdk-android-mc:~# while [ 1 ]; do
> cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
> sleep 60
> done
373530624
496242688
560435200
618070016
636719104
636719104
636719104
636719104
636719104
640253952
640622592
640655360
657563648
657563648
657563648
657563648
657563648
657563648
660316160
660316160
660643840
661180416
661180416
662335488
662335488
662335488
662335488
662335488
721186816
721186816
721186816
739287040
776536064
835862528
856977408

From Controller UI console I could see below logs:

WebKitBrowser"data: {url: https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#app:com.metrological.app.CNN, loaded: false}
"WebKitBrowser"data: {url: https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#app:com.metrological.app.CNN, loaded: true, httpstatus: 200}
"WebKitBrowser"data: {url: https://widgets.metrological.com/lightning/liberty/2e3c4fc22f0d35e3eb7fdb47eb7d4658#app:com.metrological.app.CNN, loaded: true}
"Monitor"data: {callsign: WebKitBrowser, action: Deactivate, reason: MemoryExceeded}
"Monitor"data: {callsign: WebKitBrowser, action: Activate, reason: Automatic}
"WebKitBrowser"data: {state: deactivated, reason: MemoryExceeded}
"WebKitBrowser"data: {state: activated, reason: Automatic}
"WebKitBrowser"data: {url: about:blank, loaded: false}
"WebKitBrowser"data: {suspended: false}
"WebKitBrowser"data: {url: about:blank, loaded: true, httpstatus: -1}
"WebKitBrowser"data: {url: about:blank, loaded: true}

logs:
wpeframework.log

Without webinspector open I could see memory usage(/sys/fs/cgroup/memory/memory.max_usage_in_bytes) is around 461193216 after 40 mins of CNN playback

Image tested:

root@raspberrypi4-64-rdk-android-mc:~# cat /version.txt 
imagename:rdk-generic-mediaclient-wpe-image_FBT_rdk-next_20241004052558
BRANCH=rdk-next
YOCTO_VERSION=dunfell
VERSION=6.1.10.04.24
SPIN=0
BUILD_TIME="2024-10-04 05:25:58"
WPE_WEBKIT_VERSION=2.38.6+gitAUTOINC+030740eeae-r4
WPEFRAMEWORK-VERSION=R4.4.1
Generated on Fri Oct 04  05:25:58 UTC 2024
JENKINS_JOB=0
JENKINS_BUILD_NUMBER=0
root@raspberrypi4-64-rdk-android-mc:~# 

[hridhya-narayanan-infosys]

Any update on this issue?

[pgorszkowski-igalia]

@hridhya-narayanan-infosys : I am able to reproduce the problem and I am investigating the reason of the problem.

[pgorszkowski-igalia]

@hridhya-narayanan-infosys : the problem is also visible in upstream (in specific situation). I created a fix in upstream: WebKit/WebKit#37536 which is waiting for review and merge.
In the meantime I will prepare a downstream patch which you can apply and test.

[pgorszkowski-igalia]

@hridhya-narayanan-infosys : The downstream patch you can find here: 39f3e68, please use it and test.

[hridhya-narayanan-infosys]

@pgorszkowski-igalia thanks. I will test the issue with above patch.

[hridhya-narayanan-infosys]

@pgorszkowski-igalia , I have tested the issue with above patch on RPI 4 device. I could see good improvements.
The CNN video played without any issue for about 3 hours. After this WebKitBrowser is deactivated by reason MemoryExceeded.

Currently checking on the behaviour with other apps as well.

[hridhya-narayanan-infosys]

Tested PlutoTV App(https://app-virginmedia360.pluto.tv/) on RPI4 device and we could observe crash around 30 mins.

Testing with other apps is in progress.