Fixed ImageBitmap memory leak (cvat-ai#7995)

<!-- Raise an issue to propose your change
(https://github.com/cvat-ai/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution guide](https://docs.cvat.ai/docs/contributing/).
-->

<!-- Provide a general summary of your changes in the Title above -->

### Motivation and context
Resolved cvat-ai#7909
Resolved cvat-ai#7850

### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->

### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](

https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [x] I have increased versions of npm packages if it is necessary

([cvat-canvas](https://github.com/cvat-ai/cvat/tree/develop/cvat-canvas#versioning),

[cvat-core](https://github.com/cvat-ai/cvat/tree/develop/cvat-core#versioning),

[cvat-data](https://github.com/cvat-ai/cvat/tree/develop/cvat-data#versioning)
and

[cvat-ui](https://github.com/cvat-ai/cvat/tree/develop/cvat-ui#versioning))

### License

- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/cvat-ai/cvat/blob/develop/LICENSE) that covers the
project.
  Feel free to contact the maintainers if that's a concern.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Addressed a significant memory leak issue by ensuring `ImageBitmap`
objects are properly closed after use.
- Updated various components to handle cleanup and termination of
workers and instances correctly, preventing potential resource leaks.

- **Version Updates**
  - Updated `cvat-canvas` to version 2.20.3.
  - Updated `cvat-core` to version 15.0.6.
  - Updated `cvat-data` to version 2.1.0.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

changelog.d/20240606_134734_boris_bitmap_memoryleak.md

@@ -0,0 +1,4 @@
+### Fixed
+
+- Significant memory leak related to the frames, which did not memory after become unused
+  (<https://github.com/cvat-ai/cvat/pull/7995>)

cvat-canvas/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cvat-canvas",
-  "version": "2.20.2",
+  "version": "2.20.3",
   "type": "module",
   "description": "Part of Computer Vision Annotation Tool which presents its canvas library",
   "main": "src/canvas.ts",

cvat-canvas/src/typescript/canvasView.ts

@@ -451,6 +451,7 @@ export class CanvasViewImpl implements CanvasView, Listener {
                     results.forEach((bitmap, idx) => {
                         const [curLeft, curTop] = objects[idx].points.slice(-4, -2);
                         canvas.getContext('2d').drawImage(bitmap, curLeft - left, curTop - top);
+                        bitmap.close();
                     });
 
                     const imageData = canvas.getContext('2d')

cvat-core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "cvat-core",
-  "version": "15.0.5",
+  "version": "15.0.6",
   "type": "module",
   "description": "Part of Computer Vision Tool which presents an interface for client-side integration",
   "main": "src/api.ts",

cvat-core/src/frames.ts

@@ -295,8 +295,8 @@ Object.defineProperty(FrameData.prototype.data, 'implementation', {
                     const nextChunkNumber = findTheNextNotDecodedChunk(this.number);
                     const predecodeChunksMax = Math.floor(decodedBlocksCacheSize / 2);
                     if (nextChunkNumber * chunkSize <= stopFrame &&
-                        nextChunkNumber <= chunkNumber + predecodeChunksMax) {
-                        provider.cleanup(1);
+                        nextChunkNumber <= chunkNumber + predecodeChunksMax
+                    ) {
                         frameDataCache[this.jobID].activeChunkRequest = new Promise((resolveForward) => {
                             const releasePromise = (): void => {
                                 resolveForward();
@@ -306,6 +306,14 @@ Object.defineProperty(FrameData.prototype.data, 'implementation', {
                             frameDataCache[this.jobID].getChunk(
                                 nextChunkNumber, ChunkQuality.COMPRESSED,
                             ).then((chunk: ArrayBuffer) => {
+                                if (!(this.jobID in frameDataCache)) {
+                                    // check if frameDataCache still exist
+                                    // as it may be released during chunk request
+                                    resolveForward();
+                                    return;
+                                }
+
+                                provider.cleanup(1);
                                 provider.requestDecodeBlock(
                                     chunk,
                                     nextChunkNumber * chunkSize,
@@ -333,13 +341,13 @@ Object.defineProperty(FrameData.prototype.data, 'implementation', {
             onServerRequest();
             frameDataCache[this.jobID].latestFrameDecodeRequest = requestId;
             (frameDataCache[this.jobID].activeChunkRequest || Promise.resolve()).finally(() => {
-                if (frameDataCache[this.jobID].latestFrameDecodeRequest !== requestId) {
+                if (frameDataCache[this.jobID]?.latestFrameDecodeRequest !== requestId) {
                     // not relevant request anymore
                     reject(this.number);
                     return;
                 }
 
-                // it might appear during decoding, so, check again
+                // it might appear during previous decoding, so, check again
                 const currentFrame = provider.frame(this.number);
                 if (currentFrame) {
                     resolve({
@@ -359,6 +367,14 @@ Object.defineProperty(FrameData.prototype.data, 'implementation', {
                         chunkNumber, ChunkQuality.COMPRESSED,
                     ).then((chunk: ArrayBuffer) => {
                         try {
+                            if (!(this.jobID in frameDataCache)) {
+                                // check if frameDataCache still exist
+                                // as it may be released during chunk request
+                                resolveLoadAndDecode();
+                                reject(this.number);
+                                return;
+                            }
+
                             provider
                                 .requestDecodeBlock(
                                     chunk,
@@ -703,6 +719,13 @@ export function getCachedChunks(jobID): number[] {
 
 export function clear(jobID: number): void {
     if (jobID in frameDataCache) {
+        frameDataCache[jobID].provider.close();
+        for (const contextImagesByFrame of Object.values(frameDataCache[jobID].contextCache)) {
+            for (const image of Object.values(contextImagesByFrame.data)) {
+                image.close();
+            }
+        }
+
         delete frameDataCache[jobID];
         delete frameMetaCache[jobID];
     }

cvat-data/package.json

@@ -1,7 +1,7 @@
 {
   "name": "cvat-data",
   "type": "module",
-  "version": "2.0.0",
+  "version": "2.1.0",
   "description": "",
   "main": "src/ts/cvat-data.ts",
   "scripts": {

cvat-data/src/ts/cvat-data.ts

@@ -98,7 +98,8 @@ export class FrameDecoder {
     // used for video chunks to get correct side after decoding
     private renderWidth: number;
     private renderHeight: number;
-    private zipWorker: Worker;
+    private zipWorker: Worker | null;
+    private videoWorker: Worker | null;
 
     constructor(
         blockType: BlockType,
@@ -108,6 +109,8 @@ export class FrameDecoder {
     ) {
         this.mutex = new Mutex();
         this.orderedStack = [];
+        this.zipWorker = null;
+        this.videoWorker = null;
 
         this.cachedChunksLimit = Math.max(1, cachedBlockCount);
         this.dimension = dimension;
@@ -139,6 +142,14 @@ export class FrameDecoder {
             if (typeof lastChunk === 'undefined') {
                 return;
             }
+
+            for (const frame of Object.keys(this.decodedChunks[lastChunk])) {
+                const data = this.decodedChunks[lastChunk][frame];
+                if (data instanceof ImageBitmap) {
+                    data.close();
+                }
+            }
+
             delete this.decodedChunks[lastChunk];
             length--;
         }
@@ -232,7 +243,15 @@ export class FrameDecoder {
 
     async startDecode(): Promise<void> {
         const blockToDecode = { ...this.requestedChunkToDecode };
-        const release = await this.mutex.acquire();
+        const releaseMutex = await this.mutex.acquire();
+        const release = (): void => {
+            if (this.videoWorker) {
+                this.videoWorker.terminate();
+                this.videoWorker = null;
+            }
+
+            releaseMutex();
+        };
         try {
             const { start, end, block } = this.requestedChunkToDecode;
             if (start !== blockToDecode.start) {
@@ -251,12 +270,12 @@ export class FrameDecoder {
             this.requestedChunkToDecode = null;
 
             if (this.blockType === BlockType.MP4VIDEO) {
-                const worker = new Worker(
+                this.videoWorker = new Worker(
                     new URL('./3rdparty/Decoder.worker', import.meta.url),
                 );
                 let index = start;
 
-                worker.onmessage = (e) => {
+                this.videoWorker.onmessage = (e) => {
                     if (e.data.consoleLog) {
                         // ignore initialization message
                         return;
@@ -283,22 +302,20 @@ export class FrameDecoder {
                             this.decodedChunks[chunkNumber] = decodedFrames;
                             this.chunkIsBeingDecoded.onDecodeAll();
                             this.chunkIsBeingDecoded = null;
-                            worker.terminate();
                             release();
                         }
                     });
 
                     index++;
                 };
 
-                worker.onerror = (event: ErrorEvent) => {
+                this.videoWorker.onerror = (event: ErrorEvent) => {
                     release();
-                    worker.terminate();
                     this.chunkIsBeingDecoded.onReject(event.error);
                     this.chunkIsBeingDecoded = null;
                 };
 
-                worker.postMessage({
+                this.videoWorker.postMessage({
                     type: 'Broadway.js - Worker init',
                     options: {
                         rgb: true,
@@ -314,12 +331,12 @@ export class FrameDecoder {
                 const sps = avc.sps[0];
                 const pps = avc.pps[0];
 
-                worker.postMessage({ buf: sps, offset: 0, length: sps.length });
-                worker.postMessage({ buf: pps, offset: 0, length: pps.length });
+                this.videoWorker.postMessage({ buf: sps, offset: 0, length: sps.length });
+                this.videoWorker.postMessage({ buf: pps, offset: 0, length: pps.length });
 
                 for (let sample = 0; sample < video.getSampleCount(); sample++) {
                     video.getSampleNALUnits(sample).forEach((nal) => {
-                        worker.postMessage({ buf: nal, offset: 0, length: nal.length });
+                        this.videoWorker.postMessage({ buf: nal, offset: 0, length: nal.length });
                     });
                 }
             } else {
@@ -368,6 +385,20 @@ export class FrameDecoder {
         }
     }
 
+    public close(): void {
+        if (this.zipWorker) {
+            this.zipWorker.terminate();
+            this.zipWorker = null;
+        }
+
+        if (this.videoWorker) {
+            this.videoWorker.terminate();
+            this.videoWorker = null;
+        }
+
+        this.cleanup(Number.MAX_SAFE_INTEGER);
+    }
+
     public cachedChunks(includeInProgress = false): number[] {
         const chunkIsBeingDecoded = includeInProgress && this.chunkIsBeingDecoded ?
             Math.floor(this.chunkIsBeingDecoded.start / this.chunkSize) : null;

cvat-ui/src/actions/annotation-actions.ts

@@ -872,7 +872,11 @@ export function closeJob(): ThunkAction {
     return async (dispatch: ActionCreator<Dispatch>, getState): Promise<void> => {
         const state = getState();
         const { instance: canvasInstance } = state.annotation.canvas;
-        const { jobInstance } = receiveAnnotationsParameters();
+        const { jobInstance, groundTruthInstance } = receiveAnnotationsParameters();
+
+        if (groundTruthInstance) {
+            await groundTruthInstance.close();
+        }
 
         if (jobInstance) {
             await jobInstance.close();