[camera_web] Re: Support for camera stream on web #7950

TecHaxter · 2024-10-28T20:19:35Z

This PR aims to provide support for strartImageStream and stopImageStream on Web.

#92460

Based on #6944 from the archived plugins repository

Pre-launch Checklist

POV: my first PR on a public repo
Contains required commits from the PR #6443, resolves unnecessary 202 commits

…d before expecting mock response

…m, and used window.animationFrames

…le height and width in takeFrame

…seOffScreenCanvas

ditman · 2024-10-29T06:13:59Z

I'll re-review this tomorrow!

ditman

Thanks for the contribution @TecHaxter! This is getting close. I left some comments that should improve the performance of the frame capture (maybe hit 60fps!) and the accuracy of the image data reported.

Have you used this code in any demo app compiled for the web?

ditman · 2024-10-30T22:13:51Z

packages/camera/camera_web/CHANGELOG.md

-* Updates minimum supported SDK version to Flutter 3.22/Dart 3.4.
+- Supporting camera image stream on web.
+- Updates minimum supported SDK version to Flutter 3.22/Dart 3.4.


This is probably an automatic update/fix, but please don't change all the early changelog entries of this file, stick to the * for lists. This should simplify this file greatly.

This is fixed, can be marked as resolved?

ditman · 2024-10-30T22:15:43Z

packages/camera/camera_web/example/integration_test/camera_bitrate_test.dart

@@ -106,6 +106,10 @@ void main() {
      ).thenAnswer(
          (_) => Future<MediaStream>.value(canvasElement.captureStream()));

+      when(
+        () => cameraService.hasPropertyOffScreenCanvas(),


Do we need to mock this? The code should be running on Chrome, which AFAIK has an offscreen canvas. What happens to this test if we revert this change?

The cameraService is a mock class object itself.
If we don't stub hasPropertyOffScreenCanvas, it throws an error - "TypeError: null: type 'Null' is not a subtype of type 'bool'

Can we mark this resolved?

ditman · 2024-10-30T22:27:56Z

packages/camera/camera_web/example/integration_test/camera_service_test.dart

+          await completer.future;
+          final CameraImageData cameraImageData = cameraService.takeFrame(
+            videoElement,
+            canUseOffscreenCanvas: true,


We need a test for canUseOffscreenCanvas: false

This is fixed, can be marked as resolved?

ditman · 2024-10-30T22:28:43Z

packages/camera/camera_web/example/integration_test/camera_test.dart

+      when(
+        () => cameraService.hasPropertyOffScreenCanvas(),
+      ).thenAnswer((_) => true);


Same comment as above, we need a test where offscreen canvas is available, and one where it isn't, so we test both code paths.

This is fixed, can be marked as resolved?

ditman · 2024-10-30T22:30:24Z

packages/camera/camera_web/example/integration_test/camera_test.dart

+          )..videoElement = videoElement;
+
+          when(
+            () => cameraService.takeFrame(


We're already mocking the videoElement, why are we mocking the cameraService too?

The cameraService is a mock class object itself.
If we don't stub hasPropertyOffScreenCanvas, it throws an error - "TypeError: null: type 'Null' is not a subtype of type 'bool'

Can we mark this resolved?

ditman · 2024-10-30T22:57:24Z

packages/camera/camera_web/lib/src/camera.dart

@@ -6,7 +6,6 @@ import 'dart:async';
 import 'dart:js_interop';
 import 'dart:ui';
 import 'dart:ui_web' as ui_web;
-


Do not delete this empty line please :)

This is fixed, can be marked as resolved?

ditman · 2024-10-30T23:00:42Z

packages/camera/camera_web/lib/src/camera.dart

+    Completer<void> completer = Completer<void>();
+    completer.complete();


Why do we need this? Remove?

This is fixed, can be marked as resolved?

ditman · 2024-10-30T23:01:22Z

packages/camera/camera_web/lib/src/camera.dart

+        return;
+      }
+
+      if (!completer.isCompleted) {


This is always false, isn't it? The completer is being .complete()d unconditionally in line 677?

I have changed the implementation of _triggerAnimationFramesLoop in upcoming commits.

This is fixed, can be marked as resolved?

ditman · 2024-10-30T23:03:30Z

packages/camera/camera_web/lib/src/camera.dart

+      if (!_cameraFrameStreamController.hasListener) {
+        return;
+      }


Do not control the stop of the animation like this. What you need to do is to keep the return value of window.requestAnimationFrame (which is an int), and then when the stream controller ceases to have any listeners (see StreamController.onCancel), call window.cancelAnimationFrame with the last ID returned by rAF.

This is fixed, can be marked as resolved?

ditman · 2024-10-30T23:24:29Z

packages/camera/camera_web/lib/src/camera_service.dart

+      format: const CameraImageFormat(
+        ImageFormatGroup.jpeg,
+        raw: '',
+      ),
+      planes: <CameraImagePlane>[
+        CameraImagePlane(
+          bytes: byteBuffer.asUint8List(),
+          bytesPerRow: 0,
+        ),
+      ],


Is this CameraImageData correctly constructed?

First it says that this is a jpeg (which it isn't), and then puts it in the first plane, but we say it's "0 bytesPerRow".

My suggestions are:

Make the format ImageFormatGroup.unknown (the image coming from the canvas is actually RGBA with 4 bytes per-pixel, arranged in rows, but that's not an expected ImageFormatGroup for the plugin). (Maybe called RGBA32 elsewhere?)

set a raw value that doesn't clash with anything possible with Android or iOS. "unknown" seems to be "0" on android, so we could use that too.

The bytesPerRow of the browser's getImageData seems to be width * 4.

More docs here.

I found FLEX_RGBA_8888 in Android and kCVPixelFormatType_32RGBA in iOS which looks promising for compatibility with RGBA 4 bytes per-pixel arranged image coming from ImageData of Canvas (Although I haven't tested it yet)

Also found a comment in \camera\camera_android_camerax\lib\src\android_camera_camerax.dart

/// [imageFormatGroup] is used to specify the image format used for image /// streaming, but CameraX currently only supports YUV_420_888 (supported by /// Flutter) and RGBA (not supported by Flutter). CameraX uses YUV_420_888 /// by default, so [imageFormatGroup] is not used.

Seeing this, I went with the values you suggested

I have tested this in the example app of google_mediapipe_face_detection (code not pushed yet). It works like charm ✨✨

Can we mark this resolved?

…into camera_web_stream

…Group, 0 as raw value and fixed the bytes per row value

…nFrame for cancelAnimationFrame

…p and improved readability

…enCanvasRenderingContext2D

…alculation on high resolution timestamp

…m takeFrame function of camera service, using the function directly

…or instead of intialize()

…Frequently

…t OffscreenCanvas in camera service

…t OffscreenCanvas in camera test

… accurate fps

…ter_packages into camera_web_stream

TecHaxter · 2024-11-17T12:14:52Z

Hi @ditman The PR is ready for another review, Thanks :)

stuartmorgan-g · 2025-01-14T20:51:37Z

From triage: Ping @ditman on this re-review.

stuartmorgan-g · 2025-03-25T19:39:19Z

Update from triage: we're working through trying to find a web reviewer with availability to look at this. Apologies for the delay.

Piinks · 2025-06-04T23:29:23Z

@yjbanov can you suggest a reviewer for this change? It has been blocked on a web review.

TecHaxter added 30 commits October 29, 2024 00:57

[camera/camera_web] Supporting camera stream on web

2552bdc

[camera_web] Version and Changelog updated

72a7127

[camera_web] - integration test updated: mock videoElement initialize…

d366566

…d before expecting mock response

[camera_web] - integration test updated: mock videoElement initialize…

4a18e22

…d before expecting mock response

[camera_web] camera service takeFrame with better exception handelling

b12f7e7

[camera_web] - integration test updated: mock videoElement initialize…

8697046

…d before expecting mock response

[camera_web] camera service takeFrame with better exception handelling

b42fbb9

[camera_web] tested takeFrame camera service function

b7b9342

[camera_web] - integration test updated: mock videoElement initialize…

8ec3a04

…d before expecting mock response

[camera_web] camera service takeFrame with better exception handelling

622841c

[camera_web] tested takeFrame camera service function

096ac4d

[camera_web] tested cameraFrameStream

ed6f086

[camera_web] - integration test updated: mock videoElement initialize…

d1244a0

…d before expecting mock response

[camera_web] camera service takeFrame with better exception handelling

d891a36

[camera_web] tested takeFrame camera service function

7dfb194

[camera_web] tested cameraFrameStream

b9111cb

[camera_web] platform version updated according to guidelines

f80cb03

[camera_web] - integration test updated: mock videoElement initialize…

41ac242

…d before expecting mock response

[camera_web] camera service takeFrame with better exception handelling

3edea8c

[camera_web] tested takeFrame camera service function

d1e4d95

[camera_web] tested cameraFrameStream

86da719

[camera_web] platform version updated according to guidelines

dcf48f9

style: removed late from late final_cameraFrameStreamController

7e01244

refactor: cameraFrameStream handles looping of animation frames

6d20e38

refactor: saperated _triggerAnimationFramesLoop from cameraFrameStrea…

f56879e

…m, and used window.animationFrames

chore: can use off screen canvas constant

1e7c869

chore: removed unnecessary string interpolation from videoElement sty…

5b10bce

…le height and width in takeFrame

chore: reusing the OffscreenCanvas or CanvasElement depending on canU…

90bb72e

…seOffScreenCanvas

refactor: removed getCameraImageDataFromBytes to be used in-line

b196180

fix: setup hasPropertyOffScreenCanvas in tests

5c1c662

github-actions bot removed the p: interactive_media_ads Plugin for IMA SDK label Oct 28, 2024

fix: import organized in camera_test

380c35b

TecHaxter mentioned this pull request Oct 28, 2024

[camera_web] Support for camera stream on web #6443

Closed

11 tasks

ditman requested changes Oct 30, 2024

View reviewed changes

TecHaxter added 13 commits November 1, 2024 13:12

Merge branch 'main' of https://github.com/TecHaxter/flutter_packages …

e31fbf2

…into camera_web_stream

[camera_web] fix: Changelog.md

cc818ba

[camera_web] chore: imports organized for camera.dart

a04a472

[camera_web] fix: CameraImageData constructed with unkown ImageFormat…

aa9a819

…Group, 0 as raw value and fixed the bytes per row value

[camera_web]: chore: preserving animationFrameId from requestAnimatio…

5941119

…nFrame for cancelAnimationFrame

[camera_web] chore: removed completer from _triggerAnimationFramesLoo…

3756949

…p and improved readability

[camera_web] fix: jsified willReadFrequently in getContext of Offscre…

74bb74e

…enCanvasRenderingContext2D

[camera_web] chore: optimized frame rate by using simple arithmatic c…

54aa92e

…alculation on high resolution timestamp

[camera_web] chore: removed canUseOffScreenCanvas named paramater fro…

e46ef6f

…m takeFrame function of camera service, using the function directly

[camera_web] chore: initializing canUseOffscreenCanvas from construct…

5e7bb5e

…or instead of intialize()

[camera_web]: replaced dynamic with Object? in map value for willRead…

c48418b

…Frequently

[camera_web] test: camera image stream test cases for with and withou…

a0cd373

…t OffscreenCanvas in camera service

[camera_web] test: camera frame stream test cases for with and withou…

d7b7d5c

…t OffscreenCanvas in camera test

TecHaxter force-pushed the camera_web_stream branch from 5721618 to d7b7d5c Compare November 6, 2024 18:57

Merge branch 'main' into camera_web_stream

6cd955a

TecHaxter requested a review from ditman November 6, 2024 19:10

TecHaxter added 3 commits November 12, 2024 14:17

chore: updated animate function in _triggerAnimationFramesLoop to get…

a959df5

… accurate fps

Merge branch 'main' into camera_web_stream

526160b

Merge branch 'camera_web_stream' of https://github.com/TecHaxter/flut…

98af1e3

…ter_packages into camera_web_stream

stuartmorgan-g added the triage-web Should be looked at in web triage label Jan 14, 2025

mdebbar self-requested a review June 11, 2025 18:23

		Completer<void> completer = Completer<void>();
		completer.complete();

[camera_web] Re: Support for camera stream on web #7950

Are you sure you want to change the base?

[camera_web] Re: Support for camera stream on web #7950

Uh oh!

Conversation

TecHaxter commented Oct 28, 2024

Pre-launch Checklist

Uh oh!

ditman commented Oct 29, 2024

Uh oh!

ditman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TecHaxter Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ditman Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ditman Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TecHaxter commented Nov 17, 2024

Uh oh!

stuartmorgan-g commented Jan 14, 2025

Uh oh!

stuartmorgan-g commented Mar 25, 2025

Uh oh!

Piinks commented Jun 4, 2025

Uh oh!

Uh oh!

TecHaxter Nov 6, 2024 •

edited

Loading

ditman Oct 30, 2024 •

edited

Loading

ditman Oct 30, 2024 •

edited

Loading