Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add streaming functionality for 3D renders #323

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

aaronleetw
Copy link

Closes #169

This PR introduces an Express server (src/main/MJPEGServer.ts) running on ports 1181-1189 to support MJPEG streaming. Users can right-click on the 3D view to access a "Set Stream Settings..." option, which opens a mini-window to configure:

  • Stream enable/disable
  • Stream ID
  • Stream quality

Settings persist on the Hub but not on Satellites (similar to FOV behavior).

  • The Express server starts when at least one stream is enabled and stops when the user closes AdvantageScope.
  • When launched, an INFO dialog displays the server address.
  • The code checks for ID conflicts and warns the user.
  • Streams are available at http://localhost:PORT/<stream_id>.
  • There is an additional optional fps param in the URL to configure the stream FPS. The default is 30 FPS.
  • http://localhost:PORT/ lists active stream IDs.

The renderer converts the rendered canvas to a Data URL, sending it to MJPEGServer, which serves it via Express. However, the stream requires the tab to remain active (on the Hub) or the window to stay open (on a Satellite). Attempts to mitigate this by modifying the if statement in the ThreeDimension renderer to if (this.canvas.getBoundingClientRect().width === 0 && !this.streamSettings.streamEnable) {... were unsuccessful.

{CA4C82F1-FE88-403F-99D6-B0A9E9A24EE9}
{94C04739-AAF0-4014-A4E5-02E4CBE15B90}
{535FC411-4AFC-4E29-BED8-A02245ABA122}
{C6D14C82-86A2-41A1-9FB5-6B4E8F542148}
{CD126C46-D1E5-4C6F-B19A-667683F5BEF9}
{69085703-C10D-4C54-A1FF-74BE512AF2A7}

@aaronleetw aaronleetw changed the title Added streaming functionality for 3D renders Add streaming functionality for 3D renders Jan 31, 2025
@jwbonner
Copy link
Member

Thanks for all of your work on this, as well as the other camera PR. My understanding is that the primary use case for this feature is vision simulation, where the stream could be piped back into a program like PhotonVision for pose estimation. Is that accurate?

I think there's a bigger discussion that needs to be had regarding the best way to approach that kind of simulation. My concern is that it's out of scope for AdvantageScope as a data visualization tool to be involved in the actual simulation pipeline. The requirements for that are very different WRT to latency, setup, and efficiency (e.g. continuing even when the user isn't looking). It seems like the ideal setup for vision simulation is something integrated into the robot program, as in PhotonVision's current simulation. I wouldn't rule out utilizing some of AdvantageScope's code to create some kind of command line tool for vision rendering (integrated into a robot program and using a predefined config), but that's a larger project to take on.

I'll leave this PR open for now since I imagine others will be interested in experimenting with the code, but I'm not convinced at this point that the integrated MJPEG server is the way to go long-term for AdvantageScope. I'm certainly open to ideas though if you have thoughts on any of this.

@jwbonner
Copy link
Member

Settings persist on the Hub but not on Satellites (similar to FOV behavior).

This is actually a bug, thanks for finding that! I pushed a fix that makes the renderer state restore properly for 3D satellites.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Publish rendered 3D camera views for vision simulation testing?
2 participants