Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stdin stream support. #58

Open
PaulKinlan opened this issue Apr 21, 2020 · 17 comments
Open

Add stdin stream support. #58

PaulKinlan opened this issue Apr 21, 2020 · 17 comments
Labels
help wanted Extra attention is needed

Comments

@PaulKinlan
Copy link
Contributor

Many applications stream STDIN to allow ffmpeg to encode from a data source that is not a file, or a data source that will never end (i.e, livestreaming a web cam).

Right now the lib can't support it because stdin read in WASM needs to be blocking (which freezes the thread) and this means the event loop is frozen too so that data that would have been sent via postMessage can't be processed.

Describe the solution you'd like
I added stdin buffer support to a fork of the other ffmpeg.js that allows STDIN to be read inside a worker. PaulKinlan/ffmpeg.js#1

It uses SharedArrayBuffers to communicate between the window and the worker, so this limit's it to Chrome on Desktop at the moment. It does however allow ffmpeg to block on stdin and then recieve data from the window asynchronously.

@PaulKinlan PaulKinlan changed the title Add stdin stream Add stdin stream support. Apr 21, 2020
@jeromewu
Copy link
Collaborator

This is an interested one, you can check the latest version (^0.8.1) to see if we can do it after the major refectoring.
I will take a look when available, thanks.

@jeromewu jeromewu added the help wanted Extra attention is needed label Apr 29, 2020
@PaulKinlan
Copy link
Contributor Author

Yep. i can certainly have a look.

@ROBERT-MCDOWELL
Copy link

Just following up about this important feature. Is the last git can let us experiment it?

@deepesh-agarwal
Copy link

This feature would be very helpful.

@Banou26
Copy link

Banou26 commented Sep 1, 2020

Is there an ETA on that major refactoring ?

after the major refectoring

If the refactor is not gonna happen(since i have not seen any commits on this repo & your fork of FFMPEG for months), I'm really interested in the stdin stream support and would like to work on it if possible, but i have very small knowledge of any native/wasm tools that this project uses, so if you could tell me where to start/what to look for i'd really appreciate it.

@creativefctr
Copy link

@jeromewu any updates on this?

@jeromewu
Copy link
Collaborator

@creativefctr sadly, the answer is no and I would say it is hard to achieve considering the limitation we have right now.

@bradisbell
Copy link

Hello @jeromewu. The limitation you mentioned... is that lack of SharedArrayBuffer support in browsers? It seems that SharedArrayBuffer is available again. Is it now possible to add STDIN support? Or, is there some other limitation?

@jeromewu
Copy link
Collaborator

Hmm, it is not SharedArrayBuffer that introduces this limitation. It is more like the current state of WebAssembly cannot handle stdin, so I cannot find a way to support stdin at the moment.

@Banou26
Copy link

Banou26 commented Jul 13, 2021

Not sure if most people wanna get into this but you can pretty easily make a small LibAV C++ API compiled to WASM that deals with only computing when necessary to stream data in and out of WASM-land, this is obviously not as simple as this library's plug and play API but if you really want it you can manage to do smth with a little bit of work, I have almost no prior experience with C++ but still managed to make a transmuxer work with streaming in WASM.

@jeromewu
Copy link
Collaborator

Hi @Banou26, that sounds cool! Would you mind sharing a working sample code that handles streaming? Thanks.

@Banou26
Copy link

Banou26 commented Jul 14, 2021

Hey, sorry, sadly i cannot share my actual code as it's very project-specific, If/When i have more time i potentially wanna revisit it to make it an actual library to share with everyone.

Though what i can do now is give you a rough explanation on how it works.

You can basically retake this example https://github.com/leandromoreira/ffmpeg-libav-tutorial/blob/master/2_remuxing.c

And modify it in a way that it doesn't use the filesystem but in memory buffers pushed from javascript.

But most importantly, you need to replace the main loop, while(true) by something non blocking, so for example, you could make it process 5GOPs per javascript calls.
This is to leave the thread it's currently running on, time for javascript to interact with it, e.g push more buffers, get back the previously processed buffers, make more process calls, ect...

So in the end the API you end up with is something like this in javascript

const remuxer = (libavInstance ?? new libavInstance = await (await import('../dist/libav.js'))()).Remuxer(size)
remuxer.init(BUFFER_SIZE)

const process = () => {
  remuxer.clearInput()
  remuxer.push(buffer)
  remuxer.process()
  transmuxedBuffer.push(new Uint8Array(remuxer.getInt8Array()))
  remuxer.clearOutput()
}

process()

// waits for more data to download

process()

// ...ect

remuxer.close()

Which you can then conveniently turn into a Stream based API for ease of use.

Only problem with your project i can see, is that the entire codebase of FFMPEG is based around the fact that it expects to be run on a FS with the full video at its disposition, you could probably hack it into a way that it uses the remote fetching mechanism because this code probably has some IO waiting mechanism, but it still would require probably a LOT of hacking to make it work because of the nature of this project being based on the actual FFMPEG CLI codebase.

Hope this gave you ideas/a start on how you could potentially implement it for this project.
If you have any more questions i'd be happy to try to answer them as much as possible.

@jeromewu
Copy link
Collaborator

@Banou26 Thanks for the explanation, it helps a lot. Right now looks like FS is the bottleneck for the whole thing to happen, I will think about it and see if there is a way to keep most of the features while not to use FS.

@Banou26
Copy link

Banou26 commented Nov 13, 2021

@davedoesdev managed to asyncify ffmpeg.js's emscripten_read_async to get streaming outputs Kagami/ffmpeg.js#166 so we could potentially manage to asyncify emscripten's stdin/stdout's to finally get streaming support?

@vKongv
Copy link

vKongv commented Jul 21, 2022

I am planning to work on a live stream studio web app that allow our streamer to customise their stream with on the web app before they stream to Facebook. MVP require a few key features:

  • Stream from Webcam
  • Work in both web and mobile
  • Add on screen banner that showcase product information
  • Able to stream to Facebook

I am looking for ways to actually do it and is currently experimenting ffmpeg.wasm which bump into the problem of reading constant stream from webcam to ffmpeg before modifying it. Most of the solution online is recording a video and post process it but there weren't any reliable source for live streaming editing.

If anyone could shine me some light on how to achieve this would be greatly appreciated.

@Bessonov
Copy link

Bessonov commented Nov 17, 2023

I have a similar use case as @HamptonMakes in #141 , but to convert webm to mp4 container and subsequently transport it over WebSocket.

It seems like MEMFS could be used to simulate stdin/stdout, as shown in https://github.com/Kagami/ffmpeg.js#user-content-files .

@HamptonMakes @vKongv have you found a solution?

@Banou26
Copy link

Banou26 commented Dec 10, 2023

If anyone ever wants a simple streaming remuxing lib for the web, i've just opened https://github.com/Banou26/libav-wasm (for now it simply remux mkv->mp4 with seeking and streaming support)
I'll try to clean it up a little bit, publish it on npm & add features if anyone actually wants it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

9 participants