Use glutin to render to Surface/NativeWindow without winit/ndk_glue

[lattice0]

I want to render OpenGL inside Flutter. I've already made it work in Java and I know it's possible in C++.

Flutter has this external texture concept where it creates a SurfaceTexture surfaceTexture that I can render into. Given the surfaceTexture, this is how I render things in Java:

eglSurface = egl.eglCreateWindowSurface(eglDisplay, eglConfig, surfaceTexture, null);
egl.eglMakeCurrent(eglDisplay, eglSurface, eglSurface, eglContext)

after that, it's all a matter of calling opengl functions like glTexImage to send textures, glDraw to draw, etc.

Can I use this lib or even glium to simply create an eglSurface from a surfaceTexture and then draw things on it?

I don't think glium or glutin have the power to do anything with a surfaceTexture. But maybe I can create the eglSurface (from a surfaceTexture) in java calling from Rust, then it returns and then I can issue opengl calls using glutin? Something like this:

flutter engine calls java code
java code creates the surfaceTexture from flutter call and gives it an id
java starts rust code
rust code starts glutin or glium and fills vertex buffers, etc
rust code calls java with the id and thus java code can create the egl surface and make it current
java code call returns, now glium can issue opengl calls like glTexImage and glDraw, since the eglSurface is current, things will be drawn to it

[MarijnS95]

@lattice0 Wow, I didn't expect you to have an issue open on this for well over 1.5 years :)

Is it resolved now that you've submitted a SurfaceTexture implementation to android-ndk-rs? If so, please document here how you've solved it (little codesnippets using that implementation) for posterity, and feel free to close. Thanks!

[lattice0]

@MarijnS95 I didn't solve it yet. I'm implementing SurfaceTexture rendering for glium right now: https://github.com/lattice0/glium/blob/surface_texture/src/backend/surface_texture/mod.rs, using my SurfaceTexture implementation from android-ndk-rs and also using other small PRs I made, but having lots of OpenGL errors since I'm not an OpenGL expert.

I also have a Kotlin Android POC of my SurfaceTexture backend: https://github.com/lattice0/glium_surface_texture_poc/blob/master/app/src/main/rust/surface_texture_glium/src/lib.rs

I don't know to which functions should I map glium's make_current or swap_buffers, and also important functions from SurfaceTexture like update_tex_image have no equivalent on the glium backend side.

I presumed SurfaceTexture's attach_to_gl_context was glium's make_current, so I kind of made a static holder for the current SurfaceTexture and called detach on the old one when a new one should be attached, but I got

2022-05-25 20:40:42.434 8404-8404/com.example.gliumsurfacetexturepoc E/rfacetexturepo: [SurfaceTexture-0-8404-0] attachToContext: invalid current EGLDisplay
2022-05-25 20:40:42.439 8404-8404/com.example.gliumsurfacetexturepoc A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 8404 (rfacetexturepoc), pid 8404 (rfacetexturepoc)

If someone could help me with this I'd be very glad, I'm already deep into lots of small PRs to add functionality to support thtis

[MarijnS95]

Going through the trail of your SurfaceTexture PR this sparked a nice discussion that ended in maplibre/maplibre-rs#53 (comment) and finally a PR with that working: maplibre/maplibre-rs#99.

In short you probably don't want all this complication with SurfaceTexture and the "update" functions. You should be able to get hold of a Surface - which is really just a NativeWindow - and pass that into glutin. You can't yet, but that's what I'm working on while adding supporting the Suspended/Resumed events, in order to also support generic Surfaces without winit (though it may be complicated on the API front).

[lattice0]

I forgot to say. The reason I'm doing this is to render with glium in Flutter, and the only way to render in Flutter is to a SurfaceTexture, because its API gives me one 😩

…

-------- Mensagem Original -------- Ativo 27 de mai. de 2022 06:38, Marijn Suijten escreveu:

Going through the trail of your SurfaceTexture PR this sparked a nice discussion that ended in [maplibre/maplibre-rs#53 (comment)](maplibre/maplibre-rs#53 (comment)) and finally a PR with that working: [maplibre/maplibre-rs#99](maplibre/maplibre-rs#99). In short you probably don't want all this complication with SurfaceTexture and the "update" functions. You should be able to get hold of a Surface - which is really just a NativeWindow - and pass that into glutin. You can't yet, but that's what I'm working on while adding supporting the Suspended/Resumed events, in order to also support generic Surfaces without winit (though it may be complicated on the API front). — Reply to this email directly, [view it on GitHub](#1324 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABSTHAJ2DJFGNQPIPHILXP3VMCJX7ANCNFSM4R5N77YA). You are receiving this because you were mentioned.Message ID: ***@***.***>

[lattice0]

@MarijnS95 here's an example: https://stackoverflow.com/questions/58531692/how-to-render-opengl-in-flutter-for-android

Flutter gives you a SurfaceTexture to render into: https://github.com/mogol/opengl_texture_widget_example/blob/master/android/src/main/java/com/yourcompany/opengltexture/OpenglTexturePlugin.java#L35 and I think there's no other way. I think it would be nice to have SurfaceTexture as a Backend for glium

[MarijnS95]

@lattice0 Right, I'm curious about that example directly passing the SurfaceTexture into eglCreateWindowSurface: https://github.com/mogol/opengl_texture_widget_example/blob/83930bd774505dec0a909abba53bc4c12c3d4d1c/android/src/main/java/com/yourcompany/opengltexture/OpenGLRenderer.java#L77 - I thought that has to go through an ANativeWindow / Surface. To me it feels like this system should be usable one way or another without having to upload "non-GL" data to the OpenGL ES texture with updateTexImage() since you already want to write to a GL texture directly and/or move that written texture into the SurfaceTexture.

[lattice0]

@MarijnS95 yes, when I tried to glue attachToGlContext to make_current I get attachToContext: invalid current EGLDisplay and then Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 8548 (rfacetexturepoc), pid 8548 (rfacetexturepoc) then looking at this error I found https://android.googlesource.com/platform/frameworks/native/+/a0db308/libs/gui/SurfaceTexture.cpp#387 which suggests that there's no EGL display, so I had to create one using an EGL rust wrapper. I then tried using https://github.com/timothee-haudebourg/khronos-egl

If you want to see my very crazy try: https://github.com/lattice0/glium_surface_texture_poc/blob/1e8f669f9dee7d6ef8dd6589fac31754f9c0aea9/app/src/main/rust/surface_texture_glium/src/lib.rs this is where I try to create the EGL display before glium calls some stuff. Also it has my submodules with my modifications to glium and android-ndk-rs for which I'm maintaining until/if the PRs get accepted.

However I don't feel like this is correct at all

[MarijnS95]

@lattice0 I could point out many errors in your codebase, starting with:

• JNI arguments don't match up (passing a surface + surfacetexture from Kotlin, but only accepting a SurfaceTexture in the Rust signature);
• Using JNI sys types instead of the core types (I don't know if they're layout-compatible).

I will later test if/how this works with your SurfaceTexture attempt. But as said many times before, even linked to the known-working maplibre WIP PR, try using a SurfaceView + Surface instead of a SurfaceTexture (again, this seems to contain extra tooling to upload image data from various producers into a GL texture... You can render into a GL texture directly).

I've cooked up an example for you, together with a small WIP change in glutin that replaces winit::Window/winit::WindowBuilder from winit with a RawWindowHandle. This works "by magic" because I finally implemented HasRawWindowHandle on ndk::NativeWindow. Which you can create using the also-recently-added from_surface() function:

https://github.com/MarijnS95/AndroidNativeSurface

Enjoy! 🥳

[lattice0]

@MarijnS95 thank you very much for your attention and example, very helpful. On my example I modified many times the signatures so I completely forgot to fix.

Anyways, the reason I need SurfaceTexture is that I want to render to Flutter, and the only way is using a SurfaceTexture. Flutter has an API that gives me SurfaceTextures where I can render directly, this is different from having an Android widget appearing where a Flutter widget would appear. I think this is not a problem since we can create a Surface from a SurfaceTexture. With that I can get a NativeWindow (this name does not make sense for me, can I have a window for each Surface?)

Anyways, I have a Context:

let context = glutin::ContextBuilder::new()
        .build_windowed(&window)
        .unwrap();

however you seem to use pure opengl calls, while I wanted to use Glium, so I still need a custom Backend for SurfaceTexture if I'm correct? Or maybe not, because I have a window now, from which I can get a display.

So, it looks like I can get glium to work, and I can create a Window for each Surface created from a SurfaceTexture. The word Window bothers me, it seems to say that it won't work with many SurfaceTexture (I plan to use 20 of them).

[MarijnS95]

@lattice0 Fair enough, but you have enough API available to turn a SurfaceTexture into a Surface/NativeWindow: I updated the example to showcase it both in Java (new Surface(surfaceTexture)) and in Rust through your rust-mobile/ndk#267.

It seems that you don't even need to bind a GL context to the SurfaceTexture or call update_tex_image() on the SurfaceTexture explicitly, that must all be happening "internally" when rendering through a Surface as producer (ie. it gets direct access to the GL image instead of having to upload non-GL data into the texture through that function).

however you seem to use pure opengl calls, while I wanted to use Glium, so I still need a custom Backend for SurfaceTexture if I'm correct? Or maybe not, because I have a window now, from which I can get a display.

If Glium supports HasRawWindowHandle, which I hope it does, you can just pass it a NativeWindow. That to me seems like the simplest, most straightforward, and most efficient way to implement this interop.

[MarijnS95]

I didn't even know glium is (only?) backed by glutin, that makes things even "easier" - pending a proper HasRawWindowHandle rework here. It "works", see:

https://github.com/MarijnS95/AndroidNativeSurface/compare/glium

[kchibisov]

Could the latest glutin api help with that? You can create a EglSurface from the native window now just fine.

[MarijnS95]

@kchibisov That's exactly why I was waiting for the refactor, the design you proposed earlier makes sense but I'll have a more proper/in-depth look later. In #1417 at least it seems that you need the raw window handle "quite early on" even though EGL doesn't need it, making things inconvenient.

[kchibisov]

at least it seems that you need the raw window handle "quite early on" even though EGL doesn't need it

The example isn't quite optimal here, right, since only WGL needs it that early, you may adjust it to account for that. The default api that builds display with EGL doesn't require it at all, and it only required by Wgl enum variants.

[MarijnS95]

Yeah, so I'd want to make that an Option and unwrap() for Wgl inside fn create_display() if that's okay? Does make for very ugly and platform-specific code on the user side, though :/

[MarijnS95]

I updated the aforementioned project (https://github.com/MarijnS95/AndroidNativeSurface) to use the 0.30 beta by copy-pasting most of the example again; thanks to the removal of winit this now works natively.

The example should probably be improved to use some animation (spinning/transforming the triangle) to repeatedly re-render instead of only once on surface (re?)creation, also to demonstrate how/where to store the context and other glutin state while going back and forth over JNI, but at least the core issue has been addressed and demonstrated.