Allow getting the raw texture from mvRawTexture

[ProfFan]

Is your feature request related to a problem? Please describe.

Currently, the mvRawTexture item will always fetch the texture from a memory location upon updating. However, when used with 3D contents, it's more efficient to just bind to a texture on GPU instead of refilling from RAM.

Describe the solution you'd like

Either an option in mvRawTexture that disables the fetching or a new texture item that allows getting the native (i.e. GLFW) texture so we can draw scenes on it.

Describe alternatives you've considered

We can always render to a CPU array and ask mvRawTexture to fetch from it.

Additional context

I tried to implement this myself but the documentation on the mvPythonParser is really lacking. I managed to add a function that extracts the _texture as an int, but adding another function that disables the update does not work, and I don't know why.

The error is like:

Error:     [1005]
Command:   set_update_enable
Item:      0
Label:     Not found
Item Type: Unknown
Message:   Item not found: 0

but code is exactly the same like the one getting the texture id (which works). Debugged for 4hrs, no avail.

[hoffstadt]

Hi,

Currently, the mvRawTexture item will always fetch the texture from a memory location upon updating. However, when used with 3D contents, it's more efficient to just bind to a texture on GPU instead of refilling from RAM.

It depends on the intent. We designed the mvRawTexture & mvDynamicTexture to be used when the texture needs to be updated often (i.e. every frame). We optimize for this differently per platform and in the case you are referring to, we use pixel buffer objects. I believe this is considered "best practice" for frequently updated textures. Allocating memory on the GPU is expensive, so you really want to minimize how often you do that. In 0.6, for dynamic textures, we reallocated every frame and it was extremely slow (100x slower in fact).

If you only need to update occasionally, it's recommended to use mvStaticTexture which is created without CPU access flags. When you need to update it, just recreate it.

an option in mvRawTexture that disables the fetching

Again, if you are not planning on updating often, why not use a mvStaticTexture?

new texture item that allows getting the native (i.e. GLFW) texture so we can draw scenes on it

We do plan on adding this ability in the future however we have to decide on a consistent API for doing so across platforms. Its pretty straight forward for OpenGL where the "texture" is just an unsigned int handle but not for the other platforms. In windows, you need a pointer to a shader resource view, and on MacOs you need a MTLTexture and the texture descriptor I believe.

We can always render to a CPU array and ask mvRawTexture to fetch from it.

That is actually how mvRawTexture is meant to be used (look at wiki page).

I tried to implement this myself but the documentation on the mvPythonParser is really lacking. I managed to add a function that extracts the _texture as an int, but adding another function that disables the update does not work, and I don't know why.

I apologize about the internal documentation (mvPythonParser included) lacking! The backend is starting to settle down, so we are spending alot more time cleaning up and documenting. We'd like to help contributors get around.

but code is exactly the same like the one getting the texture id (which works). Debugged for 4hrs, no avail.

I'd have to see how you did it but I'm sure we can help! You can show some of the code here contributing or if you are in the discord server, let use know your username and we can send you an invite to the contributor channel!

[ProfFan]

Thanks for the detailed response @hoffstadt ! My username is the same on Discord :)

What I did -> https://github.com/ProfFan/DearPyGui/pull/2/files

Strangely get_raw_texture is returning something, but set_update_enable fails in the nullptr check.

[hoffstadt]

Just left a comment on the PR. Let me know if that works.

[ProfFan]

@hoffstadt You suggestion works! I think I can try put up a demo now. Will post when I finish...

[hoffstadt]

Awesome!

[ProfFan]

I have a working demo using the fork:

import OpenGL.GL as gl
import OpenGL.arrays.vbo as glvbo
from OpenGL import platform as gl_platform
from OpenGL.raw.GL.VERSION.GL_1_1 import glBindTexture
from OpenGL.raw.GL.VERSION.GL_1_5 import glBindBuffer
from OpenGL.raw.GL.VERSION.GL_3_0 import GL_COLOR_ATTACHMENT0, glBindFramebuffer, glGenFramebuffers
from OpenGL.raw.GL.VERSION.GL_3_2 import glFramebufferTexture

import dearpygui.dearpygui as dpg
from dearpygui.demo import show_demo
dpg.enable_docking(dock_space=True)


with dpg.font_registry():

    # add font (set as default for entire app)
    dpg.add_font(
        "/usr/share/fonts/adobe-source-han-sans/SourceHanSansCN-Regular.otf",
        40,
        default_font=True)


import numpy as np
import threading

width = 800
height = 800
grid_arr = np.ones((width, height, 4), dtype=np.float32)

rng = np.random.default_rng()

with dpg.texture_registry():
    image_id = dpg.add_raw_texture(width,
                                   height,
                                   grid_arr,
                                   format=dpg.mvFormat_Float_rgba)
    image_id_1 = dpg.add_raw_texture(width,
                                     height,
                                     grid_arr,
                                     format=dpg.mvFormat_Float_rgba)

with dpg.window(label="Tutorial"):
    # with dpg.drawlist(width=width, height=height):
    dpg.add_image(image_id)  # , (0, 0), (width, height))

should_continue = True


def setter():
    import time

    while dpg.get_raw_texture(image_id) == 0:
        time.sleep(0.01)

    texture_id = dpg.get_raw_texture(image_id)
    dpg.set_update_enable(image_id, 0)

    # fbo = gl.glGenFramebuffers(1)
    # gl.glBindFramebuffer(gl.GL_FRAMEBUFFER, fbo)
    # gl.glBindTexture(gl.GL_TEXTURE_2D, texture_id)
    # glTexImage2D(gl.GL_TEXTURE_2D, 0, gl.GL_RGB, 400, 400, 0, gl.GL_RGBA, gl.GL_FLOAT, 0)
    # glTextureParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
    # glTextureParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
    # glFramebufferTexture2D(
    #     GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_TEXTURE_2D, texture_id, 0
    # )

    while should_continue:
        rng.standard_normal(out=grid_arr, dtype=np.float32)

        # dpg.set_value(image_id, grid_arr)
        time.sleep(0.01)


t = threading.Thread(target=setter)
t.start()

def exit_handler():
    global should_continue
    print("EXIT!")
    should_continue = False

dpg.set_exit_callback(exit_handler)

class GLPlotter:
    width, height = 800, 800

    def __init__(self, texture_id) -> None:
        import numpy

        self.data = np.array(.2*numpy.random.randn(100000,2),dtype=np.float32)
        self.vbo = glvbo.VBO(self.data)
        self.texture_id = texture_id
        self.fbo = gl.GLuint()

        glGenFramebuffers(1, self.fbo)
    
    def paintGL(self):
        glBindFramebuffer(gl.GL_FRAMEBUFFER, self.fbo)
        glBindTexture(gl.GL_TEXTURE_2D, self.texture_id)
        gl.glColor(1, 1, 0, 0)
        self.vbo.bind()
        # gl.glTexSubImage2D(gl.GL_TEXTURE_2D, 0, 0, 0, width, height, gl.GL_RGBA, gl.GL_FLOAT, 0)
        gl.glFramebufferTexture2D(gl.GL_FRAMEBUFFER, gl.GL_COLOR_ATTACHMENT0, gl.GL_TEXTURE_2D, self.texture_id, 0)

        gl.glEnableClientState(gl.GL_VERTEX_ARRAY)
        gl.glVertexPointer(2, gl.GL_FLOAT, 0, self.vbo)
        gl.glDrawArrays(gl.GL_POINTS, 0, self.data.shape[0])
        self.vbo.unbind()
        glBindFramebuffer(gl.GL_FRAMEBUFFER, 0)


    

if __name__ == "__main__":
    dpg.setup_viewport()

    dpg_context = gl_platform.GetCurrentContext()

    print(f"DPG context: {dpg_context}")

    show_demo()

    glp = None
    while dpg.is_dearpygui_running():

        if glp is None:
            texture_id = dpg.get_raw_texture(image_id)
            if texture_id != 0:
                dpg.set_update_enable(image_id, 0)
                glp = GLPlotter(texture_id)
                print("Got Painter")
        else:
            glp.paintGL()

        dpg.render_dearpygui_frame()

    dpg.cleanup_dearpygui()import OpenGL.GL as gl
import OpenGL.arrays.vbo as glvbo
from OpenGL import platform as gl_platform
from OpenGL.raw.GL.VERSION.GL_1_1 import glBindTexture
from OpenGL.raw.GL.VERSION.GL_1_5 import glBindBuffer
from OpenGL.raw.GL.VERSION.GL_3_0 import GL_COLOR_ATTACHMENT0, glBindFramebuffer, glGenFramebuffers
from OpenGL.raw.GL.VERSION.GL_3_2 import glFramebufferTexture

import dearpygui.dearpygui as dpg
from dearpygui.demo import show_demo
dpg.enable_docking(dock_space=True)


with dpg.font_registry():

    # add font (set as default for entire app)
    dpg.add_font(
        "/usr/share/fonts/adobe-source-han-sans/SourceHanSansCN-Regular.otf",
        40,
        default_font=True)


import numpy as np
import threading

width = 800
height = 800
grid_arr = np.ones((width, height, 4), dtype=np.float32)

rng = np.random.default_rng()

with dpg.texture_registry():
    image_id = dpg.add_raw_texture(width,
                                   height,
                                   grid_arr,
                                   format=dpg.mvFormat_Float_rgba)
    image_id_1 = dpg.add_raw_texture(width,
                                     height,
                                     grid_arr,
                                     format=dpg.mvFormat_Float_rgba)

with dpg.window(label="Tutorial"):
    # with dpg.drawlist(width=width, height=height):
    dpg.add_image(image_id)  # , (0, 0), (width, height))

should_continue = True


def setter():
    import time

    while dpg.get_raw_texture(image_id) == 0:
        time.sleep(0.01)

    texture_id = dpg.get_raw_texture(image_id)
    dpg.set_update_enable(image_id, 0)

    # fbo = gl.glGenFramebuffers(1)
    # gl.glBindFramebuffer(gl.GL_FRAMEBUFFER, fbo)
    # gl.glBindTexture(gl.GL_TEXTURE_2D, texture_id)
    # glTexImage2D(gl.GL_TEXTURE_2D, 0, gl.GL_RGB, 400, 400, 0, gl.GL_RGBA, gl.GL_FLOAT, 0)
    # glTextureParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
    # glTextureParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
    # glFramebufferTexture2D(
    #     GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_TEXTURE_2D, texture_id, 0
    # )

    while should_continue:
        rng.standard_normal(out=grid_arr, dtype=np.float32)

        # dpg.set_value(image_id, grid_arr)
        time.sleep(0.01)


t = threading.Thread(target=setter)
t.start()

def exit_handler():
    global should_continue
    print("EXIT!")
    should_continue = False

dpg.set_exit_callback(exit_handler)

class GLPlotter:
    width, height = 800, 800

    def __init__(self, texture_id) -> None:
        import numpy

        self.data = np.array(.2*numpy.random.randn(100000,2),dtype=np.float32)
        self.vbo = glvbo.VBO(self.data)
        self.texture_id = texture_id
        self.fbo = gl.GLuint()

        glGenFramebuffers(1, self.fbo)
    
    def paintGL(self):
        glBindFramebuffer(gl.GL_FRAMEBUFFER, self.fbo)
        glBindTexture(gl.GL_TEXTURE_2D, self.texture_id)
        gl.glColor(1, 1, 0, 0)
        self.vbo.bind()
        # gl.glTexSubImage2D(gl.GL_TEXTURE_2D, 0, 0, 0, width, height, gl.GL_RGBA, gl.GL_FLOAT, 0)
        gl.glFramebufferTexture2D(gl.GL_FRAMEBUFFER, gl.GL_COLOR_ATTACHMENT0, gl.GL_TEXTURE_2D, self.texture_id, 0)

        gl.glEnableClientState(gl.GL_VERTEX_ARRAY)
        gl.glVertexPointer(2, gl.GL_FLOAT, 0, self.vbo)
        gl.glDrawArrays(gl.GL_POINTS, 0, self.data.shape[0])
        self.vbo.unbind()
        glBindFramebuffer(gl.GL_FRAMEBUFFER, 0)


    

if __name__ == "__main__":
    dpg.setup_viewport()

    dpg_context = gl_platform.GetCurrentContext()

    print(f"DPG context: {dpg_context}")

    show_demo()

    glp = None
    while dpg.is_dearpygui_running():

        if glp is None:
            texture_id = dpg.get_raw_texture(image_id)
            if texture_id != 0:
                dpg.set_update_enable(image_id, 0)
                glp = GLPlotter(texture_id)
                print("Got Painter")
        else:
            glp.paintGL()

        dpg.render_dearpygui_frame()

    dpg.cleanup_dearpygui()

[ProfFan]

This is very very dirty, but it works. Need to think about the API now.

[ProfFan]

Code is in https://github.com/ProfFan/DPG_Demo . Aiming to integrate with vispy now.

[ProfFan]

I have a demo working using ModernGL and my fork:

I think the way I patched DearPyGui is not the best solution, but it works nicely.

Still, event processing is a problem with mvRawTexture. I am thinking about implementing a new widget for this purpose, so events can be passed to the 3D rendering pipeline to enable dragging and clicking.

[hoffstadt]

This is awesome. Glad you are making some progress. I'm hoping we will follow suit sooner rather than later.

[ProfFan]

@hoffstadt Glad to hear that! What do you think about adding a new widget in DPG that is conditionally compiled (on Linux + GLFW), which can pass event to the underlying renderer and renders a raw texture? I think this will enable a lot of cool applications before the new 3D engine is ready. Also, we can later add more platforms to support matrix by similar approach to mvRawTexture and mvUtilities.

[hoffstadt]

I'm not opposed to it at all. We just need to come up with a good and scalable API for it. Converting this issue to a discussion.

[ProfFan]

My vision for the API is:

• Something like a mvRawCanvas, with a backing texture allocated outside of DPG.
• Allows all event types, click, resize, drag, drag-drop, etc, etc
• Can operate both sync and async
  • sync: draw will call a Python callback to ask the underlying lib to draw on the texture
  • async: draw is a no-op, and the drawlist will draw whatever that's on the texture at frame update

I can make a prototype in my fork, and we can evaluate it together.

For OpenGL, it's simple because the texture is just a GLuint. We can implement this first. What is the problem with Vulkan and Metal?

[hoffstadt]

For Vulkan, you are better off reading some of this to understand the issues. If you've worked with Vulkan (or even dxd12) it will make sense (otherwise it may not).

[NaivelyWritten]

Seems the upstream issue has been fixed
ocornut/imgui#914 (comment)

[Tremeschin]

+1 feature, this would also allow some primitive video playback under DPG since writing raw video frame textures from cv2 to ModernGL is quite doable from my experience. (or from some external video decoding raw FBO)

For a first iteration of this API I'm ok with just this proposed async operation mode and the event handlers (click, resize..) for interactive functionality (really, a widget like plot or node editor).

We don't want to add complexity here for the "scalable" part, so keeping MGL and DPG separate, achieving "keep coding in mgl" but with DPG integration would be the mindset? Like how moderngl hugs imgui with their from moderngl_window.integrations.imgui import ModernglWindowRenderer and it just works, we could make a headless window / moderngl context and simply render whatever FBO we pointed at from a DPG widget?

[Amorano]

Exactly this. If you want a canvas widget for OGL, great, slap it in there; I want a proper API around it -- otherwise, why aren't you adding DirectX support too -- and yes, DirectPython is a thing.

[Amorano]

100% behind the idea of a 3D Widget; not 100% behind the idea of using raw OGL commands in the API, which means the same amount of time designing it proper still exists and is required.

All the things you mentioned about extending the API with this, are things already on the roadmap. I would seriously hope not much effort is thrown into this beyond just slapping in a widget (for now).