Read contents of offscreen render target?

[danhambleton]

Is it possible to access the pixels of an offscreen render target texture? Issue #171 seems to imply that reading the contents of the screen is not supported by sokol. Does this apply to render targets? My setup is similar to the one in the offscreen example:

    /* a render pass with one color- and one depth-attachment image */
    sg_image_desc img_desc = {
        .render_target = true,
        .width = 256,
        .height = 256,
        .pixel_format = SG_PIXELFORMAT_RGBA8,
        .min_filter = SG_FILTER_LINEAR,
        .mag_filter = SG_FILTER_LINEAR,
        .sample_count = MSAA_SAMPLES,
        .label = "color-image"
    };
    sg_image color_img = sg_make_image(&img_desc);

[floooh]

No, reading resource content back to the CPU side is generally not supported in sokol_gfx.h, that's also true for render targets. I'd suggest using the extension function approach described in #171 and calling the backend 3D-API specific functions required to get the data back directly.

[danhambleton]

Yep, makes sense (it is indeed very slow). I ended up implementing this little function:

/* extensions impl*/
SOKOL_API_IMPL void sg_read_texture_data(sg_image img_id, void* pixels) {
#if defined(_SOKOL_ANY_GL)
    _sg_image_t* img = _sg_lookup_image(&_sg.pools, img_id.id);
    const GLenum gl_img_format = _sg_gl_teximage_format(img->cmn.pixel_format);
    const GLenum gl_img_type = _sg_gl_teximage_type(img->cmn.pixel_format);
    GLenum gl_img_target = img->gl.target;
    GLuint gl_img_level = 0;
    glGetTexImage(gl_img_target, abc, gl_img_format, gl_img_type, pixels);
#endif
    
}

I found that you have to call this function after the second onscreen pass. That is, the first pass renders the texture off-screen, then the second pass renders it to a full-screen quad. Right before the sg_end_pass of that second step is when I needed to read the texture (otherwise all pixels were black).

[edubart]

How to do this on other backends (Metal/DX11/WebGPU), does anyone have this extension implemented for those other backends?

Reading pixels from render target textures is useful for my use case where I need to cache heavy rendering output to the disk and also to save screenshots to disk.

[Fra-Ktus]

I needed that and ended implementing a DownloadTexture function for DirectX 11 and Metal.

You can't read the texture directly and have to copy it first.

Here is the extract for Metal:

id<MTLTexture> tex = 0;

// get the texture from the sokol internals here...

if (tex)
{
    id<MTLTexture> temp_texture = 0;
    if (_sg_mtl_cmd_queue && tex)
    {
        const MTLPixelFormat format = [tex pixelFormat];
        MTLTextureDescriptor* textureDescriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:format
                                                           width:(width)
                                                          height:(height)
                                                       mipmapped:NO];
        
        textureDescriptor.storageMode = MTLStorageModeManaged;
        textureDescriptor.resourceOptions = MTLResourceStorageModeManaged;
        textureDescriptor.usage = MTLTextureUsageShaderRead + MTLTextureUsageShaderWrite;
        temp_texture = [_sg_mtl_device newTextureWithDescriptor:textureDescriptor];
        if (temp_texture)
        {
            id<MTLCommandBuffer> cmdbuffer = [_sg_mtl_cmd_queue commandBuffer];
            id<MTLBlitCommandEncoder> blitcmd = [cmdbuffer blitCommandEncoder];
            
            [blitcmd copyFromTexture:tex
                         sourceSlice:0 sourceLevel:0 sourceOrigin:MTLOriginMake(0,0,0) sourceSize:MTLSizeMake(width,height,1)
                    toTexture:temp_texture
                        destinationSlice:0 destinationLevel:0 destinationOrigin:MTLOriginMake(0,0,0)];

            [blitcmd synchronizeTexture:temp_texture slice:0 level:0];
            
            [blitcmd endEncoding];
            
            [cmdbuffer commit];
            
            [cmdbuffer waitUntilCompleted];
        }
    }
    if (temp_texture)
    {
        MTLRegion region = MTLRegionMake2D(0, 0, width, height);
        NSUInteger rowbyte = width*4;
        [temp_texture getBytes:pixels bytesPerRow:rowbyte fromRegion:region mipmapLevel:0];
        result = 1;
    }
}

[edubart]

@Fra-Ktus Thanks for the code, I've ended up with some similar code for Metal.

@floooh What's your opnion on this being officially supported by sokol_gfx.h? I've working functions for D3D11/Metal/OpenGL called sg_query_image_pixels(sg_image img_id, void* pixels, int size) to retrieve pixel data from any sg_image including render targets and sg_query_pixels(int x, int y, int w, int h, bool origin_top_left, void *pixels, int size) to retrieve pixel data from the current framebuffer, that I could make a PR.

[floooh]

What's your opnion on this being officially supported by

No objections, a PR would be welcome. I guess it's better to have any way to extract pixel data to the CPU then none at all, even if it's slow ;)

[Fra-Ktus]

@edubart Here is my DirectX11 code to download the bitmaps:

const ID3D11ShaderResourceView * res_view = 0;

// get resource view from sokol internals here

if (res_view)
{
	ID3D11Resource* tex = 0;
	ID3D11Texture2D* texture_copy = NULL;
	ID3D11ShaderResourceView_GetResource((ID3D11ShaderResourceView *) res_view, &tex);
	if (tex)
	{
		D3D11_TEXTURE2D_DESC description = { 0 };
		ID3D11Texture2D_GetDesc((ID3D11Texture2D*) tex, &description);
		description.BindFlags = 0;
		description.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
		description.Usage = D3D11_USAGE_STAGING;
		HRESULT h = ID3D11Device_CreateTexture2D(_sg.d3d11.dev, &description, NULL, &texture_copy);
		if (h == S_OK)
		{
			ID3D11DeviceContext_CopyResource(_sg.d3d11.ctx, (ID3D11Resource*) texture_copy, tex);
		}
		else
		{
			texture_copy = NULL;
		}
	}

	if (texture_copy)
	{
		D3D11_MAPPED_SUBRESOURCE mappedResource;
		HRESULT h = ID3D11DeviceContext_Map(_sg.d3d11.ctx, (ID3D11Resource*) texture_copy, 0, D3D11_MAP_READ, 0, &mappedResource);
		if (h == S_OK)
		{
			char * src_line_ptr = mappedResource.pData;
			char * dst_line_ptr = pixels;
			size_t s = width * 4;
			for (uint32_t v = 0; v < height; v++)
			{
				memcpy(dst_line_ptr, src_line_ptr, s);
				dst_line_ptr += s;
				src_line_ptr += mappedResource.RowPitch;
			}
			result = 1;
		}
		ID3D11Texture2D_Release(texture_copy);
	}
}

[danhambleton]

FWIW, we have started to collect some of these methods in a separate sokol-ext repo. It would be great to add the metal and DirectX implementations.

[floooh]

To be honest I'm a bit undecided again on whether this should go into the core API after sleeping over it...

Maybe it would indeed make sense to put it into an extension header after all (there could be an "exts" directory in the sokol repository similar to the "utils" directory.

(the point being that reading out pixels "naively" is a blocking operation on all APIs, and a proper async version is either a lot more effort, or impossible (e.g. GL))...

[iryont]

It is not like it would cause any harm, but in some cases it is heavily important to have such ability (rendering expensive stuff which is cached within a texture).

Asynchronous reading would be difficult, indeed, but I think that overall it is not needed as the caller should know exactly how expensive such request is and that it needs to be used with care.

Anyway, @Fra-Ktus - you are forgetting about one important thing. Sokol gfx has an internal buffer which needs to be committed before attempting to read back, otherwise you might end up with gibberish data like I did experience myself (rendering to an offscreen target to capture it and cache right away after ending pass for a framebuffer).

Also, capturing cannot work inside an on-going pass in APIs with command buffers, so before capturing following example should be included:

SOKOL_ASSERT(!_sg.mtl.in_pass);
if(_sg_mtl_cmd_buffer) {
    #if defined(_SG_TARGET_MACOS)
    [_sg_mtl_uniform_buffers[_sg.mtl.cur_frame_rotate_index] didModifyRange:NSMakeRange(0, _sg.mtl.cur_ub_offset)];
    #endif
    [_sg_mtl_cmd_buffer commit];
    [_sg_mtl_cmd_buffer waitUntilCompleted];
    _sg_mtl_cmd_buffer = [_sg_mtl_cmd_queue commandBufferWithUnretainedReferences];
}

It's a piece of code from _sg_mtl_commit without encoding presentDrawable because we don't want to do that yet.

[Fra-Ktus]

@iryont - Yes, I am doing that.
For Metal I had to write a custom CommitRenderToTexture function and for DirectX, it's enough to call sg_commit() before reading the content of the buffer.

See the shader browser I build using those functions to read back the content of rendered shaders...
https://youtu.be/RsO5wQhZ3JM
I grab 30 frames for each shader and loop it on the interface to give a preview to the user.
Browsing that many shaders in parallels is not possible on most standard systems.
And by default the Sokol gfx does not have that many slots.

[iryont]

Yea, you did show a good example. It is precisely my point why reading back is an important feature in some cases since real-time rendering would be rather impossible due to required computation power.

[freneticmonkey]

Sorry for digging this post up. I'm trying to use this implementation for picking for my editor that I'm currently working on but I'm seeing a bunch of strange problems that I now suspect are due to the ordering of the calls to glgetteximage() and other operations. I suspect the function above is missing a call to bind the texture before reading it.

[MikeHart66]

Would be super cool to have something in the official distro for reading pixel from a rendertarget texture. +2000 to that

[floooh]

For now I'd prefer this in some sort of optional "extension header". I'm planning an overhaul of the entire resource management area for copying data into and between resources (might be a while though). Maybe that's an opportunity to tackle the "copy data out of resources" problem, but any solution that's created before would be broken anyway.

[SoylentGraham]

Just to add-on another read-pixel solution for opengl, I was happy to see my PBO RAII implementation (desktop only iirc, or maybe I never finished an ES) worked straight away with a couple of lines. (This is faster than glGetTexImage on FBO)
https://github.com/SoylentGraham/SoyLib/blob/master/src/SoyOpengl.cpp#L1028

Usage;
https://github.com/NewChromantics/PopEngine/blob/master/src/TApiSokol.cpp#L499

Needs putting in a nice sokol-style approach :)

In this SoyLib there's also a few other approaches (eg. apple's CPU-memory-backed-buffer for textures which is ultra fast read/write)

[freneticmonkey]

Continuing to bump this post, I’ve significantly improved performance of my implementation to limit the copy to only the pixel that the mouse is over. Copy times went from 100s of ms to <1ms. If anyone is interested I can figure out how to share my Sokol ‘additions’ to get this working.

[ylluminate]

@floooh have you locked into a solution for this yet? There's been some increased discussion for this being needed for V UI and wondering if we're any closer to having something you feel could work.

[quetzalsly]

any updates on this? it would be cool to have a simple screenshot() function that works with all API's and also readpixels() for render targets, this could be useful for people who want to render to video file.

[Sakari369]

Maybe would be nice to have just a complete example for OpenGL / Metal / DirectX available that can be copied without having to figure it out through this issue..

Going to implement as suggested in this issue, in order to write the texture contents into a file to be able to do test driven development with image comparison to reference images.

@Fra-Ktus, @iryont or @edubart care to share your code for this?
Would help out figuring this out, thanks! 🙂

Btw @Fra-Ktus the shader browser looks pretty cool, would have you have source available for that to look at the implementation?

[edubart]

@Sakari369

Take a look at https://github.com/edubart/sokol_gp/blob/beedb873724c53eb56fe3539882218e7a92aac80/sokol_gfx_ext.h

But keep in mind that I'm no longer supporting that code in this project, because it was out of scope for the project, so the file was removed in more recent commits. However, the code worked fine. The Metal part of that code was made by @iryont , so credits goes to him too.

[Sakari369]

Thank you @edubart , that's a nice and clean implementation, and works perfectly when testing with OpenGL.
Had to comment out the Metal implementation now, as it was depending on SDL2 for some parts.

Could easily clean it up though, seems there is only SDL_ConvertPixels and some constants used from SDL.
If I get to removing the SDL dependency, can post results here.For now the OpenGL implementation is enough for me, but surely will need a Metal implementation later down the road.

[iryont]

@Sakari369 You don't need SDL2 for that matter at all. I just wanted a static output format of RGBA despite either ARGB or ABGR framebuffer underlying format.

[Sakari369]

@iryont Ah okay, thank you for describing that out.
Thanks for writing the code also!

[danielchasehooper]

+1 Could use texture read back. I don't care that it is slow/blocking - I have to get the pixels in order to save as an image to disk for an "export" feature.

If you want to encourage people away from slow paths, call it something like sg_read_image_THIS_IS_SLOW() (or whatever)

[logankaser]

I have a friend using a renderer I built around sokol to generate data for an ML project.
He just wants to render offscreen with an EGL context and then save that to the disk, real-time
speed is not a priority, and having something built-in would be nice.

[nurpax]

IMO this feature would be valuable for debugging, for example being able to save the RT context to a .png.