glx backend segfaults when killing xfdesktop

[absolutelynothelix]

when killing xfdesktop new glx backend segfaults:

workstation:~/Downloads/picom$ build/src/picom --config=/dev/null --backend=glx
[ 12/23/2022 13:46:02.781 x_log_error WARN ] X error 8 MATCH request 152 minor 22 serial 586
[ 12/23/2022 13:46:02.781 x_log_error WARN ] X error 9 DRAWABLE request 156 minor 4 serial 587
[ 12/23/2022 13:46:02.781 x_log_error WARN ] X error 161 GLX_BAD_PIXMAP request 152 minor 16 serial 0
[ 12/23/2022 13:46:02.782 x_log_error WARN ] X error 161 GLX_BAD_PIXMAP request 152 minor 16 serial 0
[ 12/23/2022 13:46:02.782 x_log_error WARN ] X error 161 GLX_BAD_PIXMAP request 152 minor 23 serial 599
[ 12/23/2022 13:46:02.782 x_log_error WARN ] X error 8 MATCH request 152 minor 22 serial 602
[ 12/23/2022 13:46:02.782 x_log_error WARN ] X error 9 DRAWABLE request 156 minor 4 serial 603
[ 12/23/2022 13:46:02.782 x_log_error WARN ] X error 161 GLX_BAD_PIXMAP request 152 minor 16 serial 0
[ 12/23/2022 13:46:04.665 x_log_error WARN ] X error 161 GLX_BAD_PIXMAP request 152 minor 16 serial 0
[ 12/23/2022 13:46:04.666 x_log_error WARN ] X error 161 GLX_BAD_PIXMAP request 152 minor 23 serial 1015
[ 12/23/2022 13:46:04.667 _gl_compose ERROR ] GLX error at line 479: GL_INVALID_OPERATION
Segmentation fault (core dumped)

#0  0x000055db4c207748 in gl_compose (base=0x55db4d566d50, image_data=0x55db4da953f0, image_dst=..., mask=0x0, mask_dst=..., reg_tgt=0x7fff4166fd30, 
    reg_visible=0x7fff4166fd50) at ../src/backend/gl/gl_common.c:575
#1  0x000055db4c203e39 in paint_all_new (ps=0x55db4d4d25b0, t=0x55db4d4d9f00, ignore_damage=false) at ../src/backend/backend.c:189
#2  0x000055db4c1bb921 in draw_callback_impl (loop=0x7fc92069c060, ps=0x55db4d4d25b0, revents=8192) at ../src/picom.c:1575
#3  0x000055db4c1bba42 in draw_callback (loop=0x7fc92069c060, w=0x55db4d4d2670, revents=8192) at ../src/picom.c:1601
#4  0x00007fc9206910cb in ev_invoke_pending () from /usr/lib/libev.so.4
#5  0x00007fc920694d10 in ev_run () from /usr/lib/libev.so.4
#6  0x000055db4c1c0d89 in session_run (ps=0x55db4d4d25b0) at ../src/picom.c:2535
#7  0x000055db4c1c10e4 in main (argc=3, argv=0x7fff41670168) at ../src/picom.c:2641

doesn't happen with old glx backend and xrender backends. i believe it's the same issue as #430 and #431 is related.

i'm on arch linux, running xfce 4.18, gtx 1650 w/ nvidia-dkms 525.60.11-1
hmu if you need any additional information, i'll respond asap

[absolutelynothelix]

hm, i have an idea. as discussed in the linked pull request, looks like xfdesktop sets _XROOTPMAP_ID to a pixmap that is associated with a desktop window created by xfdesktop rather than with the root window. killing xfdesktop destroys its desktop window and _XROOTPMAP_ID now… points to a pixmap that doesn’t exist anymore?

update: looks like it.

...
[ 12/26/2022 10:57:55.457 paint_all_new FATAL ERROR ] compose ps->root_image
[ 12/26/2022 10:57:55.457 _gl_compose FATAL ERROR ] composing
[ 12/26/2022 10:57:55.457 _gl_compose FATAL ERROR ] composing
[ 12/26/2022 10:57:55.457 _gl_compose FATAL ERROR ] composing
[ 12/26/2022 10:57:55.458 _gl_compose FATAL ERROR ] composing
[ 12/26/2022 10:57:55.458 _gl_compose FATAL ERROR ] composing
[ 12/26/2022 10:57:55.458 _gl_compose FATAL ERROR ] composing
[ 12/26/2022 10:57:55.459 _gl_compose FATAL ERROR ] composing
[ 12/26/2022 10:57:55.459 _gl_compose FATAL ERROR ] composing
[ 12/26/2022 10:57:55.459 _gl_compose FATAL ERROR ] composing
[ 12/26/2022 10:57:55.459 _gl_compose FATAL ERROR ] composing
# xfdesktop killed
[ 12/26/2022 10:57:55.478 x_log_error WARN ] X error 3 WINDOW request 2 minor 0 serial 7924
[ 12/26/2022 10:57:55.479 x_log_error WARN ] X error 3 WINDOW request 2 minor 0 serial 7925
[ 12/26/2022 10:57:55.479 paint_all_new FATAL ERROR ] compose ps->root_image
Segmentation fault (core dumped)

composing is a print placed in the beggining of the _gl_compose function to see when it's called and composing ps->root_image is placed before this call:

picom/src/backend/backend.c

Lines 189 to 191 in 23a2947

	ps->backend_data->ops->compose(ps->backend_data, ps->root_image,
	(coord_t){0}, NULL, (coord_t){0},
	&reg_paint, &reg_visible);

update 2: root_damaged should free root back pixmap and won’t allocate a new one because there is no root back pixmap anymore. it’s not called then? or there is a kind of race condition? can’t investigate right now, so it’s just my thoughts.

[yshui]

this is related to #984, right? i think when xfdesktop quits it frees the pixmap it put in _XROOTPMAP_ID, which we don't handle.

this is annoying :'( i wonder what's the best way to solve all of these problems. maybe we should copy the root pixmap, into something whose visual, lifetime, etc. are controlled by us.

[absolutelynothelix]

@yshui, looks like xfdesktop is available as a standalone component, so you may want to try and debug it yourself, and compare to other wallpaper setters like feh, hsetroot, nitrogen, etc. maybe you’ll notice something that i didn’t notice.

[absolutelynothelix]

@yshui, some 5am thoughts:

1. obtaining the root back pixmap should be fixed by fix binding the root background pixmap in case of depth mismatch #984
2. obtaining the new root back pixmap (or properly freeing the previous one) may be done by calling the root_damaged function before calling the compose function on ps->root_image if pixmap held by it is not valid

i think this is the most sane solution. we still need to obtain the root back pixmap correctly (1) and notice if it was destroyed (2) to create a copy and keep it up-to-date, so it’s odd complexity.

(1) is done already in #984, i can do (2) too but later. if you consider this approach good and #984 ready you may merge the first part and i’ll do the second part later.

btw, i’m short on time now and not planning to do any pull requests anytime soon so you may want to do a bug fixing release since there was several memory leaks fixed and some new features added (e.g. handling dpms). happy holidays to you and @tryone144 ;)

[absolutelynothelix]

i figured it out and it was an easy fix. everything worked as expected, but ps->root_image was not freed properly (wasn't set to NULL after calling release_image function). that allowed freed ps->root_image to pass the check before calling compose function on it. see #985.