A new call stack with resources exhausted in opj_decompress

[pic4xiu]

Using a maliciously constructed poc, resource exhaustion will occur in the opj_t1_decode_cblks function in the tcd.c file

Expected behavior and actual behavior.

I think the program should discover the hardware resource limit, refuse to parse, or return an error directly.

But the program made the memory utilization rate of the computer rise to 100%, causing the system to crash, and was eventually killed by the system because of excessive memory allocation~

Steps to reproduce the problem.

the poc is here: poc1, poc2

Run: opj_decompress -i poc1 -o te.raw or opj_decompress -i poc2 -o te.raw

We will see that the program continues to allocate memory along this call stack:

pwndbg> bt
#0  0x00007ffff7f75de2 in opj_t1_clbl_decode_processor (user_data=<optimized out>, tls=<optimized out>) at /home/pic/Download/openjpeg/src/lib/openjp2/t1.c:1805
#1  0x00007ffff7f29cb5 in opj_thread_pool_submit_job (tp=0xeffc60, job_fn=0x7ffff7f74fe0 <opj_t1_clbl_decode_processor>, user_data=0x1a63ce30) at /home/pic/Download/openjpeg/src/lib/openjp2/thread.c:835
#2  0x00007ffff7f74cba in opj_t1_decode_cblks (tcd=<optimized out>, pret=<optimized out>, tilec=<optimized out>, tccp=<optimized out>, p_manager=<optimized out>, p_manager_mutex=<optimized out>, check_pterm=<optimized out>) at /home/pic/Download/openjpeg/src/lib/openjp2/t1.c:1942
#3  0x00007ffff7fa09d2 in opj_tcd_t1_decode (p_tcd=<optimized out>, p_manager=<optimized out>) at /home/pic/Download/openjpeg/src/lib/openjp2/tcd.c:2068
#4  opj_tcd_decode_tile (p_tcd=<optimized out>, win_x0=<optimized out>, win_y0=<optimized out>, win_x1=<optimized out>, win_y1=<optimized out>, numcomps_to_decode=<optimized out>, comps_indices=<optimized out>, p_src=<optimized out>, p_max_length=<optimized out>, p_tile_no=<optimized out>, p_cstr_index=<optimized out>, p_manager=<optimized out>) at /home/pic/Download/openjpeg/src/lib/openjp2/tcd.c:1722
#5  0x00007ffff7f51cc7 in opj_j2k_decode_tile (p_j2k=0xefd610, p_tile_index=0, p_data=0x0, p_data_size=<optimized out>, p_stream=<optimized out>, p_manager=0xefd5a8) at /home/pic/Download/openjpeg/src/lib/openjp2/j2k.c:9891
#6  0x00007ffff7f5e538 in opj_j2k_decode_tiles (p_j2k=0xefd610, p_stream=<optimized out>, p_manager=<optimized out>) at /home/pic/Download/openjpeg/src/lib/openjp2/j2k.c:11736
#7  0x00007ffff7f546bd in opj_j2k_exec (p_j2k=0xefd610, p_procedure_list=0xeffbe0, p_stream=0xefc4b0, p_manager=0xefd5a8) at /home/pic/Download/openjpeg/src/lib/openjp2/j2k.c:9035
#8  opj_j2k_decode (p_j2k=0xefd610, p_stream=0xefc4b0, p_image=0xf04720, p_manager=0xefd5a8) at /home/pic/Download/openjpeg/src/lib/openjp2/j2k.c:12039
#9  0x000000000042947f in main (argc=<optimized out>, argc@entry=5, argv=<optimized out>, argv@entry=0x7fffffffe3b8) at /home/pic/Download/openjpeg/src/bin/jp2/opj_decompress.c:1582
#10 0x00007ffff7ac8083 in __libc_start_main (main=0x428e00 <main>, argc=5, argv=0x7fffffffe3b8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe3a8) at ../csu/libc-start.c:308
#11 0x0000000000406ade in _start ()

Operating system

The system should be indifferent. I successfully reproduced it in windows16g, 32g, ubuntu16g, and macos8g. Try to make the memory larger than 8g, at least ensure that there is enough memory to ensure the successful creation of Tag-tree nodes~

openjpeg version

2.5.0

[pedrohc]

CVE-2023-39329 was assigned to this flaw. If you wish to dispute or reject please let me know.