Unit test regression after 1.5.2 - ZstdIOException: Unknown frame descriptor

[Brutus5000]

After trying to upgrade all of our dependencies this test fails on all newer versions after 1.5.2 with the error com.github.luben.zstd.ZstdIOException: Unknown frame descriptor.

The compressed zstd bytes in the file have been created with a much older version, but I can't tell you which one. This matches our production scenario where we also have these older files.

What be nice if someone can look into it. Thank you.
You can checkout this dependabot branch to run it yourself.

[luben]

Can you check with the versions between 1.5.2 and 1.5.5-11, e.g. can you check with 1.5.4-2 and 1.5.5-1?

[Brutus5000]

I tried git-bisect style with multiple versions >= 1.5.2 until I figure out that 1.5.2 was the last one working. But I can make a full compatibility test with all versions if this helps you.

[luben]

That's strange. We have unit tests that assure that can still decompress payloads compressed with zstd-0.4 version. So breakage in the format is surprising. Can you extract the compressed payload and decompress it with the CLI zstd to see if it fails or not?

[Brutus5000]

Surprisingly I can not, even with older Zstd cli versions.
The file in the test contains a line of json. If I strip it away, the file starts with the magic bytes 0x28b52ffd.
But regardless of the cli version (tried back to 1.3.8 from 2018) fails with unknown header.

But our decompression is working for years with apache commons compress for decompression -.- It detects the magic bytes (I see it in debugging) otherwise it wouldn't call zstd-jni code.

Also the compression code is quite simple but in Rust -> link

[luben]

Can you send me an example compressed message (could be with the JSON header) that shows that problem to try to find the issue?

[Brutus5000]

The file is in the repo, direct download here:
https://github.com/FAForever/faf-java-commons/raw/04e60abcce56958135793ea061126d519a499007/faf-commons-data/src/test/resources/replay/zstd_reference.fafreplay

To get the raw zstd stream I just went for tail -n +2 zstd_reference.fafreplay and piped it either into xxd (to verify the magic bytes) or zstd -d -o raw

Thank you for taking the time. ❤️