BUG: «Src size is incorrect» on correct file

[danila-schelkov]

Hello! I caught a problem while using your library.

I uploaded the problematic file named buildings.zip, because GitHub forbid to load file with extension .zstd. It is just a compressed binary file, not an archive. Python built-in module can decompress it, as far as original C library. So, the problem occured only in zstd-jni. I used latest version of the library (v1.5.7-3) from maven repo. Windows and macOS both have the same problem with this file.

Caught Exception:

com.github.luben.zstd.ZstdException: Src size is incorrect
	at com.github.luben.zstd.ZstdDecompressCtx.decompressByteArray(ZstdDecompressCtx.java:224)
	at com.github.luben.zstd.ZstdDecompressCtx.decompress(ZstdDecompressCtx.java:356)
	at com.github.luben.zstd.ZstdDecompressCtx.decompress(ZstdDecompressCtx.java:373)
	at com.github.luben.zstd.Zstd.decompress(Zstd.java:1323)
	at dev.donutquine.swf.file.compression.Zstandard.decompress(Zstandard.java:24)
	at dev.donutquine.swf.file.compression.Zstandard.decompress(Zstandard.java:10)

package dev.donutquine.swf.file.compression;

import com.github.luben.zstd.Zstd;

public final class Zstandard {
    private Zstandard() {
    }

    public static byte[] decompress(byte[] compressedData, int offset) {
        return decompress(compressedData, offset, compressedData.length - offset);
    }

    public static byte[] decompress(byte[] compressedData, int offset, int length) {
        int decompressedSize = (int) Zstd.getFrameContentSize(compressedData, offset, length);

        byte[] zstdContent;
        if (offset > 0 || length != compressedData.length - offset) {
            zstdContent = new byte[length];
            System.arraycopy(compressedData, offset, zstdContent, 0, zstdContent.length);
        } else {
            zstdContent = compressedData;
        }

        return Zstd.decompress(
            zstdContent,
            decompressedSize
        );
    }

    public static byte[] compress(byte[] data) {
        return Zstd.compress(data);
    }
}

buildings.zip

[luben]

Is the file correct? when I try to decompress it with zstd, I get an error:

$ zstd -d buildings.zst
buildings.zst        : 0 B...     zstd: buildings.zst: unsupported format

I have just renamed the file.

[danila-schelkov]

The file is correct, absolutely. I used ImHex zstd pattern to validate the file and it did find no problems when parsing it.

Then I decompressed it with Python 3.12.2 and Python module named zstandard, which is basically a bindings for Facebook's zstd library.

import zstandard

with open("buildings.zst", "rb") as file:
     compressed = file.read()

decompressor = zstandard.ZstdDecompressor()
decompressed = decompressor.decompress(compressed)

print(len(compressed))  # 2564608
print(len(decompressed))  # 19004092

I don't know for sure if zstd CLI is able to decompress the file, but my colleague said that he was able to do it using the C library.

[luben]

Can you try with the CLI?

[luben]

Ok, looked a little bit closer. Your file has a complete Zstd frame (19004092 bytes decompressed) followed by some garbage that the Zstd does not understand. The decompress method (that just delegates to ZSTD_decompressDCtx) assumes that the whole input buffer has to be decompressed, so it throws exception when it hits that unknown frame.

Here is short reproduction (in the scala console):

import java.io._
import com.github.luben.zstd.ZstdInputStream

val file = new File("buildings.zst")
val fis  = new FileInputStream(file)
val zis  = new ZstdInputStream(fis)

val buff = Array.fill[Byte](8192)(0)
var len = 0
var read = 0
do {
  read = zis.read(buff)
  len += read
  println(s"$read  $len");
} while (read > 0)
....
8192  18980864 0
8192  18989056 0
8192  18997248 0
6844  19004092 0
com.github.luben.zstd.ZstdIOException: Unknown frame descriptor
  at com.github.luben.zstd.ZstdInputStreamNoFinalizer.readInternal(ZstdInputStreamNoFinalizer.java:184)

So if you use streaming decompression, you are able to get all the data out.

I am not sure why Python wrapper hides that.

[danila-schelkov]

Oh okay, now I see, the file contains 5 extra bytes and zstd library tries to parse it as next ZSTD frame. As a solution I can use your ZstdInputStream and stop executing as soon as read buffer hits Zstd.getFrameContentSize(data) size.

Thank you very much!

Could you please expose ZSTD_findFrameCompressedSize to Java in the next release, so I don't have to use streams? It seems to be very useful.

size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize);
src should point to the start of a ZSTD frame or skippable frame.
srcSize must be >= first frame size
@return : the compressed size of the first frame starting at src,
suitable to pass as srcSize to ZSTD_decompress or similar,
or an error code if input is invalid

From https://facebook.github.io/zstd/zstd_manual.html

[danila-schelkov]

Made a pull request with exposed method and decompressFrame method in the Zstd class

[luben]

As a solution I can use your ZstdInputStream and stop executing as soon as read buffer hits Zstd.getFrameContentSize(data) size

Yes, that should work. Looks like Python-zstandard uses this approach internally to implement decompress and not the one shot ZSTD_decompress

[luben]

closing as we have now released new version with new methods to deal with that case