Parallel read for json compressed files when it should not

[yongda-fan]

Describe the bug

When reading a compressed json file, with repartition_file_scans = true (default value), datafusion try to uncompress the file with parallel read. This will cause ArrowError(IoError("invalid gzip header", Custom { kind: InvalidInput, error: "invalid gzip header" }), None) because there is no gzip header in the middle.

To Reproduce

let df = ctx.read_json(
        "C:/path/to/file.gz", 
        NdJsonReadOptions::default()
        .file_compression_type(FileCompressionType::GZIP)
        .file_extension("gz")
        .schema(&s3_user_schema())
    ).await.unwrap();

Expected behavior

the data should read correctly without errors

Additional context

by put a print statement before the JsonOpener, we can see

GetResult { payload: GetResultPayload(File), meta: ObjectMeta { location: Path { raw: "C:/some/path/foo.gz" }, last_modified: 2024-03-06T00:01:02Z, size: 149873338, e_tag: Some("0-612f2a7828b80-8eee2ba"), version: None }, range: 0..4685075 }
GetResult { payload: GetResultPayload(File), meta: ObjectMeta { location: Path { raw: "C:/some/path/foo.gz" }, last_modified: 2024-03-06T00:01:02Z, size: 149873338, e_tag: Some("0-612f2a7828b80-8eee2ba"), version: None }, range: 37468620..42151887 }
GetResult { payload: GetResultPayload(File), meta: ObjectMeta { location: Path { raw: "C:/some/path/foo.gz" }, last_modified: 2024-03-06T00:01:02Z, size: 149873338, e_tag: Some("0-6.......

which suggest it's indeed reading a compressed json file in parallel.

[Lordworms]

take

[Lordworms]

Could you provide more information about the file and the schema? Since I tried a simple gzip file and it work out fine

[alamb]

I agree it is not possible to repartition a json compressed file -- so when the file is compressed we should not repartition

You can probably reproduce this issue with a single json file (maybe repeat the same record over and over again), gzip it, and then try to read it in