How to get the file size of the parquet file written by ArrowWriter?

[RinChanNOWWW]

ArrowWriter::close will return FileMetaData. FileMetaData only contains sizes of row groups, how can I get the full size of the whole parquet file (including footer, etc)?

FYI, FileWriter in arrow2 can return the file size via method FileWriter::end().

[tustvold]

SerializedFileWriter tracks the number of bytes written, and so could definitely expose this. I guess my question would be why not just obtain this from the W: Write itself?

For example, writer.into_inner().len()?

[jun0315]

pub fn blocks_to_parquet(
    schema: impl AsRef<TableSchema>,
    blocks: Vec<DataBlock>,
    write_buffer: &mut Vec<u8>,
    compression: TableCompression,
) -> Result<(u64, FileMetaData)> {
    let start_pos = write_buffer.len() as u64;
    let data_schema: DataSchema = schema.into();
    let batches: Vec<RecordBatch> = blocks
        .into_iter()
        .map(|block| block.to_record_batch(&data_schema).unwrap())
        .collect();
    assert!(batches.len() > 0);
    let props = WriterProperties::builder()
        .set_compression(compression.into())
        .build();
    let mut writer = ArrowWriter::try_new(write_buffer, batches[0].schema(), Some(props)).unwrap();

    batches.iter().try_for_each(|batch| writer.write(batch))?;

    match writer.close() {
        Ok(meta) => Ok((writer.into_inner()?.len() as u64 - start_pos, meta)),
        Err(cause) => Err(ErrorCode::Internal(format!(
            "write_parquet_file: {:?}",
            cause,
        ))),
    }
}

error[E0382]: use of moved value: `writer`
    |
55  |     let mut writer = ArrowWriter::try_new(write_buffer, batches[0].schema(), Some(props)).unwrap();
    |         ---------- move occurs because `writer` has type `ArrowWriter<&mut Vec<u8>>`, which does not implement the `Copy` trait
...
59  |     match writer.close() {
    |                  ------- `writer` moved due to this method call
60  |         Ok(meta) => Ok((writer.into_inner()?.len() as u64-start_pos, meta)),
    |                         ^^^^^^ value used here after move
    |
note: `ArrowWriter::<W>::close` takes ownership of the receiver `self`, which moves `writer`
   --> /home/qijun/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-37.0.0/src/arrow/arrow_writer/mod.rs:243:22
    |
243 |     pub fn close(mut self) -> Result<crate::format::FileMetaData> {
    |                      ^^^^

For more information about this error, try `rustc --explain E0382`.

@tustvold If we want to obtain both writer's inner and meta, does that seem difficult?

[tustvold]

In your example why can't you do this

pub fn blocks_to_parquet(
    schema: impl AsRef<TableSchema>,
    blocks: Vec<DataBlock>,
    write_buffer: &mut Vec<u8>,
    compression: TableCompression,
) -> Result<(u64, FileMetaData)> {
    let start_pos = write_buffer.len() as u64;
    let data_schema: DataSchema = schema.into();
    let batches: Vec<RecordBatch> = blocks
        .into_iter()
        .map(|block| block.to_record_batch(&data_schema).unwrap())
        .collect();
    assert!(batches.len() > 0);
    let props = WriterProperties::builder()
        .set_compression(compression.into())
        .build();
    let mut writer = ArrowWriter::try_new(write_buffer, batches[0].schema(), Some(props)).unwrap();

    batches.iter().try_for_each(|batch| writer.write(batch))?;

    match writer.close() {
        Ok(meta) => Ok((write_buffer.len() as u64 - start_pos, meta)),
        Err(cause) => Err(ErrorCode::Internal(format!(
            "write_parquet_file: {:?}",
            cause,
        ))),
    }
}

ArrowWriter is explicitly designed to allow passing in a Write with a non-static lifetime, in this case &mut Vec<u8>

[jun0315]

    match writer.close() {
        Ok(meta) => Ok((write_buffer.len() as u64 - start_pos, meta)),
        Err(cause) => Err(ErrorCode::Internal(format!(
            "write_parquet_file: {:?}",
            cause,
        ))),
    }

error[E0382]: borrow of moved value: `write_buffer`
  --> src/query/storages/common/blocks/src/block.rs:60:25
   |
41 |     write_buffer: &mut Vec<u8>,
   |     ------------ move occurs because `write_buffer` has type `&mut Vec<u8>`, which does not implement the `Copy` trait
...
55 |     let mut writer = ArrowWriter::try_new(write_buffer, batches[0].schema(), Some(props)).unwrap();
   |                                           ------------ value moved here
...
60 |         Ok(meta) => Ok((write_buffer.len() as u64 - start_pos, meta)),
   |                         ^^^^^^^^^^^^^^^^^^ value borrowed here after move

For more information about this error, try `rustc --explain E0382`.
warning: `storages-common-blocks` (lib) generated 10 warnings

This method seems to compile errors... @tustvold

[tustvold]

Try let mut writer = ArrowWriter::try_new(&mut *write_buffer, batches[0].schema(), Some(props)).unwrap();

[jun0315]

Thanks a lot