Skip to content

Commit

Permalink
[fix](Outfile) fix bug that the fileSize is not correct when outfile …
Browse files Browse the repository at this point in the history
…is completed (#23660)

From pr: #22951

1. fix that the fileSize is not correct when outfile parquet/orc file format is completed.
2. max_file_size of orc file format must be multiple of 64MB:
The function add(RowBatch) of orc::Writer would not flush any pending data to the output stream until the strip data is filled. And the defult of strip size of orc file format is 64MB, so we can not get the correct _written_len from orcOutputStream until 64MB data has been writen.
  • Loading branch information
BePPPower authored Aug 30, 2023
1 parent c9c64ab commit 10dcb4e
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion be/src/vec/runtime/vfile_result_writer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,11 @@ Status VFileResultWriter::_create_new_file_if_exceed_size() {
Status VFileResultWriter::_close_file_writer(bool done, bool only_close) {
if (_vfile_writer) {
_vfile_writer->close();
COUNTER_UPDATE(_written_data_bytes, _current_written_bytes);
// we can not use _current_written_bytes to COUNTER_UPDATE(_written_data_bytes, _current_written_bytes)
// because it will call `write()` function of orc/parquet function in `_vfile_writer->close()`
// and the real written_len will increase
// and _current_written_bytes will less than _vfile_writer->written_len()
COUNTER_UPDATE(_written_data_bytes, _vfile_writer->written_len());
_vfile_writer.reset(nullptr);
} else if (_file_writer_impl) {
_file_writer_impl->close();
Expand Down

0 comments on commit 10dcb4e

Please sign in to comment.