Skip to content

Commit

Permalink
Fix DirectBufferedInput::read not recording runtime stats (facebookin…
Browse files Browse the repository at this point in the history
…cubator#10393)

Summary:
Pull Request resolved: facebookincubator#10393

`DirectBufferedInput::read` calls are used to read metadata, but we don't record any runtime statistics during that call, so the runtime statistics (bytes, timing) of reading metadata is missing.

Reviewed By: xiaoxmeng

Differential Revision: D59175484

fbshipit-source-id: 018a2b1dcbe58d207f5c3283b81b2d975dfa88ad
  • Loading branch information
Yuhta authored and facebook-github-bot committed Jul 4, 2024
1 parent ac92be3 commit 92d7ca3
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 1 deletion.
17 changes: 17 additions & 0 deletions velox/dwio/common/DirectBufferedInput.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,23 @@ std::shared_ptr<DirectCoalescedLoad> DirectBufferedInput::coalescedLoad(
});
}

std::unique_ptr<SeekableInputStream> DirectBufferedInput::read(
uint64_t offset,
uint64_t length,
LogType /*logType*/) const {
VELOX_CHECK_LE(offset + length, fileSize_);
return std::make_unique<DirectInputStream>(
const_cast<DirectBufferedInput*>(this),
ioStats_.get(),
Region{offset, length},
input_,
fileNum_,
nullptr,
TrackingId(),
0,
options_.loadQuantum());
}

namespace {
void appendRanges(
memory::Allocation& allocation,
Expand Down
3 changes: 3 additions & 0 deletions velox/dwio/common/DirectBufferedInput.h
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,9 @@ class DirectBufferedInput : public BufferedInput {
std::shared_ptr<DirectCoalescedLoad> coalescedLoad(
const SeekableInputStream* stream);

std::unique_ptr<SeekableInputStream>
read(uint64_t offset, uint64_t length, LogType logType) const override;

folly::Executor* executor() const override {
return executor_;
}
Expand Down
2 changes: 1 addition & 1 deletion velox/exec/tests/TableScanTest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ TEST_F(TableScanTest, directBufferInputRawInputBytes) {
ASSERT_TRUE(it != planStats.end());
auto rawInputBytes = it->second.rawInputBytes;
auto overreadBytes = getTableScanRuntimeStats(task).at("overreadBytes").sum;
ASSERT_EQ(rawInputBytes, 26);
ASSERT_GE(rawInputBytes, 500);
ASSERT_EQ(overreadBytes, 13);
ASSERT_EQ(
getTableScanRuntimeStats(task).at("storageReadBytes").sum,
Expand Down

0 comments on commit 92d7ca3

Please sign in to comment.