Skip to content

Commit

Permalink
Fix dataSourceWallNanos calculation (facebookincubator#9505)
Browse files Browse the repository at this point in the history
Summary:

We were not counting `DataSource::addSplit` time before, adding this as
`dataSourceAddSplitWallNanos` and rename `dataSourceWallNanos` to
`dataSourceReadWallNanos`.

Differential Revision: D56201634
  • Loading branch information
Yuhta authored and facebook-github-bot committed Apr 16, 2024
1 parent 1e901c9 commit 8933c4a
Show file tree
Hide file tree
Showing 3 changed files with 14 additions and 5 deletions.
3 changes: 2 additions & 1 deletion velox/exec/OperatorUtils.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,8 @@ void gatherCopy(
// per event. This function returns true for such metrics.
bool shouldAggregateRuntimeMetric(const std::string& name) {
static const folly::F14FastSet<std::string> metricNames{
"dataSourceWallNanos",
"dataSourceAddSplitWallNanos",
"dataSourceReadWallNanos",
"dataSourceLazyWallNanos",
"queuedWallNanos",
"flushTimes"};
Expand Down
10 changes: 8 additions & 2 deletions velox/exec/TableScan.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,13 @@ RowVectorPtr TableScan::getOutput() {
dataSource_->setFromDataSource(std::move(preparedDataSource));
} else {
curStatus_ = "getOutput: adding split";
const auto addSplitStartMicros = getCurrentTimeMicro();
dataSource_->addSplit(connectorSplit);
stats_.wlock()->addRuntimeStat(
"dataSourceAddSplitWallNanos",
RuntimeCounter(
(getCurrentTimeMicro() - addSplitStartMicros) * 1'000,
RuntimeCounter::Unit::kNanos));
}
curStatus_ = "getOutput: updating stats_.numSplits";
++stats_.wlock()->numSplits;
Expand Down Expand Up @@ -218,10 +224,10 @@ RowVectorPtr TableScan::getOutput() {
checkPreload();

{
curStatus_ = "getOutput: updating stats_.dataSourceWallNanos";
curStatus_ = "getOutput: updating stats_.dataSourceReadWallNanos";
auto lockedStats = stats_.wlock();
lockedStats->addRuntimeStat(
"dataSourceWallNanos",
"dataSourceReadWallNanos",
RuntimeCounter(
(getCurrentTimeMicro() - ioTimeStartMicros) * 1'000,
RuntimeCounter::Unit::kNanos));
Expand Down
6 changes: 4 additions & 2 deletions velox/exec/tests/PrintPlanWithStatsTest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,8 @@ TEST_F(PrintPlanWithStatsTest, innerJoinWithTableScan) {
{" runningGetOutputWallNanos\\s+sum: .+, count: 1, min: .+, max: .+"},
{" -- TableScan\\[table: hive_table\\] -> c0:INTEGER, c1:BIGINT"},
{" Input: 2000 rows \\(.+\\), Raw Input: 20480 rows \\(.+\\), Output: 2000 rows \\(.+\\), Cpu time: .+, Blocked wall time: .+, Peak memory: .+, Memory allocations: .+, Threads: 1, Splits: 20"},
{" dataSourceWallNanos [ ]* sum: .+, count: 1, min: .+, max: .+"},
{" dataSourceAddSplitWallNanos[ ]* sum: .+, count: 1, min: .+, max: .+"},
{" dataSourceReadWallNanos[ ]* sum: .+, count: 1, min: .+, max: .+"},
{" dynamicFiltersAccepted[ ]* sum: 1, count: 1, min: 1, max: 1"},
{" flattenStringDictionaryValues [ ]* sum: 0, count: 1, min: 0, max: 0"},
{" ioWaitNanos [ ]* sum: .+, count: .+ min: .+, max: .+"},
Expand Down Expand Up @@ -280,7 +281,8 @@ TEST_F(PrintPlanWithStatsTest, partialAggregateWithTableScan) {
{" runningGetOutputWallNanos\\s+sum: .+, count: 1, min: .+, max: .+"},
{" -- TableScan\\[table: hive_table\\] -> c0:BIGINT, c1:INTEGER, c2:SMALLINT, c3:REAL, c4:DOUBLE, c5:VARCHAR"},
{" Input: 10000 rows \\(.+\\), Output: 10000 rows \\(.+\\), Cpu time: .+, Blocked wall time: .+, Peak memory: .+, Memory allocations: .+, Threads: 1, Splits: 1"},
{" dataSourceWallNanos[ ]* sum: .+, count: 1, min: .+, max: .+"},
{" dataSourceAddSplitWallNanos[ ]* sum: .+, count: 1, min: .+, max: .+"},
{" dataSourceReadWallNanos[ ]* sum: .+, count: 1, min: .+, max: .+"},
{" flattenStringDictionaryValues [ ]* sum: 0, count: 1, min: 0, max: 0"},
{" ioWaitNanos [ ]* sum: .+, count: .+ min: .+, max: .+"},
{" localReadBytes [ ]* sum: 0B, count: 1, min: 0B, max: 0B"},
Expand Down

0 comments on commit 8933c4a

Please sign in to comment.