misc: Add toString() to OutputBuffer stats #11562

yingsu00 · 2024-11-17T04:38:12Z

Sample output:

[bufferedBytes: 0B, bufferedPages: 0, totalBytesSent: 6.69KB, totalRowsSent: 400, totalPagesSent: 4, averageBufferTimeMs: 0, numTopBuffers: 4] 
  D0: [finished: false, bytesBuffered: 0B, rowsBuffered: 0, pagesBuffered: 0, bytesSent: 1.67KB, rowsSent: 100, pagesSent:1]
  D1: [finished: false, bytesBuffered: 0B, rowsBuffered: 0, pagesBuffered: 0, bytesSent: 1.67KB, rowsSent: 100, pagesSent:1]
  D2: [finished: false, bytesBuffered: 0B, rowsBuffered: 0, pagesBuffered: 0, bytesSent: 1.67KB, rowsSent: 100, pagesSent:1]
  D3: [finished: false, bytesBuffered: 0B, rowsBuffered: 0, pagesBuffered: 0, bytesSent: 1.67KB, rowsSent: 100, pagesSent:1]

netlify · 2024-11-17T04:38:27Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`a005bbb`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/675b5e9d064f7800089bc3f2

xiaoxmeng

@yingsu00 thanks for the improvement. Do you want to have a unit test for this? thanks!

xiaoxmeng · 2024-11-18T07:52:53Z

velox/exec/OutputBuffer.h

+      return fmt::format(
+          "[{}, {}, {}, {}, {}, {}, {}]",
+          finished,
+          bytesBuffered,


use succinctBytes for stats in bytes

velox/exec/OutputBuffer.h

xiaoxmeng · 2024-11-18T07:54:13Z

velox/exec/OutputBuffer.h

+      }
+
+      return fmt::format(
+          "[ bufferedBytes: {}, bufferedPages: {}, "


[bufferedBytes

xiaoxmeng · 2024-11-18T07:54:23Z

velox/exec/OutputBuffer.h

+          "[ bufferedBytes: {}, bufferedPages: {}, "
+          "totalBytesSent: {}, totalRowsSent: {}, totalPagesSent: {}, "
+          "averageBufferTimeMs: {}, numTopBuffers: {}, {}]",
+          bufferedBytes,


xiaoxmeng · 2024-11-18T07:55:13Z

velox/exec/OutputBuffer.h

+      std::string destinationBufferStats;
+      if (!buffersStats.empty()) {
+        for (auto& destinationBufferStat : buffersStats) {
+          destinationBufferStats += destinationBufferStat.toString();


Do we want one line per each destination buffer given there would be hundreds of them for better display? Thanks!

Yes I have updated this PR to one line per destination buffer

yingsu00 · 2024-11-19T09:47:36Z

@xiaoxmeng Thanks for reviewing! I haven't finished the test yet, but here's the output what it looks like:

[bufferedBytes: 0B, bufferedPages: 0, totalBytesSent: 6.69KB, totalRowsSent: 400, totalPagesSent: 4, averageBufferTimeMs: 0, numTopBuffers: 4] 
  D0: [finished: false, bytesBuffered: 0B, rowsBuffered: 0, pagesBuffered: 0, bytesSent: 1.67KB, rowsSent: 100, pagesSent:1]
  D1: [finished: false, bytesBuffered: 0B, rowsBuffered: 0, pagesBuffered: 0, bytesSent: 1.67KB, rowsSent: 100, pagesSent:1]
  D2: [finished: false, bytesBuffered: 0B, rowsBuffered: 0, pagesBuffered: 0, bytesSent: 1.67KB, rowsSent: 100, pagesSent:1]
  D3: [finished: false, bytesBuffered: 0B, rowsBuffered: 0, pagesBuffered: 0, bytesSent: 1.67KB, rowsSent: 100, pagesSent:1]

Let me know if it looks ok.

velox/common/testutil/CMakeLists.txt

yingsu00 · 2024-11-26T03:52:08Z

@assignUser @pedroerp @xiaoxmeng The tests are green now, could you please review again? The current CMakelists.txt looks like this

velox_add_library(velox_test_util ScopedTestTime.cpp TestValue.cpp)
velox_link_libraries(velox_test_util PUBLIC velox_exception)

velox_add_library(velox_test_output_matcher OutputMatcher.cpp)
velox_link_libraries(velox_test_output_matcher PUBLIC Folly::folly GTest::gtest re2::re2)

Thank you!

assignUser · 2024-11-27T10:56:49Z

velox/common/testutil/CMakeLists.txt

+velox_add_library(velox_test_output_matcher OutputMatcher.cpp)
+velox_link_libraries(velox_test_output_matcher PUBLIC Folly::folly GTest::gtest
+                                                      re2::re2)


Like this it will still fail the build due to missing GTest when VELOX_BUILD_TESTING is not set. But because of the refactor you can just move it into the if and that should be fine (it's just used in tests right?)

assignUser

CMake looks good now, thanks!

yingsu00 · 2024-12-04T22:14:42Z

@xiaoxmeng I've addressed all comments and added test. The tests all pass now. Do you want to review again? Many thanks!

xiaoxmeng

@yingsu00 thanks for the update % nit.

xiaoxmeng · 2024-12-12T05:38:34Z

velox/exec/OutputBuffer.h

+    std::string toString() const {
+      return fmt::format(
+          "[finished: {}, bytesBuffered: {}, rowsBuffered: {}, pagesBuffered: {}, bytesSent: {}, rowsSent: {}, pagesSent:{}]",
+          //          "[{}, {}, {}, {}, {}, {}, {}]",


facebook-github-bot · 2024-12-16T22:46:15Z

@kgpai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-12-17T06:56:04Z

@kgpai merged this pull request in e46cb76.

…#11562)" This reverts commit e46cb76.

majetideepak · 2024-12-18T18:12:04Z

CI pass dropped to 37% with this change.

majetideepak · 2024-12-18T18:16:30Z

velox/common/testutil/OutputMatcher.cpp

+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/*


License header is twice here.

…kincubator#11562) Summary: Backing out D67304118: Add toString() to OutputBuffer stats since it breaks certain tests : https://github.com/facebookincubator/velox/actions/runs/12305614280/job/34345592257 . Differential Revision: D67403265

…kincubator#11562) (facebookincubator#11906) Summary: Backing out D67304118: Add toString() to OutputBuffer stats since it breaks certain tests : https://github.com/facebookincubator/velox/actions/runs/12305614280/job/34345592257 . Differential Revision: D67403265

…#11906) Summary: Pull Request resolved: #11906 Backing out D67304118: Add toString() to OutputBuffer stats since it breaks certain tests : https://github.com/facebookincubator/velox/actions/runs/12305614280/job/34345592257 . Reviewed By: bikramSingh91 Differential Revision: D67403265 fbshipit-source-id: 8c0723b2748645e88a68ddf57c2b1cc5dbbb2095

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 17, 2024

yingsu00 requested a review from xiaoxmeng November 17, 2024 04:38

xiaoxmeng reviewed Nov 18, 2024

View reviewed changes

yingsu00 force-pushed the OutputBufferToString branch from 5a4513c to 3df500d Compare November 20, 2024 00:33

yingsu00 requested review from assignUser and majetideepak as code owners November 20, 2024 00:33

yingsu00 force-pushed the OutputBufferToString branch 2 times, most recently from d246b66 to 503e0fb Compare November 20, 2024 01:13

assignUser requested changes Nov 20, 2024

View reviewed changes

velox/common/testutil/CMakeLists.txt Outdated Show resolved Hide resolved

yingsu00 force-pushed the OutputBufferToString branch 5 times, most recently from 97f067b to e9b83f4 Compare November 26, 2024 01:12

assignUser reviewed Nov 27, 2024

View reviewed changes

yingsu00 force-pushed the OutputBufferToString branch from e9b83f4 to a5e42e1 Compare November 28, 2024 00:40

assignUser approved these changes Nov 28, 2024

View reviewed changes

yingsu00 force-pushed the OutputBufferToString branch from a5e42e1 to ee05acf Compare December 3, 2024 22:13

refactor: Extract OutputMatcher class in common

96cb979

xiaoxmeng approved these changes Dec 12, 2024

View reviewed changes

yingsu00 force-pushed the OutputBufferToString branch from ee05acf to a005bbb Compare December 12, 2024 22:07

misc: Add toString() to OutputBuffer stats

a005bbb

xiaoxmeng approved these changes Dec 12, 2024

View reviewed changes

xiaoxmeng added the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Dec 12, 2024

facebook-github-bot closed this in e46cb76 Dec 17, 2024

facebook-github-bot added the Merged label Dec 17, 2024

majetideepak added a commit to majetideepak/velox that referenced this pull request Dec 18, 2024

Revert "misc: Add toString() to OutputBuffer stats (facebookincubator…

d6c7a2f

…#11562)" This reverts commit e46cb76.

majetideepak mentioned this pull request Dec 18, 2024

misc(revert): Add toString() to OutputBuffer stats (#11562) #11904

Closed

majetideepak mentioned this pull request Dec 18, 2024

CI tests are failing with subprocess getting killed #11898

Open

majetideepak reviewed Dec 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

misc: Add toString() to OutputBuffer stats #11562

misc: Add toString() to OutputBuffer stats #11562

yingsu00 commented Nov 17, 2024 •

edited

Loading

netlify bot commented Nov 17, 2024 •

edited

Loading

xiaoxmeng left a comment

xiaoxmeng Nov 18, 2024

xiaoxmeng Nov 18, 2024

xiaoxmeng Nov 18, 2024

xiaoxmeng Nov 18, 2024

yingsu00 Dec 4, 2024

yingsu00 commented Nov 19, 2024

yingsu00 commented Nov 26, 2024

assignUser Nov 27, 2024

assignUser left a comment

yingsu00 commented Dec 4, 2024

xiaoxmeng left a comment

xiaoxmeng Dec 12, 2024

facebook-github-bot commented Dec 16, 2024

facebook-github-bot commented Dec 17, 2024

majetideepak commented Dec 18, 2024

majetideepak Dec 18, 2024

misc: Add toString() to OutputBuffer stats #11562

misc: Add toString() to OutputBuffer stats #11562

Conversation

yingsu00 commented Nov 17, 2024 • edited Loading

netlify bot commented Nov 17, 2024 • edited Loading

✅ Deploy Preview for meta-velox canceled.

xiaoxmeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yingsu00 commented Nov 19, 2024

yingsu00 commented Nov 26, 2024

Choose a reason for hiding this comment

assignUser left a comment

Choose a reason for hiding this comment

yingsu00 commented Dec 4, 2024

xiaoxmeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Dec 16, 2024

facebook-github-bot commented Dec 17, 2024

majetideepak commented Dec 18, 2024

Choose a reason for hiding this comment

yingsu00 commented Nov 17, 2024 •

edited

Loading

netlify bot commented Nov 17, 2024 •

edited

Loading