Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[CELEBORN-917][GLUTEN] Record read metric should be compatible with G…
…luten shuffle serde ### What changes were proposed in this pull request? When updating record read metric, we should consider if the input record is `ColumnarBatch`. So if the serde is the Gluten columnar batch, we should use `ColumnarBatch.numRows`. ### Why are the changes needed? Make the shuffle record read metric correct. ### Does this PR introduce _any_ user-facing change? yes, the metrics changed ### How was this patch tested? manually test before: <img width="415" alt="image" src="https://github.com/apache/incubator-celeborn/assets/12025282/221ab814-4b02-4688-80ab-31f21cd900a4"> after: <img width="415" alt="image" src="https://github.com/apache/incubator-celeborn/assets/12025282/1c7257c0-2f30-41c3-9ea8-6bc5cda3de85"> Closes apache#1838 from ulysses-you/gluten. Lead-authored-by: youxiduo <[email protected]> Co-authored-by: Xiduo You <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
- Loading branch information