Support reading Iceberg positional delete files #7847

yingsu00 · 2023-12-03T16:53:20Z

In this PR we introduces IcebergSplitReader which supports reading
Iceberg splits with positional delete files

netlify · 2023-12-03T16:53:26Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`ac818fd`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/65cb6be1c9eae40007fb32d9

Yuhta · 2023-12-14T18:54:49Z

@yingsu00 Is this one going to replace #7362?

yingsu00 · 2023-12-16T12:05:49Z

@Yuhta This PR is to replace #7362 . Can you please review? Thank you!

Yuhta

I think overall it's good. Some of the utility is not really needed in delta reader (makeScanSpec, testFilters), but it does not hurt to make standalone functions of them. Can we extract the creation of HiveConnectorUtil.h and refactor of HiveDataSource into a different PR to be merged before the delta readers?

velox/connectors/hive/iceberg/IcebergMetadataColumns.h

velox/connectors/hive/iceberg/PositionalDeleteFileReader.cpp

Yuhta · 2023-12-18T18:11:02Z

velox/connectors/hive/iceberg/PositionalDeleteFileReader.cpp

+  auto deleteTableHandle = std::make_shared<HiveTableHandle>(
+      connectorId, deleteFileName, false, createFilters(), nullptr);
+
+  auto scanSpec = makeScanSpec(


You don't need makeScanSpec, just make createFilters() return the scan spec with filters set as I mentioned here: #7362 (comment)

@Yuhta I tried what you suggested, but it throws exception because of the channel out of bound for the _pos column. Note that we only project out _pos column, but when we add it through addAllChildFields(), it assigns the channel 1 for _pos column. Adding more lines to fix this may not be as neat as just calling makeScanSpec so I converted it back to makeScanSpec().

Yuhta · 2023-12-18T18:16:48Z

velox/connectors/hive/iceberg/PositionalDeleteFileReader.h

+#include "velox/connectors/hive/FileHandle.h"
+
+namespace facebook::velox {
+class BaseVector;


These forward declaration are overkill, since this is not a public interface, we can just include these dependencies in the header

yingsu00 · 2023-12-19T14:31:33Z

I think overall it's good. Some of the utility is not really needed in delta reader (makeScanSpec, testFilters), but it does not hurt to make standalone functions of them. Can we extract the creation of HiveConnectorUtil.h and refactor of HiveDataSource into a different PR to be merged before the delta readers?

Sure. Created #8104

yingsu00

@Yuhta Addressed all comments, can you please review again? Many thanks!

yingsu00 · 2023-12-20T14:07:06Z

velox/connectors/hive/iceberg/PositionalDeleteFileReader.cpp

+  auto deleteTableHandle = std::make_shared<HiveTableHandle>(
+      connectorId, deleteFileName, false, createFilters(), nullptr);
+
+  auto scanSpec = makeScanSpec(


@Yuhta I tried what you suggested, but it throws exception because of the channel out of bound for the _pos column. Note that we only project out _pos column, but when we add it through addAllChildFields(), it assigns the channel 1 for _pos column. Adding more lines to fix this may not be as neat as just calling makeScanSpec so I converted it back to makeScanSpec().

velox/connectors/hive/iceberg/PositionalDeleteFileReader.cpp

yingsu00 · 2024-02-08T16:55:45Z

The test failure is because of #8708

facebook-github-bot · 2024-02-08T22:52:49Z

@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

yingsu00 · 2024-02-09T12:54:24Z

@Yuhta All tests passed, will you please review again? Thanks!

Yuhta · 2024-02-09T16:42:39Z

@yingsu00 Some test failing under ASAN:

Note: Google Test filter = HiveIcebergTest.positionalDeletesMultipleBaseFiles
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from HiveIcebergTest
[ RUN      ] HiveIcebergTest.positionalDeletesMultipleBaseFiles

[W0209](https://www.internalfb.com/intern/workorder/0209/) 08:41:12.979076 256532 MemoryArbitrator.cpp:83] Query memory capacity[8.00GB] is set for NOOP arbitrator which has no capacity enforcement
I0209 08:41:13.349335 256532 HiveConnector.cpp:69] Hive connector test-hive created with maximum of 20000 cached file handles.
I0209 08:41:13.378745 256532 Compression.cpp:628] Initialized zstd compressor with compression level 7
I0209 08:41:13.462200 256532 Writer.cpp:546] Stripe 0: Flush overhead = 111872, data length = 43825, pre flush mem = 984384, post flush mem = 768448. Closing = true
I0209 08:41:13.462814 256532 FileSink.cpp:123] closing file: /tmp/velox_test_Si8NnM,  total size: 43.38KB
=================================================================
==256532==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000089910 at pc 0x0000005b9c92 bp 0x7ffe69215070 sp 0x7ffe69214830
READ of size 29 at 0x604000089910 thread T0
SCARINESS: 54 (multi-byte-read-heap-use-after-free)
    #0 0x5b9c91 in __asan_memcpy (/data/sandcastle/boxes/eden-trunk-hg-full-fbsource/buck-out/v2/gen/fbcode/29760d6803037a85/velox/connectors/hive/iceberg/tests/__tests__/tests+0x5b9c91)
    #1 0x7f1e15dbb4bc in facebook::velox::FlatVector<facebook::velox::StringView>::set(int, facebook::velox::StringView) fbcode/velox/vector/FlatVector.cpp:118
    #2 0x4e58b4 in std::shared_ptr<facebook::velox::FlatVector<facebook::velox::test::EvalTypeHelper<facebook::velox::StringView>::Type>> facebook::velox::test::VectorMaker::flatVector<facebook::velox::StringView>(int, std::function<facebook::velox::StringView (int)>, std::function<bool (int)>, std::shared_ptr<facebook::velox::Type const> const&) fbcode/velox/vector/tests/utils/VectorMaker.h:133
    #3 0x33dcb3 in facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long) fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:160
    #4 0x332908 in facebook::velox::connector::hive::iceberg::HiveIcebergTest::assertPositionalDeletes(std::vector<long, std::allocator<long>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool) fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:56
    #5 0x33199b in facebook::velox::connector::hive::iceberg::HiveIcebergTest::assertPositionalDeletes(std::vector<long, std::allocator<long>> const&, bool) fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:40
    #6 0x325c21 in facebook::velox::connector::hive::iceberg::HiveIcebergTest_positionalDeletesMultipleBaseFiles_Test::TestBody() fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:265
    #7 0x7f1e1920589e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) fbsource/src/gtest.cc:2675
    #8 0x7f1e19205124 in testing::Test::Run() fbsource/src/gtest.cc:2692
    #9 0x7f1e1920ad5f in testing::TestInfo::Run() fbsource/src/gtest.cc:2841
    #10 0x7f1e19212d16 in testing::TestSuite::Run() fbsource/src/gtest.cc:3020
    #11 0x7f1e1924e67b in testing::internal::UnitTestImpl::RunAllTests() fbsource/src/gtest.cc:5925
    #12 0x7f1e1924d6db in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) fbsource/src/gtest.cc:2675
    #13 0x7f1e1924cc19 in testing::UnitTest::Run() fbsource/src/gtest.cc:5489
    #14 0x7f1e190de9a0 in RUN_ALL_TESTS() fbsource/gtest/gtest.h:2317
    #15 0x7f1e190de62f in main fbcode/common/gtest/LightMain.cpp:20
    #16 0x7f1e1182c656 in __libc_start_call_main /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #17 0x7f1e1182c717 in [__libc_start_main@GLIBC_2.2.5](mailto:__libc_start_main@GLIBC_2.2.5) /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../csu/libc-start.c:409:3
    #18 0x31cd30 in _start /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86_64/start.S:116

0x604000089910 is located 0 bytes inside of 45-byte region [0x604000089910,0x60400008993d)
freed by thread T0 here:
    #0 0x72b647 in operator delete(void*, unsigned long) (/data/sandcastle/boxes/eden-trunk-hg-full-fbsource/buck-out/v2/gen/fbcode/29760d6803037a85/velox/connectors/hive/iceberg/tests/__tests__/tests+0x72b647)
    #1 0x327d14 in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/ext/new_allocator.h:146
    #2 0x327cdc in deallocate fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/allocator.h:211
    #3 0x327cdc in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/alloc_traits.h:492
    #4 0x327c4c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/basic_string.h:245
    #5 0x327686 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/basic_string.h:240
    #6 0x326a38 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/basic_string.h:671
    #7 0x4e7cd2 in auto facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)::operator()<int>(auto) const fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:164
    #8 0x4e79c0 in auto std::__invoke_impl<facebook::velox::StringView, facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int>(std::__invoke_other, facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/invoke.h:61
    #9 0x4e7770 in std::enable_if<is_invocable_r_v<auto, facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int>, auto>::type std::__invoke_r<facebook::velox::StringView, facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int>(facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/invoke.h:114
    #10 0x4e73b8 in std::_Function_handler<facebook::velox::StringView (int), facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)>::_M_invoke(std::_Any_data const&, int&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_function.h:291
    #11 0x4e7009 in std::function<facebook::velox::StringView (int)>::operator()(int) const fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_function.h:560
    #12 0x4e57ac in std::shared_ptr<facebook::velox::FlatVector<facebook::velox::test::EvalTypeHelper<facebook::velox::StringView>::Type>> facebook::velox::test::VectorMaker::flatVector<facebook::velox::StringView>(int, std::function<facebook::velox::StringView (int)>, std::function<bool (int)>, std::shared_ptr<facebook::velox::Type const> const&) fbcode/velox/vector/tests/utils/VectorMaker.h:132
    #13 0x33dcb3 in facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long) fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:160
    #14 0x332908 in facebook::velox::connector::hive::iceberg::HiveIcebergTest::assertPositionalDeletes(std::vector<long, std::allocator<long>> const&,
...
...
...
2.34/src/glibc-2.34/csu/../csu/libc-start.c:409:3
    #28 0x31cd30 in _start /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86_64/start.S:116

previously allocated by thread T0 here:
    #0 0x72a827 in operator new(unsigned long) (/data/sandcastle/boxes/eden-trunk-hg-full-fbsource/buck-out/v2/gen/fbcode/29760d6803037a85/velox/connectors/hive/iceberg/tests/__tests__/tests+0x72a827)
    #1 0x327a40 in __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/ext/new_allocator.h:128
    #2 0x327950 in allocate fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/allocator.h:197
    #3 0x327950 in std::allocator_traits<std::allocator<char>>::allocate(std::allocator<char>&, unsigned long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/alloc_traits.h:460
    #4 0x327539 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_create(unsigned long&, unsigned long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/basic_string.tcc:153
    #5 0x335364 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/basic_string.tcc:307
    #6 0x3350ba in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_append(char const*, unsigned long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/basic_string.tcc:395
    #7 0x334ded in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::append(char const*) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/basic_string.h:1259
    #8 0x337a88 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> std::operator+<char, std::char_traits<char>, std::allocator<char>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, char const*) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/basic_string.h:6135
    #9 0x4e7cae in auto facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)::operator()<int>(auto) const fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:162
    #10 0x4e79c0 in auto std::__invoke_impl<facebook::velox::StringView, facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int>(std::__invoke_other, facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/invoke.h:61
    #11 0x4e7770 in std::enable_if<is_invocable_r_v<auto, facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int>, auto>::type std::__invoke_r<facebook::velox::StringView, facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int>(facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)&, int&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/invoke.h:114
    #12 0x4e73b8 in std::_Function_handler<facebook::velox::StringView (int), facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long)::'lambda'(auto)>::_M_invoke(std::_Any_data const&, int&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_function.h:291
    #13 0x4e7009 in std::function<facebook::velox::StringView (int)>::operator()(int) const fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_function.h:560
    #14 0x4e57ac in std::shared_ptr<facebook::velox::FlatVector<facebook::velox::test::EvalTypeHelper<facebook::velox::StringView>::Type>> facebook::velox::test::VectorMaker::flatVector<facebook::velox::StringView>(int, std::function<facebook::velox::StringView (int)>, std::function<bool (int)>, std::shared_ptr<facebook::velox::Type const> const&) fbcode/velox/vector/tests/utils/VectorMaker.h:132
    #15 0x33dcb3 in facebook::velox::connector::hive::iceberg::HiveIcebergTest::writePositionDeleteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::vector<long, std::allocator<long>> const&, long, long) fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:160
    #16 0x332908 in facebook::velox::connector::hive::iceberg::HiveIcebergTest::assertPositionalDeletes(std::vector<long, std::allocator<long>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool) fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:56
    #17 0x33199b in facebook::velox::connector::hive::iceberg::HiveIcebergTest::assertPositionalDeletes(std::vector<long, std::allocator<long>> const&, bool) fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:40
    #18 0x325c21 in facebook::velox::connector::hive::iceberg::HiveIcebergTest_positionalDeletesMultipleBaseFiles_Test::TestBody() fbcode/velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp:265
    #19 0x7f1e1920589e in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) fbsource/src/gtest.cc:2675
    #20 0x7f1e19205124 in testing::Test::Run() fbsource/src/gtest.cc:2692
    #21 0x7f1e1920ad5f in testing::TestInfo::Run() fbsource/src/gtest.cc:2841
    #22 0x7f1e19212d16 in testing::TestSuite::Run() fbsource/src/gtest.cc:3020
    #23 0x7f1e1924e67b in testing::internal::UnitTestImpl::RunAllTests() fbsource/src/gtest.cc:5925
    #24 0x7f1e1924d6db in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) fbsource/src/gtest.cc:2675
    #25 0x7f1e1924cc19 in testing::UnitTest::Run() fbsource/src/gtest.cc:5489
    #26 0x7f1e190de9a0 in RUN_ALL_TESTS() fbsource/gtest/gtest.h:2317
    #27 0x7f1e190de62f in main fbcode/common/gtest/LightMain.cpp:20
    #28 0x7f1e1182c656 in __libc_start_call_main /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #29 0x7f1e1182c717 in [__libc_start_main@GLIBC_2.2.5](mailto:__libc_start_main@GLIBC_2.2.5) /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../csu/libc-start.c:409:3
    #30 0x31cd30 in _start /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86_64/start.S:116

SUMMARY: AddressSanitizer: heap-use-after-free (/data/sandcastle/boxes/eden-trunk-hg-full-fbsource/buck-out/v2/gen/fbcode/29760d6803037a85/velox/connectors/hive/iceberg/tests/__tests__/tests+0x5b9c91) in __asan_memcpy
Shadow bytes around the buggy address:
  0x0c08800092d0: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 fa
  0x0c08800092e0: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 fa
  0x0c08800092f0: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 fa
  0x0c0880009300: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
  0x0c0880009310: fa fa 00 00 00 00 00 00 fa fa fd fd fd fd fd fa
=>0x0c0880009320: fa fa[fd]fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c0880009330: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x0c0880009340: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
  0x0c0880009350: fa fa fd fd fd fd fd fd fa fa 00 00 00 00 00 00
  0x0c0880009360: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c0880009370: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==256532==ABORTING

Test was never completed. The test process might have crashed.
(Truncated to 18976 bytes by Tpx; full output here: everstore://GOBUfhaOYCG2WkwBAPENID2yrNNabswMAAAz:Full_Test_Output: - Alternatively, you can use `bunnylol everpaste GOBUfhaOYCG2WkwBAPENID2yrNNabswMAAAz`)

In this commit we introduces IcebergSplitReader which supports reading Iceberg splits with positional delete files. Add subfield filter for the delete file path column

yingsu00 · 2024-02-13T16:39:15Z

@Yuhta Thanks for the logs. I have fixed the problem, can you please try importing it again? Thanks!

facebook-github-bot · 2024-02-13T17:01:51Z

@Yuhta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-02-14T15:28:50Z

@Yuhta merged this pull request in afe819e.

conbench-facebook · 2024-02-14T16:14:03Z

Conbench analyzed the 1 benchmark run on commit afe819e8.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

…end (#4779) Velox add iceberg mor table read support in facebookincubator/velox#7847. This PR supports read iceberg mor table for Velox backend.

…end (apache#4779) Velox add iceberg mor table read support in facebookincubator/velox#7847. This PR supports read iceberg mor table for Velox backend.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 3, 2023

yingsu00 force-pushed the iceberg7RowReader branch 4 times, most recently from e65a2e1 to 12b6e7a Compare December 4, 2023 14:42

yingsu00 changed the title ~~Support reading Iceberg delete files~~ Support reading Iceberg positional delete files Dec 4, 2023

yingsu00 force-pushed the iceberg7RowReader branch 2 times, most recently from 124d5de to 75c4d21 Compare December 5, 2023 05:52

yingsu00 marked this pull request as ready for review December 6, 2023 06:33

yingsu00 requested a review from Yuhta December 6, 2023 06:33

yingsu00 force-pushed the iceberg7RowReader branch 2 times, most recently from 9f67b80 to e1111b9 Compare December 12, 2023 13:30

yingsu00 force-pushed the iceberg7RowReader branch 6 times, most recently from 100dec9 to 276994f Compare December 14, 2023 13:20

yingsu00 mentioned this pull request Dec 16, 2023

Introducing IcebergSplitReader #7362

Closed

yingsu00 force-pushed the iceberg7RowReader branch from 276994f to 6cc07d5 Compare December 16, 2023 12:23

Yuhta reviewed Dec 18, 2023

View reviewed changes

yingsu00 force-pushed the iceberg7RowReader branch from 6cc07d5 to aad4c9b Compare December 20, 2023 14:08

yingsu00 commented Dec 20, 2023

View reviewed changes

yingsu00 force-pushed the iceberg7RowReader branch 2 times, most recently from 5f49d04 to 633da9a Compare December 21, 2023 09:27

yingsu00 force-pushed the iceberg7RowReader branch 2 times, most recently from d0732ab to 9979b00 Compare February 8, 2024 15:26

Yuhta reviewed Feb 8, 2024

View reviewed changes

velox/connectors/hive/iceberg/PositionalDeleteFileReader.cpp Outdated Show resolved Hide resolved

yingsu00 force-pushed the iceberg7RowReader branch from 9979b00 to cbb8efe Compare February 8, 2024 16:47

yingsu00 force-pushed the iceberg7RowReader branch 2 times, most recently from 58eee69 to afa6151 Compare February 9, 2024 08:42

yingsu00 force-pushed the iceberg7RowReader branch 3 times, most recently from 4d5201e to f4331a1 Compare February 13, 2024 13:14

Introducing IcebergSplitReader

ac818fd

In this commit we introduces IcebergSplitReader which supports reading Iceberg splits with positional delete files. Add subfield filter for the delete file path column

yingsu00 force-pushed the iceberg7RowReader branch from f4331a1 to ac818fd Compare February 13, 2024 13:17

facebook-github-bot closed this in afe819e Feb 14, 2024

facebook-github-bot added the Merged label Feb 14, 2024

FelixYBW mentioned this pull request Feb 15, 2024

[VL] Daily Update Velox Version (2024-02-15) apache/incubator-gluten#4697

Merged

czentgr mentioned this pull request Feb 20, 2024

gtest library dependency leak into non-test velox libraries with hive connector #8807

Closed

imjalpreet mentioned this pull request Feb 21, 2024

[native] Support reading Iceberg V2 tables with Position deletes prestodb/presto#21980

Merged

6 tasks

liujiayi771 mentioned this pull request Feb 26, 2024

[GLUTEN-3378][VL] Feat: Support read iceberg mor table for Velox backend apache/incubator-gluten#4779

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support reading Iceberg positional delete files #7847

Support reading Iceberg positional delete files #7847

yingsu00 commented Dec 3, 2023 •

edited

Loading

netlify bot commented Dec 3, 2023 •

edited

Loading

Yuhta commented Dec 14, 2023

yingsu00 commented Dec 16, 2023

Yuhta left a comment

Yuhta Dec 18, 2023

yingsu00 Dec 20, 2023

Yuhta Dec 18, 2023

yingsu00 commented Dec 19, 2023

yingsu00 left a comment

yingsu00 Dec 20, 2023

yingsu00 commented Feb 8, 2024

facebook-github-bot commented Feb 8, 2024

yingsu00 commented Feb 9, 2024

Yuhta commented Feb 9, 2024

yingsu00 commented Feb 13, 2024

facebook-github-bot commented Feb 13, 2024

facebook-github-bot commented Feb 14, 2024

conbench-facebook bot commented Feb 14, 2024

Support reading Iceberg positional delete files #7847

Support reading Iceberg positional delete files #7847

Conversation

yingsu00 commented Dec 3, 2023 • edited Loading

netlify bot commented Dec 3, 2023 • edited Loading

✅ Deploy Preview for meta-velox canceled.

Yuhta commented Dec 14, 2023

yingsu00 commented Dec 16, 2023

Yuhta left a comment

Choose a reason for hiding this comment

Yuhta Dec 18, 2023

Choose a reason for hiding this comment

yingsu00 Dec 20, 2023

Choose a reason for hiding this comment

Yuhta Dec 18, 2023

Choose a reason for hiding this comment

yingsu00 commented Dec 19, 2023

yingsu00 left a comment

Choose a reason for hiding this comment

yingsu00 Dec 20, 2023

Choose a reason for hiding this comment

yingsu00 commented Feb 8, 2024

facebook-github-bot commented Feb 8, 2024

yingsu00 commented Feb 9, 2024

Yuhta commented Feb 9, 2024

yingsu00 commented Feb 13, 2024

facebook-github-bot commented Feb 13, 2024

facebook-github-bot commented Feb 14, 2024

conbench-facebook bot commented Feb 14, 2024

yingsu00 commented Dec 3, 2023 •

edited

Loading

netlify bot commented Dec 3, 2023 •

edited

Loading