Skip to content

Commit

Permalink
Add VectorFuzzer::fuzzInputFlatRow API (facebookincubator#6849)
Browse files Browse the repository at this point in the history
Summary:
It is a good practice to use the fuzzer to generate row vectors as test data
while writing UT codes (e.g. fuzzer.fuzzRow(rowType_). But in some scenarios,
we need to control the generation encoding, say the parquet writer uses arrow
parquet writer, which could not support constant and dictionary encoding
vectors. See discussion here
facebookincubator#6608 (comment).
With this new API, we could use fuzzer like the following,

```C++
VectorFuzzer fuzzer({.vectorSize = vectorSize}, leafPool_.get());
fuzzer.fuzzInputFlatRow(rowType));
```

Pull Request resolved: facebookincubator#6849

Reviewed By: kgpai

Differential Revision: D49830128

Pulled By: mbasmanova

fbshipit-source-id: 17f5f80c9c07577fd2cef32f9e4733155f1a7113
  • Loading branch information
duanmeng authored and facebook-github-bot committed Oct 2, 2023
1 parent d090149 commit e9e323f
Show file tree
Hide file tree
Showing 3 changed files with 41 additions and 0 deletions.
12 changes: 12 additions & 0 deletions velox/vector/fuzzer/VectorFuzzer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,18 @@ RowVectorPtr VectorFuzzer::fuzzInputRow(const RowTypePtr& rowType) {
return fuzzRow(rowType, opts_.vectorSize, false);
}

RowVectorPtr VectorFuzzer::fuzzInputFlatRow(const RowTypePtr& rowType) {
std::vector<VectorPtr> children;
auto size = static_cast<vector_size_t>(opts_.vectorSize);
children.reserve(rowType->size());
for (auto i = 0; i < rowType->size(); ++i) {
children.emplace_back(fuzzFlat(rowType->childAt(i), size));
}

return std::make_shared<RowVector>(
pool_, rowType, nullptr, size, std::move(children));
}

RowVectorPtr VectorFuzzer::fuzzRow(
std::vector<VectorPtr>&& children,
std::vector<std::string> childrenNames,
Expand Down
4 changes: 4 additions & 0 deletions velox/vector/fuzzer/VectorFuzzer.h
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,10 @@ class VectorFuzzer {
// elements.
RowVectorPtr fuzzInputRow(const RowTypePtr& rowType);

/// Same as the function above, but all generated vectors are flat, i.e. no
/// constant or dictionary-encoded vectors at any level.
RowVectorPtr fuzzInputFlatRow(const RowTypePtr& rowType);

// Generates a random type, including maps, vectors, and arrays. maxDepth
// limits the maximum level of nesting for complex types. maxDepth <= 1 means
// no complex types are allowed.
Expand Down
25 changes: 25 additions & 0 deletions velox/vector/fuzzer/tests/VectorFuzzerTest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -688,6 +688,31 @@ TEST_F(VectorFuzzerTest, fuzzRowChildrenToLazy) {
ASSERT_TRUE(wrappedRow->childAt(1)->as<LazyVector>()->isLoaded());
}

TEST_F(VectorFuzzerTest, flatInputRow) {
VectorFuzzer fuzzer({.vectorSize = 10}, pool());
auto vector = fuzzer.fuzzInputFlatRow(
ROW({DOUBLE(), ARRAY(BIGINT()), MAP(BIGINT(), VARCHAR())}));
ASSERT_TRUE(vector->type()->kindEquals(
ROW({DOUBLE(), ARRAY(BIGINT()), MAP(BIGINT(), VARCHAR())})));
ASSERT_EQ(VectorEncoding::Simple::FLAT, vector->childAt(0)->encoding());
ASSERT_EQ(VectorEncoding::Simple::ARRAY, vector->childAt(1)->encoding());
ASSERT_EQ(VectorEncoding::Simple::MAP, vector->childAt(2)->encoding());

// Arrays.
auto elements = vector->childAt(1)->as<ArrayVector>()->elements();
ASSERT_TRUE(elements->type()->kindEquals(BIGINT()));
ASSERT_EQ(VectorEncoding::Simple::FLAT, elements->encoding());

// Maps.
auto mapKeys = vector->childAt(2)->as<MapVector>()->mapKeys();
ASSERT_TRUE(mapKeys->type()->kindEquals(BIGINT()));
ASSERT_EQ(VectorEncoding::Simple::FLAT, mapKeys->encoding());

auto mapValues = vector->childAt(2)->as<MapVector>()->mapValues();
ASSERT_TRUE(mapValues->type()->kindEquals(VARCHAR()));
ASSERT_EQ(VectorEncoding::Simple::FLAT, mapValues->encoding());
}

void VectorFuzzerTest::validateMaxSizes(VectorPtr vector, size_t maxSize) {
if (vector->typeKind() == TypeKind::ARRAY) {
validateMaxSizes(vector->template as<ArrayVector>()->elements(), maxSize);
Expand Down

0 comments on commit e9e323f

Please sign in to comment.