[Bug fix] Append() in Nullable columns causes nulls_ array to be incorrectly appended to if nested column Append() fails

[moaazassali]

If Append() of the nested column fails, the nulls_ column will be incremented despite nothing being appended.

This causes the following issues:
1- The Size() reported by the Nullable<T> column will be wrong since it depends on the size of nulls_.
2- If you try to access this new "fake" element of the Nullable<T> column, an error will be thrown because false would be appended to nulls_ indicating that a valid non-null element exists. When At() or [] is used to access it, the actual nested column does not have this value, which causes the error to be thrown.
3- This is especially the case for columns like FixedString which could throw.

This pull request resolves that by moving the appending of nulls_ to the end of the Append() operation of the nested column. If the nested column Append() does fail, then the Append() to nulls_ is never called.

[Enmk]

This may also fail, so perhaps we can make ColumnNullable::Append(!value.has_value()); the first operation, wrap second part in try-catch (or with some RAII wrapper), and rollback last value in nulls_ in case of error. (perhaps by introducing protected method ColumnNullable::PopBackNull())

[moaazassali]

Isn't the wanted behavior for Append() to fail if the nested Append() fails too? For example, if we append a 10-character string to ColumnFixedString(5), it will throw a validation error:

string.cpp, line 38:

    if (str.size() > string_size_) {
        throw ValidationError("Expected string of length not greater than "
                                 + std::to_string(string_size_) + " bytes, received "
                                 + std::to_string(str.size()) + " bytes.");
    }

So, my assumption was that Nullable(ColumnFixedString(5)) should behave the same way as the nested column and also throw an error instead of catching the error and failing silently. By moving ColumnNullable::Append(!value.has_value()); to the end, we ensure that we throw an error early, matching the behavior of the nested column without causing unintended side effects.

However, I may have misunderstood the intended behavior. Could you please clarify further?

[Enmk]

yeah, it needs to fail, but also the behavior must be consistent. I.e. nulls_->Size() must be the same as typed_nested_data_->Size(), which is not true with current implementation (and your proposed change).

So I just propose the way to rollback changes of nulls_, since it is the easiest one to do, if there is any exception.

[moaazassali]

Could you please provide a specific example of how it leads to inconsistencies? I can't seem to think of one. This is my understanding:

We have 3 cases:
1- value is not null and nested Append() does not fail:

• we call typed_nested_data_->Append(std::move(*value)); which does not fail
• typed_nested_data_ size = N+1, nulls_ size = N
• we call ColumnNullable::Append(!value.has_value());
• typed_nested_data_ size = N+1, nulls_ size = N+1
• we have consistent sizes

2- value is not null and nested Append fails:

• we call typed_nested_data_->Append(std::move(*value)); which fails
• an error is thrown so we exit the function
• typed_nested_data_ size = N, nulls_ size = N
• we have consistent sizes

3- value is null:

• we call typed_nested_data_->Append(typename ValueType::value_type{}); which cannot fail
• we call ColumnNullable::Append(!value.has_value());
• typed_nested_data_ size = N+1, nulls_ size = N+1
• we have consistent sizes

I have also tested those 3 cases and it passed them:

const auto col = new ColumnNullableT<ColumnFixedString>(5);
col->Append("hello");
CHECK(col->Size() == 1);
CHECK(col->typed_nested_data_->Size() == 1);

CHECK_THROWS(col->Append("world!"));
CHECK(col->Size() == 1);
CHECK(col->typed_nested_data_->Size() == 1);

col->Append(std::nullopt);
CHECK(col->Size() == 2);
CHECK(col->typed_nested_data_->Size() == 2);

[Enmk]

Yeah, but what if ColumnNullable::Append(!value.has_value()); fails?

In that case we have typed_nested_data_->Size() == ColumnNullable::Size() + 1 and hence the ColumnNullableT instance is in inconsistent state.

[Enmk]

Some changes required

[Enmk]

Hi @moaazassali thank you for contribution! Please fix the issue mentioned in the review, and then we can merge it.