Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Go] type mismatch building record with fixed-length non-nullable array #43012

Closed
wkalt opened this issue Jun 23, 2024 · 1 comment
Closed

[Go] type mismatch building record with fixed-length non-nullable array #43012

wkalt opened this issue Jun 23, 2024 · 1 comment
Assignees
Labels
Component: Go Type: usage Issue is a user question

Comments

@wkalt
Copy link
Contributor

wkalt commented Jun 23, 2024

Describe the usage question you have. Please include as many useful details as possible.

Hello,
Thanks for the library. I'm wondering what is wrong with the code below. I am struggling to build a record that includes a non-nullable fixed-size list.

If I change the call to FixedSizeListOfNonNullable to ListOfNonNullable my panic goes away, so being fixed-size seems relevant.

I'm not sure if this is a usage error or a bug, I'm just getting started with arrow.

func TestExample(t *testing.T) {
	pool := memory.NewGoAllocator()

	schema := arrow.NewSchema([]arrow.Field{
		{Name: "data", Type: arrow.FixedSizeListOfNonNullable(1, arrow.PrimitiveTypes.Int32)},
	}, nil)

	builder := array.NewRecordBuilder(pool, schema)

	lb := builder.Field(0).(*array.FixedSizeListBuilder)

	// append data to the value builder
	vb := lb.ValueBuilder().(*array.Int32Builder)
	vb.Append(10)

	// why is the array that comes out of here nullable?
	//_ = lb.NewArray()

	// panic here
	_ = builder.NewRecord()
	// row/array: column "data" type mismatch: got=struct<foo: fixed_size_list<item: int32, nullable>[1]>, want=struct<foo: fixed_size_list<item: int32>[1]> [recovered]
}

Component(s)

Go

@wkalt wkalt added the Type: usage Issue is a user question label Jun 23, 2024
@wkalt wkalt changed the title [Go] type mismatch building record with fixed-length array [Go] type mismatch building record with fixed-length non-nullable array Jun 23, 2024
@wkalt
Copy link
Contributor Author

wkalt commented Jun 23, 2024

It seems to me that any nullability context is getting dropped for fixed-length arrays here:

https://github.com/apache/arrow/blob/main/go/arrow/array/builder.go#L358,L360

Regular lists don't have this problem because they pass the entire field structure down to the list builder constructor. Fixed size arrays need the same "WithField" treatment introduced in 8bff490ce37f0bb427ad865fae6c1d29c18d9fa7.

wkalt added a commit to wkalt/arrow that referenced this issue Jun 24, 2024
…ed-size lists

Prior to this commit, the builder constructor for fixed-length lists
would lose information as to whether the list elements were marked
not-nullable, which meant the data type available from the builder would
always reflect "nullable". When NewRecord was invoked, this type would
get checked against the type present in the original schema. If the
schema requested a not-nullable fixed-size array, this check would
always fail and cause a panic.

This commit introduces an alternative builder constructor
"NewFixedSizeListBuilderWithField" for fixed-size lists that takes the
entire field context, similar to what already exists for lists and large
lists.
zeroshade pushed a commit that referenced this issue Jun 25, 2024
…e lists (#43013)

Prior to this commit, the builder constructor for fixed-length lists
would lose information as to whether the list elements were marked
not-nullable, which meant the data type available from the builder would
always reflect "nullable". When NewRecord was invoked, this type would
get checked against the type present in the original schema. If the
schema requested a not-nullable fixed-size array, this check would
always fail and cause a panic.

This commit introduces an alternative builder constructor
"NewFixedSizeListBuilderWithField" for fixed-size lists that takes the
entire field context, similar to what already exists for lists and large
lists.


**This PR contains a "Critical Fix".**
* GitHub Issue: #43012
@wkalt wkalt closed this as completed Jun 26, 2024
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Jul 9, 2024
…ed-size lists (apache#43013)

Prior to this commit, the builder constructor for fixed-length lists
would lose information as to whether the list elements were marked
not-nullable, which meant the data type available from the builder would
always reflect "nullable". When NewRecord was invoked, this type would
get checked against the type present in the original schema. If the
schema requested a not-nullable fixed-size array, this check would
always fail and cause a panic.

This commit introduces an alternative builder constructor
"NewFixedSizeListBuilderWithField" for fixed-size lists that takes the
entire field context, similar to what already exists for lists and large
lists.


**This PR contains a "Critical Fix".**
* GitHub Issue: apache#43012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Go Type: usage Issue is a user question
Projects
None yet
Development

No branches or pull requests

1 participant