bug: join may result in duplicate column names
#21048
Comments
leoliu0
added
bug
MarcoGorelli
added
P-high
|
woooah In [1]: import polars as pl
In [2]: df = pl.DataFrame({'a':[1,2,3,4,5],'b':[6,6,6,6,6]})
...: df = df.join(df,on=['a']).join(df,on=['a'])
In [3]: df
Out[3]:
shape: (5, 4)
┌─────┬─────┬─────────┬─────────┐
│ a ┆ b ┆ b_right ┆ b_right │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════════╪═════════╡
│ 1 ┆ 6 ┆ 6 ┆ 6 │
│ 2 ┆ 6 ┆ 6 ┆ 6 │
│ 3 ┆ 6 ┆ 6 ┆ 6 │
│ 4 ┆ 6 ┆ 6 ┆ 6 │
│ 5 ┆ 6 ┆ 6 ┆ 6 │
└─────┴─────┴─────────┴─────────┘thanks @leoliu0 for the report, this shouldn't happen, and is (I guess) a result of some check being bypassed? |
MarcoGorelli
changed the title
join may result in duplicate column names
>>> df.lazy().join(df.lazy(),on=['a']).join(df.lazy(),on=['a']).collect(new_streaming=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/orlp/.localpython/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 2056, in collect
return wrap_df(ldf.collect(callback))
^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.DuplicateError: column with name 'b_right' already exists
You may want to try:
- renaming the column prior to joining
- using the `suffix` parameter to specify a suffix different to the default one ('_right')New-streaming already detects this 😎 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checks
Reproducible example
When df has duplicated column names, to_pandas failed
gives an error called
Result::unwrap()on anErrvalue: ComputeError(ErrString("RecordBatch requires an equal number of fields and arrays"))It seems joins would create duplicated column names, which is very weird behavior.
It would be great if join would automatically rename additional columns to xxx_right2, xxx_right3 etc?
Log output
Issue description
As described above
Expected behavior
Should complete without error
Installed versions
The text was updated successfully, but these errors were encountered: