Add support for reading CSV files with comments

[bbannier]

This PR adds support for parsing CSV files containing comment lines.

Closes #10262.

[bbannier]

This is currently a sketch for a possible implementation for #10262. The approach taken push interpretation of comment lines into arrow-csv apache/arrow-rs#5759 adding support for that; the task here is then to plug a datafusion comment config setting through to arrow-csv.

If this is a viable solution it would require a bump of at least the arrow-csv dependency in datafusion to a version containing support for comments. To at least explore that I prefixed the actual implementation patch here with two patches performing that bump. It appears that a bump to the master version of arrow-csv (or something else from the collection of crates in https://github.com/apache/arrow-rs) requires changes to datafusion; I attempted to perform that bump, but currently there are still some remaining issues,

Error: ResourcesExhausted("Failed to allocate additional 2208 bytes for GroupedHashAggregateStream[0] with 348 bytes already allocated - maximum available is 1600")

---- aggregates::tests::aggregate_source_with_yielding_with_spill stdout ----
Error: ResourcesExhausted("Failed to allocate additional 2208 bytes for GroupedHashAggregateStream[0] with 348 bytes already allocated - maximum available is 1600")

---- aggregates::tests::run_first_last_multi_partitions stdout ----
Error: ResourcesExhausted("Failed to allocate additional 3704 bytes for GroupedHashAggregateStream[0] with 437 bytes already allocated - maximum available is 3200")

@alamb, would you be open to shepherding this PR and apache/arrow-rs#5759, or alternatively could help identify someone who could?

[alamb]

If this is a viable solution it would require a bump of at least the arrow-csv dependency in datafusion to a version containing support for comments.

Yes. FWIW DataFusion typically upgrades to the latest arrow-rs (including arrow-csv) dependency so while extra time would be needed no extra work would be

[bbannier]

This is now rebased on main which recently bumped arrow.

[alamb]

Thank you very much for this contribution @bbannier -- this code looks great. The only thing I think this PR now needs is some test coverage so we don't break it in the future

Here is my suggestion for testing:

1. update csv_files.slt, see this file for info on running sql logic tests

Note I think you can programatically create a csv file with a command like

> copy (values ('column1,column2'), ('#second line is a comment'), ('2,3')) TO '/tmp/my.csv' OPTIONS ('format.delimiter' '|');
+-------+
| count |
+-------+
| 3     |
+-------+
1 row(s) fetched.
Elapsed 0.004 seconds.

That results in

$ cat /tmp/my.csv
column1,column2
#second line is a comment
2,3

[alamb]

Thank you @bbannier 🚀

[alamb]

👍 Love it