Support skipping comments in CSV files #5758
Labels
Comments
bbannier
added
the
enhancement
bbannier
added a commit
to bbannier/arrow-rs
that referenced
this issue
May 12, 2024
This patch adds reader support for a comment character for reading CSV
files. While comments like almost nothing around the CSV format are not
truly standardized, a common format supported by many CSV
readers[^1][^2] is to ignore full lines starting with a comment
character (often `#`); inline or end of line comments are not supported.
Example:
# This is a comment in a CSV file without header.
1,2
# Comment inside the data block.
11,22
The implementation of this for Arrow is pretty straight-forward as all
we need to do is expose the existing `comment` option of `csv_core` used
to read CSV files.
Closes apache#5758.
[^1]: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html
[^2]: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html
bbannier
added a commit
to bbannier/arrow-rs
that referenced
this issue
May 12, 2024
This patch adds reader support for a comment character for reading CSV
files. While comments like almost nothing around the CSV format are not
truly standardized, a common format supported by many CSV
readers[^1][^2] is to ignore full lines starting with a comment
character (often `#`); inline or end of line comments are not supported.
Example:
# This is a comment in a CSV file without header.
1,2
# Comment inside the data block.
11,22
The implementation of this for Arrow is pretty straight-forward as all
we need to do is expose the existing `comment` option of `csv_core` used
to read CSV files.
Closes apache#5758.
[^1]: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html
[^2]: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html
tustvold
pushed a commit
that referenced
this issue
May 13, 2024
This patch adds reader support for a comment character for reading CSV
files. While comments like almost nothing around the CSV format are not
truly standardized, a common format supported by many CSV
readers[^1][^2] is to ignore full lines starting with a comment
character (often `#`); inline or end of line comments are not supported.
Example:
# This is a comment in a CSV file without header.
1,2
# Comment inside the data block.
11,22
The implementation of this for Arrow is pretty straight-forward as all
we need to do is expose the existing `comment` option of `csv_core` used
to read CSV files.
Closes #5758.
[^1]: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html
[^2]: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
CSV readers often support skipping comments12 which are typically full-line comments indicated by some prefix character (often
#is used). It would be great if arrow-csv would allow parsing of such CSV files.Footnotes
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html ↩
https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html ↩
The text was updated successfully, but these errors were encountered: