Ray Dataset - improve documentation

[DevJake]

Description

To be really blunt, the documentation is of pretty low quality. I've just spent several hours trying to find how to apply a specific schema to the load_parquet() function, and it's as simple as just supplying it directly: load_parquet(schema=[Schema object]). I'm glad the solution is simple, but to spend this long trying to find a solution is indicative of a bigger issue: the documentation fails to mention that this is possible, nor what other parameters a Dataset object might take. Navigating to the actual Dataset page is difficult, since the search does not return meaningful results, nor are mentions of it hyperlinked, only highlighted.

Take this page on the read_parquet method. The listed parameters aren't exactly described or given examples, their descriptions are already clear from the name: "filesystem - The filesystem implementation to read from." This information is obvious from the name, but what types of values are accepted? I will assume string values, but what strings, exactly?

I'm sure this is a great library, I can tell there's an enormous amount of work and effort that has gone into it. But to ship with such uninformative, inaccessible and poorly structured documentation makes it incredibly frustrating for new people like me to get to grips with it. I'd love to use this library further but if anything more complex than just loading in data is equally unclear, I won't be able to...

The solution I found was from this stackoverflow post, which I only found by googling the uninformative error message I received when calling .repartition(): pyarrow.lib.check_statuspyarrow.lib.ArrowNotImplementedError: Unsupported cast from double to null using function cast_null. Perhaps such an error message should show a snippet of the record and/or column(s) in question, or mention needing to verify no null entries exist in the dataset's schema?

Link

https://docs.ray.io/en/latest/data/consuming-datasets.html#consuming-datasets
https://docs.ray.io/en/latest/data/api/input_output.html#ray.data.read_parquet

[richardliaw]

@DevJake I think this is really good feedback, and we've been trying to make the documentation better.

That being said, from our perspective it's very hard to write great docs when we're trying to write it for ourselves. I'm wondering if you'd be willing to help sit down with us, walk us through your experience from a first-person perspective, and give some better color on what you're trying to do?

[pcmoritz]

Thanks, this is super great feedback! I'll try to make a PR to improve the documentation with these points and tag you in it @DevJake

As Richard points out, we'd also love to get more feedback from you. Ray Data is still a very new library and we are actively improving the documentation -- such immediate feedback is incredibly valuable!

Thanks for your help :)

[pcmoritz]

I made a PR to improve the issues you brought up on the Ray side (#28488).

The error message will need to be improved in Apache Arrow. I'm happy to do that and have done so in the past (e.g. apache/arrow#13979, apache/arrow#14001, apache/arrow#14019) -- the Apache Arrow project is still relatively young and can use all our help to iron out usability issues like this. I'm incredibly excited that Apache Arrow is developing a (Python) native data processing stack and we should help them as best as we can by giving them feedback and helping by sending them PRs.

Sorry for your frustration and thanks for your help in surfacing and fixing these issues!

[DevJake]

Hey @pcmoritz and @richardliaw, I really appreciate your quick responses and action on this. I apologise for the bluntness in my message, but I'm sure you can understand. A lot of new, open-source software is in development in response to the AI explosion we're seeing lately, so it definitely needs time to get up to scratch with stuff that's been out there for a much longer period. I'd be more than happy to help work through the documentation, perhaps you'll see me on the Slack soon enough 👀

When I've got more time, I'll probably end up making some pull requests myself to contribute. No use knowing an issue exists without trying to also resolve it. 😅

[pcmoritz]

Thanks that sounds great! Any more help and feedback and also PRs are certainly very much appreciated :)

This very concrete feedback really helps improve things, keep it coming!

[pcmoritz]

Btw, I was digging a bit deeper on the underlying issue why you needed to specify a schema at all here and my findings are in https://issues.apache.org/jira/projects/ARROW/issues/ARROW-17719

I wonder how large the performance overhead would be if we made the default to inspect the schemata of all partitions. That seems more user friendly :)

[DevJake]

That makes complete sense, with the schema being inferred from the very first file.

I imagine inspecting all partitions might carry a considerable overhead, especially with a large number of partitions. Perhaps any or all of the following would work:

• Prompting the user to explicitly specify a schema, as I have done.
• Providing an optional parameter to infer the schema by merging across all partitions, as you suggested. Something like get_schema_from_all_partitions=False.
• Issuing clear warnings for any fields denoted as null in the schema, as this is a probable issue in most cases.

I'm glad you managed to track down the underlying issue. I suppose it's particularly unlikely that someone would load a dataset that features one or more entirely empty/null columns, but not impossible.