consider making funnel.Rmd a part of large.Rmd

[maelle]

No description provided.

[maelle]

And re-organize content.

• Big picture: big data + RAM = risky. duckplyr's design principles.
• How to input data for use with duckplyr.
• A subsection about duckdb request
• How to compute, including funneling and computation to files.

[maelle]

I'll have a go at this.

[krlmlr]

Thanks. I welcome proposals to further split up these vignettes. Right now, both "large" and "funnel" are almost too large on their own, and it would be too scary for me to attempt to read both in one sitting.

What's the motivation to merge these?

[maelle]

The motivation is that both are meant for large data.

I think that with better subsection titles, it'll be easier to skim.

[maelle]

Actually I might split them but differently!

• how to input data to duckplyr
• how to handle large data (includes highlighting Parquet etc, and funneling/lazy stuff)

[maelle]

• how to input data
• how to compute
• funnel
• special tips for large data

[maelle]

@krlmlr I'd consider making funnel a part of "how to compute" so there'd be three vignettes instead of two.

• How to input data
• How to compute
• Special tips for large data: use certain input functions, compute to files, make sure you have funneling in place. Including the "big picture" section that is currently at the end of large.Rmd.

I think the paragraph comparing duckplyr to dbplyr that is currently in the large.Rmd vignette should be moved to the README (I can make a PR) because comparisons to similar tools are sort of usual in READMEs.

I'd remove the "funnel" section from the vignette for developers.

How does this sound?

[krlmlr]

I'd need to think about it. Let's focus on the small wins for now.

[krlmlr]

In my view, data in and data out should be in one vignette.

We can add a third vignette that presents an entirely different view, perhaps a "Getting started" vignette that links to the two others. If there's then material that no longer is a good fit for "large" or "funnel" (soon to be "valve"?), we can rearrange gradually.

[krlmlr]

Would you like to contribute a "Getting started" vignette?

[maelle]

Yes, I'll try! 😸 I'll share the diagram source then, as it could go in there.