Core blocks

[nbenn]

What are core blocks?

Clear candidates

• data blocks ("browser" for datasets) others? A csv reader?
• dplyr single table verbs such as filter(), select(), mutate()
• multi-table blocks left_join() et al.
• ggplot

Further down the line

• slice()?
• union()/bind_rows() maybe bind_cols()?
• tidyr::pivot_*?
• others?

This does not have to be comprehensive and can be extended (or blocks can be moved into other packages) but we should get a better idea on what goes into the MVP.

Open questions

• how do we implement plotting: layer-by-layer or one-shot?
• is group()/ungroup() a distinct operation (i.e. implemented as separate blocks)?

[nbenn]

Related to #103

[christophsax]

As discussed:

Milestone I (Start of January)

• data blocks ("browser" for datasets) others? A csv reader?

  • CSV reader (rio?)

• dplyr single table verbs such as

  • filter(), select(), mutate(), summarize (with by)

• Some basic plotting idea (we are not clear what that is)

  • A layered ggplot? Additional Discussion needed, e.g. next week

• workspace object

  • Serializable (to JSON, from JSON)
  • Storable (as part of Superframework)

• Extendability

  • BMS developer should be able to develop their own, (e.g. admiral) blocks.
  • With documentation

Milestone II (MVP, End of February)

• left_join()
• bind_rows()

Beyond MVP

• tidyr::pivot_*
• group / ungroup
• CRAN release
• separation into separate packages

[christophsax]

@JohnCoene @DivadNojnarg trying the discussion, I guess they should have some advantage for this.

I would think that, once on CRAN, blockr would contain basic, polished dplyr blocks. With this, it would be already a nice package that solves many use cases. Seems easier to me to write vignettes etc. for such a package than for a core package that is useless by itself.

Also, you need the dplyr blocks in most applications.

Probably no need to settle this at the moment.

[nbenn]

@christophsax none of this has to be settled in the sense that we have to decide and then have to stick with that decision.

At the same time, I feel it would be good to have more clarity on some of the questions here. For the MVP stage,

• what blocks do we want to offer?
• where do they live?
• how do we do plotting?
• how do we deal with grouped data?

[nbenn]

@christophsax but yeah, none of these has to be answered right away. This is just a starting point to structure the discussion a bit.

[nbenn]

I'll try to structure the discussion aligned with these questions. I'll move remarks made as issues over here. Please feel free to add.

[nbenn]

What blocks do we want to offer?

Dataset

• dataset selector
• reading of files
• a block that returns a lazy table?

Transformation/dplyr

I assume we all agree on the following:

• arrange()
• filter()
• mutate()
• rename()
• select()
• summarise()
• mutating joins all behave similar, so if we have one, we have all: inner_join(), left_join(), right_join() and full_join()

Other single table verbs:

• What about the slice() family? Instead of head()/tail()?
• relocate()?
• the "rows API": rows_insert(), rows_append(), rows_update(), rows_patch(), rows_upsert(), rows_delete()

Multi-table verbs:

• bind_rows() and bind_cols()
• setops: intersect(), union(), union_all(), setdiff(), setequal() and symdiff()
• more exotic joins: semi_join(), anti_join(), nest_join() and cross_join()

Plotting

• ggplot block(s)

[nbenn]

Where do which blocks live?

• transformation/dplyr blocks in separate package? @JohnCoene (++), @DivadNojnarg (+), @christophsax (-)
• are datasets blocks for sure part of the core package? what's the scope here?
• what about plotting? what is the scope we are aiming for here?

[nbenn]

We could even be stricter and only keep in blockr what is related to the infrastructure (constructors, fields, ..., data block).

@DivadNojnarg

I see blockr as the package used by block developers, and move all blocks to other packages: blockr.dplyr, blockr.admiral, ...

@JohnCoene

blockr would contain basic, polished dplyr blocks. With this, it would be already a nice package that solves many use cases. Seems easier to me to write vignettes etc. for such a package than for a core package that is useless by itself. Also, you need the dplyr blocks in most applications.

@christophsax