tar_group without dplyr dependency

[robitalec]

Prework

• Read and agree to the code of conduct and contributing guidelines.
• If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.
• Format your code according to the tidyverse style guide.

Proposal

Not sure this is necessarily "new feature" but I wasn't sure exactly where to organize it.

I was just testing out the targets::tar_group and related tarchetypes::tar_group_by functions.
If one is working with a data.table, the requirement to use dplyr::group_by can be tricky.
The data.table class is lost, and downstream targets can fail.

I adapted the example _targets.R created with targets::tar_script:

library(targets)

# Adjust this summary function to use data.table and append a mean_x column
summ <- function(dataset) {
  dataset[, mean_x := mean(x)]
}

# Set target-specific options such as packages.
tar_option_set(packages = "data.table")

# End this file with a list of target objects.
# (Adjusted to a data.table, works fine)
tar1 <- list(
  tar_target(
    data, 
    data.table(x = sample.int(100),
               y = sample.int(100),
               id = LETTERS[1:5])
  ),
  tar_target(
    summary, 
    summ(data)
  )
)

This works fine as expected.

The next step was to use the group_by and tar_group functions:

# Set target-specific options such as packages.
tar_option_set(packages = c("data.table", "dplyr"))

# Using tar_group breaks because tar_group/dplyr drops data.table class
tar2 <- list(
  tar_target(
    data, 
    data.table(x = sample.int(100),
               y = sample.int(100),
               id = LETTERS[1:5]) %>%
      dplyr::group_by(id) %>%
      tar_group(),
    iteration = 'group'
  ),
  tar_target(
    summary, 
    summ(data),
    map(data)
  )
)

Since the summ function is expecting a data.table, this (unsurprisingly) breaks.

● run target data
● run branch summary_d3b1f6e1
x error branch summary_d3b1f6e1
Error : Check that is.data.table(DT) == TRUE. Otherwise, := and `:=`(...) are defined for use in j, once only and in particular ways. Se
Error: callr subprocess failed: Check that is.data.table(DT) == TRUE. Otherwise, := and `:=`(...) are defined for use in j, once only and in particular ways. 

So I had a look at the source for tar_group here

# https://github.com/ropensci/targets/blob/main/R/tar_group.R#L61
tar_group <- function(x) {
  assert_package("dplyr")
  groups <- attr(x, "groups")
  if (is.null(groups)) {
    throw_validate(
      "tar_group() must take a grouped data frame from dplyr::group_by()"
    )
  }
  x$tar_group <- tar_group_column(x, groups)
  x
}

tar_group_column <- function(x, groups) {
  out <- rep(NA_integer_, nrow(x))
  rows <- groups$.rows
  for (index in seq_along(rows)) {
    out[rows[[index]]] <- index
  }
  out
}

This looked to me similar in to using data.table's .GRP function to append a column indicating a group index. In this case named "tar_group".
So I modified the function to do that instead (obviously we'd need to replace much of the assertions and checks, but just in an exploratory mode for now)

tar_group <- function(x, by) {
  # assert_package("dplyr")
  # groups <- attr(x, "groups")
  # if (is.null(groups)) {
  #   throw_validate(
  #     "tar_group() must take a grouped data frame from dplyr::group_by()"
  #   )
  # }
  x[, tar_group := .GRP, by = by]
  # x$tar_group <- tar_group_column(x, groups)
  x
}

# tar_group_column <- function(x, groups) {
#   out <- rep(NA_integer_, nrow(x))
#   rows <- groups$.rows
#   for (index in seq_along(rows)) {
#     out[rows[[index]]] <- index
#   }
#   out
# }

Here's that target updated with the quick function.

# Data.table based tar_group, using .GRP to assign tar_group index directly
tar_group_datatable <- function(x, by) {
  x[, tar_group := .GRP, by = by]
  x
}

tar3 <- list(
  tar_target(
    data, 
    tar_group_datatable(data.table(x = sample.int(100),
                                   y = sample.int(100),
                                   id = LETTERS[1:5]),
                        by = 'id'),
    iteration = 'group'
  ),
  tar_target(
    summary, 
    summ(data),
    map(data)
  )
)

And finally, I realized that it's really just a special column name (tar_group) for targets.
This brings me to why I'm not sure that's really an issue describing a new feature, it's rather maybe just a neat workaround.
If we assign a column to the data and use iteration = 'group', we don't need a tar_group or dplyr::group_by call at all.

# Simpler yet, just user-side assign the column directly
# Here as well, extending the example to multiple columns easily
tar4 <- list(
  tar_target(
    data, 
    data.table(x = sample.int(100),
               y = sample.int(100),
               id = LETTERS[1:5],
               yr = rep(c(2012, 2013), each = 50))
  ),
  tar_target(
    grouped,
    data[, tar_group := .GRP, by = c('id', 'yr')],
    iteration = 'group'
  ),
  tar_target(
    summary, 
    summ(grouped),
    map(grouped)
  )
)

And that's it! Data.table-aware tar_group equivalent. I'm going to use it in my own target workflows. Not sure how it fits into targets, but thought you might want to hear about it.

This small example doesn't really seem necessary given the power of data.table's by. But where we pass a grouped data.table to downstream targets that require chunks of a data.table for other kinds of processing (say converting to a matrix), this kind of tar_group flexibility could be really useful. I ran into this when I was using a map over file paths, but then wanted to run on arbitrary chunks for ID and years within file paths. So I couldn't just set up a wrapper static map with a list of years for example.

Thanks!

[wlandau]

Neat use case, thanks for sharing!

Another workaround is use dplyr for grouping to coerce the output of tar_group() to a data.table using tar_target(..., format = "fst_tbl").

[robitalec]

Thanks - the use case label is better suited.

Interesting ok. I wonder about added overhead from casting between data.frames and data.tables for each group, but that's a good alternative to keep in mind.

Thanks!