[ideas] A pattern target where each branch has a dependency on the previous branch.

[rowanclarke]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I often need to know the intermediate results of performing a reduction over a pattern - something akin to purrr::accumulate - which I don't think is currently possible.

The following is an example of how to use such a feature:

tar_dir({
  tar_script(
    list(
      tar_target(x, c(1, 2, 3)),
      # create a pattern that accumulates over the buds that `map(x)` specifies.
      tar_target(y, x + y, map(x))
    )
  )
  tar_make()
  expect_equal(
    tar_read(y) |> unname(),
    purrr::accumulate(c(1, 2, 3), `+`)
  )
})

The benefit comes when we change x to c(1, 2, 4) - the first two branches of y can be skipped.

I'd be very keen to hear your thoughts on this - whether it aligns with the design of targets and how it could be implemented. I've tried implementing it myself by just inserting new dependencies before creating the junction and it seems to work as described.

[wlandau]

With dynamic branching, this would not be feasible for targets because those intermediate steps create dependencies between branches, and targets is not built to handle that. But a target factory with static branching could handle it with metaprogramming.

[rowanclarke]

Hi. Thanks for the swift response. Unless I'm mistaken, I can only use dynamic branching if the size of x cannot be determined in advance.
I'll just use a single target for now in my own application since it only takes a couple seconds to run, but I would be interested to know if you anticipate any potential developments in this area?

[wlandau]

Unless I'm mistaken, I can only use dynamic branching if the size of x cannot be determined in advance.

Right, that's a good point.

I'll just use a single target for now in my own application since it only takes a couple seconds to run, but I would be interested to know if you anticipate any potential developments in this area?

It makes sense how this would bring you into dynamic branching territory. But to implement in targets would be a huge undertaking because it breaks the assumptions of the internal design. And I think the payoff would be small because this would be such an advanced feature that few people would use. Even in those explicitly pairwise-reduce scenarios, it will make the most sense in many cases to put it all in a single target like you are doing now, c.f. https://books.ropensci.org/targets/targets.html#how-much-a-target-should-do