What is the best approach for a large project with many smaller subprojects?

[kendonB]

One of my teams has a large project that consists of many disparate analyses, with very few interactions between these subprojects.

Currently, each subproject has its own drake cache, in its subfolder.

However, to keep us sane, we made an early choice to always have the assumed working directory be the root folder of the project. The cache location is set explicitly in each drake project.

If we were to convert this project to targets or start a new subproject using targets, what would be the best approach to take?

Obviously the desirable features are:

1. The working directory is the root folder
2. We can choose to run a specific subproject without checking targets associated with all the other subprojects.

[wlandau]

If each sub-project has its own _targets.R script, then the working directory to run a given sub-project is the root directory of that particular sub-project. There is some discussion about sub-projects at #297 including the unitar package. Also relevant is #407. I recommend that each sub-project get its own _targets/ data store at the sub-project root, but this is now customizable.

[kendonB]

If I understand correctly @wlandau, to achieve desirable feature 1. this would still require the respective _targets.R files to all live in the root folder of the parent project? This of course couldn't happen unless the user could set the name of the _targets.R in the tar_make call?

[wlandau]

Does that mean you want all sub-projects to run from the same root directory? I am not sure that can be done cleanly because only one sub-project could be active at any given time. Since #407, the way to set the data store is through a top-level _targets.yaml file. Switching projects means you have to switch the data store path through _targets.yaml (see tar_config_set()). The reasons behind the _targets.yaml implementation are discussed at the top of #297 and in #407.

I recommend you keep each sub-project in its own directory with its own project root and its own _targets.R file. Separate sub-project-specific code bases and sub-project-specific Git repos also tend to make things easier to manage. Ideally, sub-projects should only join together at the conceptual level. At the implementation level, I think each sub-project should behave as an independent, fully-formed, full-fledged project of its own, rather than a piece of an intimidating monolith. Monoliths are prone to tech debt.

I have no plans to allow custom alternative names for _targets.R unless it is absolutely necessary. (For _targets/, #407 was a last resort to make targets work on RStudio Connect. There was literally no other workaround because of strict storage retention policies on Connect.) The restrictive names _targets.R and _targets.yaml deliberately enforce reproducibility standards and try to nudge users away from the perils of multi-pipeline Gordian knots.

[riazarbi]

@kendonB if you are open to using an S3 backend as your cache, you could just set all of the subprojects to use the same bucket as the backend. Then, in effect, you have a global cache for all your projects. I hope that's clear, if it is not let me know and I'll try break it down a bit.