[help] Handling a target that returns a file and a value

[multimeric]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

format = "file" requires the target to return only a vector of files. This can be annoying because the operation of saving the file might have generated additional data that you want to keep inside some R data structures.

For example, say we have an object that initialises by creating a data structure on disk:

init <- function(a, b, c){
    data <- paste0(a, b, c)
    path <- tempfile()
    write.csv(data, path)
    structure(
        list(a, b, c, path = path),
        class = "my_class"
    )
}

In this case, I want changes to the file to cause this target to rerun, which requires format = "file". However, I can't just return path here because I want to keep the new structure. I could move the file creation logic into a separate write function that is a dependency to the init target, but then I have to replicate all of my branching logic, and the logic used to combine the data across the init and write targets. If I move the write function after init, then it will never run because nothing depends on the path explicitly: all future targets use the structure returned by init.

I guess what I'm after is something analogous to Nextflow's tuple(path, val) idiom, which lets you return a path and an in-memory value, both of which get tracked appropriately: https://www.nextflow.io/docs/latest/process.html#output-type-tuple.

It seems that tar_cue doesn't offer any control here. I could make a custom format using tar_format(), but that's quite a complex solution.

[wlandau]

targets is opinionated, and it nudges users toward pure functions. Cases like this are rare and usually not encouraged in a function-oriented approach. I do realize some packages create in-memory R objects that rely on special saved files, but this is not generally good practice. I would recommend contacting the author of the package to request the ability to marshal/serialize such objects. Examples include terra::rast() and keras::serialize_model().

[wlandau]

It depends on the reason why you need a file on disk. For example, you might want to proactively "export" a strategic subset of data outside the pipeline to analyze with a different tool. In that case, I would recommend you stick with non-file targets for almost everything, then have a file target downstream of everything which saves the most important subset of those results.

[multimeric]

What I want to do is move some data to disk to reduce memory usage. For example, assuming I have sce which is a SingleCellExperiment:

tar_target(
    on_disk,
    {
        assays(sce) <- purrr::map(function(assay) HDF5Array::writeHDF5Array(assay, ...))
        sce
    }
)

Of course I want to track the on-disk versions of the array (located at HDF5Array::getHDF5DumpDir), because if it's deleted then the target needs to re-run. I equally need the sce itself, because that's the data structure I'll be operating on in future targets.

[wlandau]

The memory and storage options at https://books.ropensci.org/targets/performance.html might help equally well. Otherwise, if you need to save a file manually and only keep a light reference in memory, then a custom tar_format() fits the use case.

[multimeric]

Thanks, I just wanted to check that there wasn't anything I was missing. A custom format sounds good, or I could probably make a target factory that generates two targets and enforces a dependency between them.

[multimeric]

As a follow up to this. I tried doing this using a target factory, but it became evident that it wasn't possible. The idea was to have the user write one target that returns the data and the file, and the factory secretly adds a second target that tracks the file on its own.

Unfortunately this can't work, because if you add a file task f that follows on from the original task o, Targets will only re-run f when the file is deleted, even though o is the one that creates it and therefore the one that should be re-run. If you swap the order, and for example create an empty directory in task f, and then populate it with data in task o, then targets will think that the directory has been tampered with and re-run task f every time.

Therefore you have to solve this using formats and not factories.