Automatically generate the pipeline graph at the end of a build

[lindsayplatt]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I really love using tar_visnetwork(), tar_glimpse(), and tar_mermaid() to generate visuals of the pipeline nodes and connections before, during, and/or after a call to tar_make(). I am trying to get better at committing these graphs (either as images or Markdown chunks with mermaid.js code) to the GitHub repo that contains my pipeline code so that users can see changes to the pipeline structure over time. The issue is that everyone has to manually remember to run one of these functions after they build the pipeline.

I know that we can't have these run as targets within the pipeline because we end up with strange circular behavior. For the moment, I have a wrapper function called tar_make_and_viz() that contains the following and is sourced and run instead of tar_make():

tar_make_and_viz <- function(...) {
    tar_make(...)
    
    # Save visual output
    htmlwidgets::saveWidget(tar_glimpse(), file = 'pipeline_viz.html')
    webshot::webshot("pipeline_viz.html", "pipeline_viz.png", vwidth = 800, vheight = 400)
}

However, it would be great if there was some way to integrate this step into tar_make() itself.

• What if there was an option that allowed the build to save the visual output, perhaps which viz to create and the output file to save it to? E.g. two new arguments viz_method = c('none', 'visnetwork', 'glimpse', 'mermaid') + viz_output = 'tar_config_get("viz_output")' where viz_output is a filepath
• Or what if there was a more general option for post-build-steps? E.g. post_build_script = tar_config_get("post_build_script") where post_build_script is a filepath to an R script that is run after a successful tar_make().

[noamross]

One strategy I use is to make the _targets.R itself a "file" target, and then make a target dependent on that to generate the graph. In my case the latter target is the README.Rmd. (example in https://github.com/ecohealthalliance/sabrenet-rousettus-dynamics)

This isn't perfect, as sometimes things that change the graph fall outside of the main _targets.R file. For instance, I sometimes split the pipeline into several files that are sourced into this file.

[wlandau]

Interesting idea to make _targets.R a file dependency for this.

[lindsayplatt]

I sometimes split the pipeline into several files that are sourced into this file.

Same, this is how I keep big pipelines organized. I would be OK setting cue = 'always' or using a tarchetypes::tar_cue_age() or something to the target that builds the viz. The viz build is not an expensive step computationally, so it's not as critical to only build if something changes IMO. Since I am committing the output, I'll notice a change to the pipeline in the file to commit. Thanks for providing your example!

[wlandau]

I know that we can't have these run as targets within the pipeline because we end up with strange circular behavior.

That's a great point. Because the graph depends on everything in the pipeline, a tar_mermaid() target within that same pipeline would technically be circular. But for graphs, I don't think the consequences are all that bad. targets cannot detect this circularity, so it won't break the pipeline, and the graph visualization is informal anyway.

To mitigate or avoid the circularity, you could use tar_glimpse() instead of tar_visnetwork(), or you could set outdated = FALSE in tar_visnetwork() or tar_mermaid(). That way, the graph only depends on the structure of the pipeline, rather than the specific statuses of each of the targets. You might still see unpredictable node colors from the information from tar_progress(), but that data is less formal than what you get from tar_outdated(). If you take those steps, then it makes even more sense to depend on _targets.R and any other R scripts that define components of the pipeline or initial functions/globals.

[lindsayplatt]

Love the idea to use outdated = FALSE so that I can still use tar_mermaid().

I tried adding the following to my _targets.R file just below the targets but it errored with a message that sounds like it might be an issue with circularity.

htmlwidgets::saveWidget(tar_glimpse(), file = 'pipeline_viz.html')
webshot::webshot("pipeline_viz.html", "pipeline_viz.png", vwidth = 800, vheight = 400)

Error:
! The target script file _targets.R must not call tar_make() or similar functions that would source the target script again and cause infinite recursion.

I'd like to commit something interactive since big pipeline visualizations can get really big and the text is tiny. I'm going to test your idea of adding as a target directly but using outdated=TRUE.

[lindsayplatt]

I get the same error when I attempt to add as a target. Here is a reprex with both an attempt using tar_glimpse() and tar_visnetwork()

targets::tar_dir({ 
  targets::tar_script({
    list(
      tar_target(x, 1 + 1),
      tar_target(y, x * 2),
      tar_target(viz, tar_glimpse()),
      tar_target(viz2, tar_visnetwork(outdated=T))
    )
  })
  targets::tar_make()
})

Error:
! The target script file _targets.R must not call tar_make() or similar functions that would 
source the target script again and cause infinite recursion.

I can, however, run tar_glimpse() after tar_make() and generate the output I seek.

targets::tar_dir({ 
  targets::tar_script({
    list(
      tar_target(x, 1 + 1),
      tar_target(y, x * 2)
    )
  })
  targets::tar_make()
  tar_glimpse()
})

[wlandau]

Yeah, I forgot about that safeguard. I still stand by it because it catches infinite recursions that inexperienced users sometimes cause, but it means you may have to call tar_visnetwork() from a function defined in a script outside _targets.R.