[RFC] Post-pipeline tasks/hooks

[connorjs]

Which project is this feature idea for?

Turborepo

Describe the feature you'd like to request

I would like to run some script after all packages have executed a given task.

For example, running coverage task in each package, and then running an additional task at the root that merges/aggregates all of that coverage. I can think of similar postXxx tasks for documentation, copying/moving/bundling files, etc.

Keywords: "all packages", "star dependsOn"

Describe the solution you'd like

Due to the nature of “all packages,” I would like a special dependsOn value ("*") that only root tasks can reference.

Example:

"//#coverage": {
  "dependsOn": ["*coverage"]
}

Note: I originally thought just *, but realized *<task_name> may make more sense.

Describe alternatives you've considered

Alternatives I considered

1. Running a new turbo instance. For example, having the root package.json’s coverage:root run turbo run coverage and have a postcoverage:root execute the aggregation. Not ideal because we would have two turbo runs which limits parallelization I think.

2. Creating an umbrella @scope/all-packages package that declares every other package as a dependency. I could then use this package to execute any aggregation logic. Not ideal at first glance because I now have two places to maintain workspace configuration/workspace list, which can lead to out-of-sync issues.

   This has similarity to explicitly declaring dependencies in dependsOn for the root task, but I think Turbo prefers we use package.json dependencies to model package dependencies, and dependsOn to manage task dependencies.

[arlyon]

Thanks for raising this issue! This idea has been floating around for a while, in fact I recently bumped into it myself on a personal project. I have fed this back to the team and will get a timeline soon.

Cheers!

[mehulkar]

Couple things to think about:

• Running a task after some set of tasks has completed. We can loosely call it a "finalization" task, and it's a general purpose tool we could design around
• "Root" depending on all packages. I'm not sure ^ is appropriate for this, because we really do want that to mean a topological dependency through package.json dependencies.

That said, I'll put this into our team discussion queue.

[mehulkar]

We talked about this today at the team meeting and agreed that there are a few problems to solve here and we don't have a clear solution in mind. Some insight into what we are mulling over:

We think that allowing a root task to depend on all packages is not semantically "correct" as a root task does not topologically depend on all packages via dependencies. To denote "all runs of this task" would need some unique API or microsyntax, and we'd want to gather more use cases before designing that.

This issue needs some dedicated time and ownership from the core team.

[wesleytodd]

We think that allowing a root task to depend on all packages is not semantically "correct" as a root task does not topologically depend on all packages via dependencies.

I think what folks need here is not about the semantics of dependency relations, it is simply "run this after" (even partly called out with the postXxx callout above). If there is an alternative way y'all want to approach it which achieves the ordering folks need but keeps clean separation between the semantics of dependency relations, I would bet most folks would be satisfied.

Personally I am comfortable with the semantics not being consistent because the ergonomics are more important, but I can see why this might be concerning to y'all.

[mehulkar]

Both are valid solutions, but the postXxx approach feels more like a "hook" into Turbo's behavior and breaks the mental model a DAG / "pipeline" for me. I feel like it would also have a long tail of UX that could get complicated (when you take into account package#task syntax, Workspace Configs, etc).

[wesleytodd]

I feel like it would also have a long tail of UX that could get complicated

Yeah I very much agree with this assessment. Partly that is why I prefer to have the DAG allow for this "root task depends on all packages task" rather than adding a new concept which breaks that paradigm (even if it breaks the dependency concept from the package tree perspective).

[connorjs]

FWIW, I think I have the same issue with Nx. I experimented with the @scope/all-packages approach (in Nx, I can just use manual dependsOn in the project.json), but I did have the headache of manual syncing.

Running a task after some set of tasks has completed. We can loosely call it a "finalization" task, and it's a general purpose tool we could design around

Yes, I like this semantic focus; it matches my mental model and examples the most (merge coverage, aggregate docs).

---

Are there ways to simplify the maintenance of the @scope/all-packages package? For example, a script that reads npm workspaces and self-updates its own dependencies?

I mention this in case this train of thought preserves the "DAG/pipeline" semantics and just removes pain of a hand-maintained umbrella package.

---

Disclaimer: I’m using Nx these days, not Turbo, so do not make decisions solely for my ideas! But I like that Turbo and Nx cross pollinate often.

[alecmev]

I'm not sure if it's what you mean by "manual dependsOn", but Nx is happy with this:

"build-all": {
  "dependsOn": [
    {
      "projects": "*",
      "target": "build"
    }
  ],
  "executor": "nx:noop"
}

[Caleb-T-Owens]

Hey! i've run into this today configuring turbo for our linting setup. As we use the flat file configuration we have just the one root eslint which doesn't have any dependencies listed. That means I need to specify each package that needs to be built manually.

I'd be keen to hear if there has been any progress on this

[anthonyshew]

This doesn't sound like what's being described here. That sounds like a difficulty in the Package Graph rather than the Task Graph.

It might also be worth noting that ESLint's new Flat Config doesn't necessarily mean you have to have one configuration file. We'll be working on guidance for this once we've landed #8606.

The short version is that we will still be recommending a configuration file per-package so that caching behavior will work correctly. This is something that other folks have discussed in ESLint's Discussions: https://github.com/eslint/eslint/discussions?discussions_q=flat+monorepo

[alecmev]

I chose to go the umbrella way for now, and didn't want to worry about forgetting to add/remove packages, so made a Yarn 4+ constraint to keep track of it for me. Sharing in case somebody else wants to borrow it:

/** @type {import('@yarnpkg/types').Yarn.Config} */
module.exports = {
  async constraints({ Yarn }) {
    for (const workspace of Yarn.workspaces()) {
      // This is a loop because there might be other constraints here

      if (workspace.ident === '@repo/umbrella') {
        const packageNames = Yarn.workspaces()
          .filter((x) => x.cwd !== '.') // Root
          .map((x) => x.ident)
          .filter((x) => x !== null) // Separate for automatic type narrowing
          .filter((x) => x !== workspace.ident);

        for (const packageName of packageNames) {
          workspace.set(['dependencies', packageName], 'workspace:*');
        }

        for (const dependency of Yarn.dependencies({ workspace })) {
          if (packageNames.includes(dependency.ident)) continue;
          workspace.unset(['dependencies', dependency.ident]);
        }

        // Just in case
        workspace.unset('devDependencies');
      }
    }
  },
};