Tags - A unified solution for idempotency and observability

[pranaygp]

Tags

Tags are piece of metadata that are associated with runs, steps, hooks, and other future entities. It's a simple key/value solution that addresses two problems

1. Meaningful Visibility - on the o11y tab, we need to provide users a way to search and filter for runs, steps, etc.. Run IDs/step IDs are generated and not meaningful. Tags provide a way to add attributes to these entities which are expected to be indexed by the world for filtering and search. This way, worlds don't need to index all the run/step inputs/outputs (which contains sensitive data)
2. Idempotency - Support for atomically creating a tag, or throwing on conflict, is the most versatile way to do run idempotency (explained further down)

Let's first see it in use:

import { setTag } from "workflow";

async function myWorkflow() {
  "use workflow";

  setTag("foo", "bar"); // returns promises, but you don't need to await them
  await setTag("baz", 1);
}

This is the simplest use of tags - it upserts two tags on the active run - "bar", and "baz".

Signature/Spec:

type Value = string | number | boolean | null | Date;
type SetTagOptions = {
    // prevent multiple entities having the same key:value pair. 
    // uniqueness is enforced globally across the project
    unique: boolean
}
async function setTag(
  key: string,  // bounded to 256 characters (or left to world)
  value: Value | (current: Value | undefined) => Value, // bounded to 256 bytes (or left to world)
  options: SetTagOptions
): Promise<Value>

Key Points

• These are simple key:value pairs that are intended to be short and will show up in the observability when inspecting the entity
• The keys and values should not contain sensitive data
• Keys and values are expected to be indexed by the world to enable search and filtering in the UI
• Worlds can (and probably should) limit the number of tags and/or the size of the tags, to prevent unbounded growth and overindexing
• No nested values. Keep it simple.

Usage for idempotency

Most alternatives implement idempotency purely as an "idempotency key" that needs to be set when the run is invoked, or allow you to customize the Run ID. The main challenge here is that you don't always know what the idempotency key is when starting the run itself

Take this example of a code review workflow:

async function reviewPR(repo: string, pr: number) {
  "use workflow";

  const { checksum } = await getLatestCommit(repo, pr);
  
  // we want run idempotency on the commit SHA, not the PR itself
  // so we create a unique tag
  await setTag("commitSHA", checksum, {unique: true});
  // this throws a `TagConflictError` if there's already a run for this commit
}

Having idempotency controlled within the workflow is more flexible than leaving it to the caller only. Caller based idempotency actually comes for free if we also allow users to set tags at creation time. For instance, with runs, start can be extended to allow passing in tags at invoke time. Example syntax:

start(
  reviewPR,
  ["vercel/workflow", 42],
  {
    tags: {
      commitSHA: { value: "...", unique: true } // if we happened to have it already
    }
  }
)

Doing the "idempotencyKey" is a slippery slope to having to support various kinds of behaviors (for example, Temporal has a whole doc on ID reuse and conflicts to support multiple behaviours. I find the whole thing too complicated). setTag gives all the power to the user.

Let's model various possible behaviours

Fail the current run if an existing run already exists with the same tag

(example already shown above)

Fail the current run only if the conflicting run did not fail

async function reviewPR() {
  "use workflow";
  
  try {
    await setTag("idempotencyKey", "...", {unique: true});
  } catch (error) {
    if (error instanceof TagConflictError) {
      const status = await error.existingRun.status
     if (status !== 'failed') {
        // only throw if the conflicting run failed. otherwise we can continue this new run
        throw e;
     }
    }
  }
}

Cancel the existing run, and continue the current one

async function reviewPR() {
  "use workflow";
  
  try {
    await setTag("idempotencyKey", "...", {unique: true});
  } catch (error) {
    if (error instanceof TagConflictError) {
      await error.existingRun.cancel(); // throws again if cancel fails
    }
  }

  // if execution got here, it means the old run was in a running state, and was successfully cancelled
}

Do anything really...

async function reviewPR() {
  "use workflow";
  
  let existingRun: Run | null = null;
  try {
    await setTag("idempotencyKey", "...", {unique: true});
  } catch (error) {
    if (error instanceof TagConflictError) {
      run = getRun(error.existingRunId)
    }
  }

 // we can now continue this workflow run but adapt behaviour based on this.
 // we can also choose to not use `unique` and simply list all the runs by a given tag
 // and use that result here to model all sorts of complex/custom handling
}

Tags are really powerful 🤯

An Implementation Gotcha!

It's tempting to implement setTag, and especially getTag, directly in the runtime without forking out to a step (for instance, getTag can be completely synchronous since the run data is available already). But calls to setTag and getTag must have the return values cached in the event log to preserve determinism. Otherwise unqiue would always throw, and even simple examples like this would fail:

async function myWorkflow() {
  "use workflow";

  console.log(await getTag("foo")); // should be `undefined` on replays too
  await setTag("foo", true)
  console.log(await getTag("foo")); // should always be true
}

This is not a problems if these APIs are simply implemented as steps themselves, although an optimization later is running these as "local steps" - i.e. run in the workflow process itself while recording them in event log

Special Tags / Reserved Namespace

Tags names beginning with $ are reserved for special functionality and metadata. Here are some proposed tags -

• $name (type: string). If set, the observability CLI and UI will prefer this instead of the ID as a more "human friendly" name.
  • Open question: should these automatically enforce unique? The benefit is we can make commans like npm wf cancel <run-name> work as expected
  • Pranay's 2c - yes, we make them unique and lax that requirement in the future
    *$id (type: string. enforced uniqueness). 💡 insight: this could be particularly useful for solving the steps "stable ID" problem - whereby we would use $id over the implicit generated step IDs during replay
• Insert other ideas? $ai.model/$ai.provider/etc. (for pretty UI icons, etc),

Other examples of using tags

Steps

Since uniqueness is enforced globally across a project (across all runs/workflows), this tags solution extends naturally to providing a layer of step idempotency. This can be particularly helpful to preserve idempotency for steps that is guaranteed across re-runs/upgrades

async function sendThankYouEmail(orderID: string) {
  "use step"

  // When a service provider accepts idempotency keys, this is the natural thing people do:
  const { stepId } = getStepMetadata();
  await card.charge({idempotencyKey: stepId})

  // However, if you, for any reason, cancelled the workflow run and restarted it
  // or you upgraded a run using the "cancel and restart" strategy (RFC pending)
  // then `stepId` is not stable. However, orderId here is. So the recommended
  // approach would actually be:
  await card.charge({idempotencyKey: orderId})

  // However, not every service has an idempotency key solution. This often means
  // people have to roll their own idempotency using redis. However, tags solve that
  // already.
  await setTag("send-thank-you-email-unique-id", orderID, { unique: true })

  // now you can do whatever compute you want here and are guaranteed
  // idempotency. 
}

Another Implementation Gotcha / Open Question
Using unique setTags inside steps would cause every retry attempt to fail.
Also, what is the intended behaviour for tags if a step fails anyway? should tags accumulate across all retries, or do users only care about the final state of the tags (i.e. on the only successful attempt)

Pranay's 2c. Using setTag in a run is actually scoped to the current attempt (note: we don't currently have an "attempt" entity). The state of the tags on the step entity is just a (materialized) pointer to the latest "attempt"'s tags. unique doesn't follow the pointers, so it won't throw across attempts, but will throws across steps

(Also Pranay: Is this getting too complex? There's got to be an occam's razor solution to this that still lets us use tags across runs, steps, and hooks)

Hooks

Tagging hooks is only really meant for o11y, and can only be set when creating the hook. Unique unique when setting tags on a hook is pretty redundant since the token is already unique - the only exception is if for some reason you do want to use autogenerated hook tokens for aesthetics, but you want to fail the workflow run on a unique tag. Pretty esoteric 🤷🏼

For completeness, here's hook usage

async function documentReviewFlow() {
  "use workflow";

  const webhook = createWebhook({
    tags: {
      $name: "Pranay's NDA Review"
      document: "NDA",
      reviewer: "[email protected]"
    }
  })
  await sendEmail("[email protected]", "NDA.pdf", webhook.url);
  await webhook;
}

Now, on he o11y UI, I would have a more useful/readable view in the hook so I can identify what each hook actually is rather than the currently non-descriptive hook ID

Related APIs

getTag (get value of a specific tag inside a run)
listTags (list all the tags and values inside a run)
start(...) can be extended to allow setting tags at call time
createHook/createWebhook needs to be extended to set tegs on hook creation time
notably, steps don't have a way to set keys at invoke time, but this limitation seems fine to me since steps can always set them as the first thing. We can also solve this later by having a run(step, args, opts) api as an alternative to calling steps directly if we need it

Related World Spec Changes

<Not filling this in - should be easy to infer from the above and we can sort out as we go in PR reviews>

Open Questions

• What should the limits be? Size of keys/values? Or should that be left to the world? One benefit of limiting them in the spec instead of leaving it to the world is that the UI does not have to handle arbitrarily sizes/number of tags - but maybe that's not worth worrying about
• Should these be made available at workflow (root), or workflow/api?
  • Pranay's 2c - workflow for setTag, getTag, and listTags (which are all automatically applied to the run/step scope in which they are called), and workflow/api for listRunsByTags (or similar o11y APIs)
• Should we support different uniqueness scopes? (unique across all workflow in a project, across all runs in a workflow, all steps in a run, etc). OR, do we just do globally unique and leave it to the developer to include workflowName, runId, etc. in the key names?

[hugo082]

I really appreciate the flexibility of the tags solution—it's powerful and enables a lot of creative use cases. However, I have concerns about using this API as the primary mechanism for workflow ID/idempotency management:

1. Atomicity Issues

The try/catch pattern for implementing common idempotency policies is fundamentally non-atomic:

try {
  await setTag("idempotencyKey", "...", {unique: true});
} catch (error) {
    if (error instanceof TagConflictError) {
      // RACE CONDITION WINDOW OPENS HERE
      await error.existingRun.cancel();
      // What if the existing run completes successfully during cancel?
      // What if another deployment with the same ID starts right here?
      // What if the cancel fails but we continue anyway?
    }
}
// We are guaranteed to be alone here...

Operations like "terminate existing and start new" need to be atomic at the server level to avoid race conditions.

2. Boilerplate for Common Cases

Looking at real-world usage, 80% of idempotency use cases fall into three patterns:

• FAIL - reject if a run exists (default behavior)
• USE_EXISTING - return handle to existing run
• TERMINATE_EXISTING - cancel existing and proceed with new

Having to write 5-10 lines of try/catch boilerplate for each of these common cases seems tedious when they could be expressed as simple configuration options.

3. Explicit APIs vs Reserved Tags

For core workflow semantics like ID management and conflict resolution, I believe explicit API parameters are clearer than reserved tags. When someone reads code or configuration, seeing workflowIdConflictPolicy: TERMINATE_EXISTING immediately communicates intent, whereas the policy being buried in try/catch blocks inside the workflow makes it harder to understand the behavior at a glance.

Reserved tags like $id or $name make sense for metadata, but conflating them with behavioral policies feels like mixing concerns.

4. Future continueAsNew Considerations

Have you thought through how this interacts with a future continueAsNew API? Typically, continueAsNew keeps the same workflow ID but starts fresh execution. Questions that arise:

• Do tags carry over to the new execution?
• If tags are the idempotency mechanism, does the new execution inherit the same idempotency constraints?
• How do unique tags behave across the continuation boundary?

Just want to make sure we're designing with this use case in mind, even if continueAsNew comes later.

---

To be clear: I'm not against tags as a feature—they're great for metadata and observability. I'm specifically concerned about them being the primary solution for workflow idempotency and ID management, which feels like a core workflow primitive that deserves first-class API support.

[mikearnaldi]

In complete agreement with @hugo082 I would also add: if there is no standard way to call a workflow idempotently (and given workflows body is always idempotent it makes close to no sense ever calling a workflow non idempotently) any higher order abstraction becomes impossible, think for example an API where the user passes in a workflow and it gets executed behind the scenes, such API would have no knowledge of the specific way idempotency is implemented for the specific workflow.

Overall I think this RFC is very good, just not for idempotency of workflow invocation.

[Schniz]

Should we call it annotations instead of tags, like OpenTelemetry does?

[tobiaslins]

@Schniz I think attributes are even better?

[ethshea]

+1 annotations are just the java way of adding attributes

[Schniz]

:claude-thinking: You're absolutely right!

Attributes it is! sorry for the mixup

[pranaygp]

Yup! Aligned on attributes! will update the RFC :)

[1042422227]

Gracias