> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modulex.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Error handling & retries

> How node failures surface as error events, how to configure per-node retries with exponential backoff, how to debug a failed run, and the common causes of node errors in ModuleX workflows.

export const MediaEmbed = ({id, type = 'screenshot', caption = '', ext, ratio = '16 / 9'}) => {
  const isVideo = type === 'video' || type === 'app_video';
  const resolvedExt = ext || (isVideo ? 'mp4' : type === 'screenshot' ? 'webp' : 'svg');
  const src = 'https://media.modulex.dev/' + id + '.' + resolvedExt;
  const [status, setStatus] = useState('loading');
  const [isDev, setIsDev] = useState(false);
  const [inView, setInView] = useState(false);
  const boxRef = useRef(null);
  useEffect(() => {
    if (typeof window === 'undefined') return;
    const h = window.location.hostname;
    setIsDev(h === 'localhost' || h === '127.0.0.1' || h.endsWith('.mintlify.app'));
  }, []);
  useEffect(() => {
    if (inView) return;
    if (typeof IntersectionObserver === 'undefined') {
      setInView(true);
      return;
    }
    const el = boxRef.current;
    if (!el) return;
    const io = new IntersectionObserver(entries => {
      if (entries.some(e => e.isIntersecting)) {
        setInView(true);
        io.disconnect();
      }
    }, {
      rootMargin: '300px'
    });
    io.observe(el);
    return () => io.disconnect();
  }, [inView]);
  if (status === 'missing') {
    if (!isDev) return null;
    return <div style={{
      display: 'flex',
      flexDirection: 'column',
      justifyContent: 'center',
      gap: '0.4rem',
      padding: '1rem 1.25rem',
      margin: '1.25rem 0',
      width: '100%',
      aspectRatio: ratio,
      boxSizing: 'border-box',
      border: '1px dashed rgba(128,128,128,0.45)',
      borderRadius: '0.75rem',
      background: 'rgba(128,128,128,0.06)',
      color: 'currentColor',
      fontSize: '0.85rem',
      lineHeight: 1.45
    }}>
        <div style={{
      display: 'flex',
      alignItems: 'center',
      gap: '0.5rem',
      opacity: 0.75
    }}>
          <span aria-hidden="true">🎬</span>
          <code style={{
      fontSize: '0.75rem'
    }}>{id}</code>
          <span style={{
      fontSize: '0.65rem',
      textTransform: 'uppercase',
      letterSpacing: '0.04em',
      padding: '0.1rem 0.4rem',
      borderRadius: '0.4rem',
      background: 'rgba(128,128,128,0.18)'
    }}>
            {type}
          </span>
        </div>
        <div style={{
      opacity: 0.9
    }}>{caption || 'Media not uploaded yet.'}</div>
        <div style={{
      fontSize: '0.7rem',
      opacity: 0.5
    }}>
          Upload to R2 as <code>{id}.{resolvedExt}</code> — preview only, hidden in production.
        </div>
      </div>;
  }
  const mediaStyle = {
    display: status === 'loaded' ? 'block' : 'none',
    width: '100%',
    height: 'auto',
    borderRadius: '0.75rem'
  };
  const media = isVideo ? <video src={inView ? src : undefined} autoPlay loop muted playsInline preload="metadata" onLoadedData={() => setStatus('loaded')} onError={() => setStatus('missing')} style={mediaStyle} /> : <img src={inView ? src : undefined} alt={caption} onLoad={() => setStatus('loaded')} onError={() => setStatus('missing')} style={mediaStyle} />;
  return <figure style={{
    margin: '1.25rem 0'
  }}>
      <div ref={boxRef} style={status === 'loaded' ? {
    width: '100%'
  } : {
    width: '100%',
    aspectRatio: ratio,
    borderRadius: '0.75rem',
    background: 'rgba(128,128,128,0.06)'
  }}>
        {media}
      </div>
      {status === 'loaded' && caption ? <figcaption style={{
    marginTop: '0.5rem',
    textAlign: 'center',
    fontSize: '0.85rem',
    opacity: 0.7
  }}>
          {caption}
        </figcaption> : null}
    </figure>;
};

A node can fail for many reasons — a model timeout, an unreachable HTTP endpoint, an
exhausted credit allowance, or a bug in a reference like `{{node_id.field}}`. ModuleX wraps
most node types in a retry-and-events layer that retries transient failures with exponential
backoff, streams every attempt to the run's event stream, and stops the run with a typed
error when retries are exhausted. This page covers the retry contract, every error event,
the credit impact of a failed run, and how to debug one.

For how a run streams in general, see [Running workflows](/workflow-builder/execution/running).
For documented gaps you should not rely on, see [Known limitations](/reference/known-limitations).

## How a node failure surfaces

Every retry-wrapped node emits the same three event types to the run's stream, observed over
[SSE run streaming](/realtime/sse-streaming) on `GET /workflows/listen/{run_id}`. These are
the raw wire shapes the executor publishes — note the field is `node` (the node id), not
`node_id`.

<Steps>
  <Step title="node_started">
    Emitted once, the moment the node begins, before any attempt:
    `{type, node, name, timestamp}`.
  </Step>

  <Step title="node_retry (zero or more)">
    Emitted after a failed attempt that will be retried — that is, the error type matched
    the node's retry list and this was not the final attempt. Carries the attempt counter and
    the computed backoff delay: `{type, node, name, attempt, max_attempts, error_type, error_message, next_retry_in, timestamp}`.
  </Step>

  <Step title="node_error (terminal for the node)">
    Emitted when retries are exhausted or the error is not retryable. The node then re-raises,
    which stops the whole run: `{type, node, name, error_type, error_message, reason, attempt, max_attempts, recoverable, timestamp}`.
  </Step>
</Steps>

When a node re-raises, the executor catches it, publishes a single run-level `error` event,
and sets the run status to `failed` (see [Run-level failure](#run-level-failure)).

<Note>
  Two node types are intentionally **not** retry-wrapped. The [interrupt node](/workflow-builder/nodes/interrupt)
  is never retried — it pauses for a human and must not auto-retry. The
  [function node](/workflow-builder/nodes/function) typically converts a soft failure into a
  result value rather than raising, so it does not trigger the retry path (see
  [Common causes](#common-causes)). All other node types — `llm`, `tool`, `agent`,
  `conditional`, `transformer`, `guardrails`, `knowledge` — are wrapped, though the pure
  in-memory ones (`transformer`, `guardrails`, `conditional`) rarely raise a retryable error
  because they make no external calls.
</Note>

<MediaEmbed id="MX-MEDIA-3050" type="app_video" caption={"A workflow run with a failing node, watched live in the builder."} />

## Retry configuration

Each node carries an optional `retry_config` on its `NodeDefinition`. If you omit it, the
engine applies a built-in default identical to the values below. Set it per node in the detail
panel, or directly on the node JSON when authoring through the API.

<ParamField path="retry_config" type="object" optional>
  Per-node retry behavior. When omitted, defaults to 3 attempts (2 retries) with the standard
  transient-error list below.
</ParamField>

<ParamField path="retry_config.max_attempts" type="integer" default="3">
  Total attempts including the first, so `1` means no retry and `3` means 2 retries. Range
  `1`-`10`. Out-of-range values are rejected at validation time.
</ParamField>

<ParamField path="retry_config.initial_interval" type="number" default="1.0">
  Delay in seconds before the first retry. Range `0.1`-`60.0`.
</ParamField>

<ParamField path="retry_config.backoff_factor" type="number" default="2.0">
  Multiplier applied to the delay on each subsequent retry (exponential backoff). Range
  `1.0`-`5.0`. A factor of `1.0` makes the delay constant.
</ParamField>

<ParamField path="retry_config.retry_on_error_types" type="string[]" default={`["TimeoutError", "ConnectionError", "HTTPError"]`}>
  The error types that trigger a retry. Any error whose type is **not** in this list fails
  immediately with no retry, regardless of `max_attempts`.
</ParamField>

### How the backoff delay is computed

The delay before the retry following attempt `n` is:

```text theme={null}
delay = initial_interval * (backoff_factor ^ (attempt - 1))
```

With the defaults (`initial_interval=1.0`, `backoff_factor=2.0`, `max_attempts=3`), a node
that keeps failing on a retryable error produces this sequence:

| Attempt | Outcome           | Event                       | Wait before next attempt |
| ------- | ----------------- | --------------------------- | ------------------------ |
| 1       | fails (retryable) | `node_retry` (`attempt: 1`) | `1.0s` (`= 1.0 * 2^0`)   |
| 2       | fails (retryable) | `node_retry` (`attempt: 2`) | `2.0s` (`= 1.0 * 2^1`)   |
| 3       | fails (final)     | `node_error` (`attempt: 3`) | — (re-raises, run fails) |

The `next_retry_in` field on each `node_retry` event carries this computed delay in seconds.

### How an error is matched to the retry list

An error is retried only if both are true: it is **not** the final attempt, and the error
matches `retry_on_error_types`. Matching is by exact exception class name, plus a substring
fallback on the error message for the three default types:

* `TimeoutError` also matches any error whose message contains `timeout`.
* `ConnectionError` also matches any error whose message contains `connect`.
* `HTTPError` also matches any error whose message contains `http`.

Custom type names in `retry_on_error_types` are matched by exact class name only. A
[billing denial](#credit-impact-of-a-failed-run) raised mid-run is **not** in the default
list, so it fails fast without retrying — retrying an exhausted credit allowance would not
help.

### A worked example: a tool node with custom retries

This `tool` node calls an integration action and retries up to four times, starting at half a
second and tripling the delay each time, on connection and rate-limit failures.

```json theme={null}
{
  "id": "node_send_slack",
  "type": "tool",
  "name": "Post to Slack",
  "x": 480,
  "y": 200,
  "retry_config": {
    "max_attempts": 4,
    "initial_interval": 0.5,
    "backoff_factor": 3.0,
    "retry_on_error_types": ["ConnectionError", "HTTPError", "RateLimitError"]
  },
  "tool_config": {
    "tool": {
      "integration_name": "slack",
      "service_name": "chat_post_message",
      "credential_id": "cred_2ab9f1"
    },
    "input_mapping": {
      "channel": "#alerts",
      "text": "{{node_summarize.summary}}"
    }
  }
}
```

With `backoff_factor: 3.0` and `initial_interval: 0.5`, the waits between attempts are
`0.5s`, `1.5s`, then `4.5s`. After the fourth failed attempt the node emits `node_error` and
the run fails.

## Error events on the wire

These are the exact payloads published by the engine, in publish order. They carry `node`
(the node id) and a flat structure — they are the live wire shapes, not the typed event
models. See [SSE run streaming](/realtime/sse-streaming) for framing (`data: <json>\n\n`,
no `event:` line; switch on the JSON `type`).

<CodeGroup>
  ```json node_started theme={null}
  {
    "type": "node_started",
    "node": "node_send_slack",
    "name": "Post to Slack",
    "timestamp": 1750420800.123
  }
  ```

  ```json node_retry theme={null}
  {
    "type": "node_retry",
    "node": "node_send_slack",
    "name": "Post to Slack",
    "attempt": 1,
    "max_attempts": 4,
    "error_type": "ConnectionError",
    "error_message": "Failed to connect to slack.com",
    "next_retry_in": 0.5,
    "timestamp": 1750420800.456
  }
  ```

  ```json node_error theme={null}
  {
    "type": "node_error",
    "node": "node_send_slack",
    "name": "Post to Slack",
    "error_type": "ConnectionError",
    "error_message": "Failed to connect to slack.com",
    "reason": null,
    "attempt": 4,
    "max_attempts": 4,
    "recoverable": false,
    "timestamp": 1750420810.789
  }
  ```
</CodeGroup>

<ResponseField name="node" type="string">
  The id of the node that emitted the event (the field is `node`, not `node_id`).
</ResponseField>

<ResponseField name="name" type="string">
  The node's human-readable name, or the node id if no name is set.
</ResponseField>

<ResponseField name="attempt" type="integer">
  The 1-based attempt number this event describes.
</ResponseField>

<ResponseField name="max_attempts" type="integer">
  The node's effective `max_attempts` (from `retry_config`, or `3` by default).
</ResponseField>

<ResponseField name="error_type" type="string">
  The exception class name, for example `TimeoutError` or `ValueError`.
</ResponseField>

<ResponseField name="error_message" type="string">
  The error's string form.
</ResponseField>

<ResponseField name="next_retry_in" type="number">
  On `node_retry` only: the computed backoff delay in seconds before the next attempt.
</ResponseField>

<ResponseField name="reason" type="string | null">
  On `node_error` only: a stable machine token taken from the exception's `code` attribute
  when present, for example `credit_exhausted` for a mid-run budget stop. It is `null` for
  errors that define no code, in which case match on `error_type` and `error_message` instead.
</ResponseField>

<ResponseField name="recoverable" type="boolean">
  On `node_error` only: always `false`. A `node_error` is terminal for the node — it re-raises
  and the run fails. There is no in-engine "continue past a failed node" path.
</ResponseField>

## Run-level failure

When a node exhausts its retries (or raises a non-retryable error), it re-raises. The
executor catches the exception, publishes a single run-level `error` event, and sets the run
status to `failed`. The `error` event is flat with just a `message`:

```json error theme={null}
{
  "type": "error",
  "message": "Execution failed: Failed to connect to slack.com"
}
```

`error` is a [terminal event](/realtime/sse-streaming): the listen stream closes after it.

<Warning>
  The wire `error` event is **not** the typed `WorkflowErrorEvent` model. The runtime publishes
  the flat `{type, message}` shape above, where `message` is the prefixed exception string
  (`"Execution failed: ..."`). The earlier per-node `node_error` event carries the structured
  fields (`error_type`, `reason`, `attempt`); the run-level `error` event does not. To attribute
  a run failure to a specific node, read the last `node_error` before the `error` frame.
</Warning>

The durable run record reflects this too: the in-memory run status is `failed`, and the run's
`status` is also `failed`, with `error_message` populated. (Note the status
naming split: a successful run streams a `done` event but its durable status is `succeeded`.)
You can read the durable record back via `GET /workflow-runs/{run_pk}`.

A run can also end without a `node_error` in two other ways:

* **Timeout** — a run that exceeds the workflow execution timeout publishes an `error` event
  carrying `error_type: "timeout"` and sets status `failed`.
* **Cancellation** — a `POST /workflows/cancel/{run_id}` request sets a flag the executor
  checks between nodes; the run publishes a `cancelled` event and sets status `cancelled`.
  This is graceful — the current node finishes first.

## Credit impact of a failed run

Understanding what a failed run costs is the most common error-handling question.

<Note>
  A run is charged **one** [run credit](/billing/credits) at admission, on the
  [`POST /workflows/run`](/workflow-builder/execution/api-endpoint) call, **before** the
  background task starts. That charge is **not** refunded if the workflow later fails. A run
  that fails at the third node still costs its run credit. Resuming an interrupted run reuses the
  same `run_id` reservation, so a resume does **not** add a second run credit.
</Note>

Beyond the flat run credit:

* **Retries do not add run credits.** The single run credit covers the whole logical run,
  including every retry of every node.
* **LLM and agent calls meter tokens per call.** Each attempt of an [LLM node](/workflow-builder/nodes/llm)
  or [agent node](/workflow-builder/nodes/agent) that actually reaches the model records token
  usage in credits. A node that fails after its first model call and then retries will meter
  the tokens of each attempt that reached the model. Token metering failures never fail the
  run — usage logging is best-effort.
* **Managed knowledge retrieval is gated.** A [knowledge node](/workflow-builder/nodes/knowledge)
  using managed `modulexdb` reserves a retrieval credit before embedding and records the cost on
  success; on error it releases the reservation. The managed-retrieval gate is best-effort
  inside a run — a billing hiccup releases the reservation rather than crashing the workflow.
* **BYOK is not credited.** Bring-your-own-key model and knowledge usage is billed directly by
  your provider, not in ModuleX credits.

### Billing denials mid-run

The [billing gate](/billing/usage-gating) is live on the run surface. If your org's credit
allowance is exhausted (and no wallet overage is available), the admission gate on
`POST /workflows/run` rejects the run **before** it starts with a flat `DenialEnvelope`
([402 / 403 / 429](/api-reference/errors)) — no run record is created and no credit is charged.

If a budget limit is hit **mid-run** (for example by an LLM node's token usage), the failing
node surfaces a `node_error` whose `reason` is the stable denial token (such as
`credit_exhausted`), so a client can branch on it deterministically rather than scanning the
free-text message. See [Usage gating & limits](/billing/usage-gating) and the full envelope
reference on the [Errors page](/api-reference/errors).

## Debugging a failed node

<Steps>
  <Step title="Find the last node_error before the run error">
    Stream the run with `GET /workflows/listen/{run_id}` and read the events in order. The last
    `node_error` frame names the node (`node`), the exception (`error_type`), the message
    (`error_message`), and — for coded failures — the `reason` token. The subsequent run-level
    `error` frame only restates the message.
  </Step>

  <Step title="Inspect the checkpoint state">
    Read the checkpoint with `GET /workflows/state/{thread_id}` to see the state as
    it stood when the run stopped. Every node writes its result to state under its own id, so
    you can confirm which upstream nodes produced values and which reference a failing node
    expected. An unresolved `{{node_id.field}}` reference is left intact as a literal string in
    state rather than raising — a tell-tale sign of a typo in a reference.
  </Step>

  <Step title="Reproduce with a tighter retry budget">
    Set the node's `retry_config.max_attempts` to `1` temporarily to fail fast and surface the
    underlying error immediately without waiting through backoff delays. Restore the retry
    budget once you have identified the cause.
  </Step>

  <Step title="Check the durable run record">
    `GET /workflow-runs/{run_pk}` returns the persisted run with `status: "failed"`,
    `error_message`, `started_at`/`completed_at`, and the `input_snapshot` — useful when you
    are debugging after the SSE stream has closed.
  </Step>
</Steps>

<MediaEmbed id="MX-MEDIA-3051" type="screenshot" caption={"The detail panel showing a node's retry settings."} />

## Common causes

<AccordionGroup>
  <Accordion title="Model timeout or rate limit (llm / agent nodes)">
    A slow or rate-limited model call raises a timeout or HTTP error. These match the default
    retry list (`TimeoutError`, `HTTPError`), so they retry automatically with backoff. If the
    provider stays unavailable across all attempts, the node emits `node_error`. Raise
    `max_attempts` or `initial_interval` for flaky providers, or switch the
    [LLM provider](/integrations/llm-providers/overview).
  </Accordion>

  <Accordion title="Unreachable endpoint (tool nodes / http_request function)">
    A connection failure to an integration or HTTP endpoint raises a connection or HTTP error,
    both retryable by default. Note the difference between node types: a [tool node](/workflow-builder/nodes/tool)
    that raises is retried and can fail the run, whereas the `http_request`
    [function](/workflow-builder/nodes/function) returns a result with `success: false` instead
    of raising — see the next item.
  </Accordion>

  <Accordion title="Soft failures from function nodes (no retry, no run failure)">
    A [function node](/workflow-builder/nodes/function) does not raise on a logical failure.
    Instead it returns `{error, success: false, ...}` under the node id, so the run continues
    and the next node can branch on it. Because nothing is raised, the retry layer is never
    triggered. If you need a failing HTTP call to retry, use a `tool` node, or branch on the
    function result with a [conditional node](/workflow-builder/nodes/conditional).
  </Accordion>

  <Accordion title="Unresolved node references">
    A reference such as `{{node_id.field}}` that points at a missing node, a wrong field, or an
    out-of-range array index resolves to `None` (for nested paths) or is left intact as the
    literal string (for whole-template strings) rather than raising. The symptom is wrong or
    empty input downstream, not an error event. Verify references on
    [Variables & references](/workflow-builder/variables-and-references) and inspect state at
    the checkpoint.
  </Accordion>

  <Accordion title="Guardrails blocking content">
    A [guardrails node](/workflow-builder/nodes/guardrails) with `on_failure: block` stops the
    flow when validation fails. This is expected behavior, not a node exception — the node
    returns a result with `valid: false` and `blocked`. Use `on_failure: route` with a
    `failure_route` to send blocked content down a recovery branch instead of stopping.
  </Accordion>

  <Accordion title="Credit allowance exhausted">
    The run is rejected at admission (no run created, no charge) with a `DenialEnvelope`, or a
    mid-run node emits `node_error` with `reason: "credit_exhausted"`. Top up your
    [wallet](/billing/wallet) or upgrade your [plan](/billing/plans); see
    [Usage gating & limits](/billing/usage-gating).
  </Accordion>

  <Accordion title="Malformed workflow schema (run never starts)">
    A malformed `workflow_schema` does not surface as a node error — it fails the create/update
    or run call itself. Because the request body is an untyped dict, a schema validation failure
    surfaces as a **500** (not the usual 422), with a descriptive `detail`. Validate your graph
    before running; the `__start__` and `__end__` nodes are virtual and must not appear in the
    `nodes` array.
  </Accordion>
</AccordionGroup>

## Related

<CardGroup cols={2}>
  <Card title="Running workflows" icon="play" href="/workflow-builder/execution/running">
    Run a workflow from the builder and observe the live event stream.
  </Card>

  <Card title="Known limitations" icon="triangle-exclamation" href="/reference/known-limitations">
    Documented gaps and broken paths you should not rely on.
  </Card>

  <Card title="Variables & references" icon="brackets-curly" href="/workflow-builder/variables-and-references">
    The `{{node_id.field}}` reference system and how unresolved references behave.
  </Card>

  <Card title="Errors & status codes" icon="circle-exclamation" href="/api-reference/errors">
    The full error-envelope and HTTP status reference, including the DenialEnvelope.
  </Card>

  <Card title="Usage gating & limits" icon="shield-halved" href="/billing/usage-gating">
    The billing admission gate and its 402 / 403 / 429 responses.
  </Card>

  <Card title="SSE run streaming" icon="tower-broadcast" href="/realtime/sse-streaming">
    The run event stream that carries node\_started, node\_retry, and node\_error.
  </Card>
</CardGroup>