Skip to main content
A node can fail for many reasons — a model timeout, an unreachable HTTP endpoint, an exhausted credit allowance, or a bug in a reference like {{node_id.field}}. ModuleX wraps most node types in a retry-and-events layer that retries transient failures with exponential backoff, streams every attempt to the run’s event stream, and stops the run with a typed error when retries are exhausted. This page covers the retry contract, every error event, the credit impact of a failed run, and how to debug one. For how a run streams in general, see Running workflows. For documented gaps you should not rely on, see Known limitations.

​
How a node failure surfaces

Every retry-wrapped node emits the same three event types to the run’s stream, observed over SSE run streaming on GET /workflows/listen/{run_id}. These are the raw wire shapes the executor publishes — note the field is node (the node id), not node_id.
1

node_started

Emitted once, the moment the node begins, before any attempt: {type, node, name, timestamp}.
2

node_retry (zero or more)

Emitted after a failed attempt that will be retried — that is, the error type matched the node’s retry list and this was not the final attempt. Carries the attempt counter and the computed backoff delay: {type, node, name, attempt, max_attempts, error_type, error_message, next_retry_in, timestamp}.
3

node_error (terminal for the node)

Emitted when retries are exhausted or the error is not retryable. The node then re-raises, which stops the whole run: {type, node, name, error_type, error_message, reason, attempt, max_attempts, recoverable, timestamp}.
When a node re-raises, the executor catches it, publishes a single run-level error event, and sets the run status to failed (see Run-level failure).
Two node types are intentionally not retry-wrapped. The interrupt node is never retried — it pauses for a human and must not auto-retry. The function node typically converts a soft failure into a result value rather than raising, so it does not trigger the retry path (see Common causes). All other node types — llm, tool, agent, conditional, transformer, guardrails, knowledge — are wrapped, though the pure in-memory ones (transformer, guardrails, conditional) rarely raise a retryable error because they make no external calls.

​
Retry configuration

Each node carries an optional retry_config on its NodeDefinition. If you omit it, the engine applies a built-in default identical to the values below. Set it per node in the detail panel, or directly on the node JSON when authoring through the API.
​
object
Per-node retry behavior. When omitted, defaults to 3 attempts (2 retries) with the standard transient-error list below.
​
integer
default:"3"
Total attempts including the first, so 1 means no retry and 3 means 2 retries. Range 1-10. Out-of-range values are rejected at validation time.
​
number
default:"1.0"
Delay in seconds before the first retry. Range 0.1-60.0.
​
number
default:"2.0"
Multiplier applied to the delay on each subsequent retry (exponential backoff). Range 1.0-5.0. A factor of 1.0 makes the delay constant.
​
string[]
The error types that trigger a retry. Any error whose type is not in this list fails immediately with no retry, regardless of max_attempts.

​
How the backoff delay is computed

The delay before the retry following attempt n is:
With the defaults (initial_interval=1.0, backoff_factor=2.0, max_attempts=3), a node that keeps failing on a retryable error produces this sequence: The next_retry_in field on each node_retry event carries this computed delay in seconds.

​
How an error is matched to the retry list

An error is retried only if both are true: it is not the final attempt, and the error matches retry_on_error_types. Matching is by exact exception class name, plus a substring fallback on the error message for the three default types:
  • TimeoutError also matches any error whose message contains timeout.
  • ConnectionError also matches any error whose message contains connect.
  • HTTPError also matches any error whose message contains http.
Custom type names in retry_on_error_types are matched by exact class name only. A billing denial raised mid-run is not in the default list, so it fails fast without retrying — retrying an exhausted credit allowance would not help.

​
A worked example: a tool node with custom retries

This tool node calls an integration action and retries up to four times, starting at half a second and tripling the delay each time, on connection and rate-limit failures.
With backoff_factor: 3.0 and initial_interval: 0.5, the waits between attempts are 0.5s, 1.5s, then 4.5s. After the fourth failed attempt the node emits node_error and the run fails.

​
Error events on the wire

These are the exact payloads published by the engine, in publish order. They carry node (the node id) and a flat structure — they are the live wire shapes, not the typed event models. See SSE run streaming for framing (data: <json>\n\n, no event: line; switch on the JSON type).
​
string
The id of the node that emitted the event (the field is node, not node_id).
​
string
The node’s human-readable name, or the node id if no name is set.
​
integer
The 1-based attempt number this event describes.
​
integer
The node’s effective max_attempts (from retry_config, or 3 by default).
​
string
The exception class name, for example TimeoutError or ValueError.
​
string
The error’s string form.
​
number
On node_retry only: the computed backoff delay in seconds before the next attempt.
​
string | null
On node_error only: a stable machine token taken from the exception’s code attribute when present, for example credit_exhausted for a mid-run budget stop. It is null for errors that define no code, in which case match on error_type and error_message instead.
​
boolean
On node_error only: always false. A node_error is terminal for the node — it re-raises and the run fails. There is no in-engine “continue past a failed node” path.

​
Run-level failure

When a node exhausts its retries (or raises a non-retryable error), it re-raises. The executor catches the exception, publishes a single run-level error event, and sets the run status to failed. The error event is flat with just a message:
error
error is a terminal event: the listen stream closes after it.
The wire error event is not the typed WorkflowErrorEvent model. The runtime publishes the flat {type, message} shape above, where message is the prefixed exception string ("Execution failed: ..."). The earlier per-node node_error event carries the structured fields (error_type, reason, attempt); the run-level error event does not. To attribute a run failure to a specific node, read the last node_error before the error frame.
The durable run record reflects this too: the in-memory run status is failed, and the run’s status is also failed, with error_message populated. (Note the status naming split: a successful run streams a done event but its durable status is succeeded.) You can read the durable record back via GET /workflow-runs/{run_pk}. A run can also end without a node_error in two other ways:
  • Timeout — a run that exceeds the workflow execution timeout publishes an error event carrying error_type: "timeout" and sets status failed.
  • Cancellation — a POST /workflows/cancel/{run_id} request sets a flag the executor checks between nodes; the run publishes a cancelled event and sets status cancelled. This is graceful — the current node finishes first.

​
Credit impact of a failed run

Understanding what a failed run costs is the most common error-handling question.
A run is charged one run credit at admission, on the POST /workflows/run call, before the background task starts. That charge is not refunded if the workflow later fails. A run that fails at the third node still costs its run credit. Resuming an interrupted run reuses the same run_id reservation, so a resume does not add a second run credit.
Beyond the flat run credit:
  • Retries do not add run credits. The single run credit covers the whole logical run, including every retry of every node.
  • LLM and agent calls meter tokens per call. Each attempt of an LLM node or agent node that actually reaches the model records token usage in credits. A node that fails after its first model call and then retries will meter the tokens of each attempt that reached the model. Token metering failures never fail the run — usage logging is best-effort.
  • Managed knowledge retrieval is gated. A knowledge node using managed modulexdb reserves a retrieval credit before embedding and records the cost on success; on error it releases the reservation. The managed-retrieval gate is best-effort inside a run — a billing hiccup releases the reservation rather than crashing the workflow.
  • BYOK is not credited. Bring-your-own-key model and knowledge usage is billed directly by your provider, not in ModuleX credits.

​
Billing denials mid-run

The billing gate is live on the run surface. If your org’s credit allowance is exhausted (and no wallet overage is available), the admission gate on POST /workflows/run rejects the run before it starts with a flat DenialEnvelope (402 / 403 / 429) — no run record is created and no credit is charged. If a budget limit is hit mid-run (for example by an LLM node’s token usage), the failing node surfaces a node_error whose reason is the stable denial token (such as credit_exhausted), so a client can branch on it deterministically rather than scanning the free-text message. See Usage gating & limits and the full envelope reference on the Errors page.

​
Debugging a failed node

1

Find the last node_error before the run error

Stream the run with GET /workflows/listen/{run_id} and read the events in order. The last node_error frame names the node (node), the exception (error_type), the message (error_message), and — for coded failures — the reason token. The subsequent run-level error frame only restates the message.
2

Inspect the checkpoint state

Read the checkpoint with GET /workflows/state/{thread_id} to see the state as it stood when the run stopped. Every node writes its result to state under its own id, so you can confirm which upstream nodes produced values and which reference a failing node expected. An unresolved {{node_id.field}} reference is left intact as a literal string in state rather than raising — a tell-tale sign of a typo in a reference.
3

Reproduce with a tighter retry budget

Set the node’s retry_config.max_attempts to 1 temporarily to fail fast and surface the underlying error immediately without waiting through backoff delays. Restore the retry budget once you have identified the cause.
4

Check the durable run record

GET /workflow-runs/{run_pk} returns the persisted run with status: "failed", error_message, started_at/completed_at, and the input_snapshot — useful when you are debugging after the SSE stream has closed.

​
Common causes

A slow or rate-limited model call raises a timeout or HTTP error. These match the default retry list (TimeoutError, HTTPError), so they retry automatically with backoff. If the provider stays unavailable across all attempts, the node emits node_error. Raise max_attempts or initial_interval for flaky providers, or switch the LLM provider.
A connection failure to an integration or HTTP endpoint raises a connection or HTTP error, both retryable by default. Note the difference between node types: a tool node that raises is retried and can fail the run, whereas the http_request function returns a result with success: false instead of raising — see the next item.
A function node does not raise on a logical failure. Instead it returns {error, success: false, ...} under the node id, so the run continues and the next node can branch on it. Because nothing is raised, the retry layer is never triggered. If you need a failing HTTP call to retry, use a tool node, or branch on the function result with a conditional node.
A reference such as {{node_id.field}} that points at a missing node, a wrong field, or an out-of-range array index resolves to None (for nested paths) or is left intact as the literal string (for whole-template strings) rather than raising. The symptom is wrong or empty input downstream, not an error event. Verify references on Variables & references and inspect state at the checkpoint.
A guardrails node with on_failure: block stops the flow when validation fails. This is expected behavior, not a node exception — the node returns a result with valid: false and blocked. Use on_failure: route with a failure_route to send blocked content down a recovery branch instead of stopping.
The run is rejected at admission (no run created, no charge) with a DenialEnvelope, or a mid-run node emits node_error with reason: "credit_exhausted". Top up your wallet or upgrade your plan; see Usage gating & limits.
A malformed workflow_schema does not surface as a node error — it fails the create/update or run call itself. Because the request body is an untyped dict, a schema validation failure surfaces as a 500 (not the usual 422), with a descriptive detail. Validate your graph before running; the __start__ and __end__ nodes are virtual and must not appear in the nodes array.

Running workflows

Run a workflow from the builder and observe the live event stream.

Known limitations

Documented gaps and broken paths you should not rely on.

Variables & references

The {{node_id.field}} reference system and how unresolved references behave.

Errors & status codes

The full error-envelope and HTTP status reference, including the DenialEnvelope.

Usage gating & limits

The billing admission gate and its 402 / 403 / 429 responses.

SSE run streaming

The run event stream that carries node_started, node_retry, and node_error.