Skip to main content
This page is a tuning reference for running ModuleX well at scale. It covers four levers you can pull independently — cost (credits), latency, reliability, and the bring-your-own-key (BYOK) trade-offs that cut across all three. Every figure here is a real product constant, not a recommendation in disguise; where a value is unenforced or unconfirmed it is called out explicitly. For the exact credit unit and per-operation cost table, see Credits & metering. For per-node retry configuration and the failure event stream, see Error handling & retries. For the admission gate that turns an empty allowance into a 402 / 403 / 429, see Usage gating & limits.

How the four levers interact

The levers are not independent in practice. Choices that cut cost often also cut latency (fewer managed model calls means fewer billed token batches and fewer network round-trips), and choices that improve reliability — more retries, more guardrails — can add both cost and latency. The table below is the trade-off map this page works through.

Cost optimization (credits)

Only managed usage is metered in credits. An operation is managed when it runs through ModuleX’s own provider pool — its integration name is modulexai (managed model and tool calls) or modulexdb (managed knowledge), or its API-key source is the platform environment. Everything that runs on your own provider keys is BYOK and is never credited (see BYOK trade-offs). So the first cost question is always: which calls in this workflow are managed?

The cost surface, ranked

These are the metered operations and their charges, from Credits & metering. Optimize the rows that dominate your usage, not the cheap flat fees.
number
default:"1.05"
The system margin (+5%) applied to every token-metered LLM cost. It does not apply to the flat per-operation fees (run, retrieval, file_ingest, tool base).
integer
default:"1"
The flat credit cost of a managed integration tool action, anchored to $0.01. Multiplied by the provider pool multiplier (default 1.0).

Tactics that actually move the number

1

Pick the smallest model that passes your guardrails

Token cost is the dominant term for most workflows. Because the formula is linear in prompt and completion tokens, halving prompt size or moving from a large to a small managed model is a near-proportional credit cut. Validate quality with a guardrails node rather than reaching for the largest model by default.
2

Trim prompt and completion tokens

The unknown-model rate falls back to 0, never a silent 1.0, so cost tracks real per-model pricing rates. Shorten system prompts, cap completion length, and avoid re-sending large context on every node. For embeddings, completion tokens are forced to 0 (input-only), so only the text you embed is billed.
3

Reduce tool calls inside loops

Each managed tool action is a flat 1 × multiplier credits. A conditional node that loops a tool call N times costs N credits before tokens. Batch where the integration supports it, or gate the loop behind a condition.
4

Cache and deduplicate retrieval and ingest

Document ingest is idempotent on the document ID, so a retried or repeated ingest of the same document is not charged twice. Retrieval (1 credit) is charged per call — avoid issuing one retrieval per item in a loop when a single knowledge node search would serve the whole batch.
5

Charge once per turn, not per attempt

A run or agent turn is charged exactly once (RUN_CREDIT = 1), and a resumed turn re-enters through the resume path with source="existing", which is a no-op for charging. Retries within a node (see Reliability) do not add run credits; they may add tool or token credits if the retried call is itself managed.

Watch the allowance and the overage floor

Each plan grants a monthly credit allowance (Pro 5,000 / Max 20,000; Free 300 one-time; Enterprise custom). When the allowance is exhausted, a paid org with overage enabled spends down a prepaid wallet; a paid org without overage, or any Free org, is denied. The overage charge for a bucket is reconciled to exactly max(0, month_sum − floor) credits, where floor = max(plan_limit, overage_baseline_credits) — so enabling overage mid-month never retroactively bills the spend that preceded the toggle.
Suspended organizations (after dunning: unpaid, or past_due past its grace window) are denied with a credit_exhausted (402) and do not fall back to the Free tier. Resolve billing before relying on managed runs. See Usage gating & limits.

Latency

ModuleX runs workflows in the background and streams progress over Server-Sent Events. Understanding that path is the key to reasoning about latency.

The run path

A run does not block the HTTP response while it executes. POST /workflows/run performs the admission gate, mints a run_id, enqueues a background task, and returns immediately with {run_id, status, ...}. You then attach to the live stream with GET /workflows/listen/{run_id} over SSE. The same run_id is reused across a resume, so you can re-listen after a pause.
Response fields stay snake_case on the wire in both SDKs (run_id, created_at). The JS SDK converts request keys camelCase to snake_case but does not convert responses back, so read run.run_id, not run.runId.

Latency tactics

The run starts in the background the moment POST /workflows/run returns. Attach to GET /workflows/listen/{run_id} immediately so you observe the first node as it runs. The stream replays the run’s buffered history on connect, so a late or reconnecting listener still receives earlier events in order. A {"type":"heartbeat"} keepalive is injected every 15s of silence — treat it as liveness, not as a workflow event, and do not let an idle-connection timeout shorter than 15s drop the stream.
The terminal events are done, error, and cancelled. Stop reading when you see one rather than polling run status in a loop — both SDKs already break iteration on terminal events. Polling adds latency and round-trips without surfacing progress any sooner.
A managed model or tool call runs inside ModuleX’s provider pool. A BYOK call adds the round-trip to your own provider plus any credential resolution. If tail latency matters more than the provider bill, the managed path is usually the shorter network path; if you need a specific provider region or a model only you can reach, BYOK is the trade (see BYOK trade-offs).
When a run is wallet-funded, the run credit is committed first and the wallet-overage reconciliation runs as a background task (defer_overage_reconcile=True), so billing bookkeeping does not sit on the run’s hot path. You do not configure this — it is the default behavior — but it is why a wallet-funded run is not slower than a plan-funded one.
A failing node waits initial_interval · backoff_factor^(attempt-1) seconds between attempts. With the defaults (1.0s start, 2.0× factor, 3 attempts) a node that keeps failing adds 1.0s + 2.0s = 3.0s of pure waiting before it gives up. If a downstream service recovers quickly, lower initial_interval; if it recovers slowly, fewer attempts with a larger interval can finish sooner than many short ones. See Error handling & retries.
Token streaming is not available on the workflow /run path. The stream field on the run request is accepted and echoed back but token-level streaming was removed; you receive node-level events (node_started, node_update, node_error, …), not per-token deltas. Do not design a UI that assumes token streaming from a workflow run.

Reliability

Reliability is mostly a matter of configuring retries deliberately and failing fast on errors that retrying cannot fix.

Retry the transient, fail fast on the rest

Each node carries an optional retry_config. Omit it and the engine applies a built-in default. The full contract lives in Error handling & retries; the parameters that matter for tuning are below.
integer
default:"3"
Total attempts including the first, so 1 is no retry and 3 is two retries. Range 1-10. More attempts improve recovery but add latency and, for managed calls, credits.
number
default:"1.0"
Seconds before the first retry. Range 0.1-60.0. Lower it for fast-recovering services; raise it for rate-limited upstreams that need a cooldown.
number
default:"2.0"
Exponential multiplier applied to the delay on each subsequent retry. Range 1.0-5.0. A factor of 1.0 makes the delay constant.
string[]
Error types that trigger a retry. Any error whose type is not in this list fails immediately. A billing denial is not in the default list, so an exhausted credit allowance fails fast — retrying it would not help.
Two node types are intentionally not retry-wrapped. The interrupt node must never auto-retry — it pauses for a human. The function node usually converts a soft failure into a result value rather than raising, so it does not enter the retry path. Plan reliability around these two explicitly.

Reliability tactics

1

Add only the error types you can actually recover from

Adding RateLimitError to a tool node’s retry_on_error_types lets it ride out a brief upstream 429. Adding a non-recoverable type (an authentication failure, a malformed reference) just wastes attempts and latency. Match the list to the failures the node really sees.
2

Validate outputs with a guardrails node

A guardrails node can block, warn, transform, or route on a failed JSON-schema, regex, or PII check. Putting one after a managed LLM call catches a bad output before it propagates — cheaper and more reliable than letting a downstream node fail on malformed input.
3

Pause for a human on irreversible steps

For sensitive or one-way actions, gate the step behind an interrupt node (HITL). A run pauses, you approve or answer over HITL resume, and the same run_id continues. This trades latency and a human in the loop for the certainty that an irreversible action was confirmed.
4

Design retries to be idempotent at the SDK boundary

The SDKs retry only transient statuses (429, 500, 502, 503). The Python SDK retries GET/HEAD only; the JS SDK retries all methods. A POST /workflows/run that is network-retried could therefore start two runs from JS. Make your callers idempotent (for example, key the trigger on your own request id), because the server-side Idempotency-Key plumbing does not deduplicate runs — see the caveat below.
Idempotency-Key does not deduplicate runs. The Python SDK can attach an Idempotency-Key request header when you pass idempotency_key= to a mutating call, but POST /workflows/run mints its own internal key and reads no client-supplied Idempotency-Key header — so it is a no-op for run deduplication. The JS SDK never sends the header at all. Do not rely on it to prevent duplicate runs; enforce idempotency in your own caller. See Errors & retries.

Stay under the rate limits

Managed runs are admission-gated. The sync_exec run-rate class is consumed only when the call authenticates with an API key (auth_method == "api_key"); when that class is exceeded the gate returns a rate denial. The per-plan sync_exec limits are 10 runs/min (Free), 150 (Pro), 500 (Max), and unlimited (Enterprise). A 429 carries Retry-After and X-RateLimit-Limit / -Remaining / -Reset headers; both SDKs honor Retry-After for backoff. Smooth your call rate against these limits rather than bursting and absorbing 429s.
Treat the synchronous sync_exec limits above as the live ceiling, and design your throughput around those.
For the full status taxonomy and every error-envelope shape, see Errors & status codes and Rate limiting.

BYOK trade-offs

BYOK (bring your own key) means a model, tool, or knowledge call runs on your provider account instead of ModuleX’s managed pool. This is the single biggest cost-versus-control decision in the platform, and it cuts across all three levers above.

What changes when you go BYOK

trade-off
BYOK usage is never credited — it is analytics-only on ModuleX’s side and billed directly by your upstream provider with no ModuleX markup. You remove credit spend but take on the provider’s bill and its pricing model.
trade-off
A BYOK call adds the network round-trip to your provider plus credential resolution. A managed call runs inside ModuleX’s pool. BYOK can be faster (if your provider is closer or the model is only reachable by you) or slower — measure it for your region and model.
trade-off
With BYOK you own the upstream SLA, quota, and rate limits. ModuleX’s gate no longer meters the call, so a provider-side outage or quota exhaustion surfaces as that provider’s error inside the node — configure retry_config accordingly.
trade-off
BYOK keeps usage on your own provider account, which some teams need for data-residency or contractual reasons. The trade is that usage and analytics for BYOK calls are not part of the managed credit ledger.

A decision rule

1

Default to managed for low-volume and prototyping

Managed usage is the simplest path: one credit ledger, one gate, predictable per-operation cost, and the shortest network path inside the pool. Until volume is high enough to matter, the managed path costs less operational overhead than running your own keys.
2

Move high-volume model calls to BYOK when your provider rate is lower

Token-metered managed cost includes the 1.05 margin. If you have a negotiated provider rate or committed-use discount, a high-volume LLM node is often cheaper on BYOK — at the cost of owning that provider’s quota and latency.
3

Use BYOK when you need a specific provider, region, or model

If a model is only available on your account, or data must stay in a particular provider region, BYOK is the only path. There is no per-tier BYOK entitlement — it is available on every plan and is not gated.
4

Keep knowledge managed unless you already run a vector store

Managed knowledge (modulexdb) bills retrieval and ingest at 1 credit each. If you already operate Qdrant, Pinecone, MongoDB Atlas, or Weaviate, a BYOK knowledge provider removes those credits in exchange for running the store yourself.
BYOK is universal and ungated. Every plan can use its own keys; there is no feature.byok entitlement in the backend. The cost trade-off is purely managed-credits versus your provider bill, not a plan upgrade.

A worked tuning pass

Putting the levers together on one workflow — a daily digest that retrieves from a knowledge base, summarizes with a managed model, and posts to Slack:
1

Measure where the credits go

One run (1), one retrieval (1), one managed LLM summarize (token-metered, the dominant cost), and one Slack tool call (1 × multiplier). The LLM node is the row to optimize first.
2

Cut the dominant cost

Shorten the system prompt, cap the completion length, and validate the result with a guardrails node so a smaller managed model is safe to use. This is a near-proportional credit cut with no reliability loss.
3

Harden the external step

The Slack tool node is the only call that leaves the platform, so give it a focused retry_config["ConnectionError", "HTTPError", "RateLimitError"], a short initial_interval, and max_attempts of 3-4. Leave the in-memory nodes on the default.
4

Decide managed vs BYOK on volume

At one run per day, keep everything managed. If the same workflow fans out to thousands of runs, re-evaluate the LLM node for BYOK against your negotiated provider rate, and keep the rest managed.
5

Stay under the rate ceiling at scale

If you trigger via API key, batch starts so you stay under the sync_exec limit for your plan, and back off on Retry-After rather than retrying immediately.

Credits & metering

The credit unit, the per-operation cost table, and the reserve to charge to settle lifecycle.

Error handling & retries

The full per-node retry contract and the failure event stream.

Usage gating & limits

The admission gate and its 402 / 403 / 429 denial responses.

Recipes

Reusable patterns that combine features to solve real problems.