Workers · mcpd

What a worker is

A worker holds the bodies of the claimable steps and asks the daemon for work. It runs where your code runs, with your imports, your connection pool and your credentials, and the daemon never sees any of them. The daemon hands it a lease with the rendered input and the prior steps; the worker runs the body and reports how it went. That is the whole relationship.

Every SDK ships one. It registers the workflows it holds, long-polls for leases, heartbeats while a body runs, and answers every refusal by its reason. The example below is the same program as the one on the Workflows page; this is the part that serves it.

serve the bodies

// Serve step bodies for as long as this process lives. The worker registers
// the definition, long-polls for leases, heartbeats while a body runs, and
// answers every 409 by its reason.
const worker = new Worker(mcpd, { concurrency: 8 }).register(support)
void worker.serve()

# Serve step bodies for as long as this process lives. serve() blocks, so it
# runs on a thread of its own when the process does other work too.
worker = Worker(mcpd, concurrency=8).register(support)
threading.Thread(target=worker.serve, daemon=True).start()

// Serve step bodies for as long as this process lives: register, long-poll
// for leases, heartbeat while a body runs, answer every 409 by its reason.
let worker = Worker::builder(&mcpd).concurrency(8).register(support).build();
let serving = {
    let worker = worker.clone();
    tokio::spawn(async move { worker.serve().await })
};

// Serve step bodies for as long as this process lives: register, long-poll
// for leases, heartbeat while a body runs, answer every 409 by its reason.
worker := mcpd.NewWorker(m, mcpd.WorkerConcurrency(8)).Register(support)
go func() { _ = worker.Serve(ctx) }()

// Serve step bodies for as long as this process lives: register, long-poll
// for leases, heartbeat while a body runs, answer every 409 by its reason.
val worker = Worker(mcpd, concurrency = 8).register(support)
launch { worker.serve() }

# Serve step bodies for as long as this node lives: register, long-poll for
# leases, heartbeat while a body runs, answer every 409 by its reason.
{:ok, worker} = StepWorker.start_link(mcpd, workflows: [support], concurrency: 8)
:ok = StepWorker.serve(worker)

The protocol

Five calls, and a worker that speaks them can be written against the HTTP API without an SDK. Every SDK's worker is a client of these and nothing more.

One lease

The daemon learns that a body is running from started, that the worker is alive from the heartbeats, and how the step ended from complete or fail. Miss the heartbeats and the lease is reclaimed; a later call about it is refused with a reason, never lost.

Call	What it does
`POST /v1/api/workers/register`	Announces the worker and stores the definitions it holds. The answer carries `lease_ttl_ms` and `heartbeat_ms`, the two numbers the rest of the protocol runs on.
`POST /v1/api/steps/claim`	Offers up to `max` runnable steps, holding the request open for up to `wait_ms` when there are none. The daemon holds a claim for at most four fifths of its request timeout, 24 seconds at the default, and answers at once when it stops, so a claim can come back empty before `wait_ms`: claim again. A claim may be limited to one run, which is how a script drives its own run to the end.
`POST /v1/api/steps/{lease}/started`	Declares that the body is running. Before the body, always: it is what tells the daemon the difference between offered and running.
`POST /v1/api/steps/{lease}/heartbeat`	Extends the lease. The only evidence the daemon has that a worker is alive.
`POST /v1/api/steps/{lease}/complete`, `/fail`	Finishes the step: the output, a patch to the run's state and the usage the daemon prices; or an error with the class that decides whether it is retried.

Leases

A lease is one attempt, held by one worker, for a bounded time. It carries the rendered input, the prior steps keyed by id, the run's state and correlation, a key that is stable across retries of this step, and the trace and span ids of the step span the daemon opened. A body that records its own spans through the step context puts them under that span, so the model call inside a step lands in the run's trace at the right depth.

A lease the worker stops heartbeating expires. The daemon marks the attempt orphaned and, after a reclaim window, requeues it under at_least_once or fails it under at_most_once. A worker that comes back from a pause and posts against a reclaimed lease is not ignored: it is told.

The four refusals

A worker call the daemon cannot accept is refused with a 409 carrying one of four reasons. They are closed, and every SDK's worker handles each one by name, because a worker that collapses them into one branch abandons steps it has already finished or reports a failure for a success.

Reason	Meaning	What the worker does
`already_applied`	This lease's own earlier complete or fail landed and its reply was lost.	Treats it as success and posts nothing more.
`fenced`	A later attempt exists, or another lease ended this one.	Abandons the work and posts nothing more for this lease. A fenced heartbeat also cancels the running body through its context.
`run_terminal`	The run has ended, by finishing, failing, timing out or being cancelled.	Stops, and stops claiming for that run.
`not_started`	A complete or fail arrived for an `at_most_once` attempt whose `started` never did.	Reports it loudly as a bug in the worker. The SDK workers cannot produce it; a hand-written one can.

The body

A body receives a context, its rendered input and the prior steps. The context is where it reports what the daemon should keep: a patch to the run's state, merged on complete with top-level keys replacing; the tokens it spent, which the daemon prices with the same book it prices spans with; and a cancellation signal that fires when the lease is lost or the run ends, so a body that hands it to its own I/O stops when it should. The tokens add themselves up: a span the body records through the context, and every span under it, adds its usage to the step's when it ends, so a body that records its model calls as spans never sums by hand, and set_usage is the explicit override for one that records none.

A body ends by returning its output or by raising. Raise the SDK's retryable error to be retried within the step's budget; raise a typed step failure to name any other class; anything else is fatal. The message is sent to the daemon and shown in the journal. A stack trace is not.

A lease for a definition, a version or a step the worker does not hold is failed unservable and the body never runs. The attempt goes back to the queue for a worker that does hold it, without spending the retry budget, which is what lets two versions of a worker run side by side through a deploy.

Scaling out

Run as many workers as you like against one daemon. Each claims up to its own concurrency, the daemon hands every attempt to exactly one of them, and the fence makes a lease that two workers somehow both hold resolve to one winner. A worker's id is recorded on every attempt it took, so the journal says which process ran a step. Give each process a name, or let the SDK derive one from the host.

A claim is a long poll, so an idle fleet costs the daemon one open request per worker and no traffic. While it waits, a claim looks for work when a run starts, a step ends, a deadline fires or an event resumes a wait, and at most twice a second otherwise. A busy one is bounded by the workers' own concurrency, and a body that takes an hour costs one heartbeat every ten seconds and nothing else.

Keys

The worker calls take the worker scope; starting, signalling and cancelling runs take execute. Both fall back to the write keys when they are unset, and an admin key holds both. A worker that only serves steps needs no key that can start a run, and a webhook that only publishes events needs no key that can claim one. See Configuration for the flags.

The worker calls are not recorded in the audit log, although a lease carries the step's input and the prior steps' outputs. They are a machine loop that claims many times a minute, and a row per claim would bury the reads a person made. Reading a run back through the run endpoints is recorded; see Security.

Starting and following a run, and answering a wait: Runs.
The definition the worker serves: Workflows.
Every endpoint of the worker protocol, generated from the specification: HTTP API, Workers.