Regression testing

The loop this exists to close

An agent fails on a real question. Somebody reproduces it by hand, changes a prompt, and it looks better. Three weeks later it fails on the same question again, because nothing kept it.

A dataset keeps it. Capture the failing run, and it stops being an anecdote and becomes a case with an input, an expected answer when you have one, and a pointer back to the trace it came from. Every later version runs against it.

Capturing a run

Find the run in the interface or through the API, then hand its trace id to the dataset.

The captured case keeps source_trace_id, so months later you can open the run that motivated it and read what actually happened rather than guessing from the input.

The run's own answer is not the expected answer

use_output_as_expected is off by default, and that default is the important part. The runs worth capturing are usually the ones that went wrong, and recording a wrong answer as the reference would enshrine the bug: every later version would then be measured on its ability to reproduce it.

Turn it on when you are capturing runs that were right, to hold a behaviour you already like in place. Leave it off, and write the expected answer yourself, when you are capturing a failure.

The built-in scorers

mcpd ships thirteen deterministic scorers: ordinary code that reads a stored run and returns a number. They cost nothing, they never call a model, and they give the same answer twice, which is the property a regression test needs and a judge model does not have.

Scorer	Passes when
answered	The run produced a non-empty answer.
no_errors	No span in the run failed.
exact_match	The answer equals the expected one.
contains	The answer contains a required string.
json_valid	The answer parses as JSON.
latency_budget	The run finished inside a millisecond budget.
cost_budget	The run cost no more than a set amount.
no_tool_loop	No tool was called twice with the same input.
tool_efficiency	The run stayed inside a tool-call budget.
not_a_refusal	The answer is not a refusal or an apology.
no_leaked_secret	No key, token or credential appears in the answer.
span_present	The run recorded a span of the required kind or name.
tool_succeeded	The required tool ran, and at least one call of it ended ok.

GET /v1/api/scorers returns the same list with a needs_config flag. A scorer that needs an expected answer and is not given one declines rather than guessing, and the decline is reported.

When a model does the grading

Some questions no amount of deterministic code answers. Whether an answer actually addressed the question, whether a summary kept the meaning, whether a tone was right: those need a reader, and a model is the cheap reader.

Judged scoring is off until you configure it, and a judged scorer never runs unless you name it. Both defaults exist for the same reason: grading spends money on every run it reads, and a feature that quietly does that is a feature you find out about on an invoice.

{
  "judge": {
    "endpoint": "http://localhost:11434/v1",
    "model": "qwen2.5:14b",
    "api_key": "",
    "timeout_seconds": 30,
    "criteria": {
      "answer_relevance": "Does the answer address the question that was asked?",
      "no_hedging": "Does the answer commit, rather than hedge?"
    }
  }
}

The request is the OpenAI chat-completions shape, which is not a preference for a vendor: Ollama, vLLM, llama.cpp, LM Studio, OpenRouter and the hosted providers all accept it, so the same configuration reaches a model on your laptop and a gateway in production. Leave api_key empty for a local model that wants none. The temperature is zero, because a judge that answers differently each time cannot be compared with itself, which is the only thing scores are for.

Criteria are configured rather than passed on each request. A scorer id is stored on every score it produces and is what a later comparison groups by, so letting each call invent both the id and the question would build a column nobody can trust.

Configured criteria do not appear in GET /v1/api/scorers. That endpoint lists the thirteen built-in scorers and nothing else, so a client that builds its scorer menu from it will not offer your criteria. The one place the daemon names them back to you is the error you get for an unknown scorer id, which lists the built-ins and the configured criteria together.

timeout_seconds is how long one grading call may take, from connecting to the last byte of the reply: 30 by default, from 1 to 300. A call that runs past it is declined with the key named, and mcpd reads at most 1 MiB of a reply.

Each judged scorer is one model call per run, and the request has to finish inside --request-timeout-seconds. So one request grades at most as many runs as fit in that deadline if every call takes the whole of timeout_seconds. With the defaults, 30 and 30, and one judged scorer, that is one run. A request for more, in trace_ids or limit, is refused with a 400 that names the cap, before any model is called. Leave limit out and the request grades as many of the newest runs as fit. Raise --request-timeout-seconds to grade more in one request. Each judged score is stored as soon as the model answers, so a request cut short keeps the scores it finished.

It declines rather than guesses

A judge that returns a number it did not derive is worse than no judge, because it looks like evidence. So every one of these is reported in skipped with its reason, and none of them produces a score.

What happened	What mcpd does
The run recorded no answer	Declines. A run with no answer has not scored badly on relevance, it has not been measured.
The model replied in prose	Declines, quoting the reply. JSON wrapped in a fenced block or a sentence is still read.
The model answered outside the scale	Declines. A seven on a scale of one is not a nine tenths, and clamping it would invent a number.
The model could not be reached	Declines, with the status and the body. A rate limit is a reason, not a zero.
The reply was larger than 1 MiB	Declines without reading the rest. A verdict is a few hundred bytes, so a reply that size is not one.
The request deadline left no time for the call	Declines without calling the model. The pass answers with the scores it finished, inside the deadline, rather than a 503.

Every judged score records the model as its scorer_version. Two scores from two different judges are not comparable, and that field is what says so on the record.

Scoring happens after the fact

POST /v1/api/score runs scorers over runs that are already stored. Nothing evaluates inline. An evaluation in the request path adds its latency to the thing it is judging, and an evaluation that throws should never fail a user's request.

The response carries the scores it wrote and, separately, every scorer that declined with the reason it declined. A scorer that could not answer is more interesting than one that returned a number, so mcpd never swallows it. The project's ingest ceiling reaches this door too: a score the pass computed that would take the project over --quota-per-minute is not stored, and it is listed in skipped saying so, rather than written past the ceiling or thrown away in silence.

Send "dry_run": true to see what a pass would produce without storing anything.

Which runs a pass reads is decided by trace_ids, from and limit. Name the runs in trace_ids to score exactly those. Leave it empty and the pass takes the newest limit runs in the project, newest first, and from bounds that page: a pass sent from reads only runs that started at or after it, which is how you score everything since a deploy without listing ids.

Comparing two versions

Record an experiment for each version you want to compare, tag its scores with that experiment's id, and ask mcpd for the difference.

The answer is a per-scorer delta with the sample size on each side, and a decidable flag. Below twenty observations mcpd says the comparison cannot be decided rather than reporting a movement.

That is a deliberately blunt rule and not a significance test. The failure it prevents is the common one: a mean that moved on four runs, printed without its count, read as a result and shipped. Printing the counts beside the delta is what stops noise being mistaken for a finding.

Where this runs

All of it is the HTTP API, so the loop belongs in CI: capture failures as they happen, score the runs your test suite produces, register the build as an experiment, and fail the job when regressed comes back true on a decidable comparison.

Next

• The two signals a score sits beside: Scores and feedback.
• Watching a scorer's mean in production: Monitors and alerts.
• Every endpoint and field: HTTP API.