Redaction and retention

Redaction runs before storage

Prompts and completions are the most valuable thing mcpd holds and the most dangerous. Redaction is stage three of the ingest pipeline, ahead of the write, so a value a rule matches never reaches the database at all.

The ordering is the whole point. A value scrubbed after the fact has still been on disk, in the write-ahead log, and quite possibly in last night's backup. Never written is a different guarantee from deleted later, and it is the only one you can state plainly to somebody who asks.

Redaction is on by default. A tool that only protects you once you have configured it protects nobody. --no-redact, or MCPD_NO_REDACT, turns it off, which is sensible only when something upstream already redacts.

Redaction is the daemon's end of the wire. The SDKs hold the other end: MCPD_CAPTURE_PAYLOADS=false, or capture_payloads in code, keeps prompts and completions in the process. A span still arrives with its timing, status, usage, model, correlation and errors, and input and output are dropped before they are queued, so they never cross the network. A team that must not send the bytes at all sets that, and a span that holds nothing sensitive can opt back in for itself.

The default rules

A rule is a set of lowercase substrings and a mode. A key matches when its name, lowercased, contains any of them. Matching on a substring is why X-API-Key, authorization and customer_email are all caught without every casing and prefix being listed.

When a matched key holds an object or an array, everything underneath it is redacted too. {"credential": {"value": "hunter2", "note": "prod"}} loses both fields, not only the one whose name looked dangerous.

The limit is worth stating plainly: rules match key names, not values. An API key a user pasted into the middle of a prompt, stored under a key called content, is not matched by anything here. Redaction stops the structured capture of credentials and identifiers. It is not a content scanner, and treating it as one will disappoint you.

The four modes

The built-in rules use three of the four. Nothing is dropped by default: drop is there for a rule you add, where even the presence of a field is more than you want to keep.

$ curl -s "$MCPD/v1/api/traces/$TRACE" | jq .spans[0].input{  "question": "Where is my order A-1042?",  "api_key": "[redacted]",  "email": "al…om",  "user_id": "sha256:9f2c1b7d4a08e5c3f6b1d0a4"}

Why the hash is keyed

Hashing keeps a field useful. Equal inputs give equal digests under one key, so you can still group by a redacted user id, count how many distinct people hit a failure, and follow one of them across a session, without holding who they are.

Unkeyed, that property is a hole. Identifiers come from small, guessable spaces, and anyone with the digest and this source can hash candidates until one matches. The key is what makes the digest unrecoverable rather than merely inconvenient.

The built-in key is a constant in the source, so it is public. Set your own in any deployment that hashes anything.

Rotating the key breaks the link on purpose. Every digest changes, so rows written before the rotation can no longer be joined to rows written after it. That is what rotation is for, and it is also what it costs. Rotate when you want the link severed, not on a schedule out of habit.

Read the second half of that cost before you rotate. Erasure by subject matches the stored digest, so every row written under the old key is beyond the reach of a deletion request from the moment the new key is in force. mcpd cannot tell you that this is what happened, because nothing records which key wrote which row. All it can do is refuse to let the resulting zero pass as an answer, and the section on a zero below is that refusal.

Every signal, not only spans

A secret leaks as readily through a log line as through a prompt, so the same redactor runs over all six signals on the way in:

• Spans: input, output, attributes and metadata.
• Logs: the structured fields.
• Metrics: the labels.
• Scores: the metadata.
• Feedback: the metadata, and the free-text comment.
• Aspect notes: the sentence in text.

A span's name and a log record's message are not passed through the redactor. They are identifiers, not payloads, and rewriting them would break grouping. Do not build a span name out of a customer's email address.

The run store is not redacted

Durable runs are the exception, and it is deliberate. The run store keeps a run's input, state, output and correlation, and every step's input, output and error, exactly as they were sent, with user_id unhashed. They are the workflow's working state: a later step reads them back, and a value the redactor had rewritten would change what the run does. The spans a run writes into the trace store take the ordinary ingest path, so the trace store holds them redacted and hashed like any other span. Retention and erasure both reach the run store, as the sections below say, and every read of a run through the API is recorded in the audit log.

Long strings and deep documents

Two bounds apply to everything, whether or not a rule matched.

Any single string longer than 128 KiB is truncated, with … [N bytes truncated] appended so the reader knows the value is incomplete rather than short. The cut lands on a character boundary, never inside a multi-byte codepoint. One runaway prompt should not decide the size of your database.

Walking stops at a depth of 32, and anything below that is replaced with [too deeply nested]. A pathological or self-referencing document arriving at an open ingest port cannot pin a CPU.

Retention

--retention-days, or MCPD_RETENTION_DAYS, deletes signals older than that many days. Omit it and nothing is ever deleted. Keeping data by default is the right way round: a tool that silently discarded last month's traces would be discovered at the worst moment.

The pass runs on an hourly timer inside the daemon, not at query time. A retention sweep can delete a great many rows, and that work must never land inside somebody's request. Each pass removes, in one transaction, the spans, logs, metrics and trace rows whose received_at is older than the cutoff, and the scores, feedback and aspect notes whose own timestamp is, then logs how many rows went. Full-text index entries are cleared first, while the rows that name them still exist, so nothing is orphaned.

The same pass, on the same window, trims the run store. It deletes every durable run whose root ended before the cutoff, with its child runs and everything stored for them, in batches so that workers are not held up while it works. A run that has not ended is never deleted, however long it has been waiting. Definitions are not trimmed.

The pass covers the project named by --project and every project in the projects block of the config file, not only the default. A project that sets its own retention_days there is swept on that figure, and the rest are swept on the daemon's. A project in neither place, one a sender named on its own, has no window and is never trimmed. That matters when one daemon serves several teams: a team that agreed to keep thirty days does not silently keep them because another team's setting was the only one the sweep read, and a team nobody wrote down keeps everything.

An erasure request about a person

DELETE /v1/api/subjects/{user_id} answers a request about one person in one call. Send the identifier the way your instrumentation set it. The daemon applies the same hashing it applied at ingest, so a deployment that hashes user_id does not need you to hash it first.

The unit of erasure is the run, not the row. Only a root span usually carries a user_id, and the child spans are the ones holding the prompt and the completion, so deleting the rows that name the subject would leave the payload standing. A run is in scope when its trace projection names the subject or when any of its spans does, and then the whole run goes:

• Every span of that run, with the inputs and outputs it carried.
• The logs, metrics, scores, feedback and aspect notes hanging off it.
• Signals that name a span_id and no trace_id. A score or a log line pinned to one step of the subject's run is reached through the span ids of the runs in scope, not only through their trace ids.
• Log lines recorded against the person with no run to hang on, matched on their own user_id.
• Feedback the person wrote, matched on author_id, wherever it hangs. That row holds their own prose, and it can hang on somebody else's run, which is where it used to survive.
• Dataset cases captured from a run in scope. Those are emptied rather than deleted. The next section but one says why.
• On a daemon that serves durable runs, every run that has ended whose correlation carries the user_id, matched as you started the run with it because the run store keeps it unhashed, with its whole tree of child runs and everything the run store holds for them. A run of that person that has not ended is not touched: deleting it would corrupt the workflow it is part of. It is counted in runs_in_flight and named in note, so you can cancel it, or let it end, and send the request again.

The run store is erased first. A run's spans wait in its outbox until they are written into the trace store, so erasing the runs first deletes any span still waiting, and the trace store erasure then removes the ones already written.

What the reply carries

A count per signal, two counts for the run store, and six fields beside them carrying what a count cannot: what was deliberately left behind, and whether a small number can be read as "nothing about this person is stored here". deleted is the total of the six signal counts; the run counts and aspect_notes_erased are reported beside it, not added into it.

$ curl -sX DELETE "$MCPD/v1/api/subjects/u_8841?project=checkout" | jq{  "project": "checkout",  "traces": 1,  "spans": 2,  "logs": 1,  "metrics": 0,  "scores": 1,  "feedback": 1,  "deleted": 6,  "runs_erased": 1,  "runs_in_flight": 0,  "dataset_items_erased": 1,  "aspect_notes_erased": 0,  "datasets": [    "checkout-regressions"  ],  "deletes_pending": false,  "unmatchable_identifiers": 0,  "note": "1 captured dataset cases were emptied in place rather           than deleted, so a regression test is not destroyed           without anyone being told. They hold no payload any           more."}

Captured cases are emptied, not deleted

A dataset case captured from production holds a verbatim copy of the input, so erasing the run and leaving the copy would leave the person's words in the dataset. Deleting the case outright is a harm of a different kind: somebody captured that run to keep a bug from coming back, and destroying their regression test without a word is not a decision mcpd should make on its own.

So the payload is overwritten and the row survives. The case reads {"erased": true} where its input and metadata were, its link back to the source run is cleared, and the reply names how many cases and which datasets. The case is visibly empty, which is something an operator can act on.

One case is out of reach entirely, and tenancy is the reason. A case captured into a dataset that belongs to another project is not emptied and is not counted. The call is authorised for one project and touches one project, and an erasure that reached across that boundary would be a way to learn what is in a dataset you cannot read. Erase the person from that project too, with a key that reaches it.

What it does not do

It touches one project. ?project= names it and the daemon's default project is used when you leave it out. A key that does not reach that project is refused with a 403 and nothing is deleted. Rows for the same person in a project the key cannot open are neither touched nor counted, so a count can never disclose that the subject exists somewhere else. Erasing a person from several projects is one call per project, with a key that reaches each.

A subject with nothing stored is a 200 carrying zeroes, not a 404. The state you asked for holds, and a 404 would answer a question nobody is entitled to ask: whether that person exists anywhere this key can see. It is also why a repeat after a timeout is safe. The second call reports the smaller counts of a second pass rather than an error.

It is not a way to erase a person from your own systems. mcpd deletes what mcpd stored. Anything you forwarded to another OTLP endpoint has to be erased there too, and mcpd does not know what happened to it.

Keys and the record of it

Erasure needs an admin key when keys are configured, and falls back to the read keys when no admin key is set, so an existing deployment keeps the reach it had. It is written to the audit log with the action, the project and a digest of the key that asked, and a refused attempt is recorded the same way. Where user_id is hashed, the digest is what the row names: the audit log outlives the data it describes, so writing the raw identifier there would leave the name the person asked you to remove in a table retention does not sweep.

One run at a time is still there. DELETE /v1/api/traces/{trace_id} removes a single run and everything attached to it, in one transaction, returning the number of rows deleted.

A zero from an erasure is the value to distrust

A zero is what an operator repeats to a person as "we hold nothing about you", and it is the one number this endpoint cannot prove. Stored identifiers are hashed with the redaction key, and nothing records which key hashed which row, so a zero can equally mean the key moved after those rows were written. mcpd cannot tell the two apart. What it can do is never hand back a bare zero that reads like the first one.

Here is the same subject, erased under a redaction key that has been rotated since the run was recorded. The counts are all zero and the row is still there:

$ curl -sX DELETE "$MCPD/v1/api/subjects/u_8841?project=checkout" | jq "{deleted, unmatchable_identifiers, note}"{  "deleted": 0,  "unmatchable_identifiers": 0,  "note": "Nothing matched. Stored identifiers are hashed with           the redaction key, so this means no stored identifier           hashes to this subject under the key in force now. It           does not mean nothing about this person was ever           stored: rows written under a different redaction key           cannot be matched by this endpoint."}

What the note can prove depends on the project. When nothing in it is attributed to anybody, there is no row a key change could be hiding, and the note says exactly that instead. Otherwise a zero is unproven: restore the redaction key those rows were written under and repeat the request.

It reports rather than refuses, and that is a deliberate trade. Refusing would break a repeat after a timeout, it would make the reply for "never seen" differ from the reply for "erased an hour ago" and turn the endpoint into a way to ask whether a person is known to a project, and it would block every erasure in a deployment whose redaction setting ever changed, including the ones that correctly matched nothing.

Identifiers this request could not have matched

A rotation is invisible. A change to whether identifiers are hashed at all is not, and unmatchable_identifiers is what mcpd can see. A hashed identifier is recognisable by its sha256: prefix, so the store counts how many of the identifiers it holds are in the opposite form to the one this request carried: plaintext rows against a hashed request, or the reverse. Those rows were written under a different redaction setting, they may belong to the subject, and this request cannot address them.

The count runs on every erasure, not only on a zero, and the reply below is why. This project was written to with redaction on, then written to again with it off. The request matched the newer rows and deleted them, and the two older identifiers are hashed, so this request could never have addressed them. A count that stopped at deleted would have read as a complete answer:

$ curl -sX DELETE "$MCPD/v1/api/subjects/u_8841?project=checkout" | jq "{deleted, unmatchable_identifiers, note}"{  "deleted": 2,  "unmatchable_identifiers": 2,  "note": "2 stored identifiers in this project are in a form           this request could not match, so rows written under a           different redaction setting were not searched and rows           about this person may remain. Restore the redaction           setting those rows were written under and repeat the           request."}

The count is over every identifier the project holds, not over the subject's own rows, so those two may belong to somebody else entirely. That is the most it can say. What it rules out is reading a successful-looking erasure as a complete one.

The guarantee is not the same on all three backends

SQLite and Postgres delete inside the transaction that answered you. When the reply arrives the rows are gone, and deletes_pending is false.

ClickHouse does not. Deletion there is an asynchronous mutation: the server accepts the statement and applies it in the background, so the reply says what matched rather than what has already left the disk. Those erasures come back with deletes_pending set to true, which is mcpd telling you the weaker of the two guarantees is the one in force. The dataset cases are the exception on that backend. They are rewritten under their own key rather than mutated, so the emptied case is what any read returns from the moment the reply is sent. A mutation would have left the old text readable until it finished.

If you have to certify to somebody that the data is gone by a given moment, that is the difference to write down. On ClickHouse, confirm the mutation completed on the server before you answer.

Before the port is reachable

Redaction protects the values inside a trace. It does nothing about who can read the trace. mcpd warns at startup when it binds a non-loopback address with open ingest or an open read API, because stored prompts and completions are then readable by anyone who can reach the port. Set --ingest-key and --read-key first, and read Docker and Kubernetes.

Next

• Keeping less in the first place: Sampling.
• Every flag and environment variable: Configuration.
• Probes, backups and upgrades: Running it.