Docker and Kubernetes
One process, one volume, one file to back up. The longest section on this page is the one about what to set before anyone but you can reach it.
One container
No release yet.
The image, the chart and the packages the commands on this site name are the intended distribution. None of them has been published. What works today is a build from source, which needs Rust and Node:
Build and run it
git clone https://github.com/mcpd/mcpd && cd mcpd && make build && ./target/release/mcpd demo mcpd is a single process with its storage compiled in. There is no database to run beside it, no sidecar, and no static-file server: ingest, the query API and the interface are all served from one port.
docker run -p 4317:4317 -v mcpd-data:/data -e MCPD_DATA=/data/mcpd.db ghcr.io/mcpd/mcpd:0.1.0
The volume is the part worth reading twice. Without it the SQLite file lives
in the container's writable layer, and the traces go when the container
does. MCPD_DATA is set explicitly for the same reason: if the
path the daemon writes and the path you mounted ever drift apart, mcpd keeps
working and the volume stays empty, which is the kind of mistake you find
out about on the day you need the data.
Pin the tag. latest makes a restart into an upgrade, at a time
nobody chose. Each tag is built for linux/amd64 and
linux/arm64, and runs as uid 10001, which owns
/data in the image.
The port, and what answers on it
Everything is on 4317. That is what makes the deployment one container, and it is also what you need to know when you write an ingress rule or a network policy, because the paths do not all carry the same risk.
| Path | What it is | Key required |
|---|---|---|
/v1/traces, /v1/logs, /v1/metrics | OTLP ingest, for any OpenTelemetry SDK | Ingest |
/v1/ingest | Native ingest, the full model in one request | Ingest |
/v1/api/… | The query API: traces, spans, logs, scores, analytics | Read |
/ | The operator interface, embedded in the binary | None |
/healthz, /readyz, /metrics | Liveness, readiness, and mcpd's own Prometheus counters | None |
An ingest request is capped at 32 MiB. Prompts are large, so that limit is generous on purpose, but a proxy in front of mcpd usually has a smaller one: a reverse proxy that refuses a 3 MiB batch is the most common reason a trace never arrives.
Compose
Two decisions in this file are worth explaining, because both look like
omissions. The port is published on 127.0.0.1, so nothing
outside the host can reach it while you are still setting keys. And the file
has no healthcheck because the image has one: it requests
/healthz every 30 seconds with the busybox wget in
the image, on the port in MCPD_LISTEN, and
docker compose ps shows the result. Docker does not restart a
container for being unhealthy; the restart policy acts when the process
exits.
deploy/docker
# mcpd: one service, one volume.
#
# The volume is most of the reason this file exists. Without it the SQLite file
# lives in the container's writable layer, and `docker compose down` takes every
# trace with it.
#
# Copy .env.example to .env and fill it in before running this. The keys are not
# in this file on purpose: a compose file is committed, a .env file is not.
services:
mcpd:
image: ghcr.io/mcpd/mcpd:0.1.0
restart: unless-stopped
# Published on the loopback interface of the host, so nothing outside this
# machine can reach the port even while you are still setting up keys.
ports:
- "127.0.0.1:4317:4317"
environment:
MCPD_LISTEN: 0.0.0.0:4317
# Set the path explicitly. If this and the mount below ever disagree, the
# daemon writes to the container layer and the volume stays empty.
MCPD_DATA: /data/mcpd.db
MCPD_PROJECT: default
MCPD_RETENTION_DAYS: "30"
MCPD_LOG_FORMAT: json
MCPD_INGEST_KEYS: ${MCPD_INGEST_KEYS:?set MCPD_INGEST_KEYS in .env}
MCPD_READ_KEYS: ${MCPD_READ_KEYS:?set MCPD_READ_KEYS in .env}
MCPD_REDACT_KEY: ${MCPD_REDACT_KEY:?set MCPD_REDACT_KEY in .env}
# The run store: a second file on the same volume.
MCPD_RUNS_DATA: /data/mcpd.runs.db
# The two scopes of the run surface. Unset accepts the write keys, which
# here are the ingest keys; set them in .env once a worker runs outside
# this host. No value here on purpose: Compose passes the variable only
# when .env sets it, and `${VAR:-}` would pass an empty string, which the
# daemon reads as one empty key rather than none.
MCPD_EXECUTE_KEYS:
MCPD_WORKER_KEYS:
MCPD_REQUEST_TIMEOUT_SECONDS: "30"
# The two bounds on one request: bytes of body, and signals in it once
# decoded. Lower the first on a small host, since it is what decides how
# much memory one request can take before anything is parsed.
MCPD_MAX_BODY_BYTES: "33554432"
MCPD_MAX_BATCH_SIGNALS: "10000"
# Set MCPD_CORS_ORIGINS when the interface is served from another origin,
# MCPD_REDACT_KEY_CONTAINS to mask fields the built-in rules do not know,
# and MCPD_REDACT_MAX_STRING to lower the 128 KiB cap on a stored string.
# MCPD_SIGNUP=true opens account sign-up; the accounts go to
# MCPD_ACCOUNTS_DATA, beside the data file by default. Behind a proxy, set
# MCPD_CLIENT_IP_HEADER and MCPD_TRUSTED_PROXY_CIDRS so the sign-up and
# log-in limits count clients rather than the proxy.
volumes:
- mcpd-data:/data
# No healthcheck here, because the image has one: it requests /healthz
# every 30 seconds with the busybox wget in the image, on the port in
# MCPD_LISTEN, and `docker compose ps` shows the result. It is the liveness
# endpoint, never /readyz, so a slow query does not mark the daemon sick.
# Docker does not restart a container for being unhealthy; the restart
# policy above acts when the process exits.
# Longer than the 30-second request timeout above, so a request still
# running at `docker compose down` can finish before Docker kills it.
stop_grace_period: 45s
volumes:
mcpd-data: # Copy to .env and replace every value. Generate each one with:
# openssl rand -hex 32
# Keys permitted to write. Comma-separated, one per fleet or per team, so a
# leaked key can be revoked without re-deploying every agent.
MCPD_INGEST_KEYS=replace-me-ingest
# Keys permitted to read stored prompts and completions. Never the same value
# as an ingest key.
MCPD_READ_KEYS=replace-me-read
# Salt for the hashes redaction leaves behind. Without it the digests of short
# identifiers can be recovered by guessing.
MCPD_REDACT_KEY=replace-me-redact
# Keys for the run surface. Leave both commented out and the ingest keys are
# accepted for starting runs and claiming steps, which is fine while every
# worker runs on this host. Set them once a worker runs elsewhere: the worker
# key lives in the process that runs step bodies and opens nothing that reads a
# prompt. Uncomment a line only with a value: `MCPD_WORKER_KEYS=` with nothing
# after it passes an empty string, not an unset variable.
# MCPD_EXECUTE_KEYS=replace-me-execute
# MCPD_WORKER_KEYS=replace-me-worker docker compose -f deploy/docker/docker-compose.yml up -d
The :? in each key line is not decoration. Compose refuses to
start the service when the variable is unset, which turns a forgotten key
from a silent open port into an error message. The two run keys have no
value at all in the file, so Compose passes them only when
.env sets them: an empty value would reach the daemon as one
empty key rather than none.
Kubernetes
Four objects: a Deployment, a Service, a PersistentVolumeClaim, and a Secret
you create yourself. The Deployment runs one replica with the
Recreate strategy, and that is the one line in these manifests
you should not change. mcpd stores everything in a single SQLite file, one
process writes it, and a rolling update would start a second pod on the same
volume while the first still holds it. That is not a scaling story, it is a
corruption story.
The Secret is not one of the files kubectl apply reads.
secret.yaml.example shows its shape, and the extension keeps it
out of the apply, because a Secret applied as written would make its
placeholder strings your keys. Create it from the command line instead,
with keys generated on the spot. The pod runs as uid 10001 with a read-only
root filesystem, no capabilities and the runtime's default seccomp profile,
which is what the Pod Security restricted level asks for.
deploy/k8s
apiVersion: apps/v1
kind: Deployment
metadata:
name: mcpd
labels:
app.kubernetes.io/name: mcpd
spec:
# One SQLite file has one writer. Two pods mounting the same volume is not a
# scaling story, it is a corruption story.
replicas: 1
strategy:
# RollingUpdate would start the new pod while the old one still holds the
# database. Recreate stops the old pod first and costs a few seconds of
# downtime, which the SDK buffers cover.
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: mcpd
template:
metadata:
labels:
app.kubernetes.io/name: mcpd
spec:
securityContext:
runAsNonRoot: true
# The uid and gid the image creates its user with, so this pod and a
# plain `docker run` own files the same way.
runAsUser: 10001
runAsGroup: 10001
# The volume is created owned by root. Without fsGroup the daemon
# cannot create its database and exits on the first start.
fsGroup: 10001
# The Pod Security `restricted` level refuses a pod without it.
seccompProfile:
type: RuntimeDefault
# Longer than the 30-second request timeout below, so a request still
# running at SIGTERM can finish before the kubelet kills the process.
terminationGracePeriodSeconds: 45
containers:
- name: mcpd
image: ghcr.io/mcpd/mcpd:0.1.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 4317
env:
- name: MCPD_LISTEN
value: 0.0.0.0:4317
- name: MCPD_DATA
value: /data/mcpd.db
- name: MCPD_PROJECT
value: default
- name: MCPD_RETENTION_DAYS
value: "30"
- name: MCPD_LOG_FORMAT
value: json
- name: MCPD_INGEST_KEYS
valueFrom:
secretKeyRef:
name: mcpd-keys
key: ingest-keys
- name: MCPD_READ_KEYS
valueFrom:
secretKeyRef:
name: mcpd-keys
key: read-keys
- name: MCPD_ADMIN_KEYS
valueFrom:
secretKeyRef:
name: mcpd-keys
key: admin-keys
# The two scopes of the run surface. A worker key lives in the
# process that runs step bodies and opens nothing that reads a
# stored prompt. Optional: with no entry in the Secret the
# variable is not set at all, and the ingest keys are accepted.
- name: MCPD_EXECUTE_KEYS
valueFrom:
secretKeyRef:
name: mcpd-keys
key: execute-keys
optional: true
- name: MCPD_WORKER_KEYS
valueFrom:
secretKeyRef:
name: mcpd-keys
key: worker-keys
optional: true
# The run store is a second file on the same volume.
- name: MCPD_RUNS_DATA
value: /data/mcpd.runs.db
- name: MCPD_REQUEST_TIMEOUT_SECONDS
value: "30"
# The two bounds on one request: bytes of body, and signals in it
# once decoded. Lower the first on a small pod, since it is what
# decides how much memory one request can take.
- name: MCPD_MAX_BODY_BYTES
value: "33554432"
- name: MCPD_MAX_BATCH_SIGNALS
value: "10000"
# Set when the interface is served from another origin, as a
# comma-separated list of origins; and to mask fields the built-in
# rules do not know, or to lower the 128 KiB cap on a stored string:
# MCPD_CORS_ORIGINS, MCPD_REDACT_KEY_CONTAINS, MCPD_REDACT_MAX_STRING
- name: MCPD_REDACT_KEY
valueFrom:
secretKeyRef:
name: mcpd-keys
key: redact-key
# Liveness never touches storage, so a slow query cannot restart a
# process that is working.
livenessProbe:
httpGet:
path: /healthz
port: http
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
# Readiness pings the store, so traffic only reaches an instance that
# can actually write to its database.
readinessProbe:
httpGet:
path: /readyz
port: http
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
# Migrations run at start. On a large database that takes longer than
# the liveness probe would allow, so the startup probe holds it off.
startupProbe:
httpGet:
path: /healthz
port: http
periodSeconds: 2
failureThreshold: 45
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
# No CPU limit: ingest arrives in bursts, and throttling the
# writer is how a queue turns into dropped spans.
memory: 512Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: mcpd-data apiVersion: v1
kind: Service
metadata:
name: mcpd
labels:
app.kubernetes.io/name: mcpd
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: mcpd
ports:
- name: http
port: 4317
targetPort: http
protocol: TCP apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mcpd-data
labels:
app.kubernetes.io/name: mcpd
spec:
accessModes:
# ReadWriteOnce, and not because of a quota. SQLite locking is only correct
# on a local block device: on NFS or any other shared filesystem the locks
# are advisory at best and the database will eventually be corrupted.
- ReadWriteOnce
resources:
requests:
storage: 20Gi
# storageClassName: gp3 # A template, not part of the files beside it.
#
# The extension is .example on purpose. `kubectl apply -f deploy/k8s/` applies
# every manifest in this directory, and a Secret applied as written would put
# these placeholder strings into production as the keys. Create the Secret
# from the command line instead, with keys generated on the spot:
#
# kubectl -n mcpd create secret generic mcpd-keys \
# --from-literal=ingest-keys="$(openssl rand -hex 32)" \
# --from-literal=read-keys="$(openssl rand -hex 32)" \
# --from-literal=admin-keys="$(openssl rand -hex 32)" \
# --from-literal=redact-key="$(openssl rand -hex 32)"
#
# Add execute-keys and worker-keys the same way once a worker runs outside the
# cluster; until then the deployment accepts the ingest keys for both.
#
# Or copy this file, replace every value with the output of
# `openssl rand -hex 32`, apply it by name, and do not commit the result.
apiVersion: v1
kind: Secret
metadata:
name: mcpd-keys
labels:
app.kubernetes.io/name: mcpd
type: Opaque
stringData:
# Comma-separated. One key per fleet or per team means a leaked key can be
# revoked without redeploying every agent.
ingest-keys: REPLACE_WITH_A_RANDOM_KEY
# Never the same value as an ingest key.
read-keys: REPLACE_WITH_A_DIFFERENT_RANDOM_KEY
# Keys permitted to delete data and read the audit log. Without this entry
# the daemon accepts the read keys for admin work, so every read key can
# delete a trace; `mcpd doctor` reports that as a warning.
admin-keys: REPLACE_WITH_A_THIRD_RANDOM_KEY
# Keys permitted to start, cancel and signal runs.
execute-keys: REPLACE_WITH_A_FIFTH_RANDOM_KEY
# Keys permitted to claim steps and report their outcome. This one lives in
# the process that runs step bodies, so it is never the read key.
worker-keys: REPLACE_WITH_A_SIXTH_RANDOM_KEY
# Salt for the hashes redaction leaves behind.
redact-key: REPLACE_WITH_A_FOURTH_RANDOM_KEY export INGEST_KEY=$(openssl rand -hex 32) READ_KEY=$(openssl rand -hex 32) ADMIN_KEY=$(openssl rand -hex 32)
kubectl create namespace mcpd
kubectl -n mcpd create secret generic mcpd-keys \
--from-literal=ingest-keys=$INGEST_KEY \
--from-literal=read-keys=$READ_KEY \
--from-literal=admin-keys=$ADMIN_KEY \
--from-literal=redact-key=$(openssl rand -hex 32)
kubectl -n mcpd apply -f deploy/k8s/
Keep the three keys where your team keeps secrets: the ingest key goes to
every agent, the read key to whoever reads the interface, and the admin key
to whoever may delete data. Add execute-keys and
worker-keys the same way once a worker runs outside the
cluster; until then the ingest keys are accepted for both.
export EXECUTE_KEY=$(openssl rand -hex 32) WORKER_KEY=$(openssl rand -hex 32)
kubectl -n mcpd patch secret mcpd-keys --type merge \
-p '{"stringData":{"execute-keys":"'$EXECUTE_KEY'","worker-keys":"'$WORKER_KEY'"}}'
kubectl -n mcpd rollout restart deploy/mcpd kubectl -n mcpd port-forward svc/mcpd 4317:4317 Agents inside the cluster point at the Service. The SDKs read both of these from the environment, so instrumented code needs no change:
MCPD_ENDPOINT=http://mcpd.mcpd.svc:4317 MCPD_API_KEY=$INGEST_KEY The probes
The two endpoints do deliberately different things, and swapping them causes a specific, ugly failure.
-
/healthznever touches storage. It answers from process state alone. A liveness probe that queried the database would restart a healthy daemon every time a large query made it slow, and restarting is exactly what you do not want while a write-ahead log is being flushed. -
/readyzpings the store and returns 503 with"status": "degraded"and a line saying what is wrong when it cannot. That keeps traffic away from an instance that cannot serve it, without killing it.
The startup probe is there because migrations run at start. They are idempotent and usually instant, but on a large database they can take longer than the liveness probe would tolerate, and a pod that gets killed mid-migration on every attempt never starts at all.
What to give it
These numbers come from what the process actually does, not from a benchmark, and no benchmark is claimed.
| Resource | Value | Why |
|---|---|---|
| CPU request | 100m | Idle costs close to nothing. The request exists so the scheduler does not stack it under something noisy. |
| CPU limit | none | Ingest arrives in bursts. Throttling the writer during a burst is how a client queue fills and spans get dropped. |
| Memory request | 128Mi | A static binary, five SQLite connections and their page caches. |
| Memory limit | 512Mi | Headroom for concurrent ingest. One request can hold 32 MiB of prompt text, so memory tracks how many large batches arrive at once. |
| Volume | 20Gi, ReadWriteOnce | Sized from your own payloads. Text dominates, and the search index is not an external-content one, so indexed prompts and completions are stored a second time. |
Keep the volume on a local block device. SQLite's locking is only correct there: on NFS or another shared filesystem the locks are advisory at best, and the failure does not show up as an error, it shows up as a corrupt database weeks later.
The Helm chart
The chart renders the same four objects with values in front of them, plus six that are off until a value turns them on: an Ingress, a NetworkPolicy, a PodDisruptionBudget, a ServiceAccount, a ServiceMonitor, and a ConfigMap or Secret for the config file. Each release pushes it to the same registry as the image. Generate the keys first and keep them where your team keeps secrets, because the install will not print them back to you:
helm install mcpd oci://ghcr.io/mcpd/charts/mcpd -n mcpd --create-namespace --set keys.ingest=$INGEST_KEY --set keys.read=$READ_KEY
The install fails when keys.ingest or keys.read is
unset, and says why. A chart that let you discover an open ingest endpoint
later would not be doing its job. If you manage the Secret yourself, set
keys.existingSecret instead and the chart creates none.
The pod template carries a checksum of the Secret, so rotating a key and
running helm upgrade restarts the pod. Without that, the new
key sits in the cluster and the daemon keeps enforcing the old one.
Monitors, SLOs, the judge, per-project keys and the run store's timings are
read from the daemon's config file, not from flags. Write that JSON object
under configFile.content and the chart mounts it read-only and
points MCPD_CONFIG_FILE at it. Per-project keys, the judge's
api_key and forward headers are credentials, so the chart
refuses to put content carrying any of them in a ConfigMap: set
configFile.kind to Secret, or keep the file in a
Secret you manage and name it in configFile.existingSecret.
extraEnv, extraVolumes and
extraVolumeMounts cover anything else, such as a private CA
bundle.
Every combination the chart cannot serve is refused at install time with a message that names the value to change, never left to crash-loop after it. The chart's README lists each refusal.
deploy/helm/mcpd
# mcpd chart values.
#
# The chart refuses to install without keys. That is deliberate: a daemon on a
# cluster network with open ingest accepts writes from anything that can reach
# the Service, and a chart that lets you find that out later is not doing its
# job. Set keys.ingest and keys.read, or point keys.existingSecret at a Secret
# you manage yourself.
image:
repository: ghcr.io/mcpd/mcpd
# Empty means the chart's appVersion. Pin it in production.
tag: ""
pullPolicy: IfNotPresent
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
# Daemon configuration. Each of these maps to one environment variable.
config:
# The container port and the NetworkPolicy port are read from the number
# after the last colon, so moving the listener moves both.
listen: 0.0.0.0:4317
dataPath: /data/mcpd.db
project: default
# Days to keep signals. Empty or 0 keeps everything for ever, and the chart
# then sets no MCPD_RETENTION_DAYS at all.
retentionDays: 30
# Fraction of traces to keep, 0.0 to 1.0. Empty keeps all of them.
sample: ""
logFormat: json
# Turn off built-in redaction. Only sensible when something upstream already
# redacts.
noRedact: false
# Mask any value whose key contains one of these, on top of the built-in
# rules. Matched on lowercase substrings of the key.
redactKeyContains: []
# Longest single string stored, in bytes. Empty keeps the built-in 128 KiB.
redactMaxString: ""
# Where the run store lives, when runs.enabled is true. It must be inside the
# directory of dataPath, because that directory is the only writable mount.
# Empty derives it from dataPath: /data/mcpd.runs.db for the default. "off"
# is the same as runs.enabled false.
runsPath: ""
# Browser origins allowed to call the query API from another site, such as
# an interface served from a portal. Empty answers no cross-origin request.
corsOrigins: []
# Longest a request may take before it is answered with a 503. Empty keeps
# the built-in 30 seconds. Raise terminationGracePeriodSeconds with it.
requestTimeoutSeconds: ""
# An external database for signals, instead of the embedded file.
#
# An external store is what lets mcpd run more than one pod: the single writer
# and the single replica are limits of the SQLite file, not of mcpd. It is half
# of what it takes, though. The run store is always an embedded file (see
# `runs` below), so more than one replica also needs runs.enabled false.
#
# `type` decides the mode, never the URLs. That is what lets a URL live only in
# a Secret you manage: set the type here, point keys.existingSecret at the
# Secret, and leave the URL values empty. A URL value with type none, or with
# the other type, refuses to render rather than being ignored.
externalStore:
# none, postgres or clickhouse.
type: none
# A Postgres connection string. Reach for this first: it lifts both limits on
# a database most teams already run, back up and monitor.
postgresUrl: ""
# A ClickHouse HTTP endpoint. Reach for this when retention is measured in
# billions of spans.
clickhouseUrl: ""
clickhouseDatabase: mcpd
clickhouseUser: ""
# Both URLs and the ClickHouse password carry credentials, so they are held in
# the chart's Secret rather than in the pod spec. With keys.existingSecret,
# the chart creates no Secret: put them in yours under the keys postgres-url,
# clickhouse-url and clickhouse-password, and leave these values empty.
clickhousePassword: ""
# The durable run surface: definitions, the journal of step attempts, deadlines
# and waits.
#
# Its store is an embedded SQLite file whatever the signal store is, so while
# it is on, the pod keeps a file on a volume and runs as one replica. With an
# external store that volume holds the run store alone. Turn it off to run
# several pods against an external store; the daemon then answers the run
# endpoints with a 503 that says so, and nothing else changes.
runs:
enabled: true
# Pods to run. More than one needs both an external store and runs.enabled
# false, because each embedded SQLite file has one writer. The chart refuses to
# render a larger count otherwise, rather than pinning it to 1 behind your back.
replicaCount: 1
keys:
# Comma-separated keys permitted to write.
ingest: ""
# Comma-separated keys permitted to read stored prompts and completions.
read: ""
# Comma-separated keys permitted to change stored data: scores, feedback,
# datasets, experiments and acknowledgements. Unset means the ingest keys are
# accepted, which is what a deployment written before this scope existed did.
# The ingest key is in every container that emits a span, so separating them
# is worth doing once you have more than one team.
write: ""
# Comma-separated keys permitted to destroy data and read the audit log.
# Unset means the read keys are accepted, so a read key can delete a trace.
# `mcpd doctor` reports that as a warning and names this setting.
admin: ""
# Comma-separated keys permitted to start, cancel and signal runs. Unset
# means the write keys are accepted, and an admin key always is.
execute: ""
# Comma-separated keys permitted to claim steps and report their outcome.
# This is the key that lives in the process running step bodies. Unset means
# the write keys are accepted, and an admin key always is.
worker: ""
# Salt for the hashes redaction leaves behind. Optional, but set it anywhere
# that hashes: without it, digests of short identifiers can be guessed.
redact: ""
# Use a Secret you already manage instead of the values above. It must carry
# ingest-keys and read-keys, and optionally write-keys, admin-keys,
# execute-keys, worker-keys and redact-key.
existingSecret: ""
# The two bounds on one request. Empty means the built-in defaults, 33554432
# bytes (32 MiB) of body and 10000 signals per request.
#
# The byte cap is what bounds how much memory one request can take before
# anything is decoded, so lower it on a small pod. The signal cap is counted
# across the whole request on every ingest door, whether the signals arrive as
# one batch, an array of batches or an NDJSON stream.
maxBodyBytes: ""
maxBatchSignals: ""
# A ceiling on signals stored per project per minute. Empty means no ceiling,
# which is what a deployment that never sets it had before quotas existed.
#
# It stops one project spending the whole daemon. Signals over the ceiling are
# refused and reported with the count and the project, never dropped in silence,
# and the rest of the batch is still stored.
quotaPerMinute: ""
# Record WHICH PERSON acted on an audit row, not only which key.
#
# mcpd has no identity provider and is not getting one, so the identity comes
# from the ingress already in front of it. Both halves are required: the header
# your ingress sets, and the ranges allowed to set it. A header is a string
# anyone who can reach the port may send, so mcpd reads it only from an address
# you have named here. A request from anywhere else is served normally and its
# audit row carries no subject.
#
# Set trustedProxyCidrs to the range your ingress controller runs in, not to
# 0.0.0.0/0, which would trust every client to name itself.
identity:
# For example x-forwarded-user, x-forwarded-email or x-auth-request-user.
header: ""
# A list of CIDR ranges, for example ["10.42.0.0/16"].
trustedProxyCidrs: []
# The header your proxy puts the client's address in, for example
# cf-connecting-ip. Read only from trustedProxyCidrs. The sign-up and log-in
# limits count this address; without it they count the proxy's.
clientIpHeader: ""
# Open sign-up: anyone may create an account, which owns one new project and
# its own API keys. Off by default. On, a request with no key is refused
# everywhere except the health probes, sign-up and log-in. Set
# identity.clientIpHeader and identity.trustedProxyCidrs as well when an
# ingress is in front, so the per-address limits count clients.
signup:
enabled: false
# Where the accounts file goes. Empty puts it beside the data file, as
# mcpd.accounts.db, which is on the volume.
accountsData: ""
# Copy every stored span on to another OTLP endpoint. The local write always
# happens first, so a slow or absent destination never costs you the data.
forward:
# Base URL, not the /v1/traces path; mcpd appends it.
endpoint: ""
# Headers as "Name: value" strings. A credential here lands in the Secret.
headers: []
# The volume for whatever the pod keeps on disk: the signal store and the run
# store on the embedded store, or the run store alone with an external one.
# Not used with an external store and runs.enabled false, since nothing local
# is left to keep.
#
# With enabled false the chart mounts an emptyDir instead, so the daemon can
# still write, and everything it wrote is gone when the pod is.
persistence:
enabled: true
# Sized for the embedded signal store. A volume that holds the run store
# alone needs far less.
size: 20Gi
# Leave empty for the cluster default. SQLite locking is only correct on a
# local block device, so do not point this at NFS.
storageClass: ""
existingClaim: ""
service:
type: ClusterIP
port: 4317
annotations: {}
# Everything from here to `resources` is off by default, so an install that sets
# nothing renders the same four objects it always did. Each block is here
# because a chart review asks a question this chart could not otherwise answer.
# "What identity does this pod run as, and what can that identity do?"
serviceAccount:
create: false
# With create true, empty means the release's fullname. With create false,
# naming an account you manage yourself puts the pod on it, and leaving it
# empty keeps the namespace's `default` account, which is today's behaviour.
name: ""
annotations: {}
# mcpd never calls the Kubernetes API, so it has no use for a token.
# Turn this on only if something you add to the pod needs one.
automountServiceAccountToken: false
# "How does traffic reach mcpd from outside the cluster, and who terminates
# TLS?"
#
# Not mcpd: it speaks plain HTTP and has no TLS configuration, so the ingress
# controller has to terminate. Two things to know before you publish it.
#
# One path root. mcpd serves OTLP ingest, the query API and the operator
# interface from the same root, so a rule for "/" publishes all three, and your
# ingest keys are what stands between the open internet and a write. There is no
# path prefix that separates them.
#
# Body size. mcpd accepts a 32 MiB batch, and several ingress controllers cap a
# request body at 1 MiB by default, so a batch the daemon would have taken is
# refused before it arrives. On ingress-nginx the annotation to raise is
# `nginx.ingress.kubernetes.io/proxy-body-size`.
ingress:
enabled: false
className: ""
annotations: {}
hosts:
- host: mcpd.example.com
paths:
- path: /
pathType: Prefix
tls: []
# - secretName: mcpd-tls
# hosts:
# - mcpd.example.com
# "Only these namespaces may reach ingest."
#
# The limit belongs in the answer, so here it is. mcpd serves ingest, the query
# API, the operator interface and /metrics on ONE port, and a NetworkPolicy
# matches pods and ports, never paths. This gates the whole daemon or nothing:
# every source you allow in for ingest can also reach /v1/api and read stored
# prompts and completions. The read key is what holds those apart, not this
# policy. A split at the network layer would need a second listener, and mcpd
# does not have one.
#
# What it does do is still worth having. With no policy in the namespace, every
# pod in the cluster can open a connection to mcpd. With one, only these can.
#
# A trap worth knowing first: some CNI plugins enforce policy on traffic
# arriving from the node, and the kubelet's liveness and readiness probes
# arrive that way. On those clusters, add the node CIDR to `ipBlocks`, or the
# pod never becomes ready and the rollout stalls.
networkPolicy:
enabled: false
ingress:
# Whole namespaces, by label. Kubernetes sets
# `kubernetes.io/metadata.name` on every namespace itself, so this works
# without labelling anything first.
namespaceSelectors: []
# - matchLabels:
# kubernetes.io/metadata.name: agents
# Pods in mcpd's OWN namespace. A podSelector with no namespaceSelector
# beside it never reaches across a namespace boundary.
podSelectors: []
# - matchLabels:
# app.kubernetes.io/part-of: checkout-agents
ipBlocks: []
# - cidr: 10.42.0.0/16
# except:
# - 10.42.7.0/24
# Whole `from` entries, written out and appended as they are. Use this for
# the one thing the three lists above cannot say: a namespaceSelector and a
# podSelector inside a SINGLE entry, which means "those pods, in those
# namespaces" rather than "either of the two".
extraFrom: []
# - namespaceSelector:
# matchLabels:
# kubernetes.io/metadata.name: agents
# podSelector:
# matchLabels:
# app.kubernetes.io/name: checkout-agent
egress:
# Leave this off unless you have a reason. Restricting egress breaks four
# things and breaks them quietly, because mcpd logs a warning and carries
# on: alert webhooks, forwarding to another OTLP endpoint, the judge's model
# endpoint, and an external Postgres or ClickHouse.
enabled: false
allowDns: true
dnsNamespace: kube-system
# Complete egress rules, appended as written, because what mcpd is allowed
# to dial depends entirely on what you configured it to dial.
rules: []
# "What happens to ingest while a node is drained?"
#
# Only answerable with more than one replica, which takes an external store and
# runs.enabled false. With one replica, a budget over that pod either blocks
# every drain or permits every one. The chart refuses to render it there rather
# than let you discover which during a cluster upgrade.
podDisruptionBudget:
enabled: false
# Set one of these, never both. With replicaCount 2, minAvailable 1 lets one
# pod go at a time and keeps ingest up through a drain.
minAvailable: 1
maxUnavailable: ""
# "Is the collector itself monitored?"
#
# Needs the Prometheus Operator's CRDs in the cluster. Without them the install
# fails on a kind the API server does not know, which is why this is off rather
# than on with a capability check: a chart that silently drops the object you
# asked for is worse than one that fails.
#
# There is no credential below because mcpd serves /metrics with no key. It
# exposes counters and uptime, not stored prompts or completions.
serviceMonitor:
enabled: false
# Empty means the release's namespace.
namespace: ""
interval: 30s
scrapeTimeout: 10s
# The labels your Prometheus selects ServiceMonitors by. Without the ones it
# expects, this object is created and never scraped, and nothing reports that.
labels: {}
relabelings: []
metricRelabelings: []
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: 512Mi
# The uid and gid are the ones the image creates its user with, so the pod and
# a plain `docker run` own files the same way. fsGroup is what makes a new
# volume, created owned by root, writable to that user. The seccomp profile is
# required by the Pod Security `restricted` level, which refuses the pod
# without it.
podSecurityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
# Longer than the longest request. A request may run for the request timeout,
# 30 seconds by default, and the pod has to outlive it after SIGTERM or the
# kubelet kills it mid-write. Raise it with config.requestTimeoutSeconds.
terminationGracePeriodSeconds: 45
# The settings too structured for an environment variable: monitors, SLOs, the
# judge, per-project keys, retention and sampling under `projects`, and the run
# store's timings under `runs`. Written here as the JSON object the daemon's
# --config-file reads (see https://mcpd.sh/docs/configuration), mounted
# read-only at /etc/mcpd/mcpd.json, and named in MCPD_CONFIG_FILE. A change
# restarts the pod, as a key rotation does.
#
# Per-project keys, the judge's api_key and forward headers are credentials. A
# ConfigMap is readable by anything allowed to list ConfigMaps in the
# namespace, so the chart refuses to put content carrying any of them in one.
# Set kind to Secret, or keep the file in a Secret you manage and name it in
# existingSecret.
configFile:
content: {}
# monitors:
# - name: error-rate
# ...
# projects:
# checkout:
# retention_days: 90
# ConfigMap or Secret: the object `content` is rendered into.
kind: ConfigMap
# A Secret you manage that holds the file, instead of `content`.
existingSecret: ""
# The entry in existingSecret that holds the file.
existingSecretKey: mcpd.json
# Extra environment variables, as Kubernetes writes them, appended after the
# ones the chart sets. Use it for a setting this chart has no value for, or to
# read one from a Secret of your own with valueFrom.
extraEnv: []
# - name: MCPD_FORWARD_HEADERS
# valueFrom:
# secretKeyRef:
# name: otlp-credentials
# key: headers
# Extra volumes and where to mount them, for example a CA bundle for a
# ClickHouse or a judge endpoint behind a private certificate authority.
extraVolumes: []
extraVolumeMounts: []
podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {} Before anything else can reach it
Ingest is open by default. That is a deliberate choice for the first five minutes on a laptop, where a credential before the first trace would be friction with no benefit, and it is the wrong default the moment the port is reachable by anything else. So the daemon says so, every time it starts on a non-loopback address without keys:
$ docker run -p 4317:4317 -v mcpd-data:/data ghcr.io/mcpd/mcpd:0.1.0mcpd 0.1.0 listening on http://0.0.0.0:4317WARN listening on 0.0.0.0:4317 with open ingest. Anyone who can reach this port can write to it. Set --ingest-key (or MCPD_INGEST_KEYS) before exposing it.WARN listening on 0.0.0.0:4317 with an open read API. Stored prompts and completions are readable by anyone who can reach this port. Set --read-key to require a key. mcpd warns rather than refusing to start, because refusing would break the path that makes it worth running at all. The warning is the contract: if you see it in a deployment log, that deployment is open.
Six keys, six scopes
--ingest-key permits writing signals. --read-key
permits reading them. --write-key permits changing stored data:
scores, feedback, datasets, experiments and acknowledgements.
--execute-key permits starting, cancelling and signalling runs.
--worker-key permits claiming steps and reporting what happened
to them, and is the key that lives in the process running your step bodies.
--admin-key permits deleting a trace and reading the audit log.
An unset list accepts a wider one in its place: write accepts the ingest
keys, admin accepts the read keys, and execute and worker accept the write
keys. A deployment configured before a scope existed keeps working.
The first split is the one that decides your blast radius, because the populations are different sizes: every agent process in your fleet needs the ingest key, and a much smaller number of people need any of the others. Handing your fleet one key that also reads would put a credential for every stored prompt into every container that emits a span, including the ones running third-party code.
That is also why capturing a run into a dataset moved to the write scope. It reads the run's input and output on the way through, so authorising it with the key your fleet carries handed every one of those containers a way to copy any stored prompt out.
Each flag takes a comma-separated list and each has an environment variable:
MCPD_INGEST_KEYS, MCPD_READ_KEYS,
MCPD_WRITE_KEYS and MCPD_ADMIN_KEYS. Use more than
one: a key per team or per fleet means a leaked key is revoked by removing it
from the list, not by re-deploying everything that emits a span. A key is
presented as Authorization: Bearer … or in the
X-Mcpd-Key header, and is compared in constant time.
The asymmetry to know about is what happens when you leave the two newer
flags unset. An unset write list accepts the ingest keys and an unset admin
list accepts the read keys, so an upgrade never revokes a key that was
working. Until you set --admin-key, a read key is a key to read
and to delete, not a strictly read-only one, and the daemon says so at
startup. Give it out accordingly, or set the flag.
Who read what
The handlers that hand payloads back record the access: the trace detail, the span and log searches, feedback, dataset items, capture, and delete. A row carries the time, a short digest of the key that acted, the action, the project, the resource and whether authorisation allowed it. The key itself is never written down, so the log is not a second place your credentials live.
Read it with GET /v1/api/audit, which takes the admin scope.
Two properties are worth knowing before you rely on it. It is not trimmed by
--retention-days, because a log that deleted itself on the same
schedule as the data it describes could not say who read the data that has
since gone. And writing a row never fails the read it describes: a row that
cannot be written is counted as
mcpd_audit_write_failures_total on /metrics, which
is the counter to alert on.
What a key does not cover
- The interface. The markup at
/is served to anyone who can reach the port. The data behind it is not: every request it makes goes through the query API, which enforces the read key. There is no login screen in the daemon, so if people rather than services need access, put it behind the identity proxy you already run. - The probes and
/metrics. Unauthenticated, on purpose, because a kubelet and a Prometheus scraper cannot carry a key. They expose counters and uptime, never payloads. - Transport. mcpd speaks plain HTTP and does not terminate TLS. Terminate it at your ingress or in your mesh. Prompts and completions cross that link in full.
- Hashes in redacted payloads. Redaction is on unless you
pass
--no-redact, and where it hashes an identifier the digest is only as private as its salt. Set--redact-key, or short identifiers can be recovered by guessing. See Redaction and retention.
Check before you expose it
mcpd config prints the configuration an invocation would use
and exits. It is the cheapest way to find out that the environment variable
you set in the wrong place did nothing:
$ docker compose -f deploy/docker/docker-compose.yml run --rm mcpd configlisten 0.0.0.0:4317storage sqlite at /data/mcpd.dbconfig file nonelog format jsonproject defaultprojects noneretention 30 daysingest auth key requiredread auth key requiredwrite auth key required (ingest keys)admin auth key required (read keys)execute auth key required (ingest keys)worker auth key required (ingest keys)signup offruns sqlite at /data/mcpd.runs.dbrun settings lease_ttl_ms 30000, reclaim_window_ms 30000, stall_window_ms 300000, idempotency_window_ms 86400000, park_ttl_ms 86400000, max_depth 8, max_step_repeats 16identity offclient ip peer addresssampling keep everythingquota unlimitedredaction onredact key setcors same-origin onlytimeout 30s per requestlimits 32 MiB per request, 10000 signals per batchderived duration and token metricsaspects offmonitors noneobjectives nonejudge offforwarding off
Six lines matter here. If any auth line reads open, that scope
is open to whoever can reach the port. A line in brackets names the wider
list that scope is inheriting from: the invocation above sets no
--admin-key, so its read keys delete as well as read. It sets no
--execute-key or --worker-key either, so its ingest
keys also start runs and claim steps, and mcpd warns about that when it
starts. Every flag
this prints is documented on
Configuration.
Backups
The backup is one file. Everything mcpd stores, including the search index,
is in the file at MCPD_DATA.
Copying that file while the daemon runs is the one thing not to do. mcpd
runs SQLite in WAL mode, so recent writes live in a -wal file
beside it, and a copy of the database alone is a copy that is missing them.
Take a consistent copy instead, from any image that has the
sqlite3 tool and mounts the same volume:
sqlite3 /data/mcpd.db ".backup '/backup/mcpd-$(date -u +%Y%m%dT%H%M).db'"
Both read a consistent snapshot while writes continue. A volume snapshot
taken by your CSI driver works too, if the driver's snapshot is atomic; a
tar of a live directory is not. Restoring is stopping the
daemon, putting the file back at MCPD_DATA, deleting the
-wal and -shm files beside it, and starting the
daemon again; Running it walks
through it.
Deleted rows free pages inside the file, they do not shrink it, so a
database that has aged out a lot of data stays the size it was. Run
VACUUM when you want the disk back.
Upgrades
Change the tag and restart. Migrations run at start and are safe to run on every start, so no step happens outside the process. Take the backup first anyway: a migration is the one moment a schema changes under your data.
The restart is a short outage, not a rolling one, for the same reason there is one replica. Agents survive it: the SDKs queue in memory and retry, and the queue is bounded, so a restart that takes seconds costs nothing and one that takes an hour drops the oldest spans and says so in the client's log.
mcpd handles SIGTERM and shuts down gracefully, which is what Kubernetes
sends. terminationGracePeriodSeconds has to outlast the longest
request, or the kubelet kills the process mid-write. A request may run for
the 30-second request timeout, so the manifests and the chart give the pod
45 seconds, and the compose file gives the container the same. Raise both
together.
Where one SQLite file stops, and what to do about it
The embedded file is the right default and it has two hard limits. Both belong to SQLite, not to mcpd.
- One writer. Reads run on their own connections and never block the writer, which is what WAL mode is for, but one process owns the file. You scale that instance up, not out.
- One volume, one availability zone. A ReadWriteOnce claim pins the pod. There is no failover to another node with the data intact.
Two settings usually buy back more than a new backend would. Sampling keeps a fraction of traces and keeps each kept trace whole, deciding by hashing the trace id so every instance in a fleet agrees. Retention caps how far back the file goes. Running one instance per team, each with its own volume, also works and keeps a noisy team from filling everyone's disk.
When those run out, point mcpd at a database. Set
externalStore.type to postgres or
clickhouse in the chart, with the connection string in
externalStore.postgresUrl or
externalStore.clickhouseUrl. The string is held in the chart's
Secret, not in the pod spec. To keep it out of your values altogether, put
it in a Secret you manage under postgres-url or
clickhouse-url, name that Secret in
keys.existingSecret, and set the type all the same: the type,
not the URL, selects the backend.
That moves the signals and leaves one file on disk: the run store, which is
embedded SQLite whatever the signal store is. With
runs.enabled true, the default, the chart keeps it on the
volume and runs one replica. To run several pods, turn the run surface off.
The chart then runs replicaCount pods, switches from
Recreate to a rolling update, and drops the
PersistentVolumeClaim, because nothing local is left to fight over, and the
daemon answers the run endpoints with a 503 that says so. The chart refuses
a replicaCount above 1 while runs are on, rather than starting
pods that each hold a run store of their own.
helm install mcpd ./deploy/helm/mcpd \
--set keys.ingest=$INGEST_KEY \
--set keys.read=$READ_KEY \
--set externalStore.type=postgres \
--set externalStore.postgresUrl=postgres://mcpd:$PGPASSWORD@db:5432/mcpd \
--set runs.enabled=false \
--set replicaCount=3 Reach for Postgres first: it lifts both limits on a database most teams already run, back up and monitor. Migration takes an advisory lock and is skipped when the schema is current, so several replicas starting together is safe. Reach for ClickHouse when retention is measured in billions of spans. A URL for a backend the type does not name is refused rather than ignored.
One thing does not change with the backend.
Retention covers the projects you wrote down. The hourly
pass deletes signals older than --retention-days in the project
named by --project and in every project in the
projects block, each on its own retention_days
where it sets one. Traces sent under a project name that appears in neither
place have no window and are never trimmed, so watch the volume if your
agents set their own.
Next
- Every flag and environment variable: Configuration.
- Probes, counters and day-to-day running: Running it.
- What never reaches the disk: Redaction and retention.
- Not deployed anything yet: Quickstart.