Deployment runbook

This is the operator procedure for running the Argon runtime as a served platform — the reference artifacts and runbook of RFD 0057 D5 (deployment packaging). It assumes the deployment and trust model — read that first: it fixes the one topology in which serving is safe, and this runbook is how you stand that topology up. Reference artifacts (a Dockerfile, a systemd unit, an env-file template) live in the repository’s deploy/ directory; they are templates you adapt, not a turnkey stack.

The topology, in one picture

              ┌──────────────────────────────────────────────────┐
   client ───►│  Authenticating gateway  (same host / namespace)  │
  (mTLS /     │  • terminates client auth                          │
   OIDC /     │  • OVERWRITES X-Tenant-Id / X-Principal-Id /       │
   API key)   │    X-Standpoint  (never passes the client's)       │
              │  • rate-limits, back-pressures                     │
              └───────────────────────┬──────────────────────────┘
                                       │  loopback / private only
                                       ▼
              ┌──────────────────────────────────────────────────┐
              │  ox runtime serve   127.0.0.1:7780                 │
              │  • trusts the three headers at face value          │
              │  • /healthz (liveness)  /readyz (readiness)        │
              │  • network-isolated: only the gateway reaches it   │
              └──────────────────────────────────────────────────┘

Two processes, co-located. The gateway is the only thing that can open a connection to the runtime, and it is the only thing that sets the trust headers. If a client can reach the runtime port directly, it can read and write any tenant’s data — the runtime has no auth to stop it. This is the whole reason for the loopback bind (the runtime refuses a non-loopback host) and the network isolation.

Wiring the orchestrator to the probes

The runtime splits liveness from readiness on purpose (serving surface). Wire them to the two distinct orchestrator actions:

The discipline that matters: do not gate liveness on storage. A transient Postgres outage should drain traffic (readiness fails, the LB stops routing here), not kill and restart the process (which fixes nothing and amplifies the outage). /healthz answers 200 whenever the request pipeline is up; /readyz answers 503 when storage is unreachable. Both bypass admission control, so they keep answering under load — the point of the split is that an overloaded server is still live, and readiness reports the overload rather than being shed by it.

A Kubernetes example, for orientation only (adapt timings to your fixpoint cost):

livenessProbe:
  httpGet: { path: /healthz, port: 7780 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 7780 }
  periodSeconds: 5
  failureThreshold: 2
  timeoutSeconds: 2   # readiness must answer fast on a HUNG backend

The cross-version schema-change gate

When a freshly built .oxbin is loaded over a scope that already holds data, the runtime compares the artifact’s schema identity against the scope’s recorded identity. A mismatch is refused by default (ARGON_RUNTIME_SCHEMA_MISMATCH, R-B7) — serving a changed schema over existing individuals risks reading them back under the wrong shape. To proceed deliberately, set ARGON_ACCEPT_SCHEMA_CHANGE=1 for that load:

ARGON_ACCEPT_SCHEMA_CHANGE=1 ox runtime serve --project . --oxbin target/main.oxbin \
  --storage pg --database-url "$ARGON_DATABASE_URL"

Treat it as a migration step, never a standing default — leaving it set defeats the guard on every subsequent deploy.

Secrets

The runtime takes one secret: the Postgres --database-url (only when --storage pg). The runtime has no credentials of its own — there is nothing else to provision. Keep the URL out of image layers, out of the unit text, and out of the process table where you can:

• systemd — put it in a root-owned, group-readable EnvironmentFile (/etc/argon/runtime.env, mode 0640), referenced by the unit; see deploy/runtime.env.example.
• containers — feed it as an orchestrator secret mounted to the env, not baked with COPY or ENV.

Config surface

The runtime is configured by a layered surface — flags > env > file > defaults (RFD 0057 D4). Pass a TOML file with --config; a value there is the baseline an ARGON_* env var overrides, and a CLI flag overrides both. Omit --config and the surface is the historical `flags > env

defaults. Unknown keys in the file are a loud parse error (a typo never silently no-ops); a referenced-but-unset ${ENV}secret is a loud refusal. The committedserve.config.example.toml(in theoxc-serve` crate) documents every section.

Telemetry

Observability rides the runtime’s existing tracing bridge (RFD 0057 D2): set an OTLP collector endpoint and the runtime exports vendor-neutral traces and metrics over OTLP/gRPC. It is off by default — with no endpoint configured nothing is exported and the instrumentation is inert (no exporter, no background pipeline, no measurable overhead), so you opt in by configuring the endpoint, not by a feature flag.

ox runtime serve --project . --oxbin target/main.oxbin \
  --otel-endpoint http://otel-collector:4317
# or: ARGON_OTEL_ENDPOINT=http://otel-collector:4317, or [telemetry] in the file.

• Traces — each request is a root span; the handler phases (dispatch, reason, persist) attach as child spans, so a slow request is attributable to a phase. A trace crossing the gateway joins its span via propagated context.
• Metrics — request latency by path + tenant, mutation throughput, check-violation count, store/event-log growth, IVM-rebuild frequency, and reasoner-budget hits.

It is OTLP, not Prometheus-direct: one bridge gives both traces and metrics, vendor-neutrally. A Prometheus consumer runs an OTLP→Prometheus collector.

Lifecycle: graceful drain and hot-reload

Graceful drain on stop (RFD 0057 D3). On SIGTERM (an orchestrator stop) or SIGINT (Ctrl-C) the runtime stops accepting new connections and drains in-flight requests — bounded by the per-request deadline, so the drain is finite — then exits clean. A request in flight at the signal is finished or deadline-cut, never severed mid-commit. ox runtime serve returns only after the drain. The systemd unit’s default SIGTERM stop is therefore a clean drain; set TimeoutStopSec a hair above the request deadline (the reference unit uses 35s for the 30s default) so the longest in-flight request can finish before systemd escalates to SIGKILL.

Hot-reload (RFD 0057 D3). With the watcher on (default), the runtime watches the served .oxbin and, on change, validates the new artifact and performs an atomic module swap with no dropped in-flight requests — new requests bind the new module, in-flight requests complete against the one they started on. The cross-version schema-change gate still applies to a reload: an additive change reloads freely; a type-incompatible change against a live A-box is refused (ARGON_RUNTIME_MODULE_INCOMPATIBLE) unless ARGON_ACCEPT_SCHEMA_CHANGE=1 opts in. Reload never silently accepts a model that would invalidate persisted state. Disable the watcher (--no-watch, or watch = false) to fix the model for the process lifetime and roll a model only by restart.

Deploy on one box

A minimal end-to-end stand-up, mem storage, behind a co-located gateway.

1. Build the artifact.

   ox build           # produces target/<name>.oxbin
   

2. Run the runtime on loopback. Either directly:

   ox runtime serve --project . --oxbin target/main.oxbin \
     --host 127.0.0.1 --port 7780 --storage mem
   

   or via the reference systemd unit:

   sudo install -m 0755 "$(command -v ox)" /usr/local/bin/ox
   sudo install -d -o argon -g argon /srv/argon          # + place the .oxbin under it
   sudo install -m 0640 -o root -g argon deploy/runtime.env.example /etc/argon/runtime.env
   sudo cp deploy/argon-runtime.service /etc/systemd/system/
   sudo systemctl daemon-reload && sudo systemctl enable --now argon-runtime
   

3. Front it with a gateway that authenticates the client and overwrites the trust headers. A minimal nginx sketch (auth elided — wire your own auth_request / mTLS / OIDC):

   server {
     listen 443 ssl;
     # ... terminate client auth here ...
     location / {
       # OVERWRITE the trust headers — never pass the client's through.
       proxy_set_header X-Tenant-Id    $authenticated_tenant;
       proxy_set_header X-Principal-Id $authenticated_principal;
       proxy_set_header X-Standpoint   $resolved_standpoint;
       proxy_pass http://127.0.0.1:7780;
     }
   }
   

4. Verify the probes.

   curl -fsS http://127.0.0.1:7780/healthz   # → 200 {"status":"live"}
   curl -fsS http://127.0.0.1:7780/readyz    # → 200 {"status":"ready"} (or 503 if storage is down)
   

5. Confirm the isolation. From outside the box, the runtime port (7780) must be unreachable — only 443 (the gateway) is exposed. If a remote client can hit 7780, the deployment is unsafe regardless of the gateway.