Operations

This page covers day-two operations for the central platform and connected Kubernetes clusters or VMs.

Upgrade flow

For production-style environments:

Select one pinned stack from the stack-versions.yaml release matrix whose execution contract version is exactly 2.
Confirm Postgres and Redis availability, then explicitly recreate the pre-release control-plane and gateway databases for this greenfield schema epoch.
Deploy gateway, control plane, execution engine, deployment metadata, and documentation as one stack. Do not roll any image independently.
Smoke test OIDC sign-in, workspace listing, agent connectivity, built-in tools, approval receipt execution, workspace and individual credential lifecycles, and a read-only run.

Pin image tags during upgrades so deployments are reproducible. This version is a first-install/reset cutover, not a rolling database upgrade. The final baselines intentionally contain no pre-release data conversion paths. For external Postgres, take a provider snapshot or pg_dump, then use an administrative connection to drop and recreate the control-plane database. For local Compose, run task local-reset. Returning to an older schema requires restoring a matching database backup and complete pinned image matrix; an application-only rollback is unsafe across schema epochs. Control-plane rollouts keep the service available through multiple replicas. Connected agents reconnect to an available pod during the rollout, but commands that are already running can fail or time out and may need to be retried.

Durable state

Inspect durable state:

Store	Contains
Control-plane Postgres	Workspaces, users, members, targets, sessions, Agents, Workflows, schedules, trigger deliveries, approvals, attempts, append-only run events, dispatch intent, report sources, invitations, and webhooks
LLM-gateway Postgres	Catalog discovery, Agent- and target-owned MCP installations, reviewed tool state, provider connections, and encrypted secrets when using database-backed secrets
External secret manager	Provider keys and MCP auth secrets when Vault or another manager is enabled

Redis is used for coordination, rate limits, control-plane agent ownership, cross-pod command routing, and run event fanout. Postgres claims and leases are authoritative for automation scheduling and dispatch.

Health checks

Verify the platform from the outside in:

management console loads over TLS,
GET /api/v1/me works after sign-in,
OIDC callback returns to the management console,
a workspace can be created and listed,
target agents can connect over WebSocket,
a run streams events through GET /api/v1/runs/{runId}/stream,
the LLM gateway is ready with a fresh JWKS cache,
remote MCP discovery and workspace/individual connection tests pass when the kill switch is enabled,
webhooks record delivery attempts for subscribed events.
workspace automation diagnostics report no dispatch older than 30 seconds, scheduler lag over 60 seconds, stale approvals, or unexpected needs_review runs.
webhook backlog, retries, superseded issue notifications, terminal failures, and lease recovery remain within expected bounds.

For production, keep component API docs disabled unless you intentionally expose them on a protected network.

Logs to inspect

Symptom	Start with
Sign-in fails	Control-plane logs and OIDC provider redirect URI/client settings
Console cannot call API	Management console runtime config, CORS, ingress, and cookie settings
Target disconnected	target connector logs, control-plane WebSocket logs, credential state, host CA trust, and outbound network policy
Runs stuck pending	Control-plane dispatch logs, execution-engine logs, Redis connectivity
Agent or Workflow dispatch delayed	Workspace automation diagnostics, `control_plane_automation_runtime` gauges, control-plane outbox worker logs, execution-engine readiness
Run needs review	Approval and terminal audit records; confirm whether a write result was uncertain before authorizing resume
Model calls fail	LLM-gateway logs, effective provider credential status, run JWT claims, provider allow-list
MCP tools fail	Immutable run snapshot, exact server/tool reference, reviewed state, principal connection scopes, permission-mode decision, gateway egress policy, and call-time target identity for Targets MCP calls
Webhook verification fails	Consumer raw-body handling, timestamp, signature header, subscription secret
Webhook backlog grows	Worker pause setting, Postgres health, retry metrics, destination status

Rate limits and budgets

Use both platform and gateway limits:

run max runtime,
max steps,
max tool calls,
duplicate tool-call cap,
max output tokens,
per-window LLM request limits,
per-window tool-call limits.
per-user, per-installation MCP connect/verify limits.

Keep read-write runs limited to roles and targets where remediation is expected. AgentV exposes only restart_service, and only when the host helper and exact local allowlist are enabled.

Secret rotation

Rotate secrets with service-specific rollout plans:

Secret type	Rotation impact
OIDC client secret	Browser sign-in can fail until control-plane pods reload the new value
Internal service tokens	Rotate paired services together or support overlap during rollout
AgentK key	Use the Kubernetes agent-key rotation flow
AgentV credential	Generate a replacement enrollment command; the previous credential remains usable until the candidate connects and commits
Provider API keys	Replace the write-only credential in workspace AI settings or platform AI settings, then verify a read-only run
MCP credential	Revoke or rotate the least-privilege credential on the exact installation, then Verify; service identities require workspace-managed mode
Webhook signing secrets	Create or rotate the subscription secret and update the consumer

Do not reuse internal service tokens across environments. For Vault KV v2, MCP credential cleanup must delete metadata under the configured namespace, mount, prefix, and workspace path, not only the latest data version. Alert on durable cleanup retry failures after member or workspace removal. Membership revocation proceeds even when Vault is unavailable; queued cleanup must complete when the backend recovers. Load both observability/prometheus/alerts/llm-gateway.rules.yaml and observability/prometheus/alerts/control-plane-automation.rules.yaml from the deployment repository. The MCP rules alert on secret cleanup failure, sustained runtime 401/403 transitions, and schedules auto-paused for connection readiness. External MCP reachability is deliberately not a platform readiness dependency.

Public route drift

Keep these routes consistent across deployment, OIDC provider settings, docs, and management console runtime config:

https://console.example.com/
https://api.example.com/api/v1
https://docs.acornops.dev/
wss://api.example.com/api/v1/agent/connect

Replace example.com with domains you control. The public demo uses https://console.demo.acornops.dev/ and https://api.demo.acornops.dev. Default OIDC settings use the console-host callback, such as https://console.example.com/api/v1/auth/oidc/callback, so browser session cookies are set on the console origin. If you override the redirect URI, update the provider allow-list and deployment config together. Also register https://console.example.com/api/v1/auth/oidc/logout/callback as an exact post-logout redirect. If logout reports that only the AcornOps session was cleared, verify the provider discovery document exposes end_session_endpoint or configure the public userAccess.oidc.logout.endSessionEndpointOverride. Never use an internal service hostname for the browser-facing logout endpoint. Keep query strings out of edge and ingress access logs. OIDC callbacks and logout handoffs carry short-lived authorization codes, state values, or opaque single-use handles in the query string, while a provider-controlled referrer could include its complete logout URL. The supplied Compose Nginx proxy logs only the normalized URI path and omits referrers; configure equivalent redaction when using a separate ingress controller or external load balancer. Root acornops.dev is reserved outside the platform API surface. Public documentation is served directly from docs.acornops.dev.

Start here

Install

Configure

Govern

Operate

Upgrade flow

Durable state

Health checks

Logs to inspect

Rate limits and budgets

Secret rotation

Public route drift

​Upgrade flow

​Durable state

​Health checks

​Logs to inspect

​Rate limits and budgets

​Secret rotation

​Public route drift

Upgrade flow

Durable state

Health checks

Logs to inspect

Rate limits and budgets

Secret rotation

Public route drift