Operations - AcornOps

This page covers day-two operations for the central platform and connected Kubernetes clusters or VMs.

Upgrade flow

For production-style environments:

Review release notes and changed configuration keys.
Confirm Postgres and Redis availability and capacity.
Run service init jobs before starting updated services.
Roll the management console, control plane, execution engine, and LLM gateway.
Verify OIDC sign-in, workspace listing, agent connectivity, and a read-only run.
Roll target agents separately when chart, package, or agent behavior changes.

Pin image tags during upgrades so deployments are reproducible. Control-plane rollouts keep the service available through multiple replicas. Connected agents reconnect to an available pod during the rollout, but commands that are already running can fail or time out and may need to be retried.

Durable state

Inspect durable state:

Store	Contains
Control-plane Postgres	Workspaces, users, members, targets, sessions, Agents, Workflows, schedules, trigger deliveries, approvals, attempts, append-only run events, dispatch intent, report sources, invitations, and webhooks
LLM-gateway Postgres	MCP registry, gateway metadata, encrypted provider or MCP secrets when using database-backed secrets
External secret manager	Provider keys and MCP auth secrets when Vault or another manager is enabled

Redis is used for coordination, rate limits, control-plane agent ownership, cross-pod command routing, and run event fanout. Postgres claims and leases are authoritative for automation scheduling and dispatch.

Health checks

Verify the platform from the outside in:

management console loads over TLS,
GET /api/v1/me works after sign-in,
OIDC callback returns to the management console,
a workspace can be created and listed,
target agents can connect over WebSocket,
a run streams events through GET /api/v1/runs/{runId}/stream,
the LLM gateway is ready with a fresh JWKS cache,
remote MCP server discovery and connection tests pass for configured servers,
webhooks record delivery attempts for subscribed events.
workspace automation diagnostics report no dispatch older than 30 seconds, scheduler lag over 60 seconds, stale approvals, or unexpected needs_review runs.

For production, keep component API docs disabled unless you intentionally expose them on a protected network.

Logs to inspect

Symptom	Start with
Sign-in fails	Control-plane logs and OIDC provider redirect URI/client settings
Console cannot call API	Management console runtime config, CORS, ingress, and cookie settings
Agent disconnected	target agent logs, control-plane WebSocket logs, agent key, outbound network policy
Runs stuck pending	Control-plane dispatch logs, execution-engine logs, Redis connectivity
Agent or Workflow dispatch delayed	Workspace automation diagnostics, `control_plane_automation_runtime` gauges, control-plane outbox worker logs, execution-engine readiness
Run needs review	Approval and terminal audit records; confirm whether a write result was uncertain before authorizing resume
Model calls fail	LLM-gateway logs, provider keys, run JWT claims, provider allow-list
MCP tools fail	Gateway egress policy, MCP server discovery, public header validation, auth secret lookup
Webhook verification fails	Consumer raw-body handling, timestamp, signature header, subscription secret

Rate limits and budgets

Use both platform and gateway limits:

run max runtime,
max steps,
max tool calls,
duplicate tool-call cap,
max output tokens,
per-window LLM request limits,
per-window tool-call limits.

Keep read-write runs limited to roles and Kubernetes clusters where remediation is expected. VM troubleshooting runs are read-only.

Secret rotation

Rotate secrets with service-specific rollout plans:

Secret type	Rotation impact
OIDC client secret	Browser sign-in can fail until control-plane pods reload the new value
Internal service tokens	Rotate paired services together or support overlap during rollout
Agent keys	Use the control-plane agent-key rotation flow per target
Provider API keys	Update the gateway secret backend, then verify a read-only run
MCP server auth secrets	Rotate through the MCP server auth fields, test the server connection, then verify an enabled tool
Webhook signing secrets	Create or rotate the subscription secret and update the consumer

Do not reuse internal service tokens across environments.

Public route drift

Keep these routes consistent across deployment, OIDC provider settings, docs, and management console runtime config:

https://console.example.com/
https://api.example.com/api/v1
https://docs.acornops.dev/
wss://api.example.com/api/v1/agent/connect

Replace example.com with domains you control. The public demo uses https://console.demo.acornops.dev/ and https://api.demo.acornops.dev. Default OIDC settings use the console-host callback, such as https://console.example.com/api/v1/auth/oidc/callback, so browser session cookies are set on the console origin. If you override the redirect URI, update the provider allow-list and deployment config together. Root acornops.dev is reserved outside the platform API surface. Public documentation is served directly from docs.acornops.dev.

​Upgrade flow

​Durable state

​Health checks

​Logs to inspect

​Rate limits and budgets

​Secret rotation

​Public route drift