> ## Documentation Index > Fetch the complete documentation index at: https://docs.acornops.dev/llms.txt > Use this file to discover all available pages before exploring further. # Operations > Operate AcornOps safely after deployment This page covers day-two operations for the central platform and connected Kubernetes clusters or VMs. ## Upgrade flow For production-style environments: 1. Select one pinned stack from the [`stack-versions.yaml` release matrix](https://github.com/acornops/acornops-deployment/blob/main/release/stack-versions.yaml) whose execution contract version is exactly 2. 2. Confirm Postgres and Redis availability, then explicitly recreate the pre-release control-plane and gateway databases for this greenfield schema epoch. 3. Deploy gateway, control plane, execution engine, deployment metadata, and documentation as one stack. Do not roll any image independently. 4. Smoke test OIDC sign-in, workspace listing, agent connectivity, built-in tools, approval receipt execution, workspace and individual credential lifecycles, and a read-only run. Pin image tags during upgrades so deployments are reproducible. This version is a first-install/reset cutover, not a rolling database upgrade. The final baselines intentionally contain no pre-release data conversion paths. For external Postgres, take a provider snapshot or `pg_dump`, then use an administrative connection to drop and recreate the control-plane database. For local Compose, run `task local-reset`. Returning to an older schema requires restoring a matching database backup and complete pinned image matrix; an application-only rollback is unsafe across schema epochs. Control-plane rollouts keep the service available through multiple replicas. Connected agents reconnect to an available pod during the rollout, but commands that are already running can fail or time out and may need to be retried. ## Durable state Inspect durable state: | Store | Contains | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Control-plane Postgres | Workspaces, users, members, targets, sessions, Agents, Workflows, schedules, trigger deliveries, approvals, attempts, append-only run events, dispatch intent, report sources, invitations, and webhooks | | LLM-gateway Postgres | Catalog discovery, Agent- and target-owned MCP installations, reviewed tool state, provider connections, and encrypted secrets when using database-backed secrets | | External secret manager | Provider keys and MCP auth secrets when Vault or another manager is enabled | Redis is used for coordination, rate limits, control-plane agent ownership, cross-pod command routing, and run event fanout. Postgres claims and leases are authoritative for automation scheduling and dispatch. ## Health checks Verify the platform from the outside in: * management console loads over TLS, * `GET /api/v1/me` works after sign-in, * OIDC callback returns to the management console, * a workspace can be created and listed, * target agents can connect over WebSocket, * a run streams events through `GET /api/v1/runs/{runId}/stream`, * the LLM gateway is ready with a fresh JWKS cache, * remote MCP discovery and workspace/individual connection tests pass when the kill switch is enabled, * webhooks record delivery attempts for subscribed events. * workspace automation diagnostics report no dispatch older than 30 seconds, scheduler lag over 60 seconds, stale approvals, or unexpected `needs_review` runs. * webhook backlog, retries, superseded issue notifications, terminal failures, and lease recovery remain within expected bounds. For production, keep component API docs disabled unless you intentionally expose them on a protected network. ## Logs to inspect | Symptom | Start with | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Sign-in fails | Control-plane logs and OIDC provider redirect URI/client settings | | Console cannot call API | Management console runtime config, CORS, ingress, and cookie settings | | Target disconnected | target connector logs, control-plane WebSocket logs, credential state, host CA trust, and outbound network policy | | Runs stuck pending | Control-plane dispatch logs, execution-engine logs, Redis connectivity | | Agent or Workflow dispatch delayed | Workspace automation diagnostics, `control_plane_automation_runtime` gauges, control-plane outbox worker logs, execution-engine readiness | | Run needs review | Approval and terminal audit records; confirm whether a write result was uncertain before authorizing resume | | Model calls fail | LLM-gateway logs, effective provider credential status, run JWT claims, provider allow-list | | MCP tools fail | Immutable run snapshot, exact server/tool reference, reviewed state, principal connection scopes, permission-mode decision, gateway egress policy, and call-time target identity for Targets MCP calls | | Webhook verification fails | Consumer raw-body handling, timestamp, signature header, subscription secret | | Webhook backlog grows | Worker pause setting, Postgres health, retry metrics, destination status | ## Rate limits and budgets Use both platform and gateway limits: * run max runtime, * max steps, * max tool calls, * duplicate tool-call cap, * max output tokens, * per-window LLM request limits, * per-window tool-call limits. * per-user, per-installation MCP connect/verify limits. Keep read-write runs limited to roles and targets where remediation is expected. AgentV exposes only `restart_service`, and only when the host helper and exact local allowlist are enabled. ## Secret rotation Rotate secrets with service-specific rollout plans: | Secret type | Rotation impact | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | OIDC client secret | Browser sign-in can fail until control-plane pods reload the new value | | Internal service tokens | Rotate paired services together or support overlap during rollout | | AgentK key | Use the Kubernetes agent-key rotation flow | | AgentV credential | Generate a replacement enrollment command; the previous credential remains usable until the candidate connects and commits | | Provider API keys | Replace the write-only credential in workspace AI settings or platform AI settings, then verify a read-only run | | MCP credential | Revoke or rotate the least-privilege credential on the exact installation, then Verify; service identities require workspace-managed mode | | Webhook signing secrets | Create or rotate the subscription secret and update the consumer | Do not reuse internal service tokens across environments. For Vault KV v2, MCP credential cleanup must delete metadata under the configured namespace, mount, prefix, and workspace path, not only the latest data version. Alert on durable cleanup retry failures after member or workspace removal. Membership revocation proceeds even when Vault is unavailable; queued cleanup must complete when the backend recovers. Load both `observability/prometheus/alerts/llm-gateway.rules.yaml` and `observability/prometheus/alerts/control-plane-automation.rules.yaml` from the deployment repository. The MCP rules alert on secret cleanup failure, sustained runtime 401/403 transitions, and schedules auto-paused for connection readiness. External MCP reachability is deliberately not a platform readiness dependency. ## Public route drift Keep these routes consistent across deployment, OIDC provider settings, docs, and management console runtime config: * `https://console.example.com/` * `https://api.example.com/api/v1` * `https://docs.acornops.dev/` * `wss://api.example.com/api/v1/agent/connect` Replace `example.com` with domains you control. The public demo uses `https://console.demo.acornops.dev/` and `https://api.demo.acornops.dev`. Default OIDC settings use the console-host callback, such as `https://console.example.com/api/v1/auth/oidc/callback`, so browser session cookies are set on the console origin. If you override the redirect URI, update the provider allow-list and deployment config together. Also register `https://console.example.com/api/v1/auth/oidc/logout/callback` as an exact post-logout redirect. If logout reports that only the AcornOps session was cleared, verify the provider discovery document exposes `end_session_endpoint` or configure the public `userAccess.oidc.logout.endSessionEndpointOverride`. Never use an internal service hostname for the browser-facing logout endpoint. Keep query strings out of edge and ingress access logs. OIDC callbacks and logout handoffs carry short-lived authorization codes, state values, or opaque single-use handles in the query string, while a provider-controlled referrer could include its complete logout URL. The supplied Compose Nginx proxy logs only the normalized URI path and omits referrers; configure equivalent redaction when using a separate ingress controller or external load balancer. Root `acornops.dev` is reserved outside the platform API surface. Public documentation is served directly from `docs.acornops.dev`.