System overview
Component responsibilities
Runtime flow
- An operator signs in through OIDC and uses the management console with a cookie-backed control-plane session.
- The operator creates a workspace, registers a cluster or VM, and installs the generated agent command.
- The target agent authenticates with its agent key and opens an outbound WebSocket to the control plane.
- The agent sends heartbeats, capability metadata, and snapshots. The control plane persists target state and synchronizes builtin tools into the LLM gateway.
- When an operator sends a troubleshooting message, the control plane creates a run and dispatches it to the execution engine.
- The execution engine fetches run context from the control plane, streams model requests through the LLM gateway, and calls allowed tools.
- The control plane records run events and streams them back to the management console.
tools/list, stores discovered tools disabled by default, and forwards calls with platform scope headers plus any configured non-secret public headers.
Auth and trust boundaries
AcornOps uses separate credentials for separate trust boundaries:
Run-scoped JWTs include workspace, target, session, run, allowed provider, allowed model, allowed tool, and output-budget claims. Kubernetes clusters and virtual machines use the same target scope model. The LLM gateway rejects requests whose body scope does not match the token scope.
Remote MCP credentials are secret-backed. Non-secret
publicHeaders may be forwarded to remote MCP servers, but platform scope headers such as x-workspace-id, x-target-id, x-target-type, and x-run-id are reserved and are added by the gateway.
Workspace and role model
A workspace contains members, audit events, targets, Kubernetes clusters, VMs, MCP server settings, tool settings, sessions, runs, and webhooks. Server responses include role-derived permissions so clients can show available actions without reimplementing authorization rules. The deployment defines the supported workspace role templates.owner is always present, protected, and required. Deployments can disable non-owner built-ins (admin, operator, viewer, auditor) and define custom lowercase snake_case roles with supported workspace capabilities. Workspaces assign members and invitations from this deployment catalog; they cannot edit capability sets or role availability.
The control plane prevents membership changes that would leave a workspace without an owner. Only owners can assign protected roles.
Workspace audit logs are scoped to the workspace and validate filters before querying. Audit metadata is sanitized before it is stored, so raw tokens, secrets, message bodies, pod logs, auth headers, and full tool arguments are not retained. Audit retention removes persisted rows after the configured retention window.
Tool call audit events store only high-level details such as identity, source, target, duration, run id when available, and success or failure. Workspace summary responses hide operational counts from roles without workspace-data read permission; auditors keep member context but receive clusterCount: 0.
Troubleshooting conversations are owner-write and viewer-read. The user who starts a conversation can send follow-ups while they still have run creation permission. Other users with target read access can open the conversation and watch active run events live, but they cannot post into that conversation. When a user starts a separate conversation on a target with recent chat activity, the management console shows an inline warning that names recent investigators and highlights recent write-capable activity.
Data ownership
Agent snapshots are persisted by the control plane and exposed to the management console. Kubernetes snapshots include resources, events, and metrics when available. VM snapshots include host inventory, health findings, metrics, and bounded logs.
High availability posture
The Kubernetes chart defaults the management console, control plane, execution engine, and LLM gateway to multiple replicas when backed by external Postgres and Redis. The control plane uses Redis for agent routing, command routing between pods, run event fanout, and renewed scheduler leases. If a control-plane pod stops while an agent command is active, that command may fail; the agent reconnects and later commands continue through an available pod. The AgentK supports active-passive high availability through Kubernetes Lease leader election. WhenreplicaCount is greater than one, enable leader election so exactly one agent runtime connects at a time.