Skip to main content

Orchestrator

The orchestrator is the control plane and the single entry point for everything. It's a FastAPI service (deployed as a Kubernetes Deployment) that registers clients, builds images, provisions and tears down environment pods, forwards traffic into them, and persists all state in PostgreSQL.

Inside the orchestrator (click a node for source)​

Responsibilities​

  • Client lifecycle — register a client (with optional node pre-provisioning), track heartbeats, and on stop/finish tear down or release its servers.
  • Server lifecycle — start (or reuse a matching finished server), stop, finish (release for reuse), and restart environment pods, waiting for readiness. Reuse matches on seven fields, one of which is that the candidate is FINISHED — so a client that only ever stops its servers never reuses one. The start response reports reused either way; see Server reuse. A start request may tag the server with extra Kubernetes labels and annotations; keys IdeGYM manages are rejected, and a reused server keeps the metadata it was created with (see Tagging a sandbox).
  • Server status and keepalive — report a server's availability, pod phase and idle time, and hold an idle-but-in-use server against the watcher's inactivity reaper on request. The pod fields are best-effort: if Kubernetes cannot be reached, status is still served from the database with the pod fields left null. A hold is cleared when a finished server is handed to another client for reuse.
  • Image builds — accept image-builder YAML, compile each Image to a spec, and submit it through a pluggable build backend (Kaniko by default, or GKE Cloud Build) driven by a shared ImageBuildService (see image builder).
  • Forwarding — proxy HTTP, MCP, and WebSocket requests from clients into the right server pod, and persist every request/response for offline reward computation.
  • Async operations — long-running calls return an operation ID the client polls to completion; this keeps connections short and the API responsive.
  • Snapshots — on GKE, prepare and restore pod snapshots to skip cold-start work.

Key building blocks​

ConcernWhereNotes
App assemblymain.pycreate_app() builds the app, combines the FastAPI + MCP lifespans, mounts routers, instruments OpenTelemetry
MCP servermcp.pyA thin FastMCP layer over the same handlers as REST — no separate state
Routersrouter/One module per concern: client, server, build_images, forwarding, async_operation, snapshot, dashboard, diagnostics
Statedatabase/Async SQLAlchemy + asyncpg; Alembic migrations run on startup
Image buildimage_build_service.pyBackend-agnostic ImageBuildService drives a pluggable ImageBuilder (Kaniko / GKE Cloud Build)
K8s resourceskubernetes_client.pyCreates Kaniko Jobs (Kaniko backend) and server Deployments via the K8s API

Persistence model​

PostgreSQL holds clients, servers, build jobs, async operations, and snapshots. Because the orchestrator stores every forwarded request and response, an evaluation harness can recompute rewards offline and reproduce runs without re-executing them. Concurrency-sensitive operations use real Postgres features — FOR UPDATE SKIP LOCKED and advisory locks — to coordinate the watcher and request handlers safely.

Configuration​

Configuration is nested Pydantic models in api/src/idegym/api/config.py; load_config() builds Config from the environment, and a variable that is unset leaves the field's default. Environment variables are the only override mechanism — no config files, no CLI arguments. Notable knobs include database connection, connection limits, asyncio debug/dump, the MCP stateless_http mode, and node-pool scheduling. See deployment for the environment variables and Helm values that drive these.

View source​