evaluation-controller¶

The remote skill-evaluation engine, and the one optional controller: deployments that never submit remote evaluations simply do not run it. It serves the evolve-facing HTTP API (workspace upload, submission, snapshot, SSE monitoring, cancellation) and runs the reconcilers that expand each submitted Evaluation into EvaluationUnit children, schedule them through the sandboxed agent-Job machinery with bounded concurrency, collect each pod's result stream, and expire finished evaluations on a TTL.

evaluation-controller serve --namespace patchy --auth-config /etc/patchy/auth/config.yaml

The division of knowledge is strict: evolve owns every evaluation semantic (specs, grading, the LLM judge, baselines), co-located with the uploaded workspace bundle inside the pod, which runs evolve exec-unit from a per-harness evolve-runner image. Patchy owns scheduling, sandboxing, and state — it interprets only the pod's result and fatal events; the finished results entry is opaque payload stored in a per-unit ConfigMap the client reassembles locally.

Flags¶

The shared flags (--listen-addr is the API's own address), plus:

Credentials reuse the shared --<harness>-secret{,-key,-env} flags (investigation-controller): one Secret per model vendor in the agents namespace, whichever fleet exercises it. The evolve-runner images ship from the evolve repository (ghcr.io/bitwise-media-group/evolve-runner-<harness>), pinned by the operator like today's agent images; a submission's clientVersion is recorded and skew is warned, never enforced.

Authentication configuration¶

--auth-config points at a YAML file, conventionally a mounted Secret (patchy-evaluation-auth, key config.yaml). Unlike the status server there is no unauthenticated posture — every route except GET /api/v1/auth/info is bearer-authenticated, so a missing or invalid file is a startup error.

mode: oidc # none | oidc
oidc: # mode: oidc only
  issuerURL: https://sso.example.com
  clientID: evolve # the PUBLIC (PKCE) client evolve logs in as — no client secret
  # scopes: [openid, offline_access, profile, email, groups]
  claims: # claim NAMES, mapped onto the identity
    username: email
    groups: groups

mode: none (dev/e2e) short-circuits authentication with a fixed identity and bypasses authorization entirely.

evolve login needs zero OIDC configuration client-side: it reads GET /api/v1/auth/info (issuer, clientID, scopes) and runs the authorization-code + PKCE flow against the issuer with a localhost redirect. The operator prerequisite is a public OIDC client (no secret) allowing http://127.0.0.1:*/callback redirects.

Authorization¶

Native Kubernetes RBAC on the evaluations resource is the entire authorization surface — no custom verbs, no admission-policy involvement:

Bind users or SSO groups (as mapped by the claims configuration) to a role carrying those verbs — the kustomize base ships an example tier (patchy-evaluations-submitter in rbac.users.example.yaml), and the chart renders one behind evaluationController.rbac.userRoles.

Workspace bundles¶

Bundles are content-addressed by sha256: HEAD /api/v1/workspaces/{digest} dedupes before upload, PUT /api/v1/workspaces/{digest} streams the gzip tarball through to source-controller's NetworkPolicy-gated internal endpoint (:9791), which verifies the digest before caching. The digest is also the fetch capability the agent pod downloads the bundle with. Bundles unused for --workspace-retention (source-controller, default 168h) are swept; a needed-but-swept bundle settles its unit as WorkspaceLost, prompting the client to re-upload and resubmit.