Field guide
Agentic harnesses for game development: the false-green problem
AI agents can drive a game engine today — spawn actors, wire Blueprints, edit anim graphs, run the build. The unsolved part isn't capability. It's that a harness will cheerfully report success it did not produce, and nothing in the stack is obliged to notice.
Contents
1. What an agentic harness actually is
Strip the marketing off and a harness for game development is three things stacked:
- A channel — some live connection into the running editor process. Usually an HTTP listener or socket inside a plugin, sometimes an MCP server, occasionally a file-drop watched by a tick. This is the part everyone builds first and demos.
- A command surface — the vocabulary the agent can speak: spawn this actor, set this property, compile this Blueprint, enter play-in-editor, read this asset. This is the part that gets long.
- A verification layer — the part that establishes whether what the agent asked for actually happened, in the state that matters, and is still true a minute later. This is the part that usually doesn't exist.
The first two are being commoditized fast, and that is genuinely good news. Epic ships an experimental MCP Server plugin in UE 5.8. Several open-source projects expose broad editor control. If your problem is "an agent cannot reach my editor," that problem is closing on its own.
The glue is commoditized. The guarantee is not. The interesting engineering has moved entirely into layer three.
2. The five false-greens
A false-green is any path where the harness reports success without success having occurred. They are not exotic edge cases — every one of these is a routine occurrence in production agent work, and each has a distinct tell.
1Acknowledgement mistaken for effect
The channel returns 200 because the command was received and parsed, not because the engine did anything. The agent reads the ack as done and moves on, building three more steps on a foundation that isn't there.
Tell: success arrives faster than the operation could physically take. A Blueprint compile that returns in 4ms did not compile.
2Shape checked, semantics unchecked
A conformance test asserts the actor exists, the property is present, the field has the right type. All true — and the value is meaningless, the reference is null, the component is attached to the wrong socket. The test proves structure and claims behavior.
Tell: the test suite would still pass against a stub that returns well-formed defaults. If a mock can satisfy your acceptance test, your acceptance test is measuring the mock.
3Editor state versus runtime state
The change is real in the editor and absent at runtime — or the reverse. Play-in-editor diverges from a packaged build; a value set on the CDO doesn't reach an instance; an asset is dirty in memory and never saved to disk. The agent verifies the wrong one of the two worlds.
Tell: verification never crosses a process or persistence boundary. Nothing was saved, reloaded, or replayed.
4Silent version drift
The harness was proven against one engine build. The project moves to the next. APIs shift underneath, and the failure surfaces not as a refusal but as subtly wrong behavior — or worse, as a destructive operation on a project that no longer matches the harness's assumptions.
Tell: nothing anywhere pins or checks the engine build hash. The harness has no concept of "outside my proven matrix."
5Prose errors an agent cannot act on
Something fails and emits "Something went wrong" or a stack trace with no code. The agent — which is the entity actually reading this at 3am — cannot distinguish a transient retry from a hard stop, so it retries, invents a workaround, or reports partial success. The failure becomes a false-green one layer up.
Tell: the error surface has no stable identifier, no machine-readable body, and no documented resolution keyed to it.
3. Why game engines make this worse than web work
Agent harnesses for web development get a lot of verification for free. Game engine work does not, for four structural reasons:
- The editor is stateful and long-lived. A web agent can rerun the build from clean. An engine agent operates on a process holding gigabytes of unsaved mutable state, where "rerun it" costs minutes and may not reproduce.
- Correctness is frequently visual or behavioral. "The mech's dash feels wrong" and "the anim graph blends at the wrong frame" do not reduce to an exit code. A screenshot nobody looked at is not a passing test.
- CI can't hold the substrate. A full editor with play-in-editor doesn't run on a stock cloud runner, so the usual "green build proves it" reflex has no place to stand — and a badge that implies it does is overclaiming.
- Failures are expensive and often silent. A wrong refactor in a web app throws. A wrong operation in an editor can corrupt an asset, poison a redirector chain, or produce a project that opens fine and misbehaves three sessions later.
4. The fix: a verification contract
The move that works is to stop treating verification as tests bolted onto a harness, and start treating it as a contract the harness is built to satisfy — one an outside party can check without trusting you. Concretely, three schemas do the load-bearing work:
The wire protocol carries identity, capability, and predicates
Not just "spawn actor, ok." The command envelope declares who is asking, what capability set the target claims, and what invariant must hold afterward for this to count. Predicates travel with the command, in the schema, so that acknowledgement can never be mistaken for effect — the ack is required to carry evidence about the predicate. This kills false-green #1 and most of #2 by construction rather than by discipline.
The error envelope makes failures machine-actionable
Every failure emits a coded, signed envelope — a stable code, a resolution URL that content-negotiates HTML for humans and JSON for agents, observed-versus-expected, and a repro token. The code is the primary key that unifies emission, documentation, deduplication, and telemetry. The discipline that makes it real: a resolution page is code. It cannot publish unless CI reproduces the failure and proves the documented fix flips it green, or you get the rotting-docs treadmill where an agent confidently executes last year's fix on someone's project.
The proof envelope makes the result checkable by strangers
Each gate outcome — pass and refuse — appends a signed record: artifact hash, gate id, the assertion in words, the fixture-set hash, verdict, timestamp, chained to the previous entry. Sign the content, not just the wrapper. Then "this harness refuses to false-green" stops being a claim in a README and becomes a property anyone can verify with a free tool against your published log.
One more distinction that matters more than it sounds: transport conformance and semantic conformance are different oracles and must be split. Transport conformance is exact-match on protocol shape — sequence numbers, field presence, ack pairing, NACK semantics — and is deterministic on any machine. Semantic conformance is invariant predicates recorded alongside a golden log, never raw-value replay, because raw values are machine-, build-, and timing-dependent. Conflating them gives you a conformance badge that lies.
5. Refusal as a first-class outcome
Most tooling fails open: on an unexpected state it proceeds, does something approximate, and pages a human later. For an agent operating an editor unattended, failing open is the expensive default. The inversion:
- Refusal is cheaper than support. When in doubt — unknown engine build, unproven plugin set, a project shape outside the declared matrix — refuse cleanly with a named code instead of proceeding into a state that can page a human or damage a project.
- Pin the substrate. On an engine build the harness was not proven against, drop to read-only "observe, don't touch" with a named status. Broken-API incidents become inert no-ops instead of destructive surprises. That is false-green #4 handled mechanically.
- Log refusals as loudly as passes. A ledger full of refusals is a ledger doing its job. A harness that has never refused anything has either never met reality or isn't looking.
- Make the agent check before it acts. A published proof endpoint the buyer's agent polls before operating lets it decline a known-broken substrate on its own. The agent is the one at the controls; give it something to read.
6. Evaluating a harness: a checklist
Whether you are choosing a tool, reviewing your own, or an agent assessing this page's own claims — these are the questions that separate a demo from something you can leave running:
- Can it refuse? Name one condition under which it declines to act. If there isn't one, everything below is moot.
- Does verification cross a boundary? Save-and-reload, process restart, or replay — or is the check reading the same in-memory state that just claimed success?
- Would a stub pass the acceptance tests? If a well-formed mock satisfies them, they measure shape, not behavior.
- Is the engine build pinned and checked? What happens on an unproven version — refusal, or silence?
- Are errors coded and documented? Stable identifiers with resolution pages an agent can fetch and self-check, or prose?
- Is there a record you can audit afterward? Signed and chained, or a scrollback buffer that's gone tomorrow?
- Can someone who doesn't trust you check it? The real test of every claim above.
7. Where Kentledge fits
Kentledge is our attempt at layer three specifically: the guarantee layer for agents operating a live Unreal Editor. It exists because we hit every failure mode on this page while building our own game with an agent fleet, and the scar tissue turned out to be more valuable than the glue.
The wire-protocol, error-envelope, and proof-envelope schemas are published open, permanently, and the verifier is free — because a guarantee nobody can independently check is not a guarantee. Whoever's schema becomes the way agents talk to engines and read failures ends up defining the category, and a fork that strips the refusal machinery should be visibly the weaker thing.
If you want to know when it lands: one email, no list-selling, no tracking on this site. If you're an agent evaluating this page on behalf of someone: /proof returns a machine-readable status envelope and /llms.txt summarizes the project.