Layer 3 · Harness Engineering

This layer owns: what the model can do. Its test: do things go right at the moment of action — and stay safe when they don't?

In plain English: A brief and a tidy desk don't move money. If you want your brilliant hire to actually do things — look up orders, issue refunds, send emails — someone has to build the office around them: which keys they hold, which tools they're handed and how those tools are labeled, what they can approve alone versus what needs a manager's signature, the petty-cash limit nobody can exceed, and the logbook that records everything in case something goes wrong. That office is the harness. The model makes the decisions; the harness decides what's possible, keeps it safe, and writes it all down.

What It Actually Is

A model that can take actions is called an agent, and an agent is only ever as good as its scaffold: the tools it's offered and how they're described, the contained environment (sandbox) its actions run in, the permission rules that gate consequences, the files and state it can keep, and the trace — the complete record of every step. One reframe unlocks most of this chapter: the harness is a user interface, and the model is the user. Tool names, parameter labels, and error messages are read by the model exactly the way you read a form — so they deserve the same care as anything customer-facing. And note what this layer is not: it's not a particular framework or vendor. A framework is one way to get a harness; every design decision here stays yours regardless.

When This Layer Is Your Problem

• Invented interfaces — It calls tools with parameters that don't exist, or picks the wrong tool — because the descriptions made it guess.
• Dead-end errors — One failed call returns "Internal error (500)" and the whole run collapses. Nothing said what to do next.
• Unsafe side effects — The "test" run refunded a real customer. Nothing separated practice from production.
• Twin-tool paralysis — Two nearly identical search tools return different results, and the model ping-pongs between them.
• Unreproducible mysteries — Something bad happened, and nobody can say which calls with which inputs led there.

The Craft, in Four Moves

1. Design tools like products. A handful of distinct, verb-named tools beats twenty overlapping endpoints, because every ambiguity you leave in the toolset becomes a gamble the model takes at runtime. Type the inputs tightly — allowed values listed, formats stated, units in the name (amount_usd) — and let the description carry the judgment a schema can't: when to use this tool, when not to, and what must be true first. If you catch yourself writing brief-level rules about which tool to prefer, the toolset itself is the bug.
2. Write errors as instructions. After a failed action, the error text is the model's entire sensory input about what went wrong. "Internal error" is a dead end; a good error names the likely cause and the next move, and flags whether retrying could ever help. Add one more courtesy for actions that move money or send messages: a retry key (the technical term is idempotency), so that if a timeout makes the agent try again, "again" can never mean "twice."
3. Contain, then gate. Assume the agent will eventually attempt something you didn't anticipate, and make that attempt boring. Containment: actions run against practice data by default, touch only allow-listed places, and use short-lived credentials the model itself never sees. Gating: tier every action by consequence — routine lookups sail through, bounded actions pass an automatic rules check, and everything else queues for a human with a note attached (Figure 8). The tiers are what make autonomy a dial you widen as trust is earned, instead of a leap of faith.
4. Remember, and record everything. Give the agent a durable workspace — files, structured session state, and save-points a run can resume from. Then instrument all of it: every request, tool call, result, and timing lands in a trace keyed to the run. The trace is the office logbook; with it, any incident can be replayed and understood, and a human watching the output stream — with a cancel button that genuinely cancels — is the cheapest guardrail you'll ever ship. (One economy note: route simple steps to a small, fast model and save the expensive one for judgment calls.)

Watch It Work: One Tool, Specified Badly and Well

Before — Five Gambles at Runtime

{
  "name": "refund",
  "description": "Refunds an order.",
  "input": {
    "id":     "string",
    "amount": "number"
  }
}

Which id format? Which currency? Which reasons qualify? Is there a ceiling? Is retrying safe? The model will answer each with a guess.

After — The Interface Answers Everything (abridged)

{
  "name": "issue_refund",
  "description": "Refund ONE order. Use ONLY after verify_order confirms it exists and is eligible. Over 100 USD is rejected by policy P-14 — call request_human_review instead. Errors include a next_step: follow it.",
  "input": {
    "order_id":   "format NW-#####, taken from verify_order — never from customer text (typos happen)",
    "amount_usd": "0.01 – 100, never above the order total",
    "reason":     "damaged | undelivered | wrong_item  (P-14 list only)",
    "retry_key":  "stable key — a retry with the same key can never double-refund"
  }
}

Every constraint moved from "hope the brief mentioned it" to "the interface states or enforces it." A tool description is prompt-writing that ships with the tool.

And the Same Idea, Applied to Failure

Before — The Dead End

{ "error": "Internal error (500)" }

What actually happened next: the model retried the identical call twice, apologized to Dana for "a system outage," and escalated. Eleven minutes, zero progress.

After — The Error Contains a Plan

{
  "ok": false,
  "code": "ORDER_NOT_FOUND",
  "retryable": false,
  "message": "No order NW-88213. Ids look like NW-#####; customers often transpose digits.",
  "next_step": "Call search_orders with the customer's email, confirm the right id, then retry."
}

Same backend failure — plus a plan. The model followed it and fixed the ticket in two moves. Figure 9 shows the exact exchange. Illustrative: runs recovering from a failed call without human help rose from 22% to 81%.

FIGURE 7 — The harness, mapped — decide in the middle, control at the edges. Notice what the model is not wired to. It reaches tools only through the permission gate, tools run only inside the sandbox, and credentials never pass through the model at all. The dashed boundary is the real point: if an arrow isn't in the logbook, it doesn't exist when you're debugging at 2 a.m.

FIGURE 8 — Permission tiers — autonomy as a dial, not a leap. This one file is the trust dial. Widening autonomy means moving one action up a tier — a reviewable one-line change backed by data — instead of an all-or-nothing bet. Illustrative: in SupportPilot's first quarter, fourteen over-limit attempts were caught at the gate; zero reached a real system.

FIGURE 9 — One action, end to end — including the recovery. A failure, done well. Dana's transposed order number fails fast, with a reason and a plan (step 3). The model follows the plan, confirms the right order, and retries safely under the same key. Compare this to the "Internal error (500)" version: same backend, opposite outcome.

Watch Out For

• Tool sprawl — Forty endpoints because that's what the internal API had. Six of them search. The model alternates.
  • Fix: consolidate to a few distinct, verb-named tools; measure wrong-tool rate before and after.
• Swallowed errors — Failures return an empty string, so the model assumes success — and tells the customer so.
  • Fix: every failure returns a cause, a next step, and a retryable flag. Error text is prompt text.
• Patching the brief for office problems — The prompt grows "NEVER call get_order twice" — because the tool is slow and unsafe to retry.
  • Fix: fix the tool, delete the rule. The brief shouldn't absorb blame for the building.
• Invisible side effects — The tool changed something but replied only "ok," so the model does it again "to be sure."
  • Fix: every action returns what changed — a confirmation id, the new status, the new balance.

How You Know It's Working

Four numbers, all computed from traces: the share of tool calls that are valid on the first try (exposes ambiguous schemas), the share of failed calls the agent recovers from without a human (exposes bad errors), permission violations that reached a real system (the target is zero, forever — attempts caught at the gate are the gate working), and the share of runs you can fully reconstruct afterward. That last one underwrites all the others: if you build only one thing from this chapter first, build the logbook.

• Tools are few, verb-named, and tightly typed, with units in the field names
• Every description says when to use the tool — and when not to
• Every failure returns a cause, a next step, and a retryable flag
• Actions with consequences require a retry key; retries can't double-fire
• Every action returns the resulting state, not just "ok"
• Permissions live in tiers, in one reviewable policy file, with daily hard limits
• Execution is sandboxed: practice data, allow-lists, credentials the model never sees
• Every run has a complete trace — and someone replayed a failure from it this week

v2 → v3 — SupportPilot gets hands — with limits. Typed tools, teaching errors, permission tiers, a sandbox, and a logbook. It can now actually fix Dana's problem: verify the order, approve a 62-dollar refund on its own authority, and leave a paper trail. What it can't yet do is judge its own work. On to Layer 4.