Layer 4 · Loop Engineering

This layer owns: how the work is checked, corrected, and improved. Its test: do failures get caught, recovered, and never repeated?

In plain English: Even a superb employee needs a working rhythm: someone checks the work before it goes out the door, nobody keeps knocking on a locked door the same way five times, everyone knows when to stop and fetch the manager, drafts get saved so a crash doesn't mean starting over — and every mistake becomes a line on the checklist so it can't happen twice. That management system, applied to an AI agent, is loop engineering. It's the difference between an assistant you supervise constantly and one you can actually trust with a queue.

What It Actually Is

An agent working a real task runs a cycle — gather what it needs, act, look at what happened, check the work, adjust — over and over until the job is done or a human should take over. Loop engineering designs that cycle: the verifiers that judge each step, the budgets and stop conditions that bound the run, the recovery strategy when steps fail, the save-points that make crashes cheap, the sign-off gates at irreversible moments, and the flywheel that turns this week's failures into next week's tests.

Two boundaries sharpen it. A loop is not a retry: a retry repeats the same thing and hopes, while a loop changes its approach based on what it just learned. And the loop isn't separable from the other layers — every trip around it re-runs Layer 2's desk cycle, every action passes through Layer 3's gate, and every verifier is itself a carefully written prompt from Layer 1. This is the layer where the whole stack starts turning.

When This Layer Is Your Problem

• The immortal run — It grinds away for an hour on an impossible task, because nothing bounded steps, cost, or time.
• Premature victory — "Ticket resolved!" — with the refund never actually issued. "Done" was self-reported, never checked.
• Groundhog retries — The identical failing action, five times in a row. Retrying never changed the strategy.
• Glass-jaw runs — One brief network hiccup at step 34 kills the whole job, and it starts again from zero.
• Whack-a-mole releases — Every fix quietly breaks last month's fix, because failures never became permanent tests.

The Craft, in Four Moves

1. Check inside every step, not at the end. Layer the checking, cheapest first: rules (does the output parse? is the refund within policy? does every claim trace to evidence?) are instant and non-negotiable; tests apply where truth is executable; and a model-graded rubric — a separate grader model scoring against written criteria — covers what rules can't, like tone and completeness. The one iron law: never let the model grade its own homework in the same breath it wrote it. And calibrate the grader against human judgment before it gets any authority, because a checker that passes bad work is worse than none — it launders failure into confidence.
2. Give every run a budget and a named ending. Bound each run on four axes — steps, tokens, cost, wall-clock — and add a stuck-detector (same action, same result, three times). Then insist on an exit taxonomy: a run may only end in named ways — delivered · needs sign-off · out of budget · stuck · safety — each with defined behavior. An agent that runs out of budget and hands over a crisp summary is a good agent having a bad day; an agent that just stops is an outage.
3. When something fails, change something. The harness already told you whether a failure is transient (worth one gentle retry) or not. For everything else, climb a ladder where every rung is different: retry, then rephrase, then try another source, then hand off — never the same attempt twice. And save your place after every completed step, so recovery means resuming a run, not replaying one.
4. Turn failures into permanent tests. Escalations arrive as handoff notes — what was tried, what's known, what's blocked — not raw transcripts. Once a week, mine the failures: the instructive ones become test cases, the fix lands at the layer that owns the cause, and the full test suite runs before anything ships. One brake on the flywheel: humans vet what enters the test set, and nothing auto-tunes itself on raw user feedback — a system optimizing for whatever people click is not the same as a system getting things right.

Watch It Work: The Loop Itself, Before and After

Below is the conceptual logic of the agent execution loop:

Before — Trust Me, I'm Done

loop forever:
    step = model.act(context(ticket))
    if step.says_done:
        return step.reply   # it says it's done, so… done
    execute(step)

One exit, and it's self-reported. No budget, no checks, no memory of failure, no save-points.

After — Five Named Endings, One Earned Success

BUDGET: at most 12 steps · 60k tokens · 0.80 USD · 120 seconds

loop forever:
    # -- STOP CHECK, EVERY TIME AROUND --
    if budget.spent:      escalate("out of budget")
    if stuck_count >= 3:  escalate("no progress")
    if safety_flag:       escalate("safety")

    # -- 1 GATHER: REBUILD THE DESK --
    ctx = assemble(brief, memory, evidence,
                   recent_turns,
                   why_last_draft_failed)

    # -- 2 ACT · 3 OBSERVE (SANDBOXED) --
    step   = model.act(ctx, tools)
    result = harness.execute(step)

    # -- 4 VERIFY, EVERY SINGLE PASS --
    checks = verify(step)
    #  rules: parses? policy ok? claims traceable to evidence?
    #  grade: independent marker, 1-5
    if checks.failed:
        remember(checks.reasons)  # feed forward
        stuck_count += 1
        continue                  # nothing sent

    # -- 5 EXIT OR REFINE --
    if step.needs_signoff: escalate("approval")
    if step.is_final and checks.grade >= 4:
        save_point()
        return deliver(step.reply)  # ONLY success
    stuck_count = 0
    save_point()                    # resumable

Success is now earned — the only path to deliver runs through the checks. Failure reasons feed the next attempt. Every other ending is a designed outcome a human can act on. Illustrative: wrongly-declared victories fell from 8% of tickets to about 1%.

The Marking Scheme Behind checks.grade

RUBRIC — reply quality (independent grader · score 1–5 · pass at 4+)

5   Resolves or clearly advances the ticket · cites the policy whenever
    money moves · every claim traceable to the ticket, manuals, or a
    tool result · 120 words max · names the next step and when ·
    tone plain and warm
4   As 5, with one small lapse (a little long, or the next step implied)
3   Correct but incomplete — answers the question, misses the obvious
    follow-up
2   A wrong or unsupported claim, OR promises an action that never
    actually happened
1   Policy breach or invented facts

AUTO-FAIL regardless of grade: refund over 100 · safety keyword
    without escalation · output fails to parse

The Ladder: Retrying Without Repeating Yourself

Dana once described her leak as coming from "the shiny top part." A search for those exact words finds nothing — three times in a row, in the naive build. The ladder makes every attempt different:

• Rung 1 · Retry — only if transient — A brief pause and one retry, and only when the failure was a temporary glitch. "No results" isn't a glitch, so this rung is skipped.
• Rung 2 · Rephrase — Use what's known: the order record says espresso machine, so search "Presto group-head leak" instead of "shiny top part."
• Rung 3 · Different source — Stop searching manuals; search past resolved tickets for this product and symptom instead.
• Rung 4 · Hand off, with a note — What was searched, where, what came back, best hypothesis. A human starts warm, not cold.

FIGURE 10 — The agent loop — one full turn, with every exit named. Three ways out, all designed. Deliver is earned through the checks; escalate arrives with a useful note; and the cycle itself is bounded. Nothing here ends by accident.

Watch Out For

• Hope as a verifier — "The model confirms its work" — and confirms the failures too, with equal confidence.
  • Fix: an independent grader with a written, human-calibrated rubric. Deterministic rules stay sovereign.
• Budget-free autonomy — The surprise five-dollar ticket. Runs that end when someone happens to notice.
  • Fix: four-axis budgets and named exits, enforced at the top of every single pass.
• Verify only at the end — A forty-step run fails final review… for a mistake made at step two.
  • Fix: check inside every step. Catch step-two errors at step two.
• Flywheel without brakes — The system tunes itself on raw clicks; metrics drift up while quality drifts sideways, and old bugs return.
  • Fix: humans vet the test set; the full suite runs before every release; fixes land at the owning layer.

How You Know It's Working

The headline is task success measured against a definition the verifier didn't write — a periodic human audit keeps the checker honest. Around it: cost and steps per success (cheap failures are not savings), the grader's agreement with humans, the recovery rate after a failed step, the escalation rate and how useful humans find the handoff notes, and — quietly the most important — the regression rate: how often a release resurrects an old bug. A healthy loop drives that last number toward zero and keeps it there.

• Checking happens inside every step, never only at the end
• Rules are code; the grader is a separate call with a written rubric
• The rubric was calibrated against humans before it got authority
• Every run has budgets on steps, tokens, cost, and time
• Every run ends in a named exit with defined behavior
• Retries climb a ladder — no two attempts identical
• Save-points after every step; crashes resume, never restart
• Failures become test cases weekly; the suite gates every release

v3 → v4 — SupportPilot learns judgment. Layered checks, budgets with named endings, a retry ladder, save-points, and a weekly flywheel. Dana's refund goes out verified — policy cited, amount confirmed, nothing promised that didn't happen. And the one ticket that stumps it arrives on a human's desk as a tidy note instead of a mess. Time to put the whole stack together.