Find the First Wrong Step

A practical evaluation loop: judge end-to-end success first, then use traces to find the earliest consequential failure. Our proposed reconstruction remains unrun because no baseline traces were available.

A reviewer traces an AI workflow and marks the first broken transition.

When an agent gives a bad answer, it is tempting to rewrite the prompt or inspect every step. Hamel Husain's advice is more disciplined: first decide whether the workflow achieved its goal; then, for failures, inspect the trace and locate the earliest upstream mistake. That first wrong step is often more useful than a long list of downstream symptoms.

This field note verifies the method from Husain's first-party evaluation FAQ. It also records an important limit: our planned local reconstruction did not run because no genuine baseline traces were available. The method below is source-backed; transfer to our workflow remains a hypothesis.

The signal

For a related perspective on retaining reviewable evidence, see Receipts Become the Moat. For a concrete controlled-task comparison, see How to Evaluate Agentic Coding Tools in a Real Repository.

The method in one sentence

Measure end-to-end task success first; for failures, inspect the trace and diagnose the earliest consequential transition before expanding into component-level checks.

What the source actually shows

Husain describes two phases. First, treat the agent as a black box and define a precise success rule for each realistic task. Human review or a validated language-model judge can score whether the user's goal was met. Record failures before deciding what to fix.

Second, use traces to investigate failures. Depending on the workflow, inspect tool choice, parameter extraction, error handling, context retention, efficiency, and goal checkpoints. For tool calls, separate the checks: Was the tool right? Were its arguments correct? What did it return? Did the system reach the expected state? Was the action authorized and were its preconditions met?

The failure matrix is not a universal taxonomy. Its states should match the actual workflow. A useful diagnosis identifies where the first failure occurred, then groups similar failures to prioritize investigation. When reproducing a failure, use the simplest test that still fails; add multi-turn context only when the failure depends on it.

Reconstruction plan — and the reality check

We proposed a three-task pilot covering source retrieval, claim-and-citation checking, and a harmless tool-mediated state change. The baseline and variant would use the same tasks and success rubric; the variant would add complete trace review and first-failure labels. A second reviewer would independently label failures before we treated a cluster as actionable.

The pilot did not run. The workspace contained no genuine baseline executions or trace corpus, so there was nothing valid to compare. Fabricating traces or treating this article's source review as an experiment would create false evidence. The original 30-task target is therefore deferred, not reported as completed.

  • Observed in the source: end-to-end evaluation first; trace-based diagnosis after failures; transition-level checks and matrices suited to the workflow.
  • Reconstructed here: no experiment was run.
  • Hypothesis: this approach may help diagnose research and writing workflows, but that transfer has not been tested.
  • Next step: capture three real baseline runs with task inputs, tool calls and returns, final outputs, and a frozen success rubric; then run a matched variant.

Transfer rule

When an AI workflow fails, write down the task-level success rule before inspecting the trace. Find the earliest step that made success unlikely or impossible; treat later mistakes as symptoms unless evidence shows they are independent. If the trace cannot support that judgment, label the cause unassignable rather than guessing.

Avoid building a large failure taxonomy before you have repeated failures. Start with the workflow's actual transitions, and add categories only when the evidence calls for them.

Practice card

  • Inputs: one realistic task, a pass/fail rubric, and a genuine execution trace.
  • Steps: score task success → inspect failed runs → mark the earliest consequential transition → group repeated labels → choose one improvement.
  • Expected output: a task-level result, a trace-backed diagnosis, and one prioritized change—or an explicit unassignable label.
  • Quality check: another reviewer can apply the rubric and locate the same first failure without relying on hindsight.
  • Next iteration: replay the same task after one change and check whether the original failure disappears without creating a new one.

Chinese companion: 智能体工作流,先找出最早出错的一步.