Turn Feedback Into Agent Memory
Turn reviewed corrections into reusable agent principles, then test whether they transfer.
An agent can produce work that is almost good enough while leaving a team to do most of the judgment. The tempting response is to keep lengthening its prompt. That may handle a few known cases, but it cannot capture a team's changing preferences or explain how to act in an unfamiliar situation. A better question is whether reviewed corrections can become a durable, testable part of the agent's working method.
The signal
- Practitioner / team: Warp describes building Buzz for its Developer Experience team.
- Source: Agents Need Feedback Loops, Not Perfect Prompts, by Warp.
- Why this matters: The article says Warp faced more than a thousand weekly mentions across platforms, too many for a small team to monitor and answer manually. Public replies have no quick, reliable external correctness signal: a team cannot safely send many variants and infer brand quality from noisy reactions.
The method in one sentence
Use specific human corrections to identify a missing general principle, update a compact skill under review, and check later examples to see whether the principle transfers.
What the source actually shows
Workflow
Warp reports that Buzz monitors mentions and classifies each as reply, like, note, or skip. If a reply is appropriate, it drafts a suggestion into Slack; a person still writes the public response. The article says the first skill used a long checklist of case-specific rules. It became brittle, so Warp describes replacing that approach with a smaller set of principles, such as being helpful rather than defensive and checking factual claims against documentation.
The reported feedback cycle compares the agent's suggestion with what the human changed or wrote instead. A separate skill asks what underlying principle is missing or unclear. It tries to generalize beyond the individual correction, checks existing principles, edits or removes overlap, places the idea in the right section, and commits a concise change. For example, “too marketing-y” should not automatically become “never mention pricing in the first sentence.” A broader candidate principle might be to lead with empathy rather than a pitch when someone is venting.
| Element | Evidence from the source | What we can test |
|---|---|---|
| Input | A draft, the human's correction or alternative, and current skill | Whether reviewers can identify a concrete failure and intended outcome |
| Agent step | Find the cause, generalize cautiously, compare with existing principles | Whether a principle covers held-out cases without becoming a rule pile |
| Output | A concise, reviewed skill edit | Whether it improves decisions on examples not used to create it |
| Quality check | Human review remains; public replies are not automated | Whether quality and inappropriate-action rates meet a threshold |
These details are REPORTED by Warp, not independently measured here. The article is a practitioner account, not a controlled comparison. It describes a workflow and its rationale; it does not establish causal quality gains, general learning, or how another team would reproduce it. OBSERVED means the source explicitly describes the workflow, not that this newsletter ran it. No local reconstruction has been performed.
Reconstruction plan
Try this on a bounded internal task, such as routing source links into “read,” “ignore,” or “ask a person.” Freeze representative examples and a rubric before changing the agent.
- Task: classify incoming links and draft a short reason; take no external action.
- Baseline: run the current prompt or skill on frozen examples and save inputs, outputs, decisions, and reviewer corrections.
- Smallest useful experiment: have a reviewer annotate specific errors; ask the agent to propose one principle per recurring pattern; have a human accept, rewrite, or reject each edit. Compare baseline and revised skill on a separate held-out set.
- Artifact to save: dated skill versions, example IDs, rubric, original output, correction, proposed principle, reviewer decision, and final output. Keep evaluation examples separate from feedback examples to reduce leakage.
- Success condition: the revised skill improves a predeclared held-out decision-quality score without increasing inappropriate actions; reviewers find the change clear and reusable.
- Stop condition: stop if reviewers disagree about the target, a proposed principle is only a one-case rule, performance regresses, or the agent crosses its approved boundary. Restore the prior version and inspect the examples before retrying.
This is a HYPOTHESIS experiment. It does not imply Warp used this exact reconstruction protocol or that skill updates will improve outcomes. A correction to source triage might reveal that the agent treated a vendor claim as independent evidence. A useful candidate principle could be “label a source's relationship to the claim before weighting it”; “never cite vendor pages” would likely overfit. A reviewer should inspect both the principle and its placement. Automation should not commit its own behavioral change without approval.
Reality check
- OBSERVED: The Warp article describes Buzz's mention triage, Slack suggestions, human-written public replies, a shift from brittle rules to principles, and a feedback-to-skill workflow.
- REPORTED: Warp says the system saves time and explains its operational motivation. These are company statements, not measured results in this article.
- RECONSTRUCTED: Nothing locally; the triage comparison is proposed only.
- HYPOTHESIS: A reviewed principle may transfer better than a one-off correction, but held-out examples are necessary to test that.
- Still unknown: the article gives no controlled outcome measurements, reviewer agreement rates, error bars, maintenance costs, or cross-task generalization evidence. Feedback can encode inconsistent judgment, and examples can leak into evaluation.
- Cost / maintenance constraint: every proposed update creates review and regression work. This process is worthwhile only if repeated near-misses cost more than collecting corrections and maintaining versions.
Transfer rule
Try this when a task requires judgment, corrections recur, a human can explain why an output missed the mark, and consequential actions remain under human control. Use a frozen baseline, held-out cases, versioned edits, and a reviewer empowered to reject a change.
Avoid treating feedback as an automatic training label when reviewers disagree, outcomes are hard to inspect, or mistakes have irreversible consequences. Keep the system advisory in those settings. Improved held-out decisions without more inappropriate actions would strengthen the case; regression, unstable labels, or edits that merely restate individual examples would weaken it.
Practice card
- Inputs: current skill, bounded examples, proposed outputs, human corrections, and explicit rubric.
- Steps: freeze a baseline; collect specific corrections; identify the general cause; compare with existing principles; obtain human approval; test on held-out cases.
- Expected output: a concise versioned skill change and audit trail, not an unreviewed self-modifying agent.
- Quality check: compare baseline and variant on identical held-out cases; inspect decision quality, inappropriate actions, and reviewer agreement.
- Next iteration: keep, revise, or roll back based on saved evidence; reserve new examples for a future evaluation set.
Links / sources
- Chinese companion: 把反馈变成智能体记忆
- First-party source: Warp, Agents Need Feedback Loops, Not Perfect Prompts
- Related series article: Turn User Failures Into Tests
- Reproduction artifact: none; local reconstruction is unrun.