07
Mistake notebook learning: Batch-clustered failures for training-free agent adaptation
Editorial reading guide · 编者导读
为什么值得读
这篇最有价值的地方不是再证明一次 external memory 可以提升分数，而是把 mutation authority 写成一个可计算 gate：先生成 candidate，再用同一 batch 验收，失败就 rollback。也正因为 gate 足够具体，读者可以追问它的 population、denominator 与 lifecycle：某个 item 没形成 updated pair 时是否仍属于 B？一个 aggregate win 能否遮住 subgroup loss？proposal batch 通过后谁负责 held-out decline？这让“持续学习”从漂亮叙事变成一组可以逐层审计的 admission contracts。
Source-backed paraphrase
论文在解决什么
MNL 先按 subject 聚类一批失败，再把共享错误模式压缩成 structured guidance；真正需要审计的不是“会不会写笔记”，而是谁进入 proposal、谁被重新比较，以及一个 net-positive batch 究竟授权了多强的 promotion claim。
Source-backed paraphrase
关键抓手
- Tuning Model 使用 retrieved guidance 执行任务，Tuner Model 对整个 batch 先做 subject mapping，再只从 failure subsets 提炼 guidance；两者可以同模，也可由强模型向弱模型蒸馏。（§§3.1–3.3；Algorithm 1）
- 每条 memory 以 subject、五部分 corrective guidance 与 subject embedding 组成；相似度超过 threshold 时论文声称 fuse into existing node，否则 append。（Appendices B.1–B.3）
- Eq. 5 把同一 proposal batch 中每个 item 的 updated reward 与 baseline reward 做符号比较，只有 wins − regressions > 0 才接受；tie 为零贡献，且 gate 没有 held-out 或 per-item non-regression 合同。（§3.3；Algorithm 1）
- 单 epoch 的 KaggleDBQA test accuracy 为 28.1%，epoch 2 的 train accuracy 升至 62.1% 而 test 降至 23.2%；论文据此承认 cross-epoch overfitting，并采用 single epoch。（§4.3；Figure 4）
Source-backed argument map
论证怎样成立
01 · 把 adaptation 从 weights 移到 external memory
MNL 用 Tuning Model 执行、Tuner Model 反思与更新，把优化对象写成经 retrieval 进入 prompt 的 memory M；self-tuning 与 cross-model tuning 是同一框架的两种角色配置。
§§1、3.1–3.2；Equations 1–2，physical PDF pp. 1–3（proceedings pp. 14629–14631）
02 · 先画全 batch 的 subject map，再从失败组抽象
系统先对完整 batch 做 subject clustering，再在每个 subject 内只聚合 failed trajectories，提炼五部分 structured guidance，并按 similarity 决定 merge 或 append；目标是让 shared pattern 取代逐条 failure log。
§3.3；Equations 3–4；Algorithm 1；Appendices B.1–B.3，physical PDF pp. 3–5、12（proceedings pp. 14631–14633、14640）
03 · 用 proposal-batch net delta 决定 mutation authority
candidate memory 必须对同一 B 个 query 重新生成；Eq. 5 逐项比较 updated 与 baseline reward，只有 ΔB = wins − regressions 严格大于零才 commit，否则保留旧 M。这个合同是 aggregate、batch-local 且 verifier-relative。
§3.3；Equation 5；Algorithm 1 lines 23–34，physical PDF pp. 4–5（proceedings pp. 14632–14633）
04 · 实验支持一种 operating point，却没有给全局单调性
六个 benchmarks、batch / if-accept ablations 与 memory-length comparisons 支持 compact guidance 的可行性；但 cross-epoch test decline、Mimo-v2 no-think 反例与 Limitations 同时限定了“stability”只能在具体 verifier、batch 与 setup 内理解。
§§4–5；Tables 1–8；Figures 3–4；Limitations，physical PDF pp. 5–9（proceedings pp. 14633–14637）
Source-backed method notes
方法与监督
Two model roles and two feedback regimes
Tuning Model 负责生成，Tuner Model 负责 subject clustering、failure analysis、guidance extraction 与 open-ended judge；两者可共享 base model，也可由 stronger tuner 蒸馏 weaker tuning model。Math / Text-to-SQL 使用 supervised reward，agentic tasks 使用 proxy LLM judge 的 binary signal。
§§3.1–3.3；§4.1；Appendix B.3，physical PDF pp. 2–5、12
Structured memory and retrieval
JSONL entry 包含 subject、embedding 与五个 mandatory guidance components：corrected examples、correct approach、mistake summary、generalizable strategy、anti-patterns。Query 和 subject 使用同一 embedding model，以 threshold + top-k 进入 active context，并附 applicability assessment。
Appendices B.1–B.3、C.1–C.4，physical PDF pp. 12–15
Proposal and acceptance population
Algorithm 1 先在完整 batch 上生成 baseline、构造 candidate，再对同一 B 个 query 形成 Rnew；Eq. 5 只比较每项 reward 的方向，不保留改善幅度。论文没有定义 updated generation 缺失、held-out decline 或 subgroup veto。
§3.3；Equation 5；Algorithm 1，physical PDF pp. 3–5
Experimental objects
实验横跨 AIME 2024/2025、GSM8K、KaggleDBQA、Spider、Mind2Web 与 AppWorld；指标包括 Pass@32、execution accuracy、Task Success 与 Step Accuracy，并同时报告 memory entries 和 guidance tokens。本站没有重跑这些 datasets、models、judges 或 service-dependent agent loops。
§§4.1–4.3；Tables 1–8；Appendix D，physical PDF pp. 5–8、16–17
Paper-reported findings
报告了什么
- Table 1 中，Qwen3-8B 的 MNL 在 AIME 2024/2025 报告 33%/30%，KaggleDBQA 报告 28%；DeepSeek-V3.2-Exp 的 KaggleDBQA 从 vanilla 24% 报告为 MNL 64%。这些是 paper-reported point estimates，不是本站复算。（§4.2；Table 1，physical PDF pp. 5–6）
- Table 3 的 Qwen3-8B ablation 从 Single-Level 到 Batch Clustering，再到 Full (+ If-Accept)，在 GSM8K、KaggleDBQA 与 Mind2Web Step Accuracy 上报告更高结果与更短 memory；组件顺序同时改变，不能把最终差值全归给单一 gate。（§4.3；Table 3，physical PDF p. 7）
- Figure 4 报告 KaggleDBQA epoch 1 test accuracy 28.1%、50 entries；epoch 2 train accuracy 升至 62.1%，test accuracy 降至 23.2%。作者将其解释为 cross-epoch overfitting，而不是持续单调改善。（§4.3；Figure 4，physical PDF p. 8）
- Table 6 的 Mimo-v2-w/o-think 是明确反例：vanilla TS/SA 为 8.75/41.12，MNL 为 8.08/40.71；同表的 think-enabled variant 则从 10.77/47.63 升至 11.09/48.51。（§4.2；Table 6，physical PDF p. 7）
Evidence boundary
仍未证明什么
- Eq. 5 的 gate 使用提出 candidate 的同一个 batch，没有 independent promotion set、held-out veto 或 subgroup constraint；一个 accepted update 可以包含 losing item，并且不能推出 test distribution 单调改善。
- Self-Evolution 依赖 Tuner Model 充当 LLM judge。论文 Limitations 明确承认 biased / inconsistent verifier 可以把系统性错误传播进 memory，batch-level accept-if-improves 不能消除这一点。（physical PDF p. 9）
- Appendix A 的 decision-stability proof sketch 假设同一 semantic cluster 内 noise independent，并调用 sub-Gaussian concentration；论文没有实证证明语义相似样本的 verifier error 独立、同分布或满足该 tail condition。（physical PDF pp. 11–12）
- 主表提供 point estimates，但没有给本站可据以复算的 complete per-run outputs、confidence intervals 或统一 request / model snapshot receipt。不同 benchmark 的 split、metric、judge 与 model regime 也不构成同一 statistical population。
- Exact current official source 是 untagged commit dc7de755522ad58864c62b74ab8e9959c01b7f23；没有证据证明它生成 ACL final tables，故 source behavior 与 paper-time execution必须分层。
- 当前 source inspection 与本次 public synthetic audit都不调用模型、API 或 benchmark data，不能估计 partial failure、duplicate subject、coverage loss 的真实频率，也不能改变任何 paper headline score。
Source audit · observation and consequence kept distinct
原文内部张力
“Prevents regressions” 强于实际 gate
Introduction 把 conservative rule 概括为 prevents regressions；Eq. 5 却允许 wins 多于 losses 时接受含有 regression 的 batch，Figure 4 又报告 epoch 2 test decline，Table 6 还出现 Mimo-v2-w/o-think 低于 vanilla。
定位：Introduction，physical PDF p. 1 · §3.3；Equation 5，physical PDF p. 4 · §§4.2–4.3；Table 6；Figure 4，physical PDF pp. 7–8
编者 consequence：可验证对象应写成 proposal-batch net improvement under one reward function，而不是 per-item、subgroup、held-out 或 continual-learning non-regression。
Paper 的 full B 与 current source 的 survivor subset
Algorithm 1 对 i=1…B 重新生成并计算 ΔB；current trainer 会在 baseline generation、updated retrieval prompt 与 updated generation 三个 stage 分别删除 response 为 None、prompt 为空或 response 为 None 的 items，只把 survivors 交给 evaluator。全部 updated 缺失时会 rollback，但 partial 缺失可以缩小 candidate-construction 或 comparison population。
定位：Algorithm 1，physical PDF p. 5 · exact current source trainer.py:430–451, 482–533, 556–569
编者 consequence：任一 original ID 没有完整 baseline/updated pair 时，paper-side ΔB 应记 incomplete / undefined；source-side survivor decision、stage ledger 与 missing-as-failure sensitivity必须分别报告，不能无声宣称 full batch improved。
Current default guidance schema 短于 paper / DBQA custom schema
Current `PromptTuner` class default只要求 2–3 sentences，没有强制 paper Appendix B.2 的 corrected examples、correct approach、mistake summary、generalizable strategy 与 anti-patterns；同一 commit 的 DBQA example另行定义并注入包含这五部分的 custom template。
定位：Appendix B.2，physical PDF p. 12 · exact current source trainer.py:125–127, 181–191 · exact current source example_dbqa.py:139–173, 275–295
编者 consequence：这是 public/default API 与 experiment-style caller contract 的张力，不能反向宣称 paper 的 DBQA setup 使用短 default；对实际 prose quality 的判断仍需 model execution，本次只做 literal source comparison。
Fuse existing node 与 append duplicate 的实现张力
Appendix B.3 说 similarity 超过 threshold 时 new insight fused into existing node；current `update_entry()` 始终以 `overwrite_if_exists=False` append。Exact subject、equal embedding、stable sort、top_k=1 时，旧 entry 可继续先被取回，新 merged guidance 被遮住。
定位：Appendix B.3，physical PDF p. 12 · exact current source knowledge_base.py:70–105, 279–292 · exact current source example top_k configuration
编者 consequence：promotion audit 不只要证明 candidate 被写入，还要绑定 candidate 在 post-update prompt 中实际可达；append count 不能自动证明 refinement 生效。
Periodic evaluation 不是 held-out rollback
Current trainer 的 periodic / epoch-end eval 只记录 score 并保存 checkpoint；`best_eval_score` 初始化后没有消费，`_load_checkpoint()` 没有 caller。Current DBQA example 还把 train 与 eval variables 指向同一个 train JSONL；这只是 current example static configuration，不能叫 paper held-out split。
定位：exact current source trainer.py:42, 167, 814–823, 929–955 · exact current source example_dbqa.py:82–83, 270, 324–327 · §4.3；Figure 4，physical PDF p. 8
编者 consequence：Periodic eval 是 telemetry，不是 promotion authority。它不违反只承诺 proposal batch 的 Eq. 5，但不能被描述成 best-checkpoint 或 held-out monotonic protection。
Editorial synthesis / question
编者判断
Promotion population is part of the evidence contract · Editorial inference — not tested by the paper
一个 batch gate 的 denominator 不能只是运行时 list 的当前长度；original、baseline-valid、updated-prompt-valid、updated-response-valid 与 evaluated identities都应进入 receipt，任何 shrink 都要带 reason 与 policy。
这是由 Algorithm 1 的 full B 与 current-source filters 推出的 audit design，不是论文提出的 logging schema，也不规定 missing 必须计作 loss。
Improvement authority must be layered · Editorial inference — not tested by the paper
proposal-batch net gain、per-item non-regression、subgroup protection、held-out improvement 与 deployment safety是五个不同 gate；前一个通过不能默授权后四个。
论文只实现并讨论第一个 gate，且 Limitations承认 verifier 和 lifecycle 风险；本站没有证明哪一组额外 gate 在所有任务上最优。
A promoted memory must be reachable under the reader contract · Editorial inference — not tested by the paper
Mutation receipt 应把 candidate state 与 post-update retrieval / prompt admission绑定；否则“写入成功”可能只是 store-level事实，新 guidance并未成为验收时真正影响生成的 evidence。
Exact-subject / equal-embedding / top_k=1 是一个 bounded mechanism case，不代表所有 subject merge、retriever 或部署都会 shadow candidate。
Accuracy needs coverage when execution can disappear · Editorial inference — not tested by the paper
如果 failed generation 可从 eval denominator消失，accuracy 必须与 registered-question coverage一起发布；否则 1/1 与 1/2 会共享同一个表面 score。
这是对 current evaluation denominator 的 measurement repair direction，不证明论文表格发生过同类缺失或因此被高估。
继续追问
如果 promotion 只看 proposal-batch 的 surviving pairs，怎样同时保全原 cohort、coverage、subgroup harm 与 held-out generalization，避免一个绿色 aggregate 吞掉未被比较的 item？
继续追问与实践
同一批经验保留为案例、规则或参数时，怎样改变未见情境中的选择；反馈被纠正之后，这种改变又怎样更新？
持续研究 · 当前判断可修订
将少量纠正反馈用于继续训练分类器或其他决策部件，而此前仍有有效的任务、候选或适用条件需要保留时。
编者建议 · 目标侧待验证
Editorial transfer · 编者设计建议
读完，可以怎样借用
基于既有精读与本条明确标注的证据范围整理；下面针对新场景的迁移对照尚未执行。
- 什么时候有用
- 系统会根据失败生成经验，并准备把“这一批总体变好”当成自动更新依据。
- 可以借哪一步
- 保留更新前后的同一组任务身份、缺失原因和实际进入 prompt 的经验；把提出候选的任务与检验泛化的任务分开。
- 先做什么对照
- 在一组候选对照里故意放入一个缺失输出、一个退步任务和一条写入但检索不到的经验，检查 aggregate 是否仍掩盖它们。
- 带走时的边界
- 这组迁移试验未执行。已有 audit 支持特定过滤、检索与计分路径的边界；不能由 batch net gain 推出逐项不退步或部署安全。
阅读依据：既有札记 §3.3、Algorithm 1 与 公开测试；“编者判断”：promotion population、retrieval reachability。
Editorial protocol · proposed, not run
可复核的下一步
Proposed · not run
Failure-inclusive promotion and held-out lifecycle protocol
在 exact caller 与真实 service failure modes 下，cohort attrition、candidate reachability、subgroup loss 与 held-out decline 各自多常见，哪种 admission policy 能在 coverage 与 false rollback 之间形成可接受 operating point？
- Method
- 冻结 paper-time或正式 tagged source、model / endpoint / decoding、dataset split、reward与retriever；为每次 proposal注册 immutable cohort，显式记录每个 stage outcome。并列比较 source survivor gate、abort-on-incomplete与预先声明的 missing policy，再用独立 held-out stream和versioned checkpoint评价 lifecycle。
- Public / synthetic fixtures
- 完整 pairs、baseline failure、partial empty retrieval、partial transport failure、timeout、malformed response、exact-subject merge、equal-rank retrieval、mixed subgroup effects、verifier flip、跨 epoch distribution shift 与 sensitive-trace redaction。
- Controls
- Full-cohort positive/zero/negative controls；all-missing rollback；candidate actually admitted to prompt；stable item IDs；registered-vs-survivor denominators；verifier disagreement；clean rollback state；checkpoint restore；no private traces in public receipts。
- Measures
- 按 caller path 报告 cohort coverage、source decision、policy sensitivity、per-item / subgroup deltas、candidate retrieval rate、eval coverage、held-out delta、rollback precision / recall 与 unresolved fraction；不把 missing 自动归类成 semantic failure。
- Limitations
- 这是 proposed-not-run successor，需要本次未运行的 model endpoints、datasets 与 paper-time source identity。即使多 seed / 多 split 结果稳定，也不能自动证明 production safety、privacy compliance 或长期 user benefit。
Attributed public perspectives and tests
署名观点与测试
Source-bound promotion-cohort and coverage contract audit
- Method
- Reject a wrong or dirty checkout and wrong paper/source digests; dynamically load the exact current PromptTuner, Evaluator and KnowledgeBase methods through narrow standard-library dependency stubs; execute ten fixed promotion batches with ordered stage ledgers and explicit per-item indices/states/rewards; bind accepted and rejected in-memory plus serialized KB states; execute exact-subject equal-embedding top-1, failed-question denominator and source-owned prompt-provenance probes；invoke the exact default-template method and AST-inspect the hash-locked DBQA example without importing it；rederive aggregate decisions; run eight mutation falsifiers and a socket guard; execute the complete amended matrix twice in fresh out-of-tree roots with hash seeds 313 and 727; compare six primary receipts byte-for-byte; then run receipt-only and fresh source-bound checked verification.
- Environment
- Darwin arm64；CPython 3.13.3 standard library；UTC and C.UTF-8；exact clean official commit dc7de755522ad58864c62b74ab8e9959c01b7f23；17-page ACL final PDF；two fresh external run/output roots；no upstream dependency installation, dataset, model, API, embedding service or AppWorld runtime；zero guarded Python socket attempts，while raw syscall / OS-level network was not instrumented。
- Fixture
- 37 AMS-authored synthetic batch items across complete positive, balanced, all-tie, partial baseline-None, partial updated-None, suffix empty-prompt, middle empty-prompt prompt-provenance, all-empty-prompt, all-None-response and mixed-subgroup cases；one exact-subject/equal-embedding knowledge-base probe；two enrolled evaluation questions with one all-failed question；one static default/custom-template and current DBQA path probe；stable original, baseline-valid, prompt-valid, response-valid and evaluated identity ledgers plus explicit per-item receipts。
- Controls
- Exact paper, commit, tree, Git-blob and source-file hashes；repo-relative runtime module binding；clean checkout and allowlisted bytes before/after；complete-cohort Eq. 5 sign controls；all-missing rollback controls；rejected logical/serialized prestate equality；accepted exact one-entry delta；per-stage identity conservation；missing-as-failure labeled counterfactual only；eight fail-closed mutations for deleted identity, flipped admission, false observed loss, rejected-state change, enrolled-denominator change, top-1 substitution, relabeled baseline missing stage and hidden prompt-provenance mismatch；Python-socket-attempt guard；public-path/secret/upstream-source scan；two-run primary-receipt byte comparison；receipt-only and source-bound verifier separation。
- Raw result
- All ten batch cases matched the locked current source. Complete 3-win/1-loss accepted; 2-win/2-loss and all-tie cases rolled back. Partial baseline-None, updated-None and suffix empty-prompt cases each evaluated 2 of 4 enrolled IDs and accepted on two survivor wins; every full-cohort decision remains UNDEFINED_FROM_OBSERVED_RESULTS and each labeled missing-as-failure sensitivity was delta 0. The middle empty-prompt case evaluated 3 of 4, accepted at 2W/1L, and logged the loss with the preceding survivor's baseline prompt rather than its own. Both all-missing controls rolled back. A complete 3W/1L batch accepted despite subgroup B delta -1. Exact update_entry left two same-subject entries and top_k=1 returned the older guidance. Evaluation returned conditional accuracy 1.0 with coverage and unconditional correct/attempted both 0.5. The class default required none of the five named components; the current DBQA custom template contained and injected all five, while its current train/eval variables named one file. All eight mutations were caught; two fresh six-file primary receipt sets were byte-identical; model/API calls and guarded Python socket attempts were zero.
- Derived result
- At the locked current official revision, complete cohorts implement the strict proposal-batch wins > losses sign gate. Current source can shrink membership after baseline generation, updated-prompt retrieval or updated generation and still accept on survivors, so those admissions do not establish a full-enrolled-cohort delta. A middle prompt-filter can also misbind baseline prompt provenance in the source negative/debug record without changing evaluator arithmetic. Net batch acceptance permits a losing item/subgroup. Under the exact bounded subject/equal-embedding/top_k=1 fixture, append semantics leave the older entry first; under the exact eval fixture, an all-failed question disappears from the accuracy denominator unless coverage is reported separately. The short default/rich DBQA custom split and same-file DBQA variables are current-source static contracts, not paper-time or benchmark findings.
- Limitations
- The runner uses synthetic inputs and narrow import/adaptor stubs, not the upstream environment. It establishes behavior only for exact executed methods, one statically inspected example and the locked revision; it does not show how frequently supported callers produce None, retrieval misses, prompt provenance errors, duplicate exact subjects or all-failed evaluation questions. Bundled ordinary batch wrappers commonly convert request failure to an empty string, so the None seam is not attributed to every example path. The DBQA example is not imported or executed, and its custom template cannot establish paper-production configuration. Equal-embedding top-1 shadowing is a bounded mechanism case, not a universal retriever claim. Evaluation coverage findings do not show that paper tables omitted failures. Same-machine byte identity establishes deterministic repeatability, not independent correctness, cross-platform portability or causal impact.
Evidence basis：Paper-facing audit of the 17-page ACL 2026 Findings final and synthetic source-bound execution against the exact clean current official MistakeNotebookLearning commit dc7de755522ad58864c62b74ab8e9959c01b7f23. Three executed modules plus one statically inspected DBQA example are locked by Git blob and SHA-256; no provenance establishes this untagged commit as the paper-production revision. The original eight-case protocol and the two-case delayed-review amendment remain separately identified.
Boundary：This is a deterministic, model-free synthetic contract audit under an original protocol frozen after paper/current-source inspection and a chronology-preserving amendment frozen after delayed review. It is not an MNL benchmark or paper-experiment rerun, model evaluation, paper-score recalculation, source-to-paper reproduction, prevalence or effect estimate, security finding, production test, or real-world non-regression claim. Missing baseline or updated items remain unavailable observations; the separate missing-as-failure calculation is only an audit sensitivity. A net-positive survivor decision does not establish full-cohort, per-item, subgroup, held-out, cross-epoch or deployment improvement. Prompt-provenance misalignment is limited to the current source negative/debug record and is not an admission-arithmetic result.
method, results and claim ceiling ↗ · frozen pre-execution protocol ↗ · delayed-review amendment and disposition ↗ · source-bound deterministic runner ↗ · checked receipt and source-bound verifier ↗ · frozen synthetic cases ↗ · exact source-executed case receipts ↗ · bounded aggregate decision ↗ · paper and exact current source manifest ↗ · mutation falsifier receipts ↗ · two-run comparison receipt ↗
Shared readings
放到一起读
没有再读那段往事，它为什么还是改变了选择？ →
经验可以通过可读 guidance 影响后续行为；接受一次更新需要说明比较的是哪批任务。
同批次改善不保证留出表现；本实验的规则归纳并非 MNL 实现。
串读定位：§§3.1–3.3、Eq. 5；本站已有 promotion cohort audit
含当前正文、出处、阅读范围与接续线索；链接的全文不随包附带。