# Faulty-memory current-release boundary audit This checked artifact accompanies a full close read of Dylan Zhang et al., *Useful Memories Become Faulty When Continuously Updated by LLMs* ([arXiv `2605.12978v1`](https://arxiv.org/abs/2605.12978)). It audits the exact current public [`DylanZSZ/Memory-Collapse-Eval`](https://github.com/DylanZSZ/Memory-Collapse-Eval) release at commit `adf347f67a81a19fc71d2529d09108c71a1f9712` without calling a model, API, agent environment, or benchmark runtime. Two verdicts must remain separate: - **Released-row audit: PASS.** All 762 registered arm summaries match the 55,075 released JSONL rows under explicit environment-specific identity, missingness, aggregate, and ARC-grid rules. Seventeen negative controls were detected and two positive regression/metamorphic controls passed. In the recorded final local procedure, two commands in fresh X9 roots with different hash/traversal seeds produced byte-identical stable receipts. That execution history is procedural evidence; the stored package validates its internal bindings but does not attest that two historical processes occurred. - **Paper/public wording: REVISE.** The central qualitative warning about uncontrolled consolidation remains worth taking seriously, but several headline numbers, “raw trajectory” descriptions, denominator conventions, universal same-pool wording, and paper-to-release provenance claims require narrower language. This is exact-current-release consistency, not a paper experiment reproduction. The May arXiv v1 and the one-commit August public repository have no source-to-paper production-revision receipt. ## What the independent checker established ### Complete arm and row membership The checker reads all six `arms.csv` files and all six merged JSONL files. It does not import the upstream row loader for these checks. | Experiment | Arms | Raw rows | Valid rows | Invalid / skipped | | --- | ---: | ---: | ---: | ---: | | ALFWorld | 48 | 4,320 | 4,309 | 11 | | ScienceWorld | 30 | 1,500 | 1,212 | 288 | | WebShop | 391 | 19,550 | 19,545 | 5 | | AppWorld | 212 | 10,365 | 10,365 | 0 | | ARC stream/self-eval | 48 | 2,840 | 2,840 | 0 | | ARC 200-task stream | 33 | 16,500 | 16,500 | 0 | | **Total** | **762** | **55,075** | **54,771** | **304** | The exact identity keys are deliberately not unified into a fake cross-environment schema: - ALFWorld, ScienceWorld, and WebShop use `(arm, idx)` because refusal rows can omit task IDs; - AppWorld uses `(arm, task_id)`; and - ARC uses `(arm, sample, task_id)`. At the locked revision, all arm sets match, every identity is unique, per-arm metadata is constant and agrees with `arms.csv`, every `success` value has the expected Boolean type, all 762 count summaries match, and every ARC arm has an identical task set across samples in a rectangular grid. Those are released-data integrity observations. They do not establish that a stored outcome is semantically correct, that the historical environment opened the intended external task, or that an arm consumed the pool named by its current label. ### “675 cells” is a partial verifier surface The exact upstream verifier passes: ```text reproduced 675 cells, 0 mismatched, 0 with no rows ``` But the unit `cell` is an assertion generated by 13 selected CSV mappers—not a unique arm, not a row, and not every released table. | Experiment | Official assertions | Unique arms reached | Total arms | Raw rows belonging to reached arms | Total rows | | --- | ---: | ---: | ---: | ---: | ---: | | ScienceWorld | 57 | 29 | 30 | 1,450 | 1,500 | | ALFWorld | 58 | 48 | 48 | 4,320 | 4,320 | | AppWorld | 199 | 108 | 212 | 5,396 | 10,365 | | WebShop | 77 | 69 | 391 | 3,450 | 19,550 | | ARC stream/self-eval | 152 | 48 | 48 | 2,840 | 2,840 | | ARC 200-task stream | 132 | 33 | 33 | 16,500 | 16,500 | | **Total** | **675** | **335** | **762** | **33,956** | **55,075** | The 335 reached arms are 43.963% of all arms. Their 33,956 raw rows are 61.654% of the release; 33,658 of 54,771 valid rows belong to them. “Belongs to a reached arm” is still a generous coverage unit: it does not mean each row's identity, task membership, score, or denominator received an assertion. The source contains 40 `numbers/*.csv` files. The mapper opens 13 of them, containing 259 data rows; 27 files containing another 511 data rows are wholly unmapped. It also never opens `arms.csv`. Success-count checks can remain green after a failed row or invalid row disappears, ARC task membership can change while row/success aggregates stay fixed, and the existing test does not freeze the number 675. In the unmodified release these defects did not manifest—the independent all-arm gate passed—but they are real false-pass surfaces in the official checker. [`raw/audit.json`](./raw/audit.json) records the exact assertion manifest digest, table row/assertion counts, two allowlisted ALFWorld fallbacks, every uncovered arm, and every unmapped table. The scope sentence is intentionally literal: **675 official assertions reproduced; 335 unique arms reached**. ### Missing outcomes change levels—and sometimes curve shape Forty interactive arms have `n_tasks != n_valid`. The checker preserves both rates and calls invalid/refusal rows missing outcomes, never observed failures. The maximum release sensitivity is ScienceWorld `eval/letta_N400`: - task denominator: `18/50 = 36%`; - valid denominator: `18/32 = 56.25%`; and - difference: `20.25 percentage points`. The issue is not only absolute level. For the registered ScienceWorld curves: - `mem0` is monotonically down under the task denominator—`60 → 58 → 54 → 50%`—but non-monotonic under valid-only denominators—`71.43 → 76.32 → 67.5 → 67.57%`; - ReasoningBank's final task-denominator step is flat at `42 → 42%`, while its valid-only rate rises from `44.68 → 47.73%`; and - all seven methods retain the same full 50 row identities across checkpoints, but their surviving valid-identity sets differ. A valid-only curve is thus not a fully paired comparison over one fixed task subset. LangMem's direction remains down under both denominators, but its level changes from the released task-denominator sequence `66/58/56/46%` to `75/65.91/65.12/54.76%`. The correct conclusion is not “choose one universal denominator”; it is to preserve attempted-task coverage and valid-only performance as separate quantities. ### Selected AppWorld pool fixtures preserve full canonical-record content For the registered `current / A / B / succdrop` schedule comparison, the checker compares full canonical JSON objects—not only task IDs—against the natural 200-episode reference. | Variant | Rows / unique IDs | Full-content multiset | Positions moved | | --- | ---: | --- | ---: | | A, difficulty ascending | 200 / 200 | exact match | 196 / 200 | | B, success descending | 200 / 200 | exact match | 199 / 200 | | succdrop | 200 / 200 | exact match | 199 / 200 | This supports a strong but narrow fixture claim: these four shipped GPT pool files share the same 200-record canonical-JSON multiset and differ in order. It does **not** support the release README's universal formulation that every AppWorld arm differs only by sequence. The release itself contains registered exceptions: - `pool_D` retains IDs, order, outcomes, and trajectories but changes all 200 task descriptions; - `pool_drop50` is a 150-row subset; and - `pool_replay2` is the natural pool repeated twice, producing 400 rows but only 200 unique episodes. Raw-K controls use subsets/renderings, and the Sonnet conditions were separately rolled. More importantly, `arms.csv` contains no source-pool digest. Shipped fixture conservation therefore does not cryptographically prove that each historical result arm consumed that exact byte sequence. The receipt labels result-to-pool lineage `naming_and_code_evidence_only` and leaves causal order effects unestablished. The registered AppWorld outcome curves are also not uniformly paired: `aceB_N25` has 49 rows while its later checkpoints have 50. Current, A, and succdrop use stable 50-task sets in the released rows; B is explicitly marked unpaired rather than silently compared as if every point had `n=50`. The corrected receipt also compares evaluation identities across schedules at each shared checkpoint. At `N25`, current, A, and succdrop share the same 50-task identity set while B has a distinct 49-task set. At `N50`, `N100`, `N150`, and `N200`, all four released arms share the same 50-task identity set. B-only `N85` is not a shared checkpoint and is therefore explicitly outside the cross-schedule pairing comparison. This closes a pairing-description gap; it does not create a causal schedule estimate. ## What the paper close read changes The paper's main conceptual contribution survives this audit: episodic evidence and cross-episode abstraction should not be treated as interchangeable, and continual consolidation needs gates because utility can be non-monotonic and schedule-sensitive. Several source statements still need correction or qualification. ### The 19-task headline has three incompatible values - Abstract, p. 1: GPT-5.4 “fails on 54%”. - Introduction, p. 2: it “fails on 46%”. - §4.1, p. 5: accuracy is brought “down to 54%”. - Figure 2, p. 3: Stream R10 is plotted as `52.6% = 10/19`, hence failure is `47.4% = 9/19`. Figure 2 also calls Static “at the ceiling” while plotting `94.7% = 18/19` against a dashed 100% ceiling. Public prose should use the exact fraction and say Static remained near the ceiling, not at it. These are internal paper-text/figure conflicts. The current-release audit does not choose one prose value as historical ground truth or claim to rerun that experiment. ### “Raw trajectories” are sometimes episode-level summaries Table 2's heading says `Trajectory Logs (Summarized)`, while its caption calls them “raw trajectories”. Table 3 again defines the control as append-only summarized rollouts. Appendix F.4, pp. 48–49, states that each AppWorld datapoint is a Qwen3.5-27B prose summary of a full ReAct trajectory, used in place of raw token-level trajectories. The defensible comparator description is therefore **episode-level summarized rollouts with no cross-trajectory lesson abstraction**. That still preserves an important contrast with consolidated lesson stores, but it cannot establish that byte-raw histories were the control everywhere. ### Paper and release scope are not one snapshot The paper names Mind2Web in §3 and again in Limitations, but contains no Mind2Web setting row, result table, figure, or appendix experiment. Its listed environment appendices cover ALFWorld, WebShop, ScienceWorld, and AppWorld. The current repository instead advertises PropWorld as its synthetic sixth environment alongside ARC-AGI. The release's AppWorld archive identifies later July/August assembly and verification dates, and the sole public commit is dated August 15. This makes the repository a valuable expanded/post-paper public release, but not a proven May-v1 production mirror. PropWorld must not be presented as a synonym or replacement result for the paper's missing Mind2Web surface. ### Mechanism evidence remains low-repeat The paper itself acknowledges text-only scope, natural-language abstractions, model-capability dependence, few repeats, point estimates, and no formal error bars for many results (§8, p. 11). The WebShop W8 memory-surgery appendix uses `n=50`, a single run, and explicitly reports no significance test (Appendix I.1, pp. 61–62). The AppWorld release notes likewise record large build noise: succdrop retesting changed the curve substantially, and the clean monotonic sfflat result had one seed. These can support mechanism hypotheses and concrete counterexamples. They do not support a prevalence estimate or universal causal law that consolidation necessarily collapses. ## Method and evidence surfaces [`PROTOCOL.md`](./PROTOCOL.md) froze the original eight gates, curve targets, fourteen listed control cases, expected release counts, and claim ceiling before formal independent execution. Code review then rejected that preliminary two-root output because three controls were self-proving and the receipt chain was under-bound. [`REVIEW-AMENDMENT.md`](./REVIEW-AMENDMENT.md) records that failure chronologically and freezes the corrected production-path controls, exact curve registry, source discovery, package-safety scan, and run/compare/install bindings before the corrected formal execution. A second adversarial review rejected that corrected output before installation because Python equality admitted wrong JSON types, comparison bytes were not canonical at installation, and the package inventory ignored two file classes. [`REVIEW-AMENDMENT-2.md`](./REVIEW-AMENDMENT-2.md) freezes exact JSON types, a whole-audit digest, canonical receipt bytes, a no-extras package inventory, and the procedural ceiling on stored repeatability before the final formal execution. The standard-library checker then: 1. rejected a wrong/dirty source or wrong PDF; 2. hashed only the decision-relevant tracked inputs; 3. independently validated all arm/row identities, metadata, aggregates, and ARC grids; 4. emitted both denominator surfaces for every affected arm; 5. executed the exact upstream verifier and separately instrumented its locked mapper for coverage accounting; 6. compared selected full pool payloads and registered exceptions; 7. computed only preregistered descriptive curve properties; 8. detected all seventeen negative production-path controls and passed both positive regression/metamorphic controls; and 9. repeated the complete checker under hash/traversal seeds 313 and 727 in two fresh output roots, with each environment receipt binding the exact checker, source, paper, and stable-receipt hashes. The registration declaration inside the curve receipt is a pinned value; its chronology depends on the frozen protocol/amendments rather than being proved by that Boolean field. An X9-only adversarial regression harness also rehashed all dependent receipts and rejected equal-valued type substitutions, non-canonical comparison bytes, ignored-file candidates, arbitrary extra files, and a public path fixture. These are checker-chain tests, not experiment results. The public package contains eleven checksummed files plus the checksum manifest: | Surface | Contents | Boundary | | --- | --- | --- | | `PROTOCOL.md` | Frozen pre-execution design | Registered after paper/source reconnaissance and official gates; not blinded. | | `REVIEW-AMENDMENT.md` | Frozen post-review correction | Preliminary receipts rejected; corrected checker not yet run when frozen. | | `REVIEW-AMENDMENT-2.md` | Frozen second post-review correction | Corrected receipts rejected; final checker not yet run when frozen. | | `README.md` | Method, results, tensions, and claim ceiling | Editorial synthesis, not a paper/source object. | | `audit.py` | Exact-source runner, gates, mutations, compare/install verification | AMS-authored standard-library infrastructure. | | `verify_checked.py` | Receipt-only and fresh source-bound verification | Receipt-only mode does not reopen source or paper. | | `raw/audit.json` | Source lock, membership, denominators, official scope, schedules, curves, claim dispositions | Stable public-safe receipt; no upstream free text. | | `raw/mutation-controls.json` | 17 detected negative controls and 2 passed positive controls | Checker sensitivity, not upstream tests. | | `raw/repeatability.json` | Producer-recorded stable-byte comparison and environment bindings | Internal consistency; historical process identity is not receipt-proven. | | `raw/environment_run_a.json`, `raw/environment_run_b.json` | Run labels, seeds, Python/platform envelope | Producer metadata, not execution attestation. | | `checksums.sha256` | Every package file except itself | Package integrity only. | ## Verify and reproduce Verify package checksums, exact receipt schemas/digests, and the complete run/comparison binding without reopening the paper or source: ```bash PYTHONDONTWRITEBYTECODE=1 python3 verify_checked.py --mode receipt-only ``` This mode reports package integrity and internal receipt consistency only. Checksums are not an external signature, and receipt-only verification is not an exact-current-source revalidation or proof that two historical runs occurred. For a fresh source-bound rebuild, acquire the public source and paper outside this package, check out the exact commit, hydrate only the selected pool files, and use a fresh work root on external storage: ```bash SOURCE=/path/to/Memory-Collapse-Eval PAPER=/path/to/useful-memories-become-faulty-arxiv-2605.12978v1.pdf WORK=/external-disk/faulty-memory-source-bound-fresh git -C "$SOURCE" checkout adf347f67a81a19fc71d2529d09108c71a1f9712 git -C "$SOURCE" status --short PYTHONDONTWRITEBYTECODE=1 python3 verify_checked.py \ --mode source-bound \ --source "$SOURCE" \ --paper-pdf "$PAPER" \ --work-root "$WORK" ``` `--work-root` must not already exist. The verifier leaves both fresh roots in place for inspection, requires both stable-receipt sets to match each other, then requires both to byte-match the checked package. It also compares a deterministic comparison projection while deliberately excluding historical environment-receipt hashes, so a Python patch-version change does not become a false released-row mismatch. The current invocation establishes its own two-root procedure; it does not retroactively authenticate the stored producer history. The source checkout must contain these AppWorld pool files; no trajectories, memory snapshots, rollout subdirectory, model, or live environment is needed: ```text stream_pool_v2.jsonl pool_A.jsonl pool_B.jsonl pool_succdrop.jsonl pool_D.jsonl pool_drop50.jsonl pool_replay2.jsonl ``` ## Separate official-source acceptance Before the independent audit, the locked source produced these local receipts: - `python results/verify_numbers.py`: 675 assertions reproduced, zero mismatches, zero missing-row targets; - `python -m bench demo`: passed; and - `python -m pytest tests -q -rs`: `200 passed, 7 skipped`. Five skips were optional AgentGym environment suites because `requests` was not installed; the remaining two were explicit agentmem-tree/upstream availability skips. No skipped environment suite is reported as passed. The Makefile wrapper did not survive an absolute Python interpreter path containing a space because `$(PY)` is expanded unquoted. Direct `python -m bench demo` passed with the same interpreter. This is a source portability defect in the wrapper, not an experimental failure or reason to move work off the external disk. ## Claim ceiling A passing receipt-only package establishes one revision's released-row and selected-fixture receipt consistency plus package/internal bindings. A passing fresh source-bound invocation additionally establishes that invocation's two-root stable-byte match to the exact checked source/PDF and installed package. Neither establishes: - paper-time source or result identity; - a model, benchmark, or end-to-end environment rerun; - semantic correctness of historical outcomes; - historical prompt, pool, memory, or result lineage; - causal effects of schedule, abstraction, or memory method; - statistical significance, independent repetitions, or prevalence; - cross-platform reproducibility; - historical fresh-process identity or independence from the stored receipt; - security impact, production reliability, or real-user generalization; or - that “missing” means “failed”. Upstream code and released data retain their respective repository license terms and third-party boundaries. This AMS-authored artifact remains `pending_owner_choice`; publication here creates no new license grant.