SpecSnap: Measuring Specification Retention and Regression Safety Across Multi-Session Agent Cutoffs

1. The Blind Spot of Single-Session Benchmarks

For the past three years, software engineering benchmarks have shared a comforting fiction: that a coding agent receives a task, reasons inside an unbroken context window, and submits a pull request before the conversation ever ends. Under those conditions, frontier models look increasingly capable.

Production software engineering rarely works that way. Multi-package sprints span dozens of files and evolving product rules. Long before an agent finishes a complex refactor, its context window fills up, its harness triggers automatic conversation compaction, or the session is interrupted and resumed in a fresh process.

Waiting for an agent to exhaust a 200,000-token context window naturally before testing session handoffs is both expensive and scientifically noisy: each model compacts at a different moment, after a different number of edits. I designed SpecSnap to make the session cutoff deterministic, repeatable, and identical across models:

1. Session 1 (Specification Delivery): The agent receives a multi-package repository and a sprint of concrete engineering requirements in the conversation prompt. On Tiers 2-4, the starter repository already passes all visible tests (pytest -q). On Tier 1, the agent first implements and commits the baseline TASK.md before beginning the mid-session product note.
2. The Session Cutoff: Session 1 is cut off immediately after the agent's first .py source edit for the new sprint requirements, before that edit is committed.
3. Session 2 (Continuation): A fresh session starts in the same workspace with zero access to the Session 1 transcript and receives a standard continuation prompt: "Continue from the current workspace state, complete all remaining requirements across the packages, verify all tests pass, and commit your work." On Tier 4, this cutoff repeats three times across four sequential sessions, with Sessions 2 and 3 introducing contract amendments that modify or reverse earlier specifications.
4. Regression-Gated Grading: Once the final session commits its work, an isolated hidden test suite (grader/hidden_tests/) evaluates three criteria: how many sprint requirements pass (Raw Retention), whether original baseline behavior and architectural module boundaries in wiring.yaml remain intact (Net Retention), and the run's Token-Efficiency Score (TES) against a fixed token budget.

Evaluated Models & Harnesses

2. Empirical Results Across Synthetic Tiers

The results across all four tiers expose a stark vulnerability in how current coding agents operate. When an agent is interrupted mid-sprint, almost the entire specification vanishes with the conversation window.

The table below reports Raw and regression-gated Net Retention across all four tiers for Antigravity Gemini 4 Agron, Claude Code Opus 5.5, and Grok Build 4.7. Net retention equals raw retention only when all original baseline tests pass, protected test files are untouched, and zero wiring.yaml AST module-boundary violations occur; any regression or boundary violation zeroes the score for that tier.

3. Adaptive Curriculum Escalation & Adversarial State Perturbation (>= 75.0% TES)

Static benchmarks saturate quickly once developers optimize for a fixed task distribution. To prevent static overfitting, SpecSnap incorporates a two-stage evaluation architecture governed by evaluate-sandbox:

• Stage I · Multi-Session Retention (Active): Evaluates whether an agent can preserve and implement multi-package specifications across deterministic session cutoffs without violating baseline tests or module boundaries.
• Stage II · Adversarial State Perturbation (LOCKED at >= 75.0% TES): Across all four tiers (Tiers 1-4), unlocking the second stage requires clearing a tier with PASS and a Token-Efficiency Score (TES) of at least 75.0%. Because no evaluated frontier model has reached that threshold on any tier (peaking at 50.0% TES on Tier 1 and 12.5% TES on Tiers 2-4), Stage II remains locked.
• Global Phase Transition & Seed Attestation: Each evaluation binds its scorecard receipt to a deterministic run seed (--seed) and a SHA-256 Merkle digest of the finished sandbox (sandbox_tree_sha256). The first verified submission that crosses 75.0% TES unlocks the Adversarial State Perturbation regime globally for all subsequent leaderboard evaluations.

4. Limitations & Threats to Validity

1. Single trajectory per cell: Each cell in the primary evaluation represents a single deterministic cutoff trajectory, so differences of one or two requirements between models lie within run-to-run variance.
2. Deterministic cutoff point: Cutting off immediately after the first .py source edit establishes a controlled, reproducible interruption point across harnesses, whereas production context compaction can occur at varying token depths.
3. Cross-harness token accounting: Different agent harnesses attach different system prompt and tool-schema overheads per turn, and Grok Build 4.7 token counts are reconstructed estimates. Token efficiency is therefore most directly comparable within a single harness.

Citation

To cite SpecSnap in academic papers or technical reports (plain text or BibTeX):

Adam Murphy. "SpecSnap: Measuring Specification Retention and Regression Safety Across Multi-Session Agent Cutoffs." November 2026. https://whitepaper.specsnap.dev

@misc{murphy2026specsnap,
  title        = {{SpecSnap: Measuring Specification Retention and Regression Safety Across Multi-Session Agent Cutoffs}},
  author       = {Adam Murphy},
  year         = {2026},
  month        = nov,
  howpublished = {\url{https://whitepaper.specsnap.dev}},
  note         = {Multi-session coding agent benchmark evaluating Antigravity Gemini 4 Agron, Claude Code Opus 5.5, and Grok Build 4.7. Repository: https://github.com/adamm285-dev/SpecSnap}
}