Authoritative GitHub status audit plus a hash-pinned CSV parse, temporary one-byte repair, and adversarial exact-diff checker controls

Progress

At the 2026-07-22 audit, upstream issue #13 remained open and current main was 96fa6c4cc3a9bb7450ee7b6773b659d3a030dace. The pinned 169,537-byte CSV has 400 data records: 399 width six and only algebra-036 width five under ordinary parsing; strict parsing rejects it. A temporary one-byte quote insertion at offset 13,618 yields a 169,538-byte file with SHA-256 0c05a0d4af9cbe3e70413b250d6c9cac1bfe4d848f6c196f83ed61ebef9ced16, 400 strict-parsed six-column records, and exact Short Answer $Y(x)=A+\frac{B}{x}-x$. This mandatory baseline epoch did not edit the released dataset or submit externally.

Research-policy redirect

A field-progress claim must request candidate review and pass the fail-closed contribution gate.

Strategy and discriminator

CSV structural validation and minimal-patch certification

A standards-library CSV parser exposes the unique malformed row; the issue-supplied answer token identifies the separator; a second checker certifies all row widths, exact target fields, and equality to the baseline plus one quote byte.

Hypothesis: At upstream commit 96fa6c4cc3a9bb7450ee7b6773b659d3a030dace, algebra-036 is the unique five-field data record, strict CSV parsing rejects the malformed syntax, and inserting one quote before the answer separator restores all 400 records and the exact short-answer field without another change.

Test: Run python3 tools/baseline_answerbench_036.py on the pinned CSV and checker; require the baseline/repair signals and rejection of wrong-answer and unrelated-edit controls.

Rationale

The source snapshots establish current scope, generic CSV parsing reproduces the defect, strict parsing supplies a syntax-level negative control, the target-aware checker confirms the exact repaired fields, and exact byte comparison rejects collateral edits. This supports progress and a precise next patch step, but not candidate promotion because no actual correction or external validation exists.

Claims requiring scrutiny

As of the captured GitHub responses dated 2026-07-22T15:30:01Z through 15:30:02Z, issue #13 was open with no comments or recorded events, and repository PR searches for answerbench_v2 and imo-bench-algebra-036 returned zero results.
Upstream main resolved to commit 96fa6c4cc3a9bb7450ee7b6773b659d3a030dace; its AnswerBench v2 snapshot is 169,537 bytes with SHA-256 275877a9d988d85278fad3a5f8a41d7f83393a60bf259531ec0a5161e6b21cf9.
Under ordinary Python 3.12 CSV parsing the pinned file has 400 data records, exactly 399 with six fields and one with five; the unique malformed record is imo-bench-algebra-036. Strict parsing rejects the malformed syntax.
Inserting one double-quote at zero-based byte offset 13,618 yields a temporary candidate with all 400 rows strict-parsed to six columns and exact short answer $Y(x)=A+\frac{B}{x}-x$.
The narrow checker rejects the malformed baseline and wrong-answer control; with --reference it also rejects a semantically valid repair containing an unrelated edit.

Evidence and scope

python3 /root/proof-factory/skills/computational-researcher/scripts/run_experiment.py ... -- python3 tools/baseline_answerbench_036.py sources/96fa6c4cc3a9bb7450ee7b6773b659d3a030dace/answerbench_v2.csv tools/check_answerbench_036.py --result artifacts/baseline-audit-96fa6c4-275877a9.json; experiment 20260722-153113-23f885 returned 0
artifacts/baseline-audit-96fa6c4-275877a9.json records baseline widths {5:1,6:399}, strict rejection, repaired SHA-256, exact answer, and all control outcomes.
Two consecutive direct harness runs produced identical result SHA-256 5a70762c42caf6024b0f1169dfa628b683e88807f3a07b27edc1c3a64774dd37.
The preserved initial strict-only run 20260722-152331-c3af1d returned 1 with _csv.Error, motivating the dual-mode parser audit.

Computational experiments

.proof-experiments/20260722-152331-c3af1d: strict-parser-first harness failed with _csv.Error on the malformed baseline; preserved as a negative result.
.proof-experiments/20260722-153113-23f885: clean-provenance dual-mode harness returned 0 in 0.426 seconds; all structural and adversarial controls passed.

Independent checker

tools/check_answerbench_036.py is a target-aware semantic and exact-byte checker separate from the generic baseline/control harness in tools/baseline_answerbench_036.py; both encodings agreed on the temporary repair.

Contribution gate

not_requested

No structured gate reasons were recorded in this legacy attempt; see the adjudication ledger.

Original model outcome: progress
Public classification: progress

Cross-domain transfers tested

Software minimal-patch certification -> an exact baseline-diff guard should reject collateral changes that semantic checks miss -> the unrelated problem-ID mutation passed semantic checks but failed --reference validation.
Strict input validation -> malformed quoting may fail lexically before a schema-width assertion -> strict Python CSV parsing rejected the baseline while ordinary parsing exposed the five-field record.

Established facts

The pinned upstream CSV has exactly 400 data records under ordinary parsing, with width counts {5: 1, 6: 399}; the width-five record is algebra-036.
artifacts/baseline-audit-96fa6c4-275877a9.json and experiment 20260722-153113-23f885 · Upstream commit 96fa6c4cc3a9bb7450ee7b6773b659d3a030dace only · computed
A quote inserted at byte offset 13,618 restores six fields on every record and exact target Short Answer $Y(x)=A+\frac{B}{x}-x$.
Strict full-file parse, target assertion, and exact-diff checker in experiment 20260722-153113-23f885 · The temporary one-byte candidate derived from the pinned baseline · computed
The semantic-only checker is insufficient to exclude unrelated changes, while --reference excludes the tested unrelated mutation.
unrelated_edit control in artifacts/baseline-audit-96fa6c4-275877a9.json · The checker and pinned baseline used in this epoch · computed

Ruled out in this epoch

Use semantic-only CSV validation as the complete contribution certificate
Proposed algebra-036 repair candidates against the pinned baseline · It accepts a candidate containing an unrelated problem-ID mutation. · The unrelated_edit control has semantic_only_returncode 0 and with_reference_returncode 1. · A new checker proves full-file equality outside the target edit by another exact mechanism.
Solve or re-derive the functional equation before repairing the CSV
This source-defined dataset correction · Issue #13 and the existing raw token supply the exact stored answer literal; the failure is CSV quoting, not mathematical uncertainty. · Issue #13 body, pinned raw bytes, and successful structural repair · The authoritative source retracts or disputes the stored answer literal.
Treat issue #13 as already repaired upstream
Repository state observed on 2026-07-22 · The issue was open, main retained the malformed bytes, and both repository PR searches returned zero. · Pinned API snapshots and baseline experiment · A newer upstream commit, issue transition, or matching pull request appears.

Open leads

Apply and verify the isolated one-byte patch against a freshly resolved upstream head.
It is the cheapest path from verified baseline to the requested dataset correction. · Run git ls-remote for upstream main, refresh if needed, then invoke the checker with --reference on the materialized candidate. · high · open
Package the full-file checker as a regression test or PR verification command.
The pinned repository tree has no existing CI/test harness for CSV widths, and semantic-only checks miss collateral edits. · Run the checker on the fresh baseline and candidate from a clean checkout and record the exact output. · normal · open
Seek upstream acceptance after owner approval.
Maintainer review is the only external acceptance path identified by CONTRIBUTING.md. · Prepare a one-byte PR diff and validation transcript linked to issue #13; do not publish without human authorization. · high · open

Continuation checkpoint

Objective: Materialize and certify the exact one-byte dataset correction if the authoritative baseline is unchanged.

First action: git ls-remote https://github.com/google-deepmind/superhuman.git refs/heads/main

Stop condition: Stop or redirect if issue #13 is closed, a competing PR exists, upstream bytes changed, the checker fails, or the candidate diff is not exactly one inserted quote.

Next moves

Resolve upstream main and repeat issue/PR status searches immediately before editing.
If the target bytes and commit are unchanged, materialize only the quote insertion and run tools/check_answerbench_036.py with the pinned baseline as --reference.
Inspect the byte diff and repository tree; reject any diff larger than one inserted quote.
After human-owner approval and CLA readiness, prepare a GitHub PR linked to issue #13 with the checker receipt; do not claim resolution until maintainer acceptance.

Citations

Tool disclosure

GPT-5.6 Sol was principal investigator. A GPT-5.6-Terra delegate supplied advisory reconnaissance and the initial narrow checker; Sol independently refetched primary sources, audited the checker, corrected the delegate memo's file-size claim, and wrote the materially different baseline/control harness. Deterministic tools: CPython 3.12.3 standard-library csv/hashlib/subprocess/tempfile, Git, curl, GitHub API, py_compile, and the computational-researcher run_experiment recorder. No CAS, solver, proof assistant, package installation, lab compute, or model agreement was used as validation.; orchestration: gpt-5.6-sol principal with gpt-5.6-terra delegates.

Duration: 976.4s
Review state: not a result claim
Attempt ID: deepmind-superhuman-13-answerbench-column-20260722-153446-675bb9

Human review ledger

No human review recorded.