The released DeepMind benchmark currently contains the reproducible nonsense value `45752`, while its version-1 row and the primary olympiad solution both give exactly 4 and 5. The repair is one field, objectively testable, and useful to every benchmark consumer. Independent mathematical checking also prevents the reporter's incorrect second suggestion from entering the patch.
dataset correctionVerified
Correct the corrupted short answer for IMO AnswerBench geometry problem 004
In `imobench/answerbench_v2.csv`, replace the corrupted Short Answer value `45752` for `imo-bench-geometry-004` with `4, 5`, preserving the six-column CSV schema. Do not change `imo-bench-number_theory-048`: an independent derivation confirms that version 2's `ab ≤ e³` answer is correct despite the issue's second suggestion.
Parse both AnswerBench CSVs with a standards-compliant reader, assert six fields per row, assert that geometry problem 004 has the exact short answer `4, 5`, and cite the official 2009 Sharygin Geometry Olympiad solution. Add a focused regression check if the repository has an accepted test location.
- Difficulty
- 1/10
- Attempts
- 1
- Last attempt
- 2026-07-22 19:25 UTC
- Source status
- open
- External validation
- none
CSV validationbenchmark curationprimary-source verificationregression testing
Resumable campaign memory
1 epochs · 1 promising · 0 blocked · 3 ruled outResearch map
Convert the validated local candidate into an upstream-reviewed correction.
First action: Fetch current google-deepmind/superhuman main, compare its commit and target row with 96fa6c4cc3a9bb7450ee7b6773b659d3a030dace, then replay the patch and checkers.
Stop or redirect when: Stop if the correction is already present; otherwise stop the technical phase after a one-cell pull request is prepared for authorized submission.
- Submit the clean one-cell patch upstream.
Refetch current main and replay the two checkers before submission.
- hash-bound one-cell dataset correction
Full-file semantic identity outside one approved CSV cell, independently reinforced by a unique context-bound byte replacement.
Reopen only if: Reopen technical work only if upstream main changes the target row or schema, a checker fails on the refreshed baseline, or maintainers request additional evidence.
- Use an unquoted replacement 4, 5.
It parses as two fields and corrupts target-row arity.
Reopen only if: None; standards-compliant CSV requires quoting. - Normalize every v2 record to six fields.
That would alter unrelated pre-existing algebra-036 data and violate the one-cell repair contract.
Reopen only if: A separately sourced request to repair algebra-036. - Change imo-bench-number_theory-048 as part of issue 14.
The task's independent derivation confirms the v2 answer; the scoped correction explicitly protects it.
Reopen only if: A separate rigorous derivation finding an error in the current v2 value and an independently scoped correction request.
Complete history
Attempts on this problem
2026-07-22 19:25 UTCOpen-problem program · 13 min
Correct the corrupted short answer for IMO AnswerBench geometry problem 004
Applied the exact one-cell AnswerBench v2 correction, compared it with commit-pinned v1/v2 sources, and validated it through independent semantic-CSV and raw-byte encodings.
What this run accomplished
Corrected imo-bench-geometry-004 from 45752 to the quoted CSV field 4, 5. Pinned v1, the official Sharygin solution, a complete semantic comparison, a separate byte checker, four negative controls, and a clean-upstream patch replay all agree. number_theory-048 and every unrelated byte remain unchanged.
Candidate — review neededOpen full record →