A controlled baseline of 457 diverse problems to calibrate the unprecedented progress of AI in mathematics.
AI progress in mathematics has been absolutely extraordinary. In a little over a year, frontier models went from barely achieving gold medals at the International Math Olympiad to a claimed resolution of the Navier–Stokes Millennium problem, one of the holy grails of mathematics.
Given this rapid rate of progress, it's critical to determine if AI is merely conquering “AI shaped” problems, or truly conquering mathematics altogether, replacing the role of human insight in proof generation.
Measuring this is difficult, since frontier labs don't report failures on open problems, only successes. We need a large, diverse set of open problems which are, with high probability, important to their fields. For this, we argue Wikipedia’s public List of unsolved problems in mathematics, snapshotted in January 2024, provides a particularly robust sample.
The dataset is in beta, and was generated using LLMs and has not been reviewed by mathematicians. See About the Data for more limitations and planned adjustments.
