OpenAI posted 722 AI math papers without the disclosures its advisers urged

OpenAI last Tuesday posted 722 math papers produced by an unreleased internal model, saying it had drawn on advice from an independent group of mathematicians. The release left out several disclosures that group had asked for a week earlier.

The Advisory Group on Mathematics and Artificial Intelligence, nine mathematicians hosted at the Institute for Advanced Study, published its guidelines on Sept. 29. "For each result released, the AI lab should make public the name of the model, the prompts used, a (summarized) chain of thought, the time taken, and the estimated cost of computation," the group wrote. It also asked labs "to stop testing advanced mathematical problems on proprietary models."

OpenAI's GitHub repository does not name the model, describing it only as "an unreleased internal OpenAI model." It publishes no prompts and includes 10 reasoning summaries. It gives one average compute figure: "roughly three hours of ChatGPT Pro thinking," according to OpenAI's post.

The group also said papers should go to repositories that "should not be controlled by any AI lab." OpenAI used its own GitHub account and said it is "continuing to explore other community-hosted alternatives." About 42% of top-line results have computer-checked Lean proofs, the repository says. A day after the release, OpenAI withdrew three papers over a sign error, leaving 719.

In an Oct. 6 statement, the advisory group said its role "should not be interpreted as a judgment of the impact of these results or an endorsement of the process by which OpenAI obtained them." It added that "it is ultimately up to the mathematical community to assess the extent to which our recommendations were followed successfully." TechCrunch first reported that the release fell short of the group's standards.

OpenAI says it will fund workshops and conferences on AI-produced results and is "working to responsibly release the model." It has not responded publicly to the criticism. Group member Melanie Wood told TechCrunch: "there is not human understanding of them at the point of release, and now the work begins."

The disclosures OpenAI skipped are the ones outsiders would need to check its claims, so until the model, prompts and per-result costs are public, mathematicians have to take the lab's word for how the results were made.