Across 71 public runs, GPT-4o-labelled scores ranged from 62.5% to 97.5%. Argonne says the files cannot always distinguish model behavior from system failures.
Argonne National Laboratory is changing the public leaderboard for its ChemGraph agentic chemistry framework from daily evaluation to monthly runs, the lab told Supercomputing News.
"We initially ran the suite daily to establish baselines and better understand run-to-run variability," said Thang Duc Pham, a postdoctoral appointee at Argonne who led ChemGraph's implementation, in written responses drafted with additional input from computational scientist Murat Keçeli. "That schedule was costly, and many apparent day-to-day changes were either small or attributable to API and infrastructure issues rather than meaningful changes in model capability."
The decision trades frequency for review time. Argonne said monthly evaluation will let the team inspect traces, separate genuine task failures from access problems, and publish more reliable summaries. The lab also said leaderboard text referring to "Daily Evaluation" was outdated and would be revised.
The public data explains why that review matters. Scores for several model-labelled series moved across repeated runs of the 40-query suite. The result files do not identify whether each miss came from the language model, an endpoint, the ChemGraph framework, or another part of the evaluation pipeline.
The leaderboard therefore measures reliability across the complete agentic chemistry system, rather than the reasoning ability of a language model in isolation.
ChemGraph is an open-source framework that converts natural-language chemistry requests into tool calls. Its LangGraph-based agents can work with scientific software including NWChem, ORCA, xTB, and MACE through the Atomic Simulation Environment, according to the project's public repository. The peer-reviewed ChemGraph paper appeared in Communications Chemistry in January 2026.
The public leaderboard evaluates single-agent and multi-agent configurations on 40 queries across 12 chemistry task categories. The scoring pipeline uses structured outputs and applies a 5% tolerance to numerical answers. Argonne calculates the headline accuracy as total correct answers divided by 40, with no separate weighting of task categories.
Pham said the suite has two jobs. It is both a capability benchmark and regression testing for the ChemGraph codebase.
"It evaluates the complete end-to-end system, including the language model, prompts, tools, and agent orchestration, rather than measuring the underlying model in isolation," he said.
Supercomputing News downloaded the published multi-agent results available on August 4 and recalculated each run using Argonne's micro-average: the number of correct answers divided by 40. The snapshot was pinned to dataset commit 2bb52d0125a5b581276eee1a960edea1b71502ed.
Across 71 dated runs between April 25 and July 14, the GPT-4o-labelled series ranged from 62.5% to 97.5%, a spread of 35 percentage points. GPT-4.1 ranged from 72.5% to 97.5%. Claude Sonnet 4.5 stayed between 92.5% and 97.5%.
Those figures are micro-averages across the 40 queries, not unweighted means of the 12 category scores. They describe the endpoint-labelled series recorded in the dataset. They do not establish that frozen versions of the underlying models changed by the same amount.
The ChemGraph paper provides useful context, but its GPT-4o result is not directly comparable with the longitudinal leaderboard.
The paper reported 100% accuracy for a multi-agent GPT-4o configuration on two reaction thermochemistry tasks that required nine to 12 tool calls. It used the pinned gpt-4o-2024-08-06 model. The leaderboard covers 40 queries in 12 categories and records the mutable openai/gpt-4o endpoint alias. The task scope and model identification differ.
The authors also cautioned in the paper that "even at zero temperature, the performance of the LLM still varies across different runs." Temperature zero reduces sampling variability, but it does not freeze the larger system around the model.
That larger system was not frozen during the leaderboard period. Argonne said repeated evaluations can capture model updates, API behavior, infrastructure problems, and ChemGraph modifications. The public result files show the score and category outcomes, but the August 4 snapshot does not record a ChemGraph commit, endpoint version, benchmark hash, or failure class for each run.
Without that provenance, a low score can show that the pipeline did not complete a task successfully. It cannot always show which component was responsible.
Argonne accessed commercial models through ARGO, its internal service for authenticated access to large language model APIs. Pham said the observed variation reflects both model nondeterminism and problems outside the model, including endpoint changes, rate limits, API access, and model availability.
"The current result files may record a failed or unavailable API call as an unsuccessful evaluation, so some sharp score drops do not represent a change in the model's scientific or reasoning capabilities," he said.
The Claude 3.5 Haiku series recorded zero across all categories on July 14 after much stronger earlier results. Pham said an ARGO endpoint or model-access problem caused the zero, not a collapse in the model's capability.
"Isolated runs affected by API or infrastructure failures should not be interpreted as capability measurements," he said. Variation among otherwise successful runs, he added, is more indicative of model sampling and the nondeterminism of agentic workflows.
The daily schedule produced a dense record, but it left less time to classify those outcomes before publication. Under the monthly cadence, Argonne plans to review traces more carefully and may run additional evaluations after material changes to ChemGraph, the benchmark suite, a model endpoint, or an API.
This is an observability problem familiar to teams putting agentic AI into production infrastructure. A single score can summarize whether an end-to-end workflow succeeded. Diagnosing a failure requires records from every link in the chain.
ChemGraph connects language models to simulation tools that can consume substantial computing allocations. Pham said the framework can run machine-learning interatomic potential software, gRASPA adsorption simulations, and XANES calculations through FDMNES at scale. The team has also integrated Globus Compute and Globus Transfer to coordinate workflows on HPC systems.
The group has already demonstrated a related agent-driven workflow on the Aurora supercomputer. In that campaign, covered previously by Supercomputing News, agents coordinated 11,182 grand canonical Monte Carlo simulations across 256 nodes. The accompanying preprint reports 60 to 90 seconds of agentic overhead for individual simulation jobs that ran for roughly 1,600 to 4,400 seconds.
Those figures measure orchestration time. They do not measure the allocation consumed by jobs that execute successfully but produce an unusable scientific result.
"Short answer is, no, we haven't measured it," Pham said when asked whether the team had quantified wasted allocation hours from agent-initiated jobs. He said ChemGraph limits that risk by focusing on well-defined tasks with constrained tools and explicit success criteria. The team expects validated workflows to reduce avoidable setup and analysis mistakes, especially for less experienced users, but has not quantified that benefit either.
The boundary is intentional. "ChemGraph does not yet autonomously generate and validate arbitrary production job scripts for any scientific application or machine," Pham said. "Our strategy is to develop these expert-guided tools via collaborations instead of just relying on LLMs to generate these tools."
That approach fits a broader pattern in which simulation systems become infrastructure for AI-guided science. Reliability depends on the model, but also on the software and services that connect a prompt to a completed calculation.
Monthly evaluation will produce fewer data points. Argonne's stated goal is to make each one easier to interpret by reviewing traces and distinguishing task failures from infrastructure problems before publishing a summary.
Run-level records of the ChemGraph commit, benchmark version, model endpoint, and failure class would make future scores easier to interpret. Argonne has already identified the most important distinction. "We are working to make this distinction clearer so that infrastructure failures are not confused with genuine task failures," Pham said.
Publishing repeated results gave users a view of variance that a one-time benchmark could not provide. The next test is whether the monthly summaries make it possible to tell a model miss from a failure elsewhere in the chain.
The SCN Staff is a small AI editorial squad working under human direction. Each agent owns one job.
Scout does the research. It runs down primary sources and checks what's already been published, on SCN and everywhere else, before a story gets written. If a claim can't be traced back to a real document, Scout flags it.
Forge writes. It takes what Scout found and turns it into a draft, argument and sentences and all. Every SCN piece starts here, then gets sharpened.
Cipher handles search: the titles, descriptions, and keyphrase work that decides whether a good article ever gets found. Least glamorous job on the squad. Also one that matters more than it looks.
Pixel makes the visuals. Images, charts, the occasional diagram, all built to SCN's brand instead of pulled from a stock library. When something's easier to see than to read, it goes to Pixel.
Editorial judgment and the final call stay with the humans. So does the fact-checking.
