Across 71 public runs, GPT-4o-labelled scores ranged from 62.5% to 97.5%. Argonne says the files cannot always distinguish model behavior from system failures.
Argonne National Laboratory's ChemGraph is an open-source agentic framework that turns a plain-language chemistry request into a sequence of calls to established simulation packages, so that the numbers come from physics codes rather than from the language model itself. A researcher who wants a molecule's vibrational frequencies can ask for them instead of building the workflow by hand. The laboratory evaluates that pipeline against a fixed benchmark suite and publishes the results as a public leaderboard on Hugging Face.
Argonne is changing that leaderboard from daily evaluation to monthly runs, the lab told Supercomputing News (SCN).
"We initially ran the suite daily to establish baselines and better understand run-to-run variability," said Thang Duc Pham, a postdoctoral appointee at Argonne who led ChemGraph's implementation, in written responses drafted with additional input from computational scientist Murat Keçeli. "That schedule was costly, and many apparent day-to-day changes were either small or attributable to API and infrastructure issues rather than meaningful changes in model capability."
The decision trades frequency for review time. Argonne said monthly evaluation will let the team inspect traces, separate genuine task failures from access problems, and publish more reliable summaries. The lab also said leaderboard text referring to "Daily Evaluation" was outdated and would be revised.
The public data explains why that review matters. Scores for several model-labelled series moved across repeated runs of the same fixed suite. The result files do not identify whether each miss came from the language model, an endpoint, the ChemGraph framework, or another part of the evaluation pipeline.
ChemGraph's LangGraph-based agents can work with scientific software including NWChem, ORCA, xTB, and MACE through the Atomic Simulation Environment, according to the project's public repository. The peer-reviewed ChemGraph paper appeared in Communications Chemistry in January 2026.
The public leaderboard evaluates single-agent and multi-agent configurations on 40 queries across 12 chemistry task categories. The scoring pipeline uses structured outputs and applies a 5% tolerance to numerical answers. Argonne calculates the headline accuracy as total correct answers divided by 40, with no separate weighting of task categories.
Pham said the suite has two jobs. It is both a capability benchmark and regression testing for the ChemGraph codebase.
"It evaluates the complete end-to-end system, including the language model, prompts, tools, and agent orchestration, rather than measuring the underlying model in isolation," he said.
SCN downloaded the published multi-agent results available on August 4 and recalculated each run using Argonne's micro-average. The snapshot was pinned to dataset commit 2bb52d0125a5b581276eee1a960edea1b71502ed.
Across 71 dated runs between April 25 and July 14, the GPT-4o-labelled series ranged from 62.5% to 97.5%, a spread of 35 percentage points. GPT-4.1 ranged from 72.5% to 97.5%. Claude Sonnet 4.5 stayed between 92.5% and 97.5%.
Those figures are micro-averages across all queries, not unweighted means of the category scores. They describe the endpoint-labelled series recorded in the dataset. They do not establish that frozen versions of the underlying models changed by the same amount.
The paper reported 100% accuracy for a multi-agent GPT-4o configuration on two reaction thermochemistry tasks that required nine to 12 tool calls. It used the pinned gpt-4o-2024-08-06 model. The leaderboard covers the full suite and records the mutable openai/gpt-4o endpoint alias. The task scope and model identification differ.
The authors also cautioned in the paper that "even at zero temperature, the performance of the LLM still varies across different runs." Temperature zero reduces sampling variability, but it does not freeze the larger system around the model.
That larger system was not frozen during the leaderboard period. Argonne said repeated evaluations can capture model updates, API behavior, infrastructure problems, and ChemGraph modifications. The public result files show the score and category outcomes, but the August 4 snapshot does not record a ChemGraph commit, endpoint version, benchmark hash, or failure class for each run.
Without that provenance, a low score can show that the pipeline did not complete a task successfully, but it doesn't always show which component was responsible.
Argonne accessed commercial models through ARGO, its internal service for authenticated access to large language model APIs. Pham said the observed variation reflects both model nondeterminism and problems outside the model, including endpoint changes, rate limits, API access, and model availability.
"The current result files may record a failed or unavailable API call as an unsuccessful evaluation, so some sharp score drops do not represent a change in the model's scientific or reasoning capabilities," he said.
The Claude 3.5 Haiku series recorded zero across all categories on July 14 after much stronger earlier results. Pham said an ARGO endpoint or model-access problem caused the zero.
"Isolated runs affected by API or infrastructure failures should not be interpreted as capability measurements," he said. Variation among otherwise successful runs, he added, is more indicative of model sampling and the nondeterminism of agentic workflows.
The daily schedule produced a dense record, but it left less time to classify those outcomes before publication. Under the monthly cadence, Argonne plans to review traces more carefully and may run additional evaluations after material changes to ChemGraph, the benchmark suite, a model endpoint, or an API.
This is an observability problem familiar to teams putting agentic AI into production infrastructure. A single score can summarize whether an end-to-end workflow succeeded, however, diagnosing a failure requires records from every link in the chain.
ChemGraph connects language models to simulation tools that can consume substantial computing allocations. Pham said the framework can run machine-learning interatomic potential software, gRASPA adsorption simulations, and XANES calculations through FDMNES at scale. The team has also integrated Globus Compute and Globus Transfer to coordinate workflows on HPC systems.
The group has already demonstrated a related agent-driven workflow on the Aurora supercomputer. In that campaign, covered previously by SCN, agents coordinated 11,182 grand canonical Monte Carlo simulations across 256 nodes. The accompanying preprint reports 60 to 90 seconds of agentic overhead for individual simulation jobs that ran for roughly 1,600 to 4,400 seconds.
Those figures measure orchestration time. They do not measure the allocation consumed by jobs that execute successfully but produce an unusable scientific result.
"Short answer is, no, we haven't measured it," Pham said when asked whether the team had quantified wasted allocation hours from agent-initiated jobs. He said ChemGraph limits that risk by focusing on well-defined tasks with constrained tools and explicit success criteria. The team expects validated workflows to reduce avoidable setup and analysis mistakes, especially for less experienced users.
The boundary is intentional. "ChemGraph does not yet autonomously generate and validate arbitrary production job scripts for any scientific application or machine," Pham said. "Our strategy is to develop these expert-guided tools via collaborations instead of just relying on LLMs to generate these tools."
That approach fits a broader pattern in which simulation systems become infrastructure for AI-guided science. Reliability depends on the model, but also on the software and services that connect a prompt to a completed calculation.
Argonne manages the Transformational AI Models Consortium, one of the two platform components in DOE's Genesis Mission. Argonne's July feature story says ChemGraph "complements" that initiative. The framework predates it. The peer-reviewed work was supported by the Office of Science's Advanced Scientific Computing Research program under Argonne's operating contract, and Genesis's first competitive awards came later. SCNs has documented the same framing applied to existing work at Los Alamos. The label matters less than the scale it implies. If agentic frameworks are going to run across a national federation, separating a model failure from an endpoint failure stops being one leaderboard's bookkeeping problem.
Monthly evaluation will produce fewer data points. Argonne's stated goal is to make each one easier to interpret by reviewing traces and distinguishing task failures from infrastructure problems before publishing a summary.
Run-level records of the ChemGraph commit, benchmark version, model endpoint, and failure class would make future scores easier to interpret. Argonne has already identified the most important distinction. "We are working to make this distinction clearer so that infrastructure failures are not confused with genuine task failures," Pham said.
Publishing repeated results gave users a view of variance that a one-time benchmark could not provide. The next test is whether the monthly summaries make it possible to tell a model miss from a failure elsewhere in the chain.
The SCN Staff is a small AI editorial squad working under human direction. Each agent owns one job.
Scout does the research. It runs down primary sources and checks what's already been published, on SCN and everywhere else, before a story gets written. If a claim can't be traced back to a real document, Scout flags it.
Forge writes. It takes what Scout found and turns it into a draft, argument and sentences and all. Every SCN piece starts here, then gets sharpened.
Cipher handles search: the titles, descriptions, and keyphrase work that decides whether a good article ever gets found. Least glamorous job on the squad. Also one that matters more than it looks.
Pixel makes the visuals. Images, charts, the occasional diagram, all built to SCN's brand instead of pulled from a stock library. When something's easier to see than to read, it goes to Pixel.
Editorial judgment and the final call stay with the humans. So does the fact-checking.
