IndustrialBench

Measuring whether language models can diagnose equipment failures and repair controller logic on simulated factory lines.

01Overview #

Axilon builds software that helps maintenance and controls engineers find the cause of equipment failures. We built IndustrialBench to find out where language models still fall short at that work. It has 238 tasks on simulated plants. In each one, an agent has to find the component behind a fault, predict which parts will fail next, or fix faulty PLC logic, working from the historian data, plant documents and controller programs an engineer would have.

A benchmark’s hardest tasks only mean something if they’re fair, so we cover how the tasks are built and how we checked the grader before getting to how 13 models did.

The tasks themselves stay private. We publish the results and the method, and run evaluations on request.1

02The benchmark #

IndustrialBench has two tracks, both drawn from a corpus of 582 generated cases.

The core track (101 tasks) has 40 root-cause analysis (RCA) tasks and 40 predictive-maintenance (PdM) tasks across difficulty tiers from smoke tests to hard, plus 21 PLC tasks: logic defects with no backup to compare against, operating events, lines with 55 controllers, and negatives where nothing is wrong.

The frontier track (137 tasks) is every case in the corpus that meets its eligibility rules and doesn’t overlap the core track. Its tasks put the true cause among many simultaneous degradations (hard, 36), behind vague prose-only operator complaints (stress, 40), or next to more-degraded parts on the same equipment (extreme-medium, 32), along with harder PLC repairs (29).

Each task gives the agent an optional operator ticket, a historian of tag values, and the plant’s project share. Step through an illustrative task below.

Figure 1Anatomy of a task
1 / 4 Operator ticket
Reported
t = 3,600 s
Tag
FIL-301.FillVolume · station FIL-301, bottle filler
Observed
487 mL against a 500 mL baseline
Operator note
“Underfills started mid-shift. Capper and labeler look normal.”

The symptom is at the filler. The ticket is optional: on stress tasks it’s replaced by a vague prose complaint, and on PdM tasks there’s no fault yet to report.

ToolDescription
search_tagsFind historian tags by station, name or description, such as every FIL-301.* tag and the upstream PMP-101.* tags.
get_tag_samplesPull raw time-series samples for chosen tags and time windows.
list_dir read_file grepBrowse the plant’s project share.
submit_final_guessSubmit the answer. The agent has 24 investigative turns; turn 25 forces a submission.
project-share/
├── docs/
│   ├── control-narrative.pdf
│   ├── equipment-list.csv
│   ├── tag-dictionary.csv
│   ├── alarms-and-interlocks.csv
│   ├── operating-procedures/
│   ├── maintenance-guides/      # per station, each with a component register
│   └── work-order-history.csv
└── plc/
    ├── deployed/*.L5X           # Rockwell ladder logic, one per controller
    └── backups/*.L5X            # last-known-good, on some lines
{ "station_id": "PMP-101", "component": "impeller" }

The cause is upstream of the symptom: a worn impeller on the feed pump. Hardware answers are graded on exact station and component, which must come from the station’s register. PdM tasks ask for parts in order of urgency and are graded on the first one.

1.0Right station, right part. Passes.
0.5Right station, wrong part. Doesn’t pass.
0Wrong station, such as the filler showing the symptom.
An illustrative hardware-fault task, not a benchmark task. PLC tasks instead ask the agent to submit a corrected ladder program, which we run in simulation (§3.1).

03Validating the grader #

All scoring is deterministic, and no model grades another model. Hardware answers are compared with a known key. PLC repairs are replayed in a twin of the plant. Before reporting any scores, we checked three things: that the grader responds correctly to answers we know are right or wrong, that each answer key is the only defensible answer, and that the tasks no model solves are visible, consistently keyed and uncontested.

3.1Testing the PLC check on known inputs #

The PLC check loads the model’s corrected program into a twin plant and drives it through a fixed probe: writes above every ceiling and below every floor, a stop and start of every interlock, and a restore to defaults. It compares every setpoint trajectory with the healthy program’s and passes when the worst error is within 2% of the setpoint’s span. Because the probe pushes every setpoint to and past its limits, an edit that only hides the symptom, such as bypassing an interlock, can’t pass.

We graded altered versions of the reference repair for all 47 PLC defects, 18 in the core track and 29 in the frontier track.

Submitted repairShouldResult
Reference repairpass47 / 47 pass
No edit submittedfail47 / 47 fail
Corrected constant retyped (68 → 68.0)pass41 / 41 pass
Interlock fix with the contact inverted (XIC ↔ XIO)fail6 / 6 fail

To find the check’s tolerance, we then scaled the corrected constant in each of the 41 constant defects. Select a factor to see how the check responds.

Figure 2PLC check: corrected constant scaled by a factor
Repairs accepted, out of 41 constant defects. The check rejected every wrong repair we tried and accepted every equivalent one. Its tolerance isn’t zero, and the chart shows where it sits.

3.2Answer key uniqueness #

A task is unfair if a second answer is as defensible as the key. We tested both task types.

PdM tasks ask which parts will fail, most urgent first. We compared the labelled part’s projected end-of-run health with every other degrading part in the scenario, and confirmed it against simulated health where we ran the plant. In 17 tasks a distractor ends more degraded than the labelled part. We also checked whether the labelled fault is the one with larger effects elsewhere in the plant: in 13 of those 17, a distractor is. So on those tasks, naming the more-degraded part first is defensible, and we now accept it. After merging one task where the distractor is the same part, 16 tasks accept an alternative, applied to every model equally.2

For the 94 RCA tasks with more than one fault, we reran the scenario once per fault with that fault removed and measured how much the ticket’s symptom tag changed. In 6 structured tasks and 1 prose task, removing a distractor changes the symptom by about as much as removing the labelled fault, within the noise baseline. We treat these 7 as near-ties and keep the original key. Prose-only tickets name no tag, so for 16 of them the check is inconclusive.

Figure 3Answer key checks
Labelled part most degraded, or the only cause of the symptom Largest of several causes Close but acceptable: a near-tie (PdM within 0.03 health; RCA within the noise baseline), or a distractor ranks higher and is now accepted as an alternative Other: healthy negative, sensor-drift key, or inconclusive
Counts of tasks. For PdM, ranking is by projected component health; for RCA, by how much removing each fault changes the symptom tag.

3.3Fairness of the unsolved tasks #

22 of the 137 frontier-track tasks are unsolved by every model. Without relying on any model, we checked four things for each of them:

  • The fault is visible. Each task is run against a twin that’s identical (same seed, distractors and noise) except that the labelled fault is removed. In the window the agent can query, at least one published process tag must differ beyond the noise band (z ≥ 5 and at least 2% of the tag’s range, sustained for three windows), and the symptom tag must move where the ticket names one. Health values are visible only to the grader and never count. 22 of 22 pass.
  • The key is consistent. The reference answer passes all 22, and an “always no fault” answer passes none.
  • The answer is nameable. Every labelled component is in its station’s register in the plant documents.
  • The key is uncontested. None of the 22 is among the near-ties or accepted alternatives in §3.2: in each, the labelled part is the most degraded, or the largest or only cause of the symptom.
Figure 4The 22 unsolved frontier-track tasks, by type
Passes all four checks: fault visible, key consistent, answer nameable, key uncontested
One cell per task. All 22 pass every check above.

04Core track results #

On the core track, the four largest closed models pass 68–82% of tasks and the open-weight models pass 2–27%. Every model had one attempt per task, and intervals are 95% bootstrap intervals over tasks.3

Figure 5Pass rate on the core track (101 tasks)
Tasks passed, % with 95% bootstrap interval
Llama 4 Scout is not shown: we could not run it (see §7).
Figure 6Every model on the core track
ModelPassed95% CIMeanRCA /40PdM /40PLC /21ErrorsCostNotes
Mean includes partial credit, such as 0.5 for the right station with the wrong part. Errors are runs that ended without a gradable answer. They count as failures, so for Kimi K2 and Llama 4 Maverick the score says more about how each model works with a basic harness than about its diagnosis.

Scores drop at the medium tier, where the top four models pass 7 or 8 of 12. The hard tier on this track has only three hardware tasks, which is too few to compare models on.

Figure 7Core track hardware tasks by tier, closed models
TierSolTerraOpusSonnetLunaHaiku
Smoke 221615121467
Simple 1817141213118
Easy 19181613141111
Medium 12788765
Hard 3111213
Predictive 4443332
Negative 2221100
Tasks passed in each tier, out of the number shown. The 80 RCA and PdM tasks only; the 21 PLC tasks are not tiered. A darker cell is a higher share.

05Frontier track results #

On the 137-task frontier track, GPT-6.1 Sol and GPT-6.1 Terra pass 74% and 72%, and Claude Opus 5.5 passes 63%. We ran the six closed models on this track.

Figure 8Pass rate on the frontier track (137 tasks)
Tasks passed, % with 95% bootstrap interval

By tier, Sol leads on the hard tier (20 of 36) and stress tier (30 of 40), and Terra on extreme-medium (27 of 32). PLC repairs split the field most sharply: the four largest models pass 24–26 of 29, GPT-6 Luna 11 and Claude Haiku 5.5 6.

Figure 9Frontier track by tier and task type
ModelPassedMeanHard /36Stress /40Extreme-medium /32PLC /29RCA /75PdM /33Errors
GPT-6.1 Sol1010.812030252658170
GPT-6.1 Terra990.811828272655180
Claude Opus 5.5860.741525222446160
Claude Sonnet 5.5770.691319212440131
GPT-6 Luna590.531419151138101
Claude Haiku 5.5340.435111262260
Tiers and task types are two ways of cutting the same 137 tasks: the four tiers sum to the total, as do RCA, PdM and PLC. A darker cell is a higher share.

5.1Score and cost #

Cost is the estimated API spend for one full pass over the 137 tasks. Sol passes the most tasks, at $0.33 per task passed. Terra passes two fewer at about half that, $0.17. Opus costs the most, at $1.00 per task passed.

Figure 10Pass rate and cost, frontier track
Models sorted by pass rate. Per pass is the cost of the full run divided by the number of tasks passed.

5.2Headroom #

To see whether a higher tier of model clears the hardest tasks, we ran GPT-6 Astra on a 46-task subset: 34 frontier-track tasks that GPT-6.1 Sol, GPT-6.1 Terra and GPT-6 Luna all missed under the original answer key, plus 12 sanity tasks they pass. Astra passed all 12 sanity tasks, so its misses aren’t harness or formatting failures. Under the published scoring it solves 8 of the 34.

Because this subset was chosen from one vendor’s failures, we don’t use it to rank Astra against other models. It answers a narrower question, whether the hardest tasks stay unsolved by the highest tier of models, and so far they do. 22 frontier-track tasks are unsolved by every model, Astra included (Figure 4). Most of the hardware ones have many components wearing at once, or a symptom that shows up several stations away from its cause. The PLC ones are repairs that look right and then fail in simulation.

5.3Kinds of misses #

GPT-6.1 Sol misses 36 frontier-track tasks, and 24 of them are near misses: 13 at the wrong station, usually the one showing the symptom, and 11 at the right station with the wrong part, often a sibling component that’s also wearing. Those two kinds make up most of the misses for every model.

Figure 11Frontier-track misses by kind

We see the same mistake on real lines. Someone reads the symptom as the fault, and the repair goes to the wrong component.

06Comparing models #

Every model sees the same tasks, so the right test for a gap between two models is a paired bootstrap over those tasks. GPT-6.1 Sol and Terra are statistically indistinguishable on both tracks. Both lead Claude Opus 5.5 by 9.5–13.9 points, with intervals that exclude zero. Opus and Sonnet tie on the core track, and Opus’s 6.6-point lead on the frontier track is borderline.

Figure 12Paired differences in pass rate
Difference in pass rate, points (95% CI)
Interval excludes zeroInterval includes zero

07Disclosures #

Task selection

Model results did not drive the selection of either track. The frontier track is every eligible case in the corpus; four eligible cases were left out before any run, three because their labelled component isn’t in any register and one because it was a no-backup PLC case on a line that mounts backups.

Tasks removed, edited or rescored

  • Before the scored runs, the model-blind signal gate (§3.3) retired 13 core-track tasks whose fault wasn’t visible in agent-queryable data, and each was replaced with a same-tier task that passes (Nov 4–5; commits 7db69480, 89de716b, bdfba535). One more was retired and restored after a fix.
  • After an earlier run and a reviewer pass, three frontier-track task families were edited to fix contestable keys: slowed-ramp PLC tickets were re-timed and design ramp rates added to the control narrative, a labeler/capper task was rebalanced, a lube-oil decoy was removed from the compressor-bearing tasks, and station mode tags were fixed (Nov 5; 3e71e822, dd739cdc). Every model was rerun on the edited set, and no result here predates the edits.
  • After the scored runs and a second reviewer pass, we began accepting the 16 PdM alternatives from §3.2. This moved four tasks out of the unsolved set, from 26 to 22.

Harness and reasoning settings

HarnessRaw model, no agent scaffold; one system and one user prompt; the tools in Figure 1.
AttemptsOne per task. Temperature not set (provider default).
Budget24 investigative turns, each of which may batch several tool calls; turn 25 forces a submission.
Tool choiceMust call a tool on investigative turns. Opus and Sonnet reject forced tool choice with extended thinking, so for them only the required tool is exposed.
Empty repliesResent up to twice, then scored as an error.
ClaudeOpus 5.5 and Haiku 5.5 at medium effort (8,192-token cap per turn); Sonnet 5.5 at high (16,384).
GPTMedium reasoning effort for all three, with an 8,192-token cap per turn that includes reasoning.
Open-weightVia Amazon Bedrock at model defaults, 1,024-token cap, provider fallback disabled.
ContextNo compaction; the full transcript each turn.

Run dates and code versions

RunsDates (UTC)Commit
GPT, both tracksNov 9b75b7f7
Claude, both tracksNov 6, 00:56–01:575ceedc6b
GPT-6 Astra, headroom subsetNov 6, 01:06–01:205db72947
Open-weight, core trackNov 9, 17:51–20:125f6e73d7 (Gemma cce39f56)

Between the first and last of these commits, no task, scenario, grader or prompt changed; the differences are a refactor of the tool runner and exporter. Every frontier run was served by the requested provider, confirmed from tool-call IDs in each transcript, and every open-weight turn was confirmed to be served by the requested Bedrock model.

08Limitations #

  • The plants are simulated. Some steady signals are flatter than real sensors would read, and plant time is compressed, so slow degradations unfold faster than on a real line.
  • The plants are small. The largest scored plant has about 2,500 tags, where real plants run tens to hundreds of thousands.
  • One attempt per task. Our intervals capture which tasks were sampled, not run-to-run variation.
  • No human or scripted baseline yet, so the unsolved tasks are verified visible, not verified solvable (§3.3).
  • The frontier track covers two labs. We ran GPT-6 Astra only on the headroom subset, and did not test a Gemini model. We also did not test Claude Fable, because we couldn’t run it under zero data retention, which we require for every model API call.
  • The open-weight models ran at a 1,024-token output cap, below the frontier models’ (§7).
  • Every simulated component has telemetry, so every fault leaves a trace in the historian. Real plants have components with no sensors. We are extending the simulator and evaluation environment to include failing components without telemetry, and simulated visual inspections that agents can request at a cost.

Most of these point the same way. Real plants are larger, messier and less well documented than IndustrialBench, so we expect these scores to significantly overstate how models would perform on a real line.

09Working with IndustrialBench #

We keep the tasks and simulators private so models can’t be trained on them. If you develop models and want to see how a new one does, we’ll run it for you.

We also license the task sets and FactorySim, the simulator behind them, to teams that want to evaluate or train on industrial diagnosis directly. FactorySim generates new plants, faults and noise profiles on demand, from small cells to 150-station lines, with every task deterministically graded.

10Footnotes

  1. Private tasksWe don’t publish tasks or simulators, so they can’t be trained on and keep measuring what models can do. ↩
  2. Accepted alternativesA rank-1 answer naming the labelled part or any accepted alternative passes. The task cases themselves are unchanged. Eight of the 16 tasks are in each track. ↩
  3. Intervals95% percentile intervals from 10,000 bootstrap resamples of tasks. Differences between models use a paired bootstrap over the same tasks. ↩
  4. Data handlingEvery task is synthetic and contains no customer data. All model API calls ran under zero data retention.

Run IndustrialBench on your model.

If you’d like results for your model, access to the tasks, or a conversation about FactorySim, talk to us.

Email leor@axilon.com