IndustrialBench
Measuring whether language models can diagnose equipment failures and repair controller logic on simulated factory lines.
01Overview #
Axilon builds software that helps maintenance and controls engineers find the cause of equipment failures. We built IndustrialBench to find out where language models still fall short at that work. It has 238 tasks on simulated plants. In each one, an agent has to find the component behind a fault, predict which parts will fail next, or fix faulty PLC logic, working from the historian data, plant documents and controller programs an engineer would have.
A benchmark’s hardest tasks only mean something if they’re fair, so we cover how the tasks are built and how we checked the grader before getting to how 13 models did.
The tasks themselves stay private. We publish the results and the method, and run evaluations on request.1
02The benchmark #
IndustrialBench has two tracks, both drawn from a corpus of 582 generated cases.
The core track (101 tasks) has 40 root-cause analysis (RCA) tasks and 40 predictive-maintenance (PdM) tasks across difficulty tiers from smoke tests to hard, plus 21 PLC tasks: logic defects with no backup to compare against, operating events, lines with 55 controllers, and negatives where nothing is wrong.
The frontier track (137 tasks) is every case in the corpus that meets its eligibility rules and doesn’t overlap the core track. Its tasks put the true cause among many simultaneous degradations (hard, 36), behind vague prose-only operator complaints (stress, 40), or next to more-degraded parts on the same equipment (extreme-medium, 32), along with harder PLC repairs (29).
Each task gives the agent an optional operator ticket, a historian of tag values, and the plant’s project share. Step through an illustrative task below.
- Reported
- t = 3,600 s
- Tag
FIL-301.FillVolume· station FIL-301, bottle filler- Observed
- 487 mL against a 500 mL baseline
- Operator note
- “Underfills started mid-shift. Capper and labeler look normal.”
The symptom is at the filler. The ticket is optional: on stress tasks it’s replaced by a vague prose complaint, and on PdM tasks there’s no fault yet to report.
| Tool | Description |
|---|---|
search_tags | Find historian tags by station, name or description, such as every FIL-301.* tag and the upstream PMP-101.* tags. |
get_tag_samples | Pull raw time-series samples for chosen tags and time windows. |
list_dir read_file grep | Browse the plant’s project share. |
submit_final_guess | Submit the answer. The agent has 24 investigative turns; turn 25 forces a submission. |
project-share/ ├── docs/ │ ├── control-narrative.pdf │ ├── equipment-list.csv │ ├── tag-dictionary.csv │ ├── alarms-and-interlocks.csv │ ├── operating-procedures/ │ ├── maintenance-guides/ # per station, each with a component register │ └── work-order-history.csv └── plc/ ├── deployed/*.L5X # Rockwell ladder logic, one per controller └── backups/*.L5X # last-known-good, on some lines
{ "station_id": "PMP-101", "component": "impeller" }
The cause is upstream of the symptom: a worn impeller on the feed pump. Hardware answers are graded on exact station and component, which must come from the station’s register. PdM tasks ask for parts in order of urgency and are graded on the first one.
03Validating the grader #
All scoring is deterministic, and no model grades another model. Hardware answers are compared with a known key. PLC repairs are replayed in a twin of the plant. Before reporting any scores, we checked three things: that the grader responds correctly to answers we know are right or wrong, that each answer key is the only defensible answer, and that the tasks no model solves are visible, consistently keyed and uncontested.
3.1Testing the PLC check on known inputs #
The PLC check loads the model’s corrected program into a twin plant and drives it through a fixed probe: writes above every ceiling and below every floor, a stop and start of every interlock, and a restore to defaults. It compares every setpoint trajectory with the healthy program’s and passes when the worst error is within 2% of the setpoint’s span. Because the probe pushes every setpoint to and past its limits, an edit that only hides the symptom, such as bypassing an interlock, can’t pass.
We graded altered versions of the reference repair for all 47 PLC defects, 18 in the core track and 29 in the frontier track.
| Submitted repair | Should | Result |
|---|---|---|
| Reference repair | pass | 47 / 47 pass |
| No edit submitted | fail | 47 / 47 fail |
Corrected constant retyped (68 → 68.0) | pass | 41 / 41 pass |
Interlock fix with the contact inverted (XIC ↔ XIO) | fail | 6 / 6 fail |
To find the check’s tolerance, we then scaled the corrected constant in each of the 41 constant defects. Select a factor to see how the check responds.
3.2Answer key uniqueness #
A task is unfair if a second answer is as defensible as the key. We tested both task types.
PdM tasks ask which parts will fail, most urgent first. We compared the labelled part’s projected end-of-run health with every other degrading part in the scenario, and confirmed it against simulated health where we ran the plant. In 17 tasks a distractor ends more degraded than the labelled part. We also checked whether the labelled fault is the one with larger effects elsewhere in the plant: in 13 of those 17, a distractor is. So on those tasks, naming the more-degraded part first is defensible, and we now accept it. After merging one task where the distractor is the same part, 16 tasks accept an alternative, applied to every model equally.2
For the 94 RCA tasks with more than one fault, we reran the scenario once per fault with that fault removed and measured how much the ticket’s symptom tag changed. In 6 structured tasks and 1 prose task, removing a distractor changes the symptom by about as much as removing the labelled fault, within the noise baseline. We treat these 7 as near-ties and keep the original key. Prose-only tickets name no tag, so for 16 of them the check is inconclusive.
3.3Fairness of the unsolved tasks #
22 of the 137 frontier-track tasks are unsolved by every model. Without relying on any model, we checked four things for each of them:
- The fault is visible. Each task is run against a twin that’s identical (same seed, distractors and noise) except that the labelled fault is removed. In the window the agent can query, at least one published process tag must differ beyond the noise band (z ≥ 5 and at least 2% of the tag’s range, sustained for three windows), and the symptom tag must move where the ticket names one. Health values are visible only to the grader and never count. 22 of 22 pass.
- The key is consistent. The reference answer passes all 22, and an “always no fault” answer passes none.
- The answer is nameable. Every labelled component is in its station’s register in the plant documents.
- The key is uncontested. None of the 22 is among the near-ties or accepted alternatives in §3.2: in each, the labelled part is the most degraded, or the largest or only cause of the symptom.
04Core track results #
On the core track, the four largest closed models pass 68–82% of tasks and the open-weight models pass 2–27%. Every model had one attempt per task, and intervals are 95% bootstrap intervals over tasks.3
| Model | Passed | 95% CI | Mean | RCA /40 | PdM /40 | PLC /21 | Errors | Cost | Notes |
|---|
Scores drop at the medium tier, where the top four models pass 7 or 8 of 12. The hard tier on this track has only three hardware tasks, which is too few to compare models on.
| Tier | Sol | Terra | Opus | Sonnet | Luna | Haiku |
|---|---|---|---|---|---|---|
| Smoke 22 | 16 | 15 | 12 | 14 | 6 | 7 |
| Simple 18 | 17 | 14 | 12 | 13 | 11 | 8 |
| Easy 19 | 18 | 16 | 13 | 14 | 11 | 11 |
| Medium 12 | 7 | 8 | 8 | 7 | 6 | 5 |
| Hard 3 | 1 | 1 | 1 | 2 | 1 | 3 |
| Predictive 4 | 4 | 4 | 3 | 3 | 3 | 2 |
| Negative 2 | 2 | 2 | 1 | 1 | 0 | 0 |
05Frontier track results #
On the 137-task frontier track, GPT-6.1 Sol and GPT-6.1 Terra pass 74% and 72%, and Claude Opus 5.5 passes 63%. We ran the six closed models on this track.
By tier, Sol leads on the hard tier (20 of 36) and stress tier (30 of 40), and Terra on extreme-medium (27 of 32). PLC repairs split the field most sharply: the four largest models pass 24–26 of 29, GPT-6 Luna 11 and Claude Haiku 5.5 6.
| Model | Passed | Mean | Hard /36 | Stress /40 | Extreme-medium /32 | PLC /29 | RCA /75 | PdM /33 | Errors |
|---|---|---|---|---|---|---|---|---|---|
| GPT-6.1 Sol | 101 | 0.81 | 20 | 30 | 25 | 26 | 58 | 17 | 0 |
| GPT-6.1 Terra | 99 | 0.81 | 18 | 28 | 27 | 26 | 55 | 18 | 0 |
| Claude Opus 5.5 | 86 | 0.74 | 15 | 25 | 22 | 24 | 46 | 16 | 0 |
| Claude Sonnet 5.5 | 77 | 0.69 | 13 | 19 | 21 | 24 | 40 | 13 | 1 |
| GPT-6 Luna | 59 | 0.53 | 14 | 19 | 15 | 11 | 38 | 10 | 1 |
| Claude Haiku 5.5 | 34 | 0.43 | 5 | 11 | 12 | 6 | 22 | 6 | 0 |
5.1Score and cost #
Cost is the estimated API spend for one full pass over the 137 tasks. Sol passes the most tasks, at $0.33 per task passed. Terra passes two fewer at about half that, $0.17. Opus costs the most, at $1.00 per task passed.
5.2Headroom #
To see whether a higher tier of model clears the hardest tasks, we ran GPT-6 Astra on a 46-task subset: 34 frontier-track tasks that GPT-6.1 Sol, GPT-6.1 Terra and GPT-6 Luna all missed under the original answer key, plus 12 sanity tasks they pass. Astra passed all 12 sanity tasks, so its misses aren’t harness or formatting failures. Under the published scoring it solves 8 of the 34.
Because this subset was chosen from one vendor’s failures, we don’t use it to rank Astra against other models. It answers a narrower question, whether the hardest tasks stay unsolved by the highest tier of models, and so far they do. 22 frontier-track tasks are unsolved by every model, Astra included (Figure 4). Most of the hardware ones have many components wearing at once, or a symptom that shows up several stations away from its cause. The PLC ones are repairs that look right and then fail in simulation.
5.3Kinds of misses #
GPT-6.1 Sol misses 36 frontier-track tasks, and 24 of them are near misses: 13 at the wrong station, usually the one showing the symptom, and 11 at the right station with the wrong part, often a sibling component that’s also wearing. Those two kinds make up most of the misses for every model.
We see the same mistake on real lines. Someone reads the symptom as the fault, and the repair goes to the wrong component.
06Comparing models #
Every model sees the same tasks, so the right test for a gap between two models is a paired bootstrap over those tasks. GPT-6.1 Sol and Terra are statistically indistinguishable on both tracks. Both lead Claude Opus 5.5 by 9.5–13.9 points, with intervals that exclude zero. Opus and Sonnet tie on the core track, and Opus’s 6.6-point lead on the frontier track is borderline.
07Disclosures #
Task selection
Model results did not drive the selection of either track. The frontier track is every eligible case in the corpus; four eligible cases were left out before any run, three because their labelled component isn’t in any register and one because it was a no-backup PLC case on a line that mounts backups.
Tasks removed, edited or rescored
- Before the scored runs, the model-blind signal gate (§3.3) retired 13 core-track tasks whose fault wasn’t visible in agent-queryable data, and each was replaced with a same-tier task that passes (Nov 4–5; commits
7db69480,89de716b,bdfba535). One more was retired and restored after a fix. - After an earlier run and a reviewer pass, three frontier-track task families were edited to fix contestable keys: slowed-ramp PLC tickets were re-timed and design ramp rates added to the control narrative, a labeler/capper task was rebalanced, a lube-oil decoy was removed from the compressor-bearing tasks, and station mode tags were fixed (Nov 5;
3e71e822,dd739cdc). Every model was rerun on the edited set, and no result here predates the edits. - After the scored runs and a second reviewer pass, we began accepting the 16 PdM alternatives from §3.2. This moved four tasks out of the unsolved set, from 26 to 22.
Harness and reasoning settings
| Harness | Raw model, no agent scaffold; one system and one user prompt; the tools in Figure 1. |
|---|---|
| Attempts | One per task. Temperature not set (provider default). |
| Budget | 24 investigative turns, each of which may batch several tool calls; turn 25 forces a submission. |
| Tool choice | Must call a tool on investigative turns. Opus and Sonnet reject forced tool choice with extended thinking, so for them only the required tool is exposed. |
| Empty replies | Resent up to twice, then scored as an error. |
| Claude | Opus 5.5 and Haiku 5.5 at medium effort (8,192-token cap per turn); Sonnet 5.5 at high (16,384). |
| GPT | Medium reasoning effort for all three, with an 8,192-token cap per turn that includes reasoning. |
| Open-weight | Via Amazon Bedrock at model defaults, 1,024-token cap, provider fallback disabled. |
| Context | No compaction; the full transcript each turn. |
Run dates and code versions
| Runs | Dates (UTC) | Commit |
|---|---|---|
| GPT, both tracks | Nov 9 | b75b7f7 |
| Claude, both tracks | Nov 6, 00:56–01:57 | 5ceedc6b |
| GPT-6 Astra, headroom subset | Nov 6, 01:06–01:20 | 5db72947 |
| Open-weight, core track | Nov 9, 17:51–20:12 | 5f6e73d7 (Gemma cce39f56) |
Between the first and last of these commits, no task, scenario, grader or prompt changed; the differences are a refactor of the tool runner and exporter. Every frontier run was served by the requested provider, confirmed from tool-call IDs in each transcript, and every open-weight turn was confirmed to be served by the requested Bedrock model.
08Limitations #
- The plants are simulated. Some steady signals are flatter than real sensors would read, and plant time is compressed, so slow degradations unfold faster than on a real line.
- The plants are small. The largest scored plant has about 2,500 tags, where real plants run tens to hundreds of thousands.
- One attempt per task. Our intervals capture which tasks were sampled, not run-to-run variation.
- No human or scripted baseline yet, so the unsolved tasks are verified visible, not verified solvable (§3.3).
- The frontier track covers two labs. We ran GPT-6 Astra only on the headroom subset, and did not test a Gemini model. We also did not test Claude Fable, because we couldn’t run it under zero data retention, which we require for every model API call.
- The open-weight models ran at a 1,024-token output cap, below the frontier models’ (§7).
- Every simulated component has telemetry, so every fault leaves a trace in the historian. Real plants have components with no sensors. We are extending the simulator and evaluation environment to include failing components without telemetry, and simulated visual inspections that agents can request at a cost.
Most of these point the same way. Real plants are larger, messier and less well documented than IndustrialBench, so we expect these scores to significantly overstate how models would perform on a real line.
09Working with IndustrialBench #
We keep the tasks and simulators private so models can’t be trained on them. If you develop models and want to see how a new one does, we’ll run it for you.
We also license the task sets and FactorySim, the simulator behind them, to teams that want to evaluate or train on industrial diagnosis directly. FactorySim generates new plants, faults and noise profiles on demand, from small cells to 150-station lines, with every task deterministically graded.
10Footnotes
- Private tasksWe don’t publish tasks or simulators, so they can’t be trained on and keep measuring what models can do. ↩
- Accepted alternativesA rank-1 answer naming the labelled part or any accepted alternative passes. The task cases themselves are unchanged. Eight of the 16 tasks are in each track. ↩
- Intervals95% percentile intervals from 10,000 bootstrap resamples of tasks. Differences between models use a paired bootstrap over the same tasks. ↩
- Data handlingEvery task is synthetic and contains no customer data. All model API calls ran under zero data retention.
Run IndustrialBench on your model.
If you’d like results for your model, access to the tasks, or a conversation about FactorySim, talk to us.
Email leor@axilon.com