Overview
A large industrial processing facility, representing approximately $1B in annual output, undertook a project to better understand unexplained downtime, inconsistent throughput, and recurring equipment issues. This engagement revealed that the most significant sources of operational loss were ICS automation issues, not cyberattacks.
Automation incidents are control system failures where the electronic communication from a computer or digital device is involved. These issues include out-of-calibration, inoperable sensors, misconfigured valves, and operator-interface blind spots. ACS data shows that across all critical infrastructure sectors, these incidents account for far more deaths, damage, and financial loss than cyberattacks, yet they are rarely identified or addressed systematically.
This project demonstrated that automation incidents can produce major operational losses, including an identified 3% degradation hit in net productivity at this billion-dollar facility.
Background: Hidden Weaknesses in Legacy Instrumentation
The facility relied on legacy process sensors and Windows-based HMIs, technologies common across critical infrastructure. As documented in industry research, these devices have known limitations:
- Many field instruments lack authentication, making them vulnerable to accidental misconfiguration, malicious manipulation, or spoofed sensor signals.
- Sensors frequently operate out of calibration or with incorrect zero/span settings, due to routine maintenance errors or sensor drift.
- HMIs refresh too slowly to detect fast-moving anomalies, meaning operators cannot see short-duration disruptions or high-frequency vibration patterns.
- Network-based monitoring cannot detect issues originating before data is converted to Ethernet packets.
These weaknesses create fertile ground for automation incidents: failures that look like "equipment problems" but are actually instrumentation and field device problems.
The Project: Raw Sensor Monitoring + Machine Learning
The facility deployed a monitoring system that tapped directly into raw, unfiltered electrical sensor signals, before they were converted into network packets or displayed on the HMI. Machine-learning models were trained to recognize normal operating behavior across pumps, valves, and process sensors.
During the learning period, the system immediately began flagging issues that had not been visible to operators:
- Sensors inoperable
- Sensors not calibrated
- Improperly ranged instruments
- Controller tuning errors
- Short-duration disruptions (<1 minute)
- Pressure drops indicating equipment degradation
- Air addition and other process anomalies missed by traditional logs
These findings aligned with broader industry observations that sensor drift, miscalibration, and maintenance mistakes are not commonly identified when monitored through traditional OT monitoring alone.
Operational Impact
Within hours of going live, the monitoring system began to identify multiple previously undetected issues affecting throughput, including sensor wiring problems, calibration issues, controller tuning errors, and pump-related disruptions.
The analysis found that these automation-incident drivers accounted for approximately 3% lost production opportunity at this facility producing around $1B annually. This identified opportunity represented tens of millions of dollars in potential additional output, not through capital expansion, but by addressing avoidable automation-incident losses. Field-related issues, which are typically instrument failure or improper ranging of scale, could also reduce nuisance alarms by 50%.
Cybersecurity Impact
Although the project focused on operations, it also strengthened cybersecurity:
- The raw-signal monitoring system detected spoofed HMI readings during a simulated man-in-the-middle attack, similar to Stuxnet.
- Because the system operated offline from IT/OT networks, it could continue functioning even when the Windows-based SCADA system experienced outages.
- The approach provided a physics-based validation layer, ensuring that network-level cybersecurity tools were not protecting untrusted or incorrect data.
This reinforced a key insight: automation-incident detection improves cybersecurity by ensuring the integrity of the physical signals themselves.
Key Takeaways
- Confirmation once again that most industrial harm comes from automation incidents, not cyberattacks. Joe's dataset of 30,000+ deaths and $100B+ in damage confirms this industry-wide reality.
- Legacy sensors are a major blind spot. They are often misconfigured or even offline, drift over time and lack authentication, creating silent failures.
- Machine learning on raw sensor data exposes issues operators cannot see. HMIs and historians miss fast anomalies and instrument drift.
- Identifying automation incidents revealed substantial financial upside. The project found a 3% production opportunity at this $1B facility, showing that these initiatives can uncover value, not just prevent loss.
- Operational reliability and cybersecurity are intertwined. Ensuring sensor integrity improves plant performance, maintenance, and cyber resilience.