Uncertainty in Automation: Anomaly Detection in Event-Based Systems Dawn Tilbury Linday Allen (PhD)...
-
Upload
samantha-fowler -
Category
Documents
-
view
216 -
download
1
Transcript of Uncertainty in Automation: Anomaly Detection in Event-Based Systems Dawn Tilbury Linday Allen (PhD)...
Uncertainty in Automation:Anomaly Detection in Event-Based Systems
Dawn TilburyLinday Allen (PhD) and John Broderick
University of Michigan
2
Outline
• Example problem: inconsistent logical behavior
• Developed solution: Anomaly detection– Model generation using observed data– Performance assessment of models using known
“good” and “bad” behavior– On-line anomaly detection
• Industrial Application– Academic assumptions meet industry realities– Resolution and results
• Lessons learned
3
Example from our testbed
s1
Part1
PartReadyRelease Pallet
LoadPart1Part1
PartReady
Correct, typical behavior Incorrect behavior
• No model of entire system’s correct behavior• Manual inspection required to find this anomaly
• Laborious, offline
ResponseOPC Tag ResponseOPC Tag
4
Approach: Anomaly detection using model generation
• Goal: diagnosis of system level event-based faults in mfg systems without use of pre-existing formal model
• Method: - Generate models based on training data- Detect anomalies on-line by comparing traces
to models- Advise the operator when anomaly occurs
PlantPlant
Logic Control System
FaultdiagnosisAnomalydetection
Operator
ControllerController
All Events
Events
Op. Feedback
Anomalies
Knowns & Unknowns
• Known: – Resources in
system• Robots• CNC machines• Pallets
• Measurable:– OPC tag changes
• Communication events between controllers
• Unknown: – Formal model of the
systemCould be constructed but is time-consuming and error prone
– Logic control codeWritten by different people at different times in different languages
– Correct event orderMany different orders may be acceptable
6
Solution approach
Model Generation
adbe...
efad...
abfc...
ok
ok
ok
Resource Info
eafd...
Performance Assessment
acea...
ok
not
22316
Fault Detection
abfe...
okabfe...
abfe...
not
OR
Given resource information and
strings of “ok” events
Create a set of models that can generate these
strings
Given some “ok” and “not ok” strings, compute the
performance of each model
Given a new string, determine whether the
models accept it (weight by model performance)
If not, where is the anomaly
Anomaly Detection Method
• Inputs:– Streams of events from system in operation– Resource information, including mapping of
events to resources
• Outputs:– Set of models that represent the system
behavior– Model performance on training data
• On-line detection:– Score for each string – anomalous or not
7
Machining Cell: Physical Set-Up
• Problem: G2 will have raw parts and at least one CNC available, but G2 incorrectly waits
• Resources: – Gantries, CNCs, buffer at hand-off
• Events: PLC data recorded via Ford data collection system
G1 G2 M1 M62 3
4
1
Entry Hand-off
Reject
Exit
Data collection set-up
• Data from each machine & gantry– Bits include: Cycle
End, Good/Bad Cycle, Wait Aux, Blocked, and Starved
– PLC message generated each time particular bit changes occur
• Approx. 11,000 parts worth of data
9
IT SystemIT System
PLC
DrivingLogic
DrivingLogic
FunctionBlock
FunctionBlock
PLC
Driving logic
FunctionBlock
FunctionBlock
DrivingLogic
DrivingLogic
(270,000 PLC messages)
Identified Inconsistencies
• What we thought we would get: – Well-defined strings
of events
– Events that acquire/release resources recorded
– Unique mapping of PLC bits to events
– Many strings, starting from the initial state, labeled as “good” or “bad
• What we got: – Not every event
triggers a message multi-bit change (order is uncertain)
– Not all resource events captured in data collection
– Some bits used for multiple purposes
– One huge log file with no defined “beginning”
Resolution of Inconsistencies
Academic Assumptions
Industry Realities Resolution
1 Resource events available
Some events filtered in data collection
I: Logic changed
2 String of ordered events
Multiple bit changes per message possible
A: Heuristic decision algorithm
3 Consistent bit-meaning mapping
Inconsistent bit-meaning mapping
I, A: Logic changed, pre-process data
4 Event streams start in initial state
System runs continuously
A: Nec. condition to create stream
5 Separate, labeled streams
Continuous, unlabeled stream
A: Splitting, labeling algorithm
Lessons learned
• Sometimes you can adapt/improve your method to handle given uncertainties– Multiple models when system model unknown– Multiple bit changes uncertain event order– Initial state unknown– Advise operator instead of closing the loop
• Sometimes you have to decrease the uncertainty by improving the system– Consistent bit/event mapping– Unobservable events for acquiring resources
13