Systematic Analysis, Testing, and Improvement of CPSMLH. Niederreiter, “Random Number Generation...
Transcript of Systematic Analysis, Testing, and Improvement of CPSMLH. Niederreiter, “Random Number Generation...
Systematic Analysis, Testing, and Improvement of CPSML
Tommaso Dreossi
1
Joint work with:Daniel Fremont, Shromona Ghosh, Xiangyu Yue, Alexandre Donze
Kurt Keutzer, Alberto Sangiovanni-Vincentelli, Sanjit A. Seshia
UC Berkeley
Cyber-Physical Systems (CPS)Integration of computation with physical processes
Building systems Factory automation Automotive
Smart citiesPower generation Avionics 2
Cyber-Physical Systems + ML/AI (CPSML)Growing use of Machine Learning/AI in CPS
20152010200520001995
200
400
StartupsdevelopingAIsystems
600
Year
Activestartups
2015201020052000Year
5K
10K
15K
Pape
rs
AnnuallypublishedAIpapers
Source:AIindex.org,Scopus,Crunchbase,VentureSource,SandHillsEconometrics 3
Cyber-Physical Systems + ML/AI (CPSML)Growing use of Machine Learning/AI in CPS
Many safety-critical applications
Source:DMVCA
0 5 10 15 20
Recklesslybehavingroaduser
Incorrectbehaviorprediction
Softwarediscrepancy
Hardwarediscrepancy
Perceptiondiscrepancy
Unwantedmaneuver
Waymo disengagementreportCalifornia,2017
Disengagement4
Formal methods approach
𝑆 ∥ 𝐸 ⊨ 𝜑SystemS
EnvironmentESpecification𝜑
Yes(proof)
No(counterexample)
Challenges for Verified AI
5
Challenges for Verified AIFormal methods approach
𝑆 ∥ 𝐸 ⊨ 𝜑SystemS
EnvironmentESpecification𝜑
Yes(proof)
No(counterexample)
• Complexmodels• E.g.,AlexNet,60Mparameters,650Kneurons)
• Largeinputspaces• E.g.,KITTI images:256^(1392x512x3)
NeednewmethodsforAbstraction andModularReasoning6
Challenges for Verified AIFormal methods approach
𝑆 ∥ 𝐸 ⊨ 𝜑SystemS
EnvironmentESpecification𝜑
Yes(proof)
No(counterexample)
• Interactionwithcomplexenvironments/agents
Needforrepresentingenvironmentscenarios7
Challenges for Verified AIFormal methods approach
𝑆 ∥ 𝐸 ⊨ 𝜑SystemS
EnvironmentESpecification𝜑
Yes(proof)
No(counterexample)
• Howdoyouformalizeperceptiontasks?
Needfornewspecificationformalisms8
Challenges for Verified AIFormal methods approach
𝑆 ∥ 𝐸 ⊨ 𝜑SystemS
EnvironmentESpecification𝜑
Yes(proof)
No(counterexample)
Ourapproach:• System:
• CompositionalanalysisofCPS-ML• AbstractionofMLmodulesinputspace
• Environment• Scenic– Scenariodescriptionlanguage
• Specification• System-levelspecifications 9
Outline
1. RunningCPSML example– Automaticemergencybrakingsystem2. Specification
• System- vsModule-levelspecification3. System
• Compositionalfalsification• MLinputabstraction• Counterexample-guidedaugmentation
4. Environment• Scenic:Scenariodescriptionlanguage
5. Conclusion
10
CPSML ExampleAutomatic Emergency Braking System (AEBS)
Environment
ControllerPlant
ML perception
• Goal: brake when an obstacle is near • Challenges:
• How to explore distance/velocity?
• How to analyze images?
• How to combine distance, velocity, images?
distance, velocity
11
SpecificationSystem- vs Module-level Specification
Environment
ControllerPlant
ML perception
distance, velocity
• Goal: brake when an obstacle is near• Specifications:
• “Never collide” (distance > 0)• “Correctly detect obstacles” 12
Outline
1. RunningCPSML example– Automaticemergencybrakingsystem2. Specification
• System- vsModule-levelspecification3. System
• Compositionalfalsification• MLinputabstraction• Counterexample-guidedaugmentation
4. Environment• Scenic:Scenariodescriptionlanguage
5. Conclusion
13
CPSML input space intractable• Idea: focus on meaningful CPS+ML input
combinations• Intuition: “If car is far, misclassification won’t
affect our system”
Strategy:1. Analyze CPS gathering info on ML role2. Use collected info to target ML3. Compose CPS + ML narrowed input spaces4. Perform targeted falsification
Compositional Falsification
Environment
ControllerPlant
ML perception
14Dreossiet.al,CompositionalFalsificationofCyber-PhysicalSystemswithMachineLearningComponents,NFM 2017
Compositional Falsification• Identifying regions of interest for AEBS• Perform optimistic/pessimistic analyses of NN
MLcorrect MLwrong Potentiallyunsaferegion(dependingonML)
15Dreossiet.al,CompositionalFalsificationofCyber-PhysicalSystemswithMachineLearningComponents,NFM 2017
ML Analyzer
• How analyze ML feature space?• E.g., image classifier: a lot of pictures to analyze
• Idea: Focus on semantic alterations
✕ ✓✓
Plausiblealterations16Dreossiet.al,SystematicTestingofConvolutionalNeuralNetworksforAutonomousDriving,RMLW 2017
brightness car z-pos
car x-posModification space
Modificationspace
✓
Neural network𝑦 ∈ {𝑐𝑎𝑟, ¬𝑐𝑎𝑟}
✓ ✕
✕
✕
✓
✓
✓
✓
✕
ML AnalyzerSystematically analyze modifications of interest
Picture spaceSystematicsampling
17
ML AnalyzerSampling methods
H.Niederreiter,“RandomNumberGenerationandQuasi-Monte-CarloMethods”,1992R.Y.Rubinstein etal.,“TheCross-EntropyMethod,AUnifiedApproachtoCombinatorialOptimization,Monte-CarloSimulation,andMachineLearning”,200418
Method Sampling speed Diversity Counterexamplefinding
Uniform random ✓ ✕ ✕
Uniform random + distance constraint ✕ − ✕
Low-discrepancy ✕ ✓ −
Cross entropy ✕ ✕ ✓
Sample Results
Inception-v3 Neural Network(pre-trained on ImageNet using TensorFlow)
This misclassificationmay not be of concern But this one
is a real hazard
Corner caseMisclassification
cluster
AEBS
19
Sample Results
Example of counterexamples
squeezeDet(trained on synthetic images)
Blind spot
squeezeDet
20
• Whattodowiththegeneratedcounterexamples?1. Analyzethemandprovideexplanations(errortables)2. Augmenttrainingsets
Misclassifications
Id Carcolor Background Orientation
1 Red Countryside Front
2 Orange Forest Back
3 White Forest Front
4 Green Forest Back
Counterexample-guided augmentation
Train Test Test Aug
Error table
Counterexamples 21Dreossiet.al,Counterexample-GuidedDataAugmentation,IJCAI2018
Sampler Imagegenerator Modelf Counter
example?
Yes
No
𝕄 m x ŷ
1. Growaugmentationset 𝔸2. Updateerrortable
𝔸 largeenough?
No
Yes
Start
End
Id Carcolor Background Orientation
1 Red Countryside Front
2 Orange Forest Back
3 White Forest Front
4 Green Forest Back
22
FindcounterexamplesandaugmenttrainingsetCounterexample-guided augmentation
Dreossiet.al,Counterexample-GuidedDataAugmentation,IJCAI2018
AugmentationComparisonAugmentation
Train - 1.5k Test - 0.75k
Counterexamples
Samplingmethodscomparison
Model Precision Recall t (sec)Original .61 .75Standard augmentation .69 .80Uniform random .76 .87 ~30Constrain .75 .86 ~92Low-discrepancy .79 .87 ~55Cross-entropy .78 .78 ~70
Test - 0.75k Aug - 0.75k
23
Outline
1. RunningCPSML example– Automaticemergencybrakingsystem2. Specification
• System- vsModule-levelspecification3. System
• Compositionalfalsification• MLinputabstraction• Counterexample-guidedaugmentation
4. Environment• Scenic:Scenariodescriptionlanguage
5. Conclusion
24
Environment Description
Problem• Largeandunstructuredinputspace• Generatemeaningfulscenes(fortestingortraining)
Car Model Car Location Car Orientation
Number of Cars Reference Scene Background
Car Color Weather Time of Day
Idea: Use simulators to model environment (e.g., GTAV)
25
• Scenic: probabilistic programming language defining distributions over scenes
• Example: a badly parked car
ScenicA Scenario Description Language
26Fremontet.al,Scenic:Language-BasedSceneGeneration,underreview
Scenic ApplicationsTesting
Exploring the behavior of the system under different conditions:
Bright and clear weather Dark and rainy weather
27
Scenic ApplicationsTraining
Generate hard cases, e.g., one car partially occluding another:
28
Scenic ApplicationsReasoning
29
Scenic ApplicationsReasoning
30
Scenic ApplicationsReasoning
Scenic makes it easy to generalize along different dimensions:
Add noise Change car model Change global position
31
Conclusion
32
Futurework• Mixreal/syntheticdata• Domainadaptation/randomization• Morecomplexdata:lidar,radar,etc.
Summary• Frameworkforsystem-levelcounterexamples• CNNanalyzer(simulationbased)• Counter-exampleguidedaugmentation• Scenic:Scenariodescriptionlanguage