ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …
Transcript of ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …
![Page 1: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/1.jpg)
UNCLAS: Dist A. Approved for public release
ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION METHOD
Yogita Pai, Michael Kokkolaras, Greg Hulbert, Panos Papalambros, Univ. of Michigan Michael K. Pozolo, US Army RDECOM-TARDEC Yan Fu, Ren-Jye Yang, Saeed Barbat, Ford Motor Company
August11,09 UNCLAS: Dist A. Approved for public release
![Page 2: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/2.jpg)
Report Documentation Page Form ApprovedOMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.
1. REPORT DATE 10 AUG 2009
2. REPORT TYPE N/A
3. DATES COVERED -
4. TITLE AND SUBTITLE Assessment of a Bayesian Model and Test Validation Method
5a. CONTRACT NUMBER
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) Yogita Pia; Michael Kokkolaras; Greg Hulbert; Panos Papalambros;Micheal K. Pozolo; Yan Fu; Ren-Jye Yang; Saeed Barbat
5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) University of Michigan US Army RDECOM-TARDEC 6501 E 11 MileRd Warren, MI 48397-5000 Ford Motor Company
8. PERFORMING ORGANIZATION REPORT NUMBER 20152
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) TACOM/TARDEC
11. SPONSOR/MONITOR’S REPORT NUMBER(S) 20152
12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release, distribution unlimited
13. SUPPLEMENTARY NOTES Presented at NDIAs Ground Vehicle Systems Engineering and Technology Symposium (GVSETS), 17 22August 2009,Troy, Michigan, USA, The original document contains color images.
14. ABSTRACT
15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: 17. LIMITATIONOF ABSTRACT
SAR
18. NUMBEROF PAGES
16
19a. NAME OFRESPONSIBLE PERSON
a. REPORT unclassified
b. ABSTRACT unclassified
c. THIS PAGE unclassified
Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18
![Page 3: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/3.jpg)
UNCLAS: Dist A. Approved for public release
Need for Validation Methodology
• Systematic method for validation necessary – Modeling and Simulation – Laboratory test – Validation of designs
• Reduce need for Subject Matter Experts • Reduce number of field tests • Assess cost of validation and certification • Use existing data mines of tests, M&S, and designs
August11,09 2
![Page 4: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/4.jpg)
UNCLAS: Dist A. Approved for public release
VV&A of Army M&S
August11,09 3
Dept. of Army pamphlet 5-11: VV&A of Army M&S
![Page 5: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/5.jpg)
UNCLAS: Dist A. Approved for public release
Bayesian Confidence Method
• Model validation under uncertainty – Uncertainty in field data – Uncertainty in model data – Validation of designs
• Multiple, incompatible data channels can be evaluated • Interval-based method provide more robust evaluation
August11,09 4
![Page 6: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/6.jpg)
UNCLAS: Dist A. Approved for public release
Bayesian Confidence Method
August11,09 5
Physical test CAE model
Multivariate CAE results Multivariate test data
Normalization
Reduced CAE results Reduced test data
BF Confidence
Normalized CAE results Normalized test data
Interval-based hypothesis testing and Bayes factor (BF) calculation
Probabilistic Principal Component Analysis
Jiang, Fu, Yang, Barbat, Li, Zhan, SAE 2009 World Congress
![Page 7: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/7.jpg)
UNCLAS: Dist A. Approved for public release
Comparison of Model and Test
• Model 1, Course 1
August11,09 6
Blue = model 1 Red = test
![Page 8: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/8.jpg)
UNCLAS: Dist A. Approved for public release
Comparison of Model and Test
• Model 2, Course 1
August11,09 7
Blue = model 2 Red = test
![Page 9: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/9.jpg)
UNCLAS: Dist A. Approved for public release
Data Reconstruction
• Course 1 • First principal component, 62% total variability captured
August11,09 8
![Page 10: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/10.jpg)
UNCLAS: Dist A. Approved for public release
Data Reconstruction
• Course 1 • First 2 principal components, 86% total variability
captured
August11,09 9
![Page 11: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/11.jpg)
UNCLAS: Dist A. Approved for public release
Data Reconstruction
• Course 1 • First 3 principal components, 99.9% total variability
captured
August11,09 10
![Page 12: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/12.jpg)
UNCLAS: Dist A. Approved for public release
Bayesian Hypothesis Testing
August11,09 11
Reduced test data, xt with variability Σt
Reduced CAE results, xc with variability Σc
Difference d = xc – xt sample statistics:
Multivariate hypothesis test: Assuming prior d ~ N(µ,∑) Ho:|µ| ≤ ε (accept) versus Ha:|µ| > ε (reject)
Bayesian factor calculation BM = P(d|Ho) / P(d|Ha) (likelihood ratio)
BF confidence quantification к = BM / (1+BM) ×100
![Page 13: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/13.jpg)
UNCLAS: Dist A. Approved for public release
Calibration Parameter Selection
August11,09 12
![Page 14: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/14.jpg)
UNCLAS: Dist A. Approved for public release
Calibration Parameter Selection
August11,09 13
p = # of principal components
% of variability captured
![Page 15: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/15.jpg)
UNCLAS: Dist A. Approved for public release
Effect of Principal Components
• Course 1
August11,09 14
Blue = model 1 Red = model 2
![Page 16: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/16.jpg)
UNCLAS: Dist A. Approved for public release
• Course 2
August11,09 15
Blue = model 1 Black = model 2
Effect of Principal Components
![Page 17: ASSESSMENT OF A BAYESIAN MODEL AND TEST VALIDATION …](https://reader030.fdocuments.net/reader030/viewer/2022032609/6238b8333c25e31bab2ace48/html5/thumbnails/17.jpg)
UNCLAS: Dist A. Approved for public release
Closing Remarks
• Bayesian framework promising for validation – Incorporates statistics of field data – Incorporates statistics of M&S – Enables systematic evaluation of data variability
• Systematic method for accepting M&S • Systematic method for comparing M&S • Further refinement needed for calibration and sensitivity • Further research required for accreditation use
August11,09 16