Approaches to Large, Unstructured, Complex Problems The ... · •The scientific method is a...
Transcript of Approaches to Large, Unstructured, Complex Problems The ... · •The scientific method is a...
Approaches to Large, Unstructured, Complex Problems
The Big PictureGeoff Vining
Professor of Statistics
Virginia TechTÍTULO
Outline
• Big Picture: How to Provide Value to an Organization
• Large, Unstructured, Complex Problems
• Scientific Method
• The Science for Solving Problems with Data
• NASA Carbon Overwrapped Pressure Vessel Example
• The Future
Big Picture
• What Is Ultimately Valuable to an Organization?
• Understand Constancy of Purpose
• Why Does the Organization Exist
– Whale Oiling Companies No Longer Exist (from Deming)
– Did Not Understand the Real Reason They Existed!
• Modern Example – Kodak
Big Picture
• Leaders Concentrate on the Major Opportunities for the Organization
• The Sexy – New Markets
– The Next I-Phone
– Disruptive Innovation
Big Picture
• Not so Sexy – Improving What the Organization Does
– More than Lean Six Sigma, Continuous Improvement
– Requires Critical Focus on What the Organization Does and Why
– Often (Not Always!) Incremental Innovation
Big Picture
• Both Involve Problems that “Keep You Awake at Night”
• Value Comes from Opportunities!
Big Picture: Major Opportunities
• Are:
– Large
– Unstructured
– Complex
• Cut Across the Organization
Big Picture: Major Opportunities
• Require the Proper Data!
– Properly Collected
– Properly Analyzed
– Properly Interpreted
• Proper Solutions Must Be Sustainable
Large, Unstructured, Complex Problems
• Examples:
– DuPont
– Scott Paper
• Solutions Combine
– Academic – Basic Tools
– Practical Experience – How to Apply the Tools
Large, Unstructured, Complex Problems
• Currently, No Text Discusses Large, Unstructured, Complex Problems!
• Emerging New Discipline: Statistical Engineering
Solutions
• Interdisciplinary Teams
• Proper Blend of Talents
– Technical – Engineering/Scientific
– Statistical/Analytical
– Organizational Psychology/Anthropology
– Project Management
Solutions
• Strong Leadership!
– Form the Right Team
– Provide the Appropriate Resources
– Engaged in the Process
Scientific Method
• The heart of Statistical Engineering is the scientific method.
• Most theories underlying statistical engineering involve strategic application of the scientific method.
– Deming-Shewhart PDCA (Plan, Do, Check, Act)
– DMAIC (Define, Measure, Analyze, Improve, Control)
Scientific Method
• The Scientific Method Is
– Fundamental Approach for Discovery and Problem Solving
– Inductive/deductive problem solving process
Scientific Method: Basic Process
1. Define the problem (deductive)
2. Propose an educated theory, idea or model (deductive)
3. Collect data to test the theory (inductive)
4. Analyze the results (inductive)
5. Interpret the data and draw conclusions (inductive)
Scientific Method: Basic Process
• This process continues until a reasonable solution emerges.
• The scientific method is a sequential learning strategy!
Summary of the Scientific Method
• Understand the Real Problem at Hand
• Define the Problem
Summary of the Scientific Method
• Discover Solutions
– Abstract from the concrete to the abstract
– Develop a theory
– Test the theory using data
– Modify the theory as necessary
• Strong Need for Interdisciplinary Collaboration
Scientific Method and Data
• Data are the keys to the successful application of the scientific method
– Data collection
– Data analysis
– Data Interpretation
Scientific Method and Data
• Quality Engineering/Industrial Statistics are the handmaiden.
• Very important role in solving large, unstructured, complex problems.
Strategy of Statistical Engineering
• Identify Problem
• Provide Structure
• Understand Context
Strategy of Statistical Engineering
• Develop the Solution Strategy
• Develop and Execute Tactics
• Deploy Final Solution
Tactics of Statistical Engineering
• Data Acquisition
• Data Exploration
• Analysis
– Traditional Statistical Methodologies
– Modern Analytics (Big Data)
Tactics of Statistical Engineering
• Inference to the Process/Problem
• Deployment of Tentative Solution
– Does It Work?
– Is It Sustainable?
Overarching Methodologies
• Data Visualization
• Project Management
• Teamwork
• Organizational Culture
More on Tactics
• The Basic Tactical Tools Are Taught in Statistics/Industrial Engineering
• Data Acquisition:
– Planning the Data Collection
– Designing Experiments
– Data Cleansing
More on Tactics
• Data Exploration:
– Exploratory Data Analysis
– Analytics!
– Preliminary Models
– Follow Up
More on Tactics
• Analysis
– Statistical Inference: Estimation and Testing
– Formal Model Building
– Predictive Analytics
– Critical Point:
• Right Tool
• Properly Applied
More on Tactics
• Inference Back to the Problem – What Did We Discover?
• Propose, Evaluate, Deploy Solutions
• Statistical Engineering: The Science for Deploying the Tools!
NASA Example - COPVs
• Relatively Small Statistical Engineering Project
• COPV – Carbon Overwrapped Pressure Vessel
• Overarching Question of Interest: Reliability of COPVs at Use Conditions for Expected Life of Mission
NASA Example - COPVs
• Issues:
– Many different types of COPVs used in spacecraft
– Vessel tests are very expensive: money and time
• NASA Engineering Safety Center (NESC) Project
Core NESC Analytics Team:
• Reliability Engineers:
– JPL
– Langley Research Center
– Glenn Research Center
• Statisticians:
– Marshall Space Flight Center
– Virginia Tech
NASA Example - COPVs
• NASA Team’s Approach: Focus on Strands Used to Wrap Vessels
– Less expensive
– Can have many more experimental units than for vessels
• Still Issue with Time to Test
• Problem: How Do Strands Predict Vessel Behavior?
Previous Strand and Vessel Tests
• Relevant strand study conducted at a national lab:
– 57 strands at high loads for 10 years
– Net information learned: Strands either fail very early or last more than 10 years
• Vessel studies:
– Also 10 years
– Weibull model parameters similar to strands
Team’s Initial Concept
• Much larger study
• Censor very early
– Reduces time
– Allows the larger study in a practical amount of time
• Proceed in phases
• Have detailed data records to track any problems
NASA Example - COPVs
• Phase A: Conducted During Shake-Out of Equipment
• Small study (bigger than the national lab study!)
• Statistical goal: Determine if the parameters from the national lab study are valid as the basis for planning the larger study!
• Note: Phase A gave the team an opportunity to re-plan the larger experiment, if necessary!
NASA Example - COPVs
• Phase B: “Gold Standard” Experiment
• Planned time required: 1 year
• Used 4 “blocks” of equal numbers of strands
– Allowed the team to correct for time effects
– Allowed the team to mitigate problems
• Size assured ability to assess model
NASA Example - COPVs
• Total Size of the Database: Huge
• Kept data from start of specific strand test to failure on the second
• Kept the last 2 minutes at the .01 second from buffer
• Buffer allowed team to investigate unusual phenomena at failure
• Essential for proper data cleansing
NASA Example - COPVs
• Parallel Vessel Study
• Reasonably large ISS study targeted to end early (< 10 yrs)
• Opportunity to step up loads to mimic strands
• Censored but longer censor time than strands
Results to Date
• Phase A: Surprisingly similar to national lab study
• Phase B:
– Serious problem occurred in the first block
– Serious conversations with possibility of replacing!
– Other three blocks produced better than the planned precision for the estimates
– Residual analysis confirmed the Weibull model
Why is COPVs Statistical Engineering?
• Application of Scientific Method to a Complex Problem
• Sequential Data Collection/Experimentation
• Each Phase Targeted Different Questions
• Clearly Documented Assumptions, Assessed via Data
• Took Proper Steps to Cleanse Data
• Real Research Question Involves System of Systems
The Future
• Success Requires Providing Value to Organizations
• Solving Large, Unstructured, Complex Problems Provides Great Value
• Current State: People Know Tools, Often Limited Number of Tools
The Future
• Future State: How Do We Use the Tools Most Effectively
– Most Understand the Full Range of Tools Required
– Most Understand Best Practices for Using the Tools
– Must Understand Leadership in the Broad Sense
• The Future Is a Journey!