Measuring Team Shared Understanding: Using Analysis-Constructed Shared Mental Model Methodology...
-
date post
21-Dec-2015 -
Category
Documents
-
view
221 -
download
1
Transcript of Measuring Team Shared Understanding: Using Analysis-Constructed Shared Mental Model Methodology...
Measuring Team Shared Understanding: Using Analysis-
Constructed Shared Mental Model Methodology
Measuring Team Shared Understanding: Using Analysis-
Constructed Shared Mental Model MethodologyTristan E. Johnson
Learning Systems Institute, Florida State University, Tallahassee, FL, USA
International Workshop and Mini-conference onExtending Cognitive Load Theory and Instructional Design
to the Development of Expert Performance
August 29-30, 2005 Open University of the Netherlands
BackgroundBackground1. Team Performance
• Team Cognition Link between SMM and Team Performance
2. Shared Understanding and Shared Mental Models
Development of SMM and its relation to team performance
Team CognitionTeam Cognition
Elaborated view of team cognition including team interactions and SMM development
Shared Knowledge TypesShared Knowledge Types
1. Task Knowledge—domain specific2. Team Knowledge—5 factors
Team Knowledge Factors Team Knowledge Factors 1. Team Knowledge
• Knowledge about team members and tasks that they need to perform• Teammates knowledge, Task knowledge
2. Team Skills• Abilities associated with successful job performance• Communication skills, Interpersonal skills, Leadership skills, Skills to
deal with conflict and team cohesion
3. Team Attitudes• Internal state that influences team members’ choices or decision to
act in a certain way under particular circumstances • Shared belief, Shared value
4. Team Dynamics• Combination of dynamic processes of team coordination and team
cohesion• Team coordination, Team cohesion
5. Team Environment• External conditions affecting the foundation of the team mental model• Technology, Organization, Synchrony & Geographic dispersion
Measuring Task KnowledgeMeasuring Task Knowledge1. Measuring Shared Understanding—
measuring concept relatedness • Card sorting, cognitive interviewing, MDS,
Pathfinder, surveys, casual maps (Langan-Fox, Code, Langfield-Smith, 2000; Trochim, 1989)
• Concept Mapping Statistical analysis Descriptive analysis
2. Analysis Constructed - Shared Mental Model (AC-SMM)
SMM Elicitation TechniquesSMM Elicitation Techniques1. TmC-SMM—Whole team elicitation (1 map)2. AC-SMM—Individual elicitation with aggregation (n maps)
SMM i— desired shared mental model stateTmC-SMM — involves team negotiation and interactionSMM∂— altered team shared mental model stateAC-SMM —retains the initial ICMM state
AC-SMM Methodology RationaleAC-SMM Methodology Rationale1. Knowledge Elicitation
• Process allows simultaneous consideration of concepts• Reflection and changes during elicitation
2. Analysis• Allows for explication of implicit relationships—
considering 1) logic, 2) structure, and 3) spatial orientation
3. Relatedness• Specific to three levels
Concepts Links Clusters
4. Appropriate for studying shared understanding in applied settings
AC-SMM Methodology Overview AC-SMM Methodology Overview 1. Instrument Design
• Structured/Semi-Structured/Unstructured• Task Analysis (Generate Concepts)
2. Data Collection• Guided Practice • Individually Constructed Mental Model (ICMM) Elicitation
3. Data Analysis• Phase I: ICMM Analysis/Coding
Relatedness at concepts, links, clusters levels Allows for explication of implicit relationships Implicit coding has [logic and spatial] or [logic and structural]
support• Phase II: Shared Analysis
Determine sharedness level—number or percentage of team members
• Phase III: AC-SMM Construction Generates SMM
Phase I: ICMM AnalysisPhase I: ICMM AnalysisFactor 1: Concepts
• Explicit individual nodes
Factor 2: Links• Two concepts joined explicitly [connector] or
implicitly
Factor 3: Clusters• Two or more connectors explicitly bridging three
or more concepts• May have implicit connections with evidence• Combination of clusters—Sub- and Super-
clusters
Factor 4: Emphasis and Sequence• Explicit notation of node emphasis or node order
ICMM Coding—LinksICMM Coding—Links
ICMM Coding—ClustersICMM Coding—Clusters
ICMM Coding—Emphasis and SequenceICMM Coding—Emphasis and Sequence
ICMM Coding ExampleICMM Coding Example
Phase II: Shared AnalysisPhase II: Shared Analysis1. Determine Sharedness Level Criterion—Number or
Percentage2. Shared Data Used for AC-SMM Construction
Phase III: AC-SMM ConstructionPhase III: AC-SMM Construction
ResearchResearch1. General Research Focus
• What task knowledge is shared?• How does shared understanding change over
time?• What are the patterns of change? • What is the affect of task performance on the
shared understanding of the team?
Timeline of Data Collection and Hours Worked During PQS Workshop Sessions
Day 1 Day 2 Day 3 Day 4 Workshop
(C) a Data (#) b Hrs Hrs
Data (#) b Hrs Hrs
Data (#) b Hrs Hrs
Data (#) b
Total Hrs
Workshop 1 (3)
Pre (4) 0.00 3.50 None 0.00 0.00
Mid (4) 1.08 1.88 Post (3) 6.47
Workshop 2 (10) Pre (11) 4.25 8.00 None 0.00 0.00 Mid (11) 4.75 2.00 Post (10) 19.00
Workshop 3 (10) Pre (12) 6.00 1.67
Mid (13) 3.67 4.87 None 5.00 4.00 Post (15) 25.21
Workshop 4 (7) Pre (13) 7.83 7.75 Post (7) 0.00 0.00 None 0.00 0.00 None 15.58
Note. a C = Number of team members submitting complete datasets. b # = Number of team members submitting ICMM for data collection period.
Data Collection TimelineData Collection Timeline
Concepts: [A] - Fleet notification of change [L] - Section 300 - Watchstations [B] - Analysis - Determine level of change [M] - Content revisions [C] - Verify corrections [N] - Add line item/task [D] - Model manager [O] - Add new references [E] - Subject matter expert [P] - Formatting [F] - Non-content related revisions [Q] - Keep line item/task [G] - Consensus [R] - Summary of changes [H] - Ve rify existing references [S] - Referencing [I] - Facilitator [T] - Delete line item/task [J] - Section 100 - Fundamentals [U] - Sequencing [K] - Section 200 - Systems [V] - Semantics (wording)
ConceptsConcepts
Team Profiles FindingsTeam Profiles Findings
Profile of Workshop 1 PQS Team
Like / Dislike Concept Mapping b
Participant by Role
PQS Workshop
Exp a
Familiar with
Others
Previous Concept
Mapping Exp
a Pre- task Post-task
Year s in Navy
FAC 01 Yes No Yes 7.00 6.00 17.00 FAC 02 c Yes No No -- -- 22.00 SME 01/MM No Yes No -- 5.00 17.00 SME02 No Yes No -- 8.00 17.00 Average All Workshop Participants 7.00 6.33 18.25 Average Complete Datasets 7.00 6.33 17.00 Note. a Exp = Experience. b Scale = 0 (Dislike) to 5 (Neither Like nor Dislike) to 10 (Like). c Indicates participant(s) did not submit all data. -- Indicates no response.
Profile of Workshop 2 PQS Team
Like / Dislike Concept Mapping b
Participant by Role
PQS Workshop
Exp a
Familiar with
Others
Previous Concept
Mapping Exp
a Pre- task Post-task
Year s in Navy
FAC 01 Yes No No -- 5.00 18.00 SME 01/ MM No Yes No -- 5.00 20.00 SME 02 c No Yes No 5.00 6.00 SME 03 No Yes No -- 7.00 19.00 SME 04 No Yes Yes 5.00 9.00 19.00 SME 05 Yes Yes No -- 5.00 19.00 SME 06 -- -- No -- 8.00 15.00 SME 07 Yes Yes No -- 0.00 17.00 SME 08 No Yes No -- -- 11.00 SME 09 No -- No -- -- 10.00 SME 10 No No No -- 7.00 18.00 Average All Workshop Participants 5.00 5.75 15.64 Average Complete Datasets 5.00 5.75 16.60 Note. a Exp = Experience. b Scale = 0 (Dislike) to 5 (Neither Like nor Dislike) to 10 (Like). c Indicates participant(s) did not submit all concept maps. -- Indicates no response
Profile of Workshop 3 PQS Team
Like / Dislike Concept Mapping b Participant
by Role
PQS Workshop
Exp a
Familiar with
Others
Previous Concept Mapping
Exp a Pre- task Post-task
Years in Navy
FAC 01 Yes No Yes 6.00 5.00 19.00 SME 01 /MM c Yes Yes Yes 6.00 5.00 18.00 SME 02 c Yes Yes No -- -- 26.00 SME 03 c No Yes No -- -- 23.00 SME 04 c No Yes No -- 6.00 19.00 SME 05 No Yes No -- 0.00 12.00 SME 06 Yes Yes No -- -- 26.00 SME 07 No Yes No -- 7.00 12.00 SME 08 c No Yes No -- 8.00 14.00 SME 09 c No Yes No -- 7.00 8.50 SME 10 c No No No -- 8.00 16.00 SME 11 c No -- No -- -- 7.00 SME 12 No -- No 8.00 10.00 4.00 SME 13 -- No No -- -- 3.00 SME 14 -- -- No -- -- 2.50 SME 15 No No No -- 7.00 15.00 SME 16 -- -- -- -- -- -- SME 17 -- -- -- -- 9.00 -- SME 18 -- -- -- -- -- -- Average All Workshop Participants 6.67 6.55 14.06 Average Complete Datasets 6.00 6.50 16.72 Note. a Exp = Experience. b Scale = 0 (Dislike) to 5 (Neither Like nor Dislike) to 10 (Like). c Indicates participant(s) did not submit all concept maps. -- Indicates no response
Profile of Workshop 4 PQS Team
Like / Dislike Concept Mapping b
Participant by Role
PQS Workshop
Exp a
Familiar with
Others
Previous Concept Mapping
Exp a Pre- task Post-task
Year s in Navy
FAC 01 Yes Yes Yes 5.00 1.00 22.00 SME 01/MM c Yes Yes Yes 5.00 -- 21.00 SME 02 No Yes No -- 5.00 19.00 SME 03 c No Yes No -- -- 27.00 SME 04 Yes Yes No -- 6.00 8.00 SME 05 c No Yes No -- -- 19.00 SME 06 c No Yes No -- -- 19.50 SME 07 Yes Yes Yes 7.00 8.00 25.00 SME 08 c No -- No -- -- 14.00 SME 09 No Yes No -- -- 21.00 SME 10 c -- Yes No -- -- 23.00 SME 11 -- -- -- -- -- -- SME 12 -- -- -- -- -- 24.00
Average All Workshop Participants 5.67 5.00 20.21 Average Complete Datasets 6.00 5.00 19.60 Note. a Exp = Experience. b Scale = 0 (Dislike) to 5 (Neither Like nor Dislike) to 10 (Like). c Indicate participant(s) did not submit all ICMMs. -- Indicates no response.
Shared Data Findings, Team 1 Only Shared Data Findings, Team 1 Only
Shared Data (³ 5 0%) Ğ Workshop 1
Workshop 01 Complete Datasets ICMM Pre-task Mid-task Post-task
# Team Members Submitting 3 3 3 Component # % # % # %
S = Shared Across ICMMs
Concepts
[A] Ğ Fleet notification of change 3 100% 2 67%
[B] Ğ Analysis-Determine level of change 3 100% 2 67%
[C] Ğ Verify corrections 3 100% 3 100% 3 100% S
[D] Ğ Model manager 3 100% 2 67% 3 100% S
[E] Ğ Subject matter expert 3 100% 2 67% 3 100% S
[F] Ğ Non-content related revisions 3 100% 2 67%
[G] Ğ Consensus 3 100% 2 67% 3 100% S
[H] Ğ Ve rify existing references 3 100% 2 67% 3 100% S
[I] Ğ Facilitator 3 100% 3 100% 3 100% S
[J] Ğ Section 100 Ğ Fundamentals 3 100% 2 67% 3 100% S
[K] Ğ Section 200 Ğ Systems 3 100% 2 67% 3 100% S
[L] Ğ Section 300 Ğ Watchstations 3 100% 2 67% 3 100% S
[M] Ğ Content revisions 3 100% 3 100% 2 67% S
[N] Ğ Add line item/task 3 100% 3 100% 3 100% S
[O] Ğ Add new references 3 100% 2 67% 3 100% S
[P] Ğ Formatting 3 100% 2 67% 3 100% S
[Q] Ğ Keep line item/task 3 100% 3 100% 3 100% S
[R] Ğ Summary of changes 3 100% 2 67%
[S] Ğ Referencing 3 100% 2 67% 3 100% S
[T] Ğ Delete line item/task 3 100% 3 100% 3 100% S
U Ğ Sequencing 3 100% 3 100% 3 100% S
V Ğ Semantics (wording) 3 100% 3 100% 2 67% S
Shared Data (³ 5 0%) Ğ Workshop 1 Workshop 01 Complete Datasets ICMM Pre-task Mid-task Post-task # Team Members Submitting 3 3 3 Component # % # % # %
S = Shared Across ICMMs
Directional Links
[D>A] 2 67%
[I>D] 2 67%
[I>E] 2 67%
[I>J] 2 67%
[J>C] 2 67%
[K>J] 2 67%
[L>K] 2 67%
[M>F] 2 67%
[P>L] 2 67%
Directional Links [U>V] 2 67%
Non-Directional Links
[E,I] 2 67%
[J,K] 2 67%
[K,L] 2 67%
[N,Q] 2 67%
[Q,T] 2 67%
Shared Data Summary Per TeamShared Data Summary Per Team
Shared Data Summary - Workshop 1
Workshop 01 Complete Datasets ICMM Datasets Pre-task Mid-task Post-task
# Team Members Submitting 3 3 3 Concepts 29 21 24 Shared (50% or more) 22 20 21 % Shared 75.86% 95.24% 87.50% Sequence Components 0 3 1 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Clusters 17 18 22 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Directional Links 68 59 52 Shared (50% or more) 3 3 4 % Shared 4.41% 5.08% 7.69% Non-Directional Links 32 17 19 Shared (50% or more) 1 2 2 % Shared 3.13% 11.76% 10.53% Important Concepts 0 6 4 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Total Components 146 124 121 Shared (50% or more) 26 25 27 % Shared 17.81% 20.16% 22.31%
Shared Data Summary - Workshop 2
Workshop 02 Complete Datasets ICMM Datasets Pre-task Mid-task Post-task
# Team Members Submitting 10 10 10 Concepts 26 23 22 Shared (50% or more) 22 22 22 % Shared 84.62% 95.65% 100.00% Sequence Components 8 0 0 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Clusters 54 56 45 Shared (50% or more) 0 1 1 % Shared 0.00% 1.79% 2.22% Directional Links 108 98 93 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Non-Directional Links 79 84 83 Shared (50% or more) 1 0 2 % Shared 1.27% 0.00%- 2.41% Important Concepts 13 7 5 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Total Components 290 268 248 Shared (50% or more) 23 23 25 % Shared 7.93% 8.58% 10.09%
Shared Data Summary (³ 50%) Ğ Workshop 3
Workshop 03 Complete Datasets ICMM Datasets Pre-task Mid-task Post-task
# Team Members Submitting 10 10 10 Concepts 24 22 22 Shared (50% or more) 21 21 21 % Shared 87.50% 95.45% 95.45% Sequence Components 3 0 0 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Clusters 56 53 53 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Directional Links 142 129 129 Shared (50% or more) 3 1 4 % Shared 2.11% 0.78% 3.10% Non-Directional Links 30 39 42 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Important Concepts 7 17 3 Shared (50% or more) 0 0 0 % Shared 0.00% 0.00% 0.00% Total Components 262 260 251 Shared (50% or more) 24 22 25 % Shared 9.16% 8.46% 9.96%
Shared Data Summary (³ 50%) Ğ Workshop 4
Workshop 4 Complete Datasets ICMM Datasets Pre Mid Post
# Team Members 7 0 7 Concepts 25 -- 24 Shared (50% or more) 19 -- 16 % Shared 76.00% -- 66.67% Sequence Components 8 -- 10 Shared (50% or more) 0 -- 0 % Shared 0.00% -- 0.00% Clusters 39 -- 44 Shared (50% or more) 0 -- 1 % Shared 0.00% -- 2,27% Directional Links 128 -- 45 Shared (50% or more) 0 -- 0 % Shared 0.00% -- 0.00% Non-Directional Links 53 -- 127 Shared (50% or more) 2 -- 1 % Shared 3.77% -- 0.79% Important Concepts 2 -- 0 Shared (50% or more) 0 -- 0 % Shared 0.00% -- -- Total Components 255 -- 250 Shared (50% or more) 21 -- 18 % Shared 8.24% -- 7.20%
Cross Case FindingsCross Case Findings
Concepts from ACSMMs Shared ³ 50% as Compared across Workshops
ACSMM Pre-Task Mid-Task Post-Task Workshop 1 2 3 4 1 2 3 1 2 3 4 Members 3 10 10 7 3 10 10 3 10 10 7
Concept % % % % % % % % % % % [A] 100 100 100 71 0 100 100 67 100 90 57 [B] 100 100 100 100 0 100 90 67 90 80 71 [C] 100 100 100 71 100 90 90 100 100 80 86 [D] 100 90 80 86 67 100 90 100 100 90 57 [E] 100 100 100 86 67 100 100 100 100 100 71 [F] 100 80 0 0 67 70 0 0 50 0 0 [G] 100 80 100 71 67 90 90 100 100 90 86 [H] 100 80 100 100 67 90 100 100 100 100 86 [I] 100 100 90 71 100 100 100 100 100 90 0 [J] 100 80 100 71 67 90 100 100 90 100 71 [K] 100 80 100 71 67 90 100 100 90 100 71 [L] 100 80 100 71 67 90 100 100 90 100 71 [M] 100 100 60 71 100 80 60 67 70 70 57 [N] 100 80 90 71 100 70 90 100 70 90 57 [O] 100 80 100 100 67 90 100 100 100 100 86 [P] 100 100 70 86 67 80 80 100 90 80 57 [Q] 100 90 80 71 100 90 80 100 90 90 57 [R] 100 100 100 86 0 90 100 67 100 80 0 [S] 100 100 80 57 67 100 70 100 100 90 0 [T] 100 60 90 71 100 60 90 100 70 90 57 [U] 100 80 80 0 100 70 60 100 70 60 0 [V] 100 70 80 0 100 70 80 67 70 90 0
ACSMM Scores per Wo rkshop
ACSMM Factor Workshop 1 Workshop 2 Workshop 3 Workshop 4
Pre-task ACSMM Score (Points) Concept 22 22 21 19 Sequence 7 9 9 4 Cluster 10 15 15 15 Link 20 20 18 8 Important Concept 0 0 0 0 Total ACSMM 59 66 63 46
Post-task ACSMM Score (Points)
Concept 21 22 21 16 Sequence 7 8 8 4 Cluster 15 25 20 20 Link 20 14 22 12 Important Concept 0 0 0 0 Total ACSMM 63 69 71 52
Change in Score from Pre-task to Post-task (Points)
Concept -1 0 0 -3 Sequence 0 -1 -1 0 Cluster 5 10 5 5 Link 0 -6 4 4 Important Concept 0 0 0 0 Total ACSMM 4 3 8 6
Pre, Mid, Post AnalysisPre, Mid, Post Analysis
ACSMM ScoresACSMM Scores
ACSMM Scores and Team Performance Outcomes
Measures of SMM and Team Performance Workshop 1 Workshop 2 Workshop 3 Workshop 4 Pre-Task ACSMM Score 59 66 63 46 Post-Task ACSMM Score 63 69 71 52 Change in ACSMM Score from Pre to Post 4 3 8 6 % Change from Pre to Post 6.78% 4.55% 12.70% 13.04% Hours Worked 6.47 19.00 25.21 15.58 Major Revisions 103 180 285 219 Major Revisions as % of Al l Revisions 41.04% 26.05% 13.08% 75.26% Minor Revisions 148 511 1894 72 Minor Revisions as $ of All Revisions 58.96% 73.95% 86.92% 24.74% Total Revisions 251 691 2179 291 Revisions per Hour 38.79 36.37 86.43 18.68
General FindingsGeneral Findings
1. Similarity among ICMMs tends to increase as does the number of clustered concepts, the tendency is for the number of concepts used to decrease.
2. ICMMs were becoming more structured and more representative of the team task
3. These ideas are not yet proven. 4. We have designed a set of studies to try and
validate our hypothesis5. This work is intended to not only learn about
teams that work in the various settings, but to validate the AC-SMM analysis model
SummarySummary1. Richer description of shared understanding in
teams2. AC-SMMs compared over time to determine
change in shared understanding3. Lacks weighted measures and precise distances
between concepts, but future work will include descriptive statistics of the key factors
4. Lack of prepositional descriptors5. As we become more precise and descriptive we
can utilize this new knowledge to better explain and understand team cognition
6. Facilitate team training with intent to improve team performance outcomes
Thanks for your attention.
Questions?
Thanks for your attention.
Questions?
Findings and Extra SlidesFindings and Extra Slides
Findings Across Pre, Mid, PostFindings Across Pre, Mid, Post
Participants & ContextParticipants & Context
1. Participants• Personnel Qualification Standards (PQS) Team• Team Task • Team Members • Team Member Roles
2. Context• U.S. Navy Training Center, Pensacola, FL• Face to face workshop• Equipment
Findings From AC-SMM AnalysisFindings From AC-SMM Analysis
• What is shared?• Does shared understanding change over time?
Secondary AnalysisSecondary Analysis
1. Sequence• Where concepts were placed within each ICMM• Focus on key concepts
Team member roles Sections of PQS book Referencing Questions
2. Links• Relationships between concepts without directionality
3. Clusters• Relationships between concepts without directionality• Commonalities between related clusters of concepts
Secondary AnalysisSecondary Analysis
• Links - Relationships between concepts without directionality • Clusters - Relationships between concepts without
directionality and commonalities between related clusters of concepts
Example of Cluster
[[B,D], [B,E], [B,I]]
with Secondary Clusters [B,D,E,I] and [D,E,I]
without Related Links
[D,E], [D,I], and [E,I]
All Data, Shared by ≥ 2
All Data, Shared by ≥ 50%
Follow-up AnalysisFollow-up Analysis
1. Sequence• Started with original data submitted by each team member• Where concepts were placed within each ICMM• Focus on key concepts
Team member roles (concepts [D], [E], [I])
Sections of PQS book (concepts [J], [K], [L])
Referencing (concepts [H], [O], [S])
Questions (concepts [N], [Q], [T])
All Data, Shared by ≥ 2Secondary Analysis
All Data, Shared by ≥ 50%Secondary Analysis
Complete Datasets, Shared by ≥ 50%
Complete Datasets, Shared by ≥ 2Secondary Analysis
Complete Datasets, Shared by ≥ 50%Secondary Analysis