Visual Analytics as a Cognitive Science

Brian Fisher

SFU School of Interactive Arts & Technology and Program in Cognitive Science

UBC Media & Graphics Interdisciplinary Centre (MAGIC)

My Background• UG Biology, Medical Biophysics tech at CWRU Med • Scientific Programmer, Varian • Ph.D Experimental Psychology, UCSC • UWO & Rutgers Centres for Cognitive Science • Human-Information Interaction AKA “Cyberpsychology”

• Institute for Robotics and Intelligent Systems Networks of Centres of Excellence “a Cognitive Basis for the Design of Intelligent User Interfaces to Complex Systems” (SFU-UWO)

• 1999- Associate Director, UBC Media And Graphics Interdisciplinary Centre, Computer Science, Psychology, Institute for Computing, Information and Cognitive Systems

• 2004- SFU School of Interactive Arts and Technology and Program in Cognitive Science

My double life

• Ph.D in Experimental Psych

• Postdoc w Cognitive

Science society founder &

president

• Psychonomics Fellow

• VIS-related symposia at

Cogsci, & APS, papers on

cogsci of interaction

• Fuzzy-logic/Bayes models

• Postdoc funded by Inst. for

Robotics and AI

• VAST SC, VEC, VACCINE

• Led Dagstuhl “Interaction with

Information for Visual

Reasoning”, Cogsci-based

papers at VIS, CHI, BELIV.

Participants

“Big Data”: Volume, Velocity & Variety

• In 2011, data expected to be about 1.8 zettabytes (1021).

• In 2013, Internet traffic to reach 667 exabytes (1018)/yr.

• Comparison: US Library of Congress is ~10 petabytes (1015).

• “By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.” (McKinsey Global Institute 2011)

Challenges for computational approaches

• “3 Vs” challenge • Relevance, validity, reliability of data uncertain • Insufficient time to reach solution

• Model challenges • Multiple models to chose from • Assumptions may or may not hold

• “Wicked problems” challenge (Rittel) • Lack criteria to evaluate solution • Each problem is unique (no population) • Problem is not understood until solution is found

The information needed to understand the problem depends upon one’s idea for solving it. -- Rittel & Webber 1973

The ultimate challengeIn business, government & the professions specific people are responsible for the decisions that are made and how they are executed. These people are derelict in their duties if they only accept what the model tells them. Either we re-engineer society to accept a computer model as the ultimate authority or we find a way that human decision makers can exercise due diligence for computational as well as informational aspects of the problem.

An prehistory of visual analytics

Visualization history

• NSF meeting: “Visualization in Scientific Computing” • Nov. 1987 Computer

Graphics • First IEEE Visualization (now

SciVis) conference in 1990

“The purpose of [scientific] computing is insight, not numbers.” Richard Hamming

“Visualization is a method of computing.” Authors of report

Information visualization • 1990 Conference

on diagrammatic reasoning

• 1995 InfoVis Conference • “Information

Visualization: Wings for the Mind” Keynote by Stuart Card

Stuart Card’s view• Increase the memory & processing

resources available to users •Reduce the search for information by using

visual representations to enhance the detection of patterns

•Engage perceptual inference operations •Use perceptual attention mechanisms for

monitoring •Support manipulation of information

Infographics represent information (data, knowledge, opinions, etc.) in context to support understanding

Visual thinking: Infographics

Visualization literacy

• Build a “language” of collections of images that support thinking

• Diagrammatic reasoning science of how we understand complex diagrams

••iThinking as “smart seeing and Projecting”

Actively looking at external representations and projecting onto them makes us more powerful thinkers than thinking in our heads alone.

DDavid Kirsh example

Design Approach: Bertin

6

Bertin Semiologie Graphique (1967)

Cartographer, built description of how data should be represented visually

Jock Mackinlay, Stanford Ph.D. dissertation

Tableau software, used for our work with Boeing etc.

Design Approach: Edward Tufte

Big Data Example (Amaral)Metabolism

Experts are overwhelmed by sheer volume and complexity of data

Cartographic Representation (Amaral)

Guimera & Amaral, Nature 433, 895 (2005)

http://www.visual-literacy.org/periodic_table/periodic_table.html

Role of psychology in VIS

• Design based on theories (but no effective eval of those theories)

• Adaptation of methods from cogsci (but original methods not well understood)

• Rarely, ongoing collaboration with cognitive scientists

On the Death of Visualization (Lorensen 2004)

Can It Survive Without Customers?

• Visualization, alone, is not a solution.

• Visualization is a critical part of many applications.

• Visualization, the Community, lacks application domain knowledge.

• Visualization has become a commodity.

• Visualization is not having an impact in applications.

Visual analytics origins

Battelle PNNL R&D Agenda Panel

• In US, Panel meeting in 2004 •Brown, GMU, Georgia Tech, OSU, Penn State, Purdue,

SFU , Stanford, UC, UI, UM, UNC, UU, WPI •Boeing, Microsoft, PARC, Sandia Labs •CIA, DHS, FBI, NIST, NSA, unspecified

• Gave rise to • Industry Consortium •DHS Centre of Excellence

• Ccicada (Rutgers DIMACS) • VACCINE (Purdue et al)

• In Europe, EU Vismaster Coordination action • DFG Scalable Visual Analytics Priority program

Visual analytics

“This science must be built on integrated perceptual and cognitive theories that embrace the dynamic interaction between cognition, perception, and action. It must provide insight on fundamental cognitive concepts such as attention and memory. It must build basic knowledge about the psychological foundations of concepts such as ‘meaning,’ ‘flow,’ ‘confidence,’ and ‘abstraction.’ “ “Illuminating the Path” (IEEE Press)

“The science of analytical reasoning facilitated by interactive visual interfaces”

How are VA Information systems different?

• Development based on understanding of expert cognition in situ • Informed by current cognitive & social science • Engagement with community of experts • Emergent cognitive science of expert reasoning

• Clear support for analytical processes-- reasoning, collaboration & interaction • Graphical analog for analytic processes • Support “Human-information discourse” • Integrated across roles in the community

Visual analytics

Pg. 4 in Thomas, J., Cook, K., Institute of Electrical and Electronics Engineers, Dept of Homeland Security, & United States. (2008). Illuminating the Path: The Research and Development Agenda for Visual Analytics. IEEE Press. Retrieved from http://www.osti.gov/energycitations/product.biblio.jsp?osti_id=912515

Cognitive & Perceptual Sciences

Visual Information Systems

Graphic & Interaction Design

Mathematical & Statistical Methods

Effective Situated

R&D

http://www.vacommunity.org/HomePage

Visual analytics as a translational cognitive science

How to bridge informatics & psychology?

• VIS offers: • Implementations • Funding • Awesome

Research Questions

• Psych offers • Methodology • Theory • Phenomena • Cheap talent

Challenges: defining boundary objects, culture clash, publication venues, academic jobs for Cogs grads…

My approach: start in the middle!• Develop bridging

“Cyberpsychology” theory & methods

• Hope is that… • They are building

from each shore • Somehow we will be

able to align things

http://www.magic.ubc.cahttp://www.icics.ubc.ca

http://interaction-science.iat.sfu.ca

Bridging ideas from D-Cog

• Visualization literacy is a form of “Smart seeing and projecting” w external representation

• We propose 2 additional D-cog perspectives: • Agent-machine coupling: coordination of thought and

action in dynamic artificial environments •Cognitive architecture modeling and characterizing

“personal equation” of individual differences (e.g. perceptual expertise)

• Socially-distributed cognition •Grounded theory in Clark’s Joint Activity Theory

framework analysis of pair/group collaborative decision making

Agent-machine coupling in air traffic control

• Cognitive architecture from psychology • Extend to expert human performance

• Cognitive expertise • Visual expertise • Visuomotor expertise • Multimodality & modularity

• Test human capabilities in dynamic display environments

Controller/display systems in air traffic control

• NextGen ATC “fishtank” projection

• Change camera position for better view

• How will global motion affect tracking?

Liu, G. Austen, E. L., Booth, K.S. Fisher, B., Argue, R. Rempel, M.I., & Enns, J. (2005) Multiple Object Tracking Is Based On Scene, Not Retinal, Coordinates. Journal of Experimental Psychology: Human Perception and Performance. 31(2), Apr 2005, 235-247.

http://www.youtube.com/watch?v=tKJVB4id_TY

FINST theory of spatial indexing

Multiple object tracking (Pylyshyn)

Fit human tracking function (Lui)

... Then add display motion

Tracking vs object speed

Tracking in warped space

Tracking in warped space

Conclusion: We track in allocentric space

• Retinal speed of targets does not determine performance

• Motion of targets relative to each other does • But only if motion preserves good metric

characteristics of space • Explanation is at the level of a human -

display cognitive system

• “Pair analytics” sessions • Student visual analyst & trained

domain expert collaborate on analytic task

• Student “drives”, expert “navigates”

• Video session & capture screen

Social D-Cog in safety analysis

Joint Activity Theory (Clark)

• Language is an essentially collaborative activity, like playing duet or paddling canoe

• We work to build common ground so as to communicate effectively and efficiently

• Clark’s theory: • Defines kinds of common ground • Formalizes the notion of activity as a “joint action” • Describes the processes by which common ground is

developed through joint action

Bird Strikes

• http://www.youtube.com/watch?feature=player_detailpage&v=5jN0bqL9cM0

Tableau Bird Strike Pair Analysis

\IN-SPIRE Bird Strike Pair Analysis

Wade Internship•Video recorded and screen captured over 10

Paired Analysis sessions using both Tableau and IN-SPIRE

• Influenced design decisions on: • 777 •P8-A • 787 • 747-8

•Changes to pilot training manual

Wade Internship• Presented work to:

• 787 Engineers • Aviation Safety Community of Practice • Aerodynamics, Performance, Stability and Control

flight data recorder analysis group • Advanced Analytics group • UW Aeronautics and Astronautics students • Boeing Educational Network webcast (400+)

• 500+ people exposed to Visual Analytics, Paired Analysis for Aviation Safety

Joint activity analysis• Observe joint attention management by

project markers • Multimodal markers — mouse gesture & verbal

(content & prosody) • Vertical and horizontal markers

• We define event structure for analysis based on markers & actions

• Social constraints seen in self-talk incidents, provides effective verbal protocols for protocol analysis methods

Metrics  include:  

• Fluidity  of  the  process.  

• Management  of  joint  attention.    

• Situational  awareness.  

• Cognitive  economics.

Publications• Kaastra, L. T. and Fisher, B. (2014) Tracking Joint Activities in Visual Analyses.

Proceedings of the Psychonomics Society meeting

• Kaastra, L. T. and Fisher, B. (2014) Pair Analytics as a Field Experiment Methodology.. In Proceedings of the 2014 BELIV Workshop: Beyond Time and Errors-Novel Evaluation Methods for Visualization

• Kaastra, L. T., Arias-Hernandez, R., & Fisher, B. (2012). Evaluating analytic performance. In Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors-Novel Evaluation Methods for Visualization

• Arias-Hernandez, R., Green, T. M., & Fisher, B. (2012). From Cognitive Amplifiers to Cognitive Prostheses: Understandings of the Material Basis of Cognition in Visual Analytics. In Interdisciplinary Science Reviews, Vol. 37 No. 1.

• Arias-Hernandez, R., Kaastra, L. T., & Fisher, B. (2011). Joint action theory and pair analytics: In- vivo studies of cognition and social interaction in collaborative visual analytics. In Proceedings of the 33rd Annual Conference of the Cognitive Science Society

• Arias-Hernandez, R., Kaastra, L. T., Green, T. M., & Fisher, B. (2011). Pair Analytics: Capturing Reasoning Processes in Collaborative Visual Analytics. In (HICSS), 2011 44th Hawaii International Conference On System Sciences.

Cognition Perceptual

Science Methods

Social Science Methods

Computation and Visualization

Methods

Graphic & Interaction

Design Methods

Analysis “in the wild”

What we need

How to do it• Institutes/centres/programs that combine:

• Application experts • VIS people • Cognitive scientists • Designers

• Publication venues for translational research • Two-part conference approach

• Outreach to application venues • Return to VIS community

Thanks to SCIENCElab!• Dr Richard Arias-

Hernández • Dr. Nathalie Prevost • Dr. Linda Kaastra • Dr. Payam Rahmdel • Samar Al-Hajj • Nadya Calderón • Tera Marie Green • Ethan Soutar-Rau • Ali Khalili

• Barry Po • Aaron Smith • Andrew Wade • Doug Mackenzie

Cogs/IAT 885 Visually Enabled Reasoning

• Cognitive theory • Psychology of human reasoning • Modelling causality • Alternative (modal/hybrid) logics

• Analytic practice • Problems and datasets from SEMVAST, Boeing • Teams w paper, IN-SPIRE, Tableau Jigsaw

• Outcomes • Learn reflective analytic practice • Prepare for internship as analyst

Textbooks

• How We Reason. Philip Johnson-Laird, Oxford Press

• Causal Models: How People Think about the World and Its Alternatives. S. Sloman, Oxford Press R

• Human Reasoning and Cognitive Science. Keith Stenning & Michiel van Lambalgen, MIT Press

Cogsci society logo