Action, Complexity and Cognition Matthias Rauterberg Industrial Design Technical University...

Action, Complexity and Cognition

Matthias RauterbergIndustrial Design

Technical University Eindhoven

2003

© M. Rauterberg, 20032/35

Possible Interpretations of 'Information'

1.) 'Information' as a message (syntax)

2.) 'Information' as the meaning of a message (semantic)

3.) 'Information' as the effect of a message (pragmatic)

4.) 'Information' as a process

5.) 'Information' as knowledge

6.) 'Information' as an entity of the world

Ref: Folberth, O. & Hackl, C. (1986, eds.) Der Informationsbegriff in Technik und Wissenschaft. München: Oldenbourg.


“Information” for Learning Systems

before reception after reception Author

dof of the decision content of the decision HARTLEY 1928

uncertainty certainty SHANNON 1949 uncertainty information BRILLOUIN 1964 potential information actual information ZUCKER 1974 entropy amount of information TOPSØE 1974


context human

mental model

situation-1

situation-2

complexity

positive incongruity

negative incongruity

Incongruity and Learning

Incongruity = Complexitycontext – Complexityhuman

learning

Ref: Rauterberg, M. (1995). About a framework for information and information processing of learning systems. In: E. Falkenberg, W. Hesse & A. Olive (eds.), Information System Concepts--Towards a consolidation of views (IFIP Working Group 8.1, pp. 54-69). London: Chapman&Hall.

http://www.idemployee.id.tue.nl/g.w.m.rauterberg/publications/ISCO95paper.pdf


physical operation

feedback control of action

goal-, subgoal-setting

mental operation

task(s)

planning of execution selection of means

The Complete Action Cycle

synchronisation in time

synchronisation in space


s0

d

s1

h

s0

a

s2

F3

s3

CR

s3

F9

s1

_

s3

TAB

s3

F2

s3

_

s3

TAB

s3

_

s3

The Idea

Any human task solving process can be described in a finite state-transition chain, if the task can be described in an ‘action space’, specified by a finite set of states ( ) and transitions [ ].

State description: s0 : main menu s1 : modul "data" s2 : routine "browse" s3 : "wrong input" state

Action description: _ : ascii key "BLANK" a : ascii key "a" d : ascii key "d" h : ascii key "h" CR: carriage return F2: function key "2" F9: function key "9" TAB:tabulator key


s0 d s1

s1 h s0

s1 a s2

s2 F3 s3

s3 CR s3

s3 F9 s1

s3 _ s3

s3 TAB s3

s3 F2 s3

elementary processes Petri-Net

s0

d h

s1

a

s2s3

F3

F9

CR

_

TAB

F2

folding

The Folding Operation in Petri Nets


Task Description

In the experiment all 12 users had to play the role of a camping place manager. This manager uses a database system with a data base consisting of three data files: PLACE, GROUP, and ADDRESS. All users had to solve the following four different tasks operating the database system:

Task 1: "How many data records are in the file ADDRESS, in the file PLACE, and in the file GROUP? Find out, please."

The user has to activate a specific menu option ("Datafile" in module "Info" of the menu interface) and to read the file size (solutions: PLACE = 17 data records, GROUP = 27 data records, ADDRESS = 280 data records).

Task 2: "Delete only the last data record of the file ADDRESS, the file PLACE, and the file GROUP (sorted by the attribute 'namekey')."

The user has to open (sorted according to the given attribute), select and delete the last data record (file: PLACE, GROUP, ADDRESS).

Task 3: "Search and select the data record with the namekey 'D..8000C O M' in the file ADDRESS, and show the content of all attributes of this data record on the screen. Correct this data record for the following attributes: State: Germany, Place number: 07. Remarks: Database system dealer can give a demonstration."

The user must select a certain data record (file: ADDRESS), update the data record with regard to the three attributes: State, Place number, Remarks.

Task 4: "Define a filter for the file PLACE with the following condition: all holidaymakers arrived on date 02/07/87. Apply this filter to the file PLACE, and show the content of all selected data records in the mask browsing mode on the screen."

The user must define a filter for the attribute "arrival date", apply the filter to the data file PLACE, and display the content of each data record found on the screen.


System Description

The dialog system was the relational data base system ADIMENS version 2.21 with a character oriented user interface (CUI) running on standard IBM PC's with standard keyboard.

The whole dialog structure is strictly hierarchical organized with three levels:

(1) the main menu has 7 dialog operations (ordinary ASCII characters chosen from a menu) to go down to 7 different modules, and 5 function keys with specific semantics;

(2) at the module level each module has exactly 4 different dialog operations to change to routines and on average 4.1 (±1.7; range: 0-5) function keys with specific semantics;

(3) at the routine level the user has only on average 3.7 (±2.9; range: 0-10) different function keys to control the dialog (additionally all ASCII keys and the 4 cursor keys are usable).

The number of all ordinary dialog contexts (main menu, modules, routines) is 1+7*4=29.

But to describe the complete dialog structure with all help, error and additional dialog states we need at least 144 different system states.

To change from one state to the other the system offers overall 358 different dialog operations (transitions).


physical execution

evaluation and control

goal-, subgoal setting

mental execution

task description

action planning selection of means

goal

system state

selected action

result

Observable Data


G_2

Start menu

M_3

Main menu Main menu. F file

F_3

User key press

1

User key press

Main menu

i

User key press

Info

Info.file Info.screen1 Info.screen2

d

M_22 M_22 M_22

DB content display

User key press

Info.screen3 Info

Info.screen1 stopped

Info Main menu Start menu Start menu Start menu

M_11 d

Automatic transition

User key press

M_22 M_22 M_22 M_11 d

User key press

M_22 BL F_10 M_22 M_22

M_11 h h

User key press

User key press

G_2 F_10

User key press

... continues

... continues

... continues

... continues






Main menu

marked

Info.file Info.screen1 Info.screen2 Info.screen3 Info

Info.screen1Info.file

Info.screen3

Info.screen2Info.screen1

DB content display

DB content display

DB content display

DB content display

DB content display

User key press

User key press

DB content display

DB content display

DB content display

Example of a task solving process


s0

d

s1

h

s0

b

s2

F3

s3

CR

s3

F9

s1

_

s3

TAB

s3

F2

s3

_

s3

TAB

s3

_

s3

structure as a

Petri net

s0

d h

s1

b

s2

s3

F3

F9 CR

_

TAB

F2

FOLDING

observable process

unknown structure (e.g., mental model)

?

main menu level

module level

routine level

How to Extract the User’s Mental Model?


How to measure complexity? In Computer Science... • algorithmic information (Solomonoff-Kolmogorov-Chaitin) • computational universality • computational time/space • according McCabe in graph theory In Physics... • thermodynamics potentials • long-range order • long-range mutual information • self-similar structures • thermodynamic depth • logical depth In Psychology... • properties of objects (e.g. valence) • properties of attributes (e.g. ordinality) • properties of cognitive structure (e.g. centrality)

http://www.idemployee.id.tue.nl/g.w.m.rauterberg/publications/COMPLEXITY95paper.pdf


Net Complexity Metrics

McCabe (1976):

Kornwachs (1987):

Stevens, Myers and Constantine (1974):

[with P=1]

Validation study:

Ccycle from McCabe outperforms all other metrics!

state-1

state-2

transition-2transition-1

Simple Petri Net:

Ref: Rauterberg, M. (1992). A method of a quantitative measurement of cognitive complexity. In: G. van der Veer, M. Tauber, S. Bagnara & M. Antalovits (eds.), Human-Computer Interaction: Tasks and Organisation--ECCE'92 (pp. 295-307). Roma: CUD.

T = number of transitionsS = number of states

http://www.idemployee.id.tue.nl/g.w.m.rauterberg/publications/ECCE92paper.pdf


state transition net

interactive dialog system

automatic recorded process

system description

adjacency matrixfrequency matrix

the analyzing program AMME

ascii text outputfile with

quantitative measures

graphic outputfile in PostScript

format

0 1 2 3 0 0 1 0 0 1 3 0 1 0 2 0 0 0 1 3 0 2 0 7

0 1 2 3 0 0 1 0 0 1 1 0 1 0 2 0 0 0 1 3 0 1 0 1

• simulation • task-subtask

• similarity • learning • MDS

• distances • personal styles • MDS

• complexity • routine

• interface design • deadlocks

USER

"defaultp.ps"

Petri net simulator PACE

"*.str""*.log"

Path finder KNOT

Markov analyzer SEQUENZ

transformation to a syntactical correct logfile

v2132 13 S'initial_state' 15@-12

390@330 S 0 0 Tnil nil 480@420

S 1 2 nil 0 cS CSCS cSsS SSRS rS 5 tftt 10 ft

"*.net" "*.ptf" "*.mkv" "*.pro""*.ps"

any Postscript interpreter

any text processor

The AMME Program Structure

http://www.idemployee.id.tue.nl/g.w.m.rauterberg/amme/AMMEmanual.pdf

http://www.ipo.tue.nl/homepages/mrauterb/amme.html


BCcycle = T – S + 1

BCcycle

Box Plot Box Plot

1 20

10

20

30

40

*

*

*

beginners - experts1 2 3 4

0

10

20

30

40

*

*

task no.

SOURCE SUM-OF-SQUARES DF MEAN-SQUARE F-RATIO P

experience 275.521 1 275.521 10.337 0.003

tasks 259.563 3 86.521 3.246 0.032

exp. x tasks 25.729 3 8.576 0.322 0.810

ERROR 1066.167 40 26.654

Behavioral Complexity (BC) àla McCabe (1976)

Experiment:

N=6 novices; N=6 experts4 tasks with a databaseMetric BC=Ccycle

Ref: Rauterberg, M. (1993). AMME: an Automatic Mental Model Evaluation to analyze user behaviour traced in a finite, discrete state space. Ergonomics, vol. 36(11), pp. 1369-1380.


Validation of the functional equivalence, computed by the similarity ratio (SR)

Adding goal setting structure

Reconstructed mental task modelHuman mental model

?

Observation of human behaviour

Folding

Adding sequential and temporal information Model

execution

original behavioural sequence simulated behavioural sequence

Device model

2

G_2

Start menu

M_3

Main menu

F_3

User key press



MsDOS

G_2

Main menu

F_3

Start menu

M_3



User key press

MsDOS

Main menu Main menu

F-fileStart menu

F_10 M_3 h F_3 1 i hG_2

MsDOS

Main menu Main menu F-file

Start menu

F_10 M_3 h F_3 1 i hG_2

MsDOS

1

3

4

5

6

i

Validation of extracted Mental Models


SR 1 org ,tR sim, tR max orgR

sim1N

orgN

t1

simN

org

2N

* 100%

The Similarity Ratio SR

Legend: R is the absolute rank position in the original or simulated process

Ref: Rauterberg, M. (1995). From novice to expert decision behaviour: a qualitative modelling approach with Petri nets. In: Y. Anzai, K. Ogawa & H. Mori (eds.), Symbiosis of Human and Artifact: Human and Social Aspects of Human-Computer Interaction--HCI'95 (Advances in Human Factors/Ergonomics, Vol. 20B, pp. 449-454). Amsterdam: Elsevier.


d

a

F3

space

space

TAB

F2

TAB

CR

space

F9

h

original

d

a

F3

TAB

F9

h

d

a

F3

TAB

space

F2

F2

CR

F9

h

d

a

F3

space

CR

space

F9

a

F3

TAB

F9

a

...

d

h

d

h

d

h

d

h

d

a

F3

TAB

CR

CR

TAB

space

F2

F9

h

d

a

F3

space

space

CR

space

F9

h

d

a

F3

F9

a

F3

TAB

CR

space

space

F9

h

40% 77% 76% 10% 10% 10%67%79% 10% 83%

Simulated logfiles with Model-1

43%

d

h

10%

Simulation Results: Model-1

SR


d

a

F3

space

space

TAB

F2

TAB

CR

space

F9

h

original

d

a

F3

space

TAB

space

TAB

space

F2

CR

F9

h

d

a

F3

space

space

TAB

TAB

space

F2

CR

F9

h

d

a

F3

space

space

TAB

space

TAB

F2

CR

F9

h

d

a

F3

space

TAB

space

TAB

space

F2

CR

F9

h

d

a

F3

space

space

space

TAB

TAB

F2

CR

F9

h

d

a

F3

space

TAB

space

space

TAB

F2

CR

F9

h

d

a

F3

space

TAB

space

TAB

space

F2

CR

F9

h

d

a

F3

space

space

TAB

space

TAB

F2

CR

F9

h

d

a

F3

space

space

TAB

TAB

space

F2

CR

F9

h

d

a

F3

space

TAB

space

TAB

space

F2

CR

F9

h

d

a

F3

space

space

space

TAB

TAB

F2

CR

F9

h

d

a

F3

space

TAB

space

space

TAB

F2

CR

F9

h

94% 94% 96% 96% 94% 94%94%96% 94% 94% 94% 96%

Simulated logfiles with Model-4

95%

Simulation Results: Model-4

SR


Model-2 (first part): Event-driven goal setting strategy

Main menuMain menu

F-fileStart menu

F_10 M_3 h F_3 1 i hG_2

MsDOS

Syste

m le

vel

Co

gn

itive le

vel

Go

al

insta

ncia

tion

le

vel

Actio

n le

vel

task description

observable action

goal instanciation

goal selection

Model-2: event-driven goal setting

Ref: Rauterberg, M., Fjeld, M. & Schluep S. (1997). Parallel or event-driven goal setting mechanism in Petri net based models of expert decision behaviour. In: S. Bagnara, E. Hollnagel, M. Mariani & L. Norros (eds.), Time and Space in Process Control--CSAPC'97 (Sixth European Conference on Cognitive Science Approaches to Process Control, pp. 98-102). Roma: CNR.


Model-3: parallel goal setting without feedbackS

ystem

leve

lC

og

nitive

leve

lG

oa

l in

stan

ciatio

n

leve

l

Actio

n le

vel

Model-3 (first part): Parallel goal setting strategy

M_3 h F_3 1 i hG_2F_10

Main menuMain menu

F-fileStart menuMsDOS

cognitive processtask

description

goal instanciation

observable action


Go

al

insta

ncia

tion

le

vel

Actio

n le

vel

M_3 h F_3 1 i hG_2F_10

Main menuMain menu

F-fileStart menuMsDOS

Syste

m le

vel

Co

gn

itive le

vel

Fe

ed

ba

ck le

vel

Model-4 (first part): Parallel goal setting with feedback

task description

feedback

cognitive process

observable action

goal instanciation

Model-4: parallel goal setting with feedback


SR 1 org ,tR sim, tR max orgR

sim1N

orgN

t1

simN

org

2N

* 100%

Table 1: The model complexity (Ccycle) and similarity ratio (SR) of the modelingapproaches-1, -2, -3 and -4 [std:=standard deviation].

modelingapproachno. 1

modeling approachno. 2

modelingapproachno. 3

modeling approachno. 4

Ccycle: (mean ± std): 13 ± 5 43 ± 17 57 ± 25 101 ± 43

Ccycle: (min…max.): 6…18 22…68 30…97 55…170

SR (mean % ± std): 41 ± 28 66 ± 21 88 ± 11 100 ± 0

SR (min…max. %): 3…79 36…98 67…100 100…100

# simulated sequences 5*6=30 5*6=30 5*6=30 5*6=30

The Similarity Ratio SR

Legend: R is the absolute rank position in the original or simulated process


#MTT = #TST / #DS

Measuring 'Personality Styles'

#R = #AT / #DT

Measuring 'Routinization'

Ccycle = #T – #S+P with #S =< #T and P=1

C'cycle = #F–(#T + #S)+P with #S > #T and P=1

Measuring Complexity

Overview over different measures

T = number of transitionsS = number of states

F = number of connectors

TST = task solving time

AT = all used transitions

DT = all different transitions

DS = all different states


The common AI assumption

observable behaviour

mental model

learning

Ref: Rauterberg, M. (1996). About faults, errors, and other dangerous things. In: C. Ntuen & E. Park (eds.), Human Interaction with Complex Systems: Conceptual Principles and Design Practice (pp. 291-305). Norwell: Kluwer.


complex system

operator

beginner

advanced

expert

learning time

interaction

SC BC CC

We found a negative correlation between Behavior-Complexity BC and [assumed] Cognitive-Complexity CC

Experiment:

N=6 novices; N=6 experts4 tasks with a databaseMetric BC=Ccycle

The reality: what we found!


observable behaviour

mental model

The reality: how to interpret?

learning


Mental Knowledge Structures: a Metaphor

s0

d h

s1

b

s2 s3F3

F9

CR

_

TAB

F2

"wall“: knowledge about unsuccessful behavior

"dales“: knowledge about successful behavior

This conclusion would have major impact e.g. on training procedures of operators of complex systems!


Learning: the traditional understanding

Before learning phase After learning phase


Mental decision making for concrete actions is like rolling a ball between hills

Decision and Action: a new View


Learning and experience

task complexity

time-1 time-2 time-3

time

task-1 task-1' task-1''

Ref: Rauterberg, M. & Aeppli, R. (1995). Learning in man-machine systems: the measurement of behavioural and cognitive complexity. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics--SMC'95 (Vol. 5, IEEE Catalog Number 95CH3576-7, pp. 4685-4690). Piscataway: Institute of Electrical and Electronics Engineers.


2

4

6

8

10

12

14

16

18

20

task 1

week-3week-2week-1

task 1' task 1 task 1' task 1 task 1'21

22

23

24

25

26

27

28

task 1

week-3week-2week-1

task 1' task 1 task 1' task 1 task 1'

The Learning Experiment

Time structure and knowledge structure are different!

Task solving time Behavioral complexity

N=6 men (average age of 25 ± 3 years)


Conclusions

• A valid metric for task complexity based on task structure allows an objective comparison

• Automatic analysis for unconstrained task solving behavior allows analysis with applied statistics

• A new analysis and modeling approach leads to new insights…


Thank you for your attention.

Action, Complexity and Cognition Matthias Rauterberg Industrial Design Technical University...

Documents

Transcript of Action, Complexity and Cognition Matthias Rauterberg Industrial Design Technical University...