CHAPTER 52 AUGMENTED COGNITION IN HUMAN–SYSTEM...

CHAPTER 52AUGMENTED COGNITION IN HUMAN–SYSTEMINTERACTION

Dylan SchmorrowOffice of Naval ResearchArlington, Virginia

Kay M. StanneyUniversity of Central FloridaOrlando, Florida

Glenn WilsonAir Force Research LaboratoryWright-Patterson Air Force Base, Ohio

Peter YoungColorado State UniversityFort Collins, Colorado

1 INTRODUCTION 1364

1.1 Human Information Processing Limitations 1365

2 COGNITIVE STATE ASSESSORS 1367

2.1 Psychophysiological Techniques forCapturing a Cognitive State 1367

2.2 Transforming Sensors into Cognitive-StateGauges 1368

3 HUMAN–SYSTEM AUGMENTATION 1369

3.1 Augmentation Strategies 1369

4 ROBUST CONTROLLERS 1372

4.1 Control System Models 1372

4.2 Controller Analysis and Design 1375

5 APPLICATION DOMAINS 1378

6 CONCLUSIONS 1380

ACKNOWLEDGMENTS 1380

REFERENCES 1380

1 INTRODUCTIONThe fig tree is pollinated only by the insectBlastophaga grossorun. The larva of the insectlives in the ovary of the fig tree, and there itgets its food. The tree and the insect are thusheavily interdependent: the tree cannot reproducewithout the insect; the insect cannot eat withoutthe tree; together, they constitute not only a viablebut a productive and thriving partnership. Thiscooperative “living together in intimate association,or even close union, of two dissimilar organisms”is called symbiosis. . . . “Man-computer symbiosis”is a subclass of man-machine systems. There aremany man-machine systems. At present, however,there are no man-computer symbioses. . . . The hopeis that, in not too many years, human brainsand computing machines will be coupled togethervery tightly, and that the resulting partnershipwill think as no human brain has ever thoughtand process data in a way not approached bythe information-handling machines we know today.(Licklider, 1960, pp. 4–5)

Although this was written over 40 years ago,Licklider’s vision fully characterizes the current sta-tus of interactive computing and contemporary aspi-rations for its future. Historically, visionaries suchas Licklider (1960) and Engelbart (1963) suggestedthat human–computer symbiosis should augmenthuman intelligence and extend human cognitive abil-ities. Yet such intelligence augmentation has so farproved elusive for interactive system developers. Thereis a burgeoning paradigm shift in interactive computingthat has the potential to realize these visionary projec-tions; it is called augmented cognition.

Augmented cognition is a constellation of desires,concepts and goals aimed at maximizing human cog-nitive abilities through the unification of humansand computational systems (Schmorrow and McBride,2004). As Licklider (1960) suggested, human brainsand computing machines should be coupled togethervery tightly. The essence of augmented cognition isto achieve such coupling by leveraging the latest inpowerful imaging techniques that enable mapping of

1364

Handbook of Human Factors and Ergonomics, Third Edition. Edited by Gavriel SalvendyCopyright # 2006 John Wiley & Sons, Inc.

AUGMENTED COGNITION IN HUMAN–SYSTEM INTERACTION 1365

distinct and detailed functions of the brain. Specifi-cally, augmented cognition seeks to revolutionize theway that humans interact with computers by cou-pling traditional electromechanical interaction devices(e.g., mouse, joystick) with psychophysiological inter-action (e.g., eye blinks, respiration, heart rate, elec-troencephalogram), such that subtle human physiolog-ical indicators can be used to direct human–systeminteraction. Fundamental research and related technol-ogy developments in this domain have centered onleveraging data from physiological indicators to alle-viate and maximize throughput of human information-processing bottlenecks [e.g., sensory, working memory(WM), attention, executive function (EF)]. The basisfor much of this work is grounded in the view thathuman information-processing capabilities are funda-mentally the weak link in the symbiotic relationshipbetween humans and computers. As computationalprowess continues to increase, human and computercapabilities are ever more reliant on each other toachieve maximal performance. Demanding conditions,such as those associated with homeland security ormilitary operations, call for expertise not from a spe-cific human or computer system, but from a linkedhuman–machine dyad. A dyad that is functionally ahuman and their computational system, which throughshared experience and insight into how they bothfunction, will jointly deliver solutions at a previouslyunimagined rate far surpassing that of a solitary entity.

Common within a majority of augmented cogni-tion endeavors is the attempt to understand intrinsicallyhow human information processing works so that aug-mentation schemes might be developed and effectivelyexploited to enhance human processing capacity. Thus,the central vision of augmented cognition is to extendhuman abilities substantially via computational tech-nologies designed explicitly to address human infor-mation processing limitations.

1.1 Human Information ProcessingLimitationsCurrent understanding of human information process-ing suggests that information is perceived throughmultiple sensory processors. This information is then

perceptually encoded (i.e., stimulus is identified andrecognized), processed by a WM subsystem that isregulated and controlled by attention via the EF,which may be supported by long-term memory (LTM),to arrive at a decision, which in turn triggers ahuman response (Baddeley, 1986, 1990, 2000; Wick-ens, 1992). Within human information processing thereare thus several “bottlenecks” or points of limitedprocessing capacity, including sensory memory, WM,attention, and executive function.

1.1.1 Sensory Memory Bottleneck

Sensory memory is responsible for encoding informa-tion and converting it to a usable mental form (Atkinsonand Shiffrin, 1968, 1971). There is a different sensorymemory system for each of the human senses, includingvisual, auditory, tactile (haptic), olfactory, and gusta-tory. Behavioral studies suggest that human informationprocessing begins with information being perceived onaverage in about 100 ms (Cheatham and White, 1954;Harter, 1967) by one of the sensory processors. Thevisual iconic sensory memory modality has been sug-gested to have an average capacity of about 17 items,and this iconic percept is fleeting, decaying completely,on average, in about 200 ms if it does not transferto WM (Sperling, 1960, 1963; Averbach and Coriell,1961; Neisser, 1967). Audition, or echoic sensory mem-ory, is suggested to have an average capacity of fiveitems and is a bit more persistent, with the “internalecho” lasting an average of about 1.5 seconds (Neisser,1967; Darwin et al., 1972). Haptic sensory memory isvery limited in terms of capacity (Watkins and Watkins,1974; Mahrer and Miles, 2002) and has a decay ratebetween 2 and 8 seconds (Bliss et al., 1966; Posnerand Konick, 1966; Lachman et al., 1979). Little isknown about olfactory and gustatory sensory memories.In general, a considerable amount of information canbe perceived if it is allocated across multiple sensorysystems. Thus, given the limited capacity of sensorymemory, augmented cognition seeks to enhance sensoryperception by exploiting multiple sensory channels forincreased input capacity (see Table 1). Sensory stimulithat have passed the sensory memory bottleneck and are

Table 1 Tenets of Augmented Cognition

Human Information-ProcessingBottleneck Tenet

Sensory memory Augmented cognition seeks to enhance sensory perception by exploiting multiplesensory channels for increased input capacity.

Working memory Augmented cognition seeks to support simultaneous processing of competing tasks byallocating data streams strategically to various multimodal sensory systems whilemaintaining multimodal information demands within working memory capacity.

Attention Augmented cognition seeks to equip computers such that they become aware of subtlecues emanating from humans indicating how they are prioritizing incominginformation (i.e., directing attention) and capitalize on these cues to enhance humaninformation processing.

Executive function Augmented cognition seeks to enhance information processing by directing the recallof contextual information that cues the optimal interpretation of incoming informationand moderates the effects of modality switching.

1366 HUMAN–COMPUTER INTERACTION

rapidly decaying must then compete for the drasticallylimited resources of WM and attention.

1.1.2 Working Memory Bottleneck

Working memory allows people to maintain andmanipulate information that has been perceived bysensory memory and is currently available in a short-term memory store. In general, WM is described asa functional multiple component of cognition “thatallows humans to comprehend and mentally representtheir immediate environment, to retain informationabout their immediate past experience, to support theacquisition of new knowledge, to solve problems,and to formulate, relate, and act on current goals”(Baddeley and Logie, 1999, p. 29). It is considereda temporary active storage area where information ismanipulated and maintained for executing simple andcomplex tasks (e.g., serial recall, problem solving).Working memory is divided into separate processesthat are required for short-term storage (accordingto Baddeley and Logie’s (1999) model, these includethe phonological loop and visuospatial sketchpad) andfor allocating attention and coordinating maintainedinformation (i.e., the executive function).

Working memory is still being defined, and recentresearch has suggested dissociations in both the phono-logical loop (i.e., phonological store vs. articulatoryrehearsal mechanism) (Baddeley and Logie, 1999) andvisuospatial sketchpad (visual form and color recogni-tion vs. localization) (Carlesimo et al., 2001; Mendez,2001; Pickering, 2001). In general, WM is said tohave a limited capacity of about seven chunks, arapid decay rate of about 200 ms, and a recognize-actprocessing time of 70 ms, on average (Miller, 1956;Card et al., 1983). Recent research suggests, however,that presenting information multimodally can in factenhance human information processing via an increasein WM capacity, with gains on the order of threetimes Miller’s (1956) “magic number” of seven beingrealized in one recent study (Samman et al., 2004).These gains could be tempered if the costs for modal-ity switching are high; this is discussed in the next twosections.

Given separable WM components and WM capac-ity enhancements based on modality, Wickens’s (1984)Multiple Resource Theory (MRT) can be expandedto suggest that modality-based resources can be uti-lized strategically at different points in user interactionto streamline a user’s cognitive load (Stanney et al.,2004). In such a case, total WM capacity will dependon how dissimilar streams of information are in termsof modality. An expanded MRT would address how toallocate multimodal WM resources, particularly dur-ing multitasking, in such a way as to allow attentionto be time-shared among various tasks. Thus, aug-mented cognition seeks to support simultaneous pro-cessing of competing tasks by strategically allocatingdata streams to various multimodal sensory systemswhile maintaining multimodal information demandswithin WM capacity (see Table 1).

1.1.3 Attention BottleneckThree general categories of attention theories can befound in the literature: (1) “cause” theories, in whichattention is suggested to modulate information pro-cessing (e.g., via a spotlight that functions as a serialscanning mechanism or via limited resource pools); (2)“effect” theories, in which attention is suggested to bea by-product of information processing among mul-tiple systems (e.g., stimulus representations competefor neuronal activation); and (3) hybrids that com-bine cause-and-effect theories (Fernandez-Duque andJohnson, 2002). In general, attention is suggested tobe a selective process via which stimulus represen-tations are transferred between sensory memory andWM and then contributes to the processing of infor-mation once in working memory. Attention improveshuman performance on a wide range of tasks, mini-mizes distractions, and facilitates access to awareness(i.e., focused attention). In the best case, attention helpsto filter out irrelevant multimodal stimuli. In the worstcase, critical information is lost due to overload ofincoming information, stimulus competition, or dis-tractions. Thus, if one were to try to enhance WM viamultimodal interaction, such stimulation would imposea trade-off between the benefits of incorporating addi-tional sensory systems and the costs associated withdividing attention between various sensory modali-ties. Attention must thus be moderated judiciously toenhance human–computer symbiosis. Augmented cog-nition seeks to “build systems that sense, and sharewith users, natural signals about attention to support. . . fluid mixed-initiative collaboration with computers. . . an assessment of a user’s current and future atten-tion (could thus) be employed to triage computationalresources” (Horvitz et al., 2003, p. 52). Thus, with aug-mented cognition, computers will become aware of sub-tle cues emanating from humans indicating how theyare prioritizing incoming information (i.e., directingattention) and will capitalize on these cues to enhancehuman information processing (see Table 1).

1.1.4 Executive Function BottleneckThe EF system is suggested to be responsible forselection, initiation, and termination of human infor-mation processing routines (e.g., encoding, storing,and retrieving) (Matlin, 1998; Baddeley, 2003). It con-trols (i.e., focuses, divides, and switches) attention,integrates information from WM subcomponents, andconnects WM with contextually triggered informationfrom LTM. The EF is thus associated with regulatoryprocesses underlying the control of human informationprocessing and sheds light on operational costs associ-ated with these control activities (Zakay and Block,2004). The EF is thought to be especially activein handling novel situations (i.e., those with contex-tual ambiguity), such as those involving planning ordecision making, error correction or troubleshooting,novel sequences of actions or responses, danger ortechnical difficulty, or the need to overcome habit-ual responses (Norman and Shallice, 1980; Shallice,1982). When a person faces such contextual ambigu-ity during human information processing, high-level


control functions of the EF become engaged. Duringsuch processing, a person will retrieve the multipleinterpretations associated with a given uncertain situ-ation, choose the more likely interpretation based oncontext and frequency of occurrence, discard alterna-tive interpretations, and mark that point in their infor-mation representation as a choice point (Zakay andBlock, 2004). Reducing contextual ambiguity, and thuseffortful EF processing, would involve easing selec-tion among multiple interpretations by increasing thenumber of contextual cues associated with any givenalternative.

As indicated previously, frequent switching bet-ween one modality or task and another will incur a costof switching that will be associated with inhibitionsof responses to the previous modality stimuli ortask, selection and activation of the response bestassociated with the new modality or task context, andresequencing of these stimuli. Since more frequentswitching may entail greater contextual changes, it isexpected to engage effortful EF processing. Thus, itis important during modality switching to consider thecost of such contextual changes. Augmented cognitionseeks to enhance information processing by directingrecall of contextual information that cues optimalinterpretation of incoming information and moderatesthe effects of modality switching (see Table 1).

2 COGNITIVE STATE ASSESSORS

Augmented cognition seeks to enhance human–systeminteraction substantially by adopting a paradigm shiftfrom primarily passive systems dependent on userinput to proactive systems that gauge and detect, viadiagnostic psychophysiological sensors, human infor-mation processing bottlenecks and then employingaugmentation strategies to overcome these limitations.To realize this paradigm shift, one must first be ableto characterize cognitive state such that the notedbottlenecks can be monitored and regulated appro-priately. Research in psychophysiology, principallythrough brain-imaging techniques, has established acorrespondence between cognitive processors and par-ticular brain structures that have an identifiable locusin the brain. This allows use of neural signals fromthose structures as a diagnostic tool of cognitive load,which can be measured in real time while a person isengaged with an interactive system. Such psychophys-iological data streams can be used to characterize cog-nitive state, specifically current load on informationprocessing bottlenecks.

2.1 Psychophysiological Techniques forCapturing a Cognitive State

Many human–system interactive situations do notprovide sufficient human performance informationthat can be used to infer cognitive state or whatshall herein be called an operator’s functional state(OFS). This is especially true of highly automatedsystems, which for the most part put the human ina monitoring role (Byrne and Parasuraman, 1996).Because system monitoring does not require overt

behavioral responses, it is difficult to assess user state.Thus, a user may not be in an optimal state at all times,and system corrections or malfunctions may not bedetected and responded to correctly. A methodologyis needed that provides accurate assessment of OFSin the absence of overt performance data and toprovide additional information when performance dataare available. Psychophysiological measures have beensuggested to fill this role.

Psychophysiological signals are always present andcan often be collected unobtrusively, thereby pro-viding a source of uninterrupted information aboutuser state (Kramer, 1991; Wilson and Eggemeier,1991; Scerbo et al., 2001; Wilson, 2002a). Correlationsbetween psychophysiological measures and OFS havebeen described (Wilson and Schlegel, 2003). Althoughthese correlations do not prove causality, they do sug-gest that psychophysiological measures can be usedto assess OFS and further, that this information can beused to modify system parameters to meet the momen-tary needs of users (i.e., cognitive augmentation viaadaptive aiding). Of the several criteria for implemen-tation of OFS driven adaptive aiding, three crucialones are that (1) significant and meaningful systemperformance improvements must be demonstrated; (2)the sensors used must be nonintrusive to a user’s pri-mary task, as this would hinder human–system perfor-mance; and (3) their use must be acceptable to users.

For widespread adoption, it must be demonstratedthat OFS assessment and aiding either (1) improvehuman performance and enhance job success for work-related applications, or (2) enhance the interactiveexperience for entertainment-based applications. Anexample of a successful application of adaptive aidingis the use of antigravity (anti-g) suits, which requirewearing additional gear that inflates at predeterminedg-levels. These suits have been proven to savelives because they can prevent g-induced loss ofconsciousness in jet pilots and have therefore met withwide acceptance.

2.1.1 Current Status

In the past, the typical approach when using psy-chophysiological measures to assess OFS was to col-lect one or more measures and demonstrate that sta-tistically significant differences exist between at leasttwo levels of task demand or human state such asfatigue. Most of this research has been conducted inthe laboratory. However, a growing body of research isexpanding into operational environments. Psychophys-iological measures have been applied successfully indriving, flight, and other test and evaluation environ-ments (Wilson, 2002a). For example, heart rate hasbeen shown to be increased significantly under highmental workload conditions compared to low men-tal workload conditions during flight (Hankins andWilson, 1988; Wilson, 2002b). Electroencephalogra-phy (EEG), a physiological measure of the momen-tary functional state of cerebral structures, providesuseful information about both high cognitive work-load and inattention (Kramer, 1991; Wilson and Egge-meier, 1991; Gundel and Wilson, 1992; Sterman and


Mann, 1995). For example, theta-band EEG activ-ity has been reported to increase with increased taskdemands (Gundel and Wilson, 1992; Gevins et al.,1998; Hankins and Wilson, 1998).

2.1.2 Current Technology for RecordingPsychophysiological Data

Numerous psychophysiological measures have beenshown to provide valuable information concerningOFS in real-world operational environments (Wilson,2002a; Wilson and Schlegel, 2003). Because of therestrictions of the operational environment, somepsychophysiological methods cannot be used. Forexample, positron emission tomography (PET), func-tional magnetic resonance imaging (fMRI), and mag-netoencephalography (MEG) are not practical OFSgauges because the associated recording equipment istoo restrictive, too large, and requires special shield-ing, among other prohibiting conditions. Even thosemeasures that are less prohibitive have drawbacks.Almost all currently available, operationally usefulpsychophysiological sensors require contact with auser’s body and use some form of electrolyte sen-sors. This is the case for EEG, electrocardiography(ECG), electrooculography (EOG), and electromyog-raphy (EMG). Users typically do not like to wear suchsensors and associated equipment. Further, the sensorsare usually attached to the skin with some type of adhe-sive, and repeated application in a day-to-day oper-ational environment may cause skin irritation. Thereare less invasive options, such as pupillometry andeye point of regard, which are typically recorded withhead-mounted sensors. Off-head eye point of regarddevices are available but they restrict head movement,which limits their applicability in real-world environ-ments.

2.1.3 New Sensor Technologies

New sensor technologies promise to provide users withmore acceptable recording methods and valuable OFSdata. Sensors that require only “dry” (no electrolyteor adhesive) contact with the skin have been devel-oped (Kingsley et al., 2002; Trejo et al., 2003). Twoapproaches that are being explored for dry EEG sen-sors are capacitive coupled and optical sensors. Thesetechnologies can also be used to record ECG, EMG,and EOG. Currently, EEG can be recorded from non-hairy skin areas such as the forehead, but the goal isto be able to record EEG from anywhere on the scalpusing these sensors. Eye activity can be recorded usingvideo cameras that image the face from a distance,requiring no actual contact with users. Additionally,new sensor technology has been developed that pro-vides measures of brain activity using blood flow tech-nology. For example, functional near-infrared (fNIR)sensors provide information about brain oxygen levels,cortical blood volume, and neuronal activity.

2.1.4 Functional Near-Infrared Sensors

Using near-infrared light emitters, near-infrared energycan be directed through the scalp and skull and

Detected lightDetected light

Incidentlight

Incidentlight

~3 cm~3 cm

Figure 1 Locations of the infrared emitter and detectorarea important to ensure that cortical tissue is imaged.(From Downs and Downs, 2004.)

reflected from underlying cerebral tissue. Two types ofcerebral information can be obtained from fNIR. Thefirst type is hemodynamic response, reflecting oxy-hemoglobin and deoxyhemoglobin concentrations inthe brain. The consensus is that increased brain activ-ity results in increased levels of local oxyhemoglobinand decreased levels of deoxyhemoglobin (Gratton andFabiani, 2001). These responses have been used toinvestigate cognitive activity (Hock et al., 1997; Vill-ringer and Chance, 1997; Takeuchi, 1999). The secondtype of information that can be obtained from fNIR isto detect changes in the optical characteristics in braintissue that are related to neuronal activity (Grattonand Fabiani, 2001). The exact cause of these opticalchanges is not totally understood. This latter methodis said to provide millisecond temporal resolution; thefirst method is much slower. For either procedure theinfrared emitters and sensors have only to touch thescalp rather than being affixed to it (see Figure 1).The emitter–sensor unit can be held in place usinga strap or cap arrangement. fNIR systems have beendeveloped that function on hairy areas of the scalpand so are not restricted to the forehead region. Thisdeveloping technology holds a great deal of promisefor advancing our understanding of cognition and maybe used more readily in operational environments thansensor technologies that require adhesives.

2.2 Transforming Sensors intoCognitive-State Gauges

To be useful, real-time assessment of cognitive activ-ity using psychophysiological measures must be trans-formed from individual measures to cognitive gauges.Whereas consideration of individual measures providesvaluable information, augmented cognition requiresgauges that are composite estimates characterizing thefunctional state of a user (such as those to gaugeload on the human information-processing bottlenecks,as well as others, such as Kolmogorov entropy ofEEG signals and task load, which are mentioned inSections 3.1.3 and 3.1.4). Given the complexity inher-ent to most operational environments, it is not suf-ficient simply to be aware that statistical changesexist in several measures. Measures or gauges mustbe able to characterize the functional state of a usersuch that this information can be used to implement


adaptive aiding (i.e., triggering of augmentation strate-gies) in real time in real-world situations. In 2003, theU.S. Defense Department Defense Advanced ResearchProject Agency (DARPA) conducted a technologyintegration experiment (TIE) with various psychophys-iological sensors (i.e., EEG, event-related potential,fNIR, pupil dilation, heart rate variability, arousal, gal-vanic skin response) to demonstrate the feasibility ofsimultaneous data collection (Morrison et al., 2003).The TIE demonstrated that real-time computation ofsensor data to produce online gauge information wasfeasible, and further confirmed that several sensor tech-nologies could be combined with minimal interference.However, substantial variability between human par-ticipants in gauge sensitivity suggested the need foradditional research. Additional research also needs tobe focused on how to transform sensors to specificOFS gauges, such as gauges to measure the load on thehuman information processing bottlenecks. Thus, aug-mented cognition seeks to leverage a set of psychophys-iological gauges that allow for real-time assessment ofcognitive state, particularly current load on informa-tion processing bottlenecks, which can then be trans-formed directly into computer control commands fortriggering implementation of augmentation strategies.

3 HUMAN–SYSTEM AUGMENTATIONIn Section 1, various human information processingbottlenecks were discussed (i.e., sensory memory,WM, EF, attention). In Section 2, means of gaugingthe current cognitive load on a person were considered.Augmented cognition seeks to overcome the notedpoints of limited capacity processing through theutilization of human–system augmentation strategies,which will be triggered by cognitive state gauges.It is suggested that through augmentation strategies,the cost of these bottlenecks (e.g., degraded humanperformance due to overload, underload, stress, lossesin situational awareness, or emotional state) can beovercome.

3.1 Augmentation StrategiesIn conventional human–system interaction, an exces-sive amount of cognitively demanding tasks can beimposed on a user. In such situations, human infor-mation processing can break down at any of thebottlenecks. Instead of overloading users, interactivesystems should seek to achieve cognitive congenial-ity (Kirsh, 1996) by (1) presenting an optimal levelof task-relevant information and ensuring that it isreadily perceived, (2) minimizing cognitive load onWM by sequencing and pacing tasks appropriately,and (3) reducing the number and cost of mental com-putations required for task success by delegating taskswhen appropriate. Taken together, these strategiesshould increase the speed, accuracy, and robustnessof human–system interaction. Each of these augmen-tation strategies (i.e., task presentation, sequencing,pacing, and delegation) is discussed below. It shouldbe noted that other such strategies can and should beidentified. Additional augmentation strategies to con-sider include but are not limited to techniques for

supporting information filtering and triage, multitask-ing, mixed-initiative interaction, and context-sensitiveinteraction (Horvitz et al., 2003).

3.1.1 Task PresentationWhen designing interactive systems, a central questionis which information should be conveyed via whichmodality. Conventional interactive systems presentinformation to users primarily via visual cues, some-times offering auditory accessories. Yet to optimizesensory processing, thereby relieving the sensorymemory bottleneck, one should consider the types ofinformation each modality is particularly suited to dis-play. Table 2 presents theorized suitability of sensorymodalities for conveying various information sources.In addition to suitability, one must consider capac-ity. As aforementioned, Samman et al. (2004) demon-strated that multimodal WM capacity can reach lev-els nearly three times that of Miller’s (1956) magicnumber seven. Thus, rather than overloading a singlemodality, by distributing information across multiplemodalities the WM bottleneck can be relieved. Table 3represents the WM capacity of various modalitiesbased on several studies (Bliss et al., 1966; Sullivanand Turvey, 1974; Smyth and Pendleton, 1990; Kelleret al., 1995; Livermore and Laing, 1996; Woodin andHeil, 1996; Feyereisen and Van der Linden, 1997;Matsuda, 1998; Jinks and Laing, 1999; Laska andTeubner, 1999; Frenchman et al., 2003). The numbersin Table 3 suggest the upper limit on the number ofitems that should be presented via each modality, asindividual modality capacity tends to decline duringmultimodal multitasking even though overall capacityincreases (Samman et al., 2004). Thus, with knowl-edge of the information sources constituting a givenapplication, a determination of optimal modalities canbe made to direct multimodal task presentation. Morespecifically, after characterizing a given application’sinformation sources via a task analysis, first a match-ing to the optimal modality can be determined usingTable 2. Then, given the outcome of the related OFSgauges (i.e., current load on sensory and WM bot-tlenecks), a determination of reserve capacity can beestimated using Table 3 and a selection of the optimalmodality made (i.e., the one with the best match fromTable 2 and adequate reserve capacity). The appliedimplication is that in cognitively demanding task envi-ronments, not only should information be presented ina modality that is most suitable but also in one that isnot currently fully loaded, thereby easing the sensorymemory and WM bottlenecks. Thus, the first augmen-tation strategy is to identify the optimal modality bywhich to present information based on considerationof suitability principles as well as current psychophys-iological measures of cognitive load (see Table 4).

3.1.2 Task SequencingOnce the modality by which to present an infor-mation source is determined, the information eventcan be scheduled. The MRT (Wickens, 1984) sug-gests that people are more efficient in time-sharingtasks when different resources are utilized in terms of


Table 2 Theorized Suitability of Modalities for Conveying Various Information Sourcesa

Sensory Modality

Information Source Visual Verbal Tactile Kinesthetic Tonal Olfactory

Spatial acuity (size,distance, position)

++ � + + −−2D localization

(absolute/relativelocation in 2D)

++ + + � + −−

3D localization(absolute/relativelocation in 3D)

� + � + −−

Change over time ++ + + � −−Persistent attention ++ −− ++ −−Absolute quantitative

parameters++ + −− −− −− −−

Temporal (e.g., duration,interval, rhythm)

� + + ++ −−Instructions + ++Rapid cuing (e.g., alerts,

warning)+ + ++ ++

Surface characteristics(e.g., roughness,texture)

+ ++

Hand–eye coordination(e.g., objectmanipulation)

++

Memory aid (e.g.,recognition of a formerlyperceived object)

+ + − − ++ ++

Affective or ambientinformation

� � + ++

a Key: ++, best modality; +, next best; �, neutral; −, not well suited, but possible; −−, unsuitable.Source: Adapted from ETSI (2002).

Table 3 WM Capacity ofVarious SensoryModalities

WM Subsystem Capacity

Visual 2–5Verbal 4–7Spatial 5–7Tactile 3–5Kinesthetic 3–5Tonal 4–6Olfactory 3–4

sensory stimuli modality (e.g., visual, auditory), WMprocessing codes (e.g., spatial, verbal), and responsemodality (e.g., vocal, manual). For example, vari-ous studies have suggested that a person can recallmore in two tasks with different types of materi-als combined than in a single task, especially if themodalities or types of representation are very differ-ent (Klapp and Netick, 1988; Penney, 1989; Baddeley,1990; Cowan, 2001; Sulzen, 2001). More recent MRTefforts have suggested that task interference can beminimized by leveraging opposite ends of four taskdimensions, including processing stages (perception,

cognition, response), perceptual (sensory), modality(visual, verbal, spatial, tactile, kinesthetic, tonal, olfac-tory), visual processing channels (focal, ambient),and WM processing codes (spatial, verbal) (Wickens,2002). An applied implication of this theory is thattime sharing of tasks should be more effective withcross-modal as compared to intramodal informationdisplays. Thus, through systematic sequencing of tasks,simultaneous processing of competing tasks can beallocated strategically across various multimodal sen-sory systems in an effort to maintain multimodalinformation demands within WM capacity. Beyondaddressing the WM bottleneck, this augmentation strat-egy can assist in prioritizing incoming information bysequencing cues according to priority, thereby direct-ing attention. When applying this strategy, it is essen-tial to ensure that there is a means to avoid the adap-tive state from oscillating too frequently. This can bedone through the application of robust controllers (seeSection 4). Through systematic control of the adaptivestate, this strategy also addresses the EF bottleneck bymoderating the effects of modality switching.

To determine task sequencing (i.e., ordering andcombining of tasks), a conflict matrix could be calcu-lated following Wickens’s (2002) approach, in whichthe amount of conflict between resource pairs for


Table 4 Augmentation Strategies

AugmentationStrategy Description

Human Information-ProcessingBottleneck Addressed

Task presentation Identify optimal modality by which to presentinformation based on consideration of suitabilityprinciples and current psychophysiologicalmeasures of cognitive load

Sensory and working memory

Task sequencing Assign modalities to information sources andschedule them, considering priority, such thatthey minimize interference over the performanceperiod while leveraging robust controllers tomoderate effects of modality switching

Sensory and working memory,attention, executive function

Task pacing Provide external pacing of tasks, which could beachieved by monitoring behavioral entropy

Working memory, attention,executive function

Task delegation Direct assisted explicit task delegation based onpsychophysiological indexes of task load

Attention, executive function

task couplings is determined. This calculation fac-tors in both conflict and task difficulty (i.e., resourcedemands), resulting in a task interference value. Thiscould be done in conjunction with a time-line anal-ysis (Sarno and Wickens, 1995), which calculatesresource demand levels of time-shared tasks over thetime during which the tasks are to be performed. Inallocating resources, these principles could be cou-pled with a scheme of task priorities (as derivedthrough an a priori task analysis), which taken togethercould guide task ordering and combining given cur-rent resource constraints (i.e., task interference valuesand OFS gauge outputs from all four bottlenecks).The second augmentation strategy is thus to assignmodalities to information sources and then schedulethem, considering priority, such that they minimizeinterference over the performance period while lever-aging robust controllers to moderate the effects ofmodality switching (see Table 4). This should helprelieve the sensory, WM, attention, and EF bottle-necks.

3.1.3 Task PacingTime management is an essential component of manydynamic task situations (and is also critical to feed-back stability of closed-loop systems, see Section 4).Yet, in cognitively demanding task environments, pac-ing skills can decline rapidly, as temporal judgmentsdepend on the amount of attentional resources allo-cated to a temporal processor (Casini and Macar,1999). Further, internal (self) pacing has been shownvia EEG signals to impose higher human informationprocessing demands compared to externally (e.g., viametronome) paced tasks (Gerloff et al., 1998). Disrup-tion of an orderly rhythm is thought to increase theentropy of the human information-processing system,thereby increasing information content due purely toasynchronous pacing of a task. Such disruption canoccur when a person becomes overloaded with infor-mation, as this often results in delayed event detectionand more corrective responses (Boer, 2001). Interest-ingly, Boer (2001) developed a simple but highly pre-dictive linear model based on Wickens and Hollands’s

(2000) MRT, which predicted the effect of varioustasks on steering entropy and driver performance. Themodel demonstrated that steering entropy was affectedprimarily by loading of spatial tasks, as would be pre-dicted by MRT because driving is a highly spatial task.Thus, to achieve effective time management, a poten-tial augmentation strategy would be to provide externalpacing of tasks, which could be achieved by monitoringbehavioral entropy (see Table 4). Specifically, the Kol-mogorov entropy (K-entropy) of EEG signals can beused to assess information flow (Pravitha et al., 2003).K-entropy is proportional to the rate at which infor-mation about the state of a dynamical system is lostin the course of time. This entropy index has beenshown to fluctuate with changes in the complexity ofhuman information processing, such as that imposedby fatigue (leading to a lesser extent of informationflow through particular brain regions) (Rekha et al.,2003) or information overload (King, 1991) whileremaining quite stable during performance of demand-ing cognitive tasks (Pravitha et al., 2003). Thus, usingK-entropy of EEG signals to direct task pacing shouldhelp relieve the WM, attention, and EF bottlenecks, asit could help minimize pace the processing of incominginformation and minimizes disruptions.

3.1.4 Task DelegationIn the context of augmented cognition, the purpose ofdynamic task delegation would be to increase informa-tion throughput by balancing the utilization of humanresources across a network of users. Task delegationallows for distribution of task demands across indi-viduals as well as coordination between humans andautomated systems. In task delegation, certain actionsrequired by a particular task performer are delegated toanother performer or back to the system itself once taskload gets above some threshold (Dearden et al., 2000;Hoc, 2001; Debernard et al., 2002). Such handing offcan be implicit (i.e., imposing an allocation based oncurrent OFS load predictions) or explicit, in that itrequires an action from the task performer prior to allo-cation. Although implicit allocation has been shownto lead to better performance than explicit, implicit


allocation does not always meet with user acceptance,as humans like to maintain control of dynamic tasksituations and become anxious when they lose con-trol (Hock et al., 2002). This, in turn, could affectbehavioral entropy, thereby affecting system pacing.Taken together, this could affect system stability prop-erties negatively (see Section 4). Assisted explicit allo-cation is a compromise, where after detecting an over-load using an OFS gauge of task load, such as thetask engagement index used by Prinzel et al., (2000,2003), the interactive system would make an alloca-tion proposal which the human would be able to vetobut would not be in charge of allocating. This coopera-tive task allocation strategy generally leads to effectiveperformance while avoiding complacency by requiringthe human to cooperate in the allocation process. Thus,a fourth potential augmentation strategy would be todirect assisted explicit task delegation based on psy-chophysiological indexes of task load (see Table 4).This should help relieve the attention and EF bot-tlenecks, as it eases the need to determine what toattend to.

4 ROBUST CONTROLLERSAlthough augmentation strategies have the potentialto enhance human performance through reducing theload on human information processing bottlenecks,they could also lead to an adaptive state that oscillatestoo frequently, thereby destabilizing human–systeminteraction over time. Thus, there is a need to identifytechniques for ensuring that changes requested throughthe augmentation strategies are implemented so asto maintain system stability and enhance humanperformance. Mathematical system theory deals withthe modeling, analysis, and design of complex dynamicsystems. Robust control theory is a discipline ofmathematical system theory that is concerned withthe analysis and design of feedback controllers forsituations where there is only partial or incompleteknowledge of the underlying system dynamics. Inthe work discussed in this chapter, whereby a user’sdisplay/input is adapted based on his or her measuredcognitive load, it is important to note that a feedbackloop is being closed around the human. Moreover,since the underlying system dynamics involve thehuman, it is certainly true that only partial knowledgeconcerning a user’s state will be available, hence theneed for this section on robust control.

4.1 Control System ModelsRecent developments in the field of cognitive neu-roscience have heralded a great deal of change inwhat is known about human mental operations (Posnerand DiGirolamo, 2000). As has been discussed, theseadvances have the potential to allow psychophysi-ological indicators to direct human–system interac-tion (Farwell and Donchin, 1988). The ability to usesensors to measure the cognitive performance of a userimmediately through psychophysiological characteris-tics, and virtually instantly adapt a system to meet userneeds, presents an exciting new paradigm in interactivesystems. The introduction of such real-time adaptive

aiding offers the prospect of radically altering howhumans interact with computer technology. However,one important aspect of such a potential change in thenature of human–system interaction is the inherent dif-ference between open- and closed-loop systems.

Even well-understood, stable open-loop systemswill show very different performance under closed-loop operation. A simple example of this effect canbe seen when bringing a speaker and a microphone(connected to each other) too close together. A well-known audio feedback effect occurs as the signalfrom the speaker runs through the microphone, backout of the speaker, back into the microphone, andso on. The resulting feedback loop is (typically)unstable and produces a familiar (and unpleasant)sound. The volume of this sound may grow ordecay (corresponding to unstable and stable feedbacksystems, respectively), depending on the proximity ofthe microphone to the speaker (which implicitly setsthe loop gain in the feedback system). Thus, twoperfectly well-behaved open-loop systems (speakerand microphone) may or may not be closed-loopstable, depending on how feedback is applied. A moreprecise quantitative example of such behavior for anaugmented cognition system will be provided later,where it is shown that a stable open-loop systemmay generate a stable or unstable closed-loop system,depending on how feedback is designed.

Although a great deal about human performancemay be understood, the nature of the shift from anopen- to a closed-loop system is a unique type ofchange. As a result, many standard predictable aspectsof cognitive and motor performance may operate indrastically different ways in closed-loop systems. Aprime candidate for understanding such closed-loopcircumstances is through the use of engineering controlsystems theory. [For a discussion of the pros and consof various types of models, see Baron et al. (1990).]Control systems theory deals with fundamental prop-erties of systems as described (typically) by mathe-matical models. It provides a framework and tools foranalyzing fundamental system properties, such as per-formance, noise rejection, and stability, and offers sys-tematic approaches for designing systems with thesedesired properties.

The idea of applying control theory to humans hassome history, with Wiener (1948) widely consideredto be the first person to draw parallels between controlsystems in machines and the organization presentwithin some living systems. However, few attemptshave been made to apply control systems theoryto human–system interaction (Flach, 1999; Jagacinskiand Flach, 2003; Young et al., 2004), and thus thisis an exciting area of research where much remainsto be done. One notable exception that the currenteffort draws from is Card et al.’s (1983) Model HumanProcessor (MHP). The MHP is a human information-processing model consisting of a basic block diagraminterconnect model of a human, with an associatedestimate of the time taken by each processing stageto process relevant data. For augmented cognitionpurposes, the three most relevant stages (i.e., blocks)


are probably the perceptual, cognitive, and motorprocessors. This is illustrated in Figure 2, which showsa human operator piloting a vehicle. In this example,information from the operator’s system display wouldfirst pass through the operator’s perceptual (i.e.,sensory) processor, being perceived, on average, inabout 100 ms (Cheatham and White, 1954; Harter,1967). Perceived information would then be availableto the cognitive processor, which has an averagecycle time of 70 ms. The cognitive processor wouldthen make a decision, and that decision would beimplemented by the motor processor, which has anaverage cycle time of 70 ms, with a resulting actionon the vehicle controls. Note that these three blocksprovide an internal model of the operator’s interactionwith the external vehicle displays and controls. Thisblock diagram model not only characterizes the flowof information and commands between the vehicleand operator but also enables us to access theinternal state of the operator at various stages in theprocess. This allows modeling of what an augmentedcognition system might have access to (internal to thehuman; e.g., load on human information processingbottlenecks) and how those data might be used to directclosed-loop human–system interaction.

If one considers a control systems model incorpo-rating the flow of human information processing, thetime taken by each block adds time delay to the model.However, it does much more than that. As indicated

in the early discussion on bottlenecks, it also impliesa certain bandwidth for the system, both in terms ofchannel capacity and because signals that vary morerapidly than the time constant of the system (i.e., high-frequency signals) do not pass through it. Hence theprocessing blocks act as low-pass filters, only allowingthrough signals that are below the system bandwidth.For example, humans do not generally perceive theflicker on a computer monitor because it typicallyoccurs at a frequency (100 Hz) higher than that of theperceptual processor’s bandwidth of only about 10 Hz.As a first attempt at modeling such phenomenon, theeffects of time lags in human perceptual, cognitive, andmotor processing blocks are considered. This results ina dynamic model of the form shown in Figure 3.

Note that the setup depicted in Figure 3 is ageneric dynamic model of any one of the MHPcomponents (perceptual, cognitive, motor) shown inFigure 2 (although the model parameters will bedifferent for each). The dynamic models associatedwith each MHP component (“first-order lag” and “timedelay”) of the block model are given, respectively, inthe time domain (i.e., convolution representation) as

y(t) = 1

τ

∫ t

0e−(t−γ)/τu(γ) dγ

z(t) = y(t − τ)

Sensory andPerceptualProcessing

CognitiveProcessing

MotorProcessing

Figure 2 Human information-processing model.


First -OrderLag

Time Delayu(t) y(t ) z(t )

Figure 3 Block model for each component.

for each processing block [with overall input u(t) andoutput z(t)], with the time constant τ taken from therelevant processing time in the MHP model. The first-order lag models the dynamic relationship betweeninput and output signals, which captures the bandwidtheffect described earlier. This is most easily seen usingthe Laplace transform to transform this model fromthe time domain to an equivalent frequency-domainrepresentation:

Y (s) = G(s)U(s)

where the function G(s) is given as

G(s) = 1

1 + sτ

This is known as the transfer function of the sys-tem. [See Phillips and Parr (1999) for an overview oftransform methods for signals and systems; see Ogata(2002) for an overview of the application of these tech-niques to dynamic systems and feedback control.] Akey point is that the time-domain convolution opera-tor has been transformed into a simple multiplicationoperator in the frequency domain. That multiplicationoperator, G(s), is both complex-valued and frequency-varying. The function G(s) captures the frequencyresponse of the system in both magnitude and phase.

To see this, one can evaluate the transfer functionalong the imaginary axis, that is, substitute s = jωinto the model (equivalent to specializing the Laplace

transform to a Fourier transform) to yield

G(jω) = 1

1 + jωτ= 1√

1 + (ωτ)2e−j tan−1 ωτ

which is the frequency response of the system (withω the real-valued frequency). This has the desiredlow-pass frequency response. Low-frequency (slowlyvarying) signals pass through almost unattenuated, buthigher-frequency (rapidly varying) signals are moreand more attenuated until hardly any of the signalpasses through the system at all. This variation of themagnitude response with frequency in the first-orderlag block is what accounts for the computer monitoreffect (i.e., lack of perceiving flicker) described earlier(one could not account for this effect with a time-delayblock alone because the frequency response of a puretime delay is flat, i.e., no variation of magnitude withfrequency).

Note that this magnitude response comes with anassociated phase response. Low-frequency signals passthough this system with almost undistorted phase.However, as frequency increases, the signals start toincur phase lag, which ultimately reaches 90◦ at highfrequency. Phase lag has a destabilizing effect onclosed-loop feedback systems, so understanding therelationship between magnitude and phase of differentfrequency signals as they pass through the system is ofcrucial importance in designing any feedback controlsystem.

These various steps have provided the separatepieces necessary to build a model of an entire open-loop system. Since transfer functions operate bymultiplication, models for the individual blocks can becascaded. These are linear models and therefore theycommute, so the order of cascade can be changed, andhence time delays can be accumulated into a singleblock if desired. This now provides a quantitativedynamic model for the human as illustrated in Figure 4.

PereceptionFilter

CognitionFilter

ProcessingDelay

MotorFilter

MotorDelay

Cognitive Workload Model Action Model

10.1s + 1

10.07s + 1

Tp + 2Tc = 240 ms 10.07s + 1

TM = 70 ms

Figure 4 Dynamic control system model of the human.


Note that, as discussed above, this model captures thegain–phase relationship with frequency, which is crucialif the model is to be used in a feedback control loop.

This model should allow accurate predictions ofopen-loop performance and other properties of thesystem to be made. However, it is important to notethat this control theory–based model is in a form thatwill also allow for prediction of how performance andproperties are modified when transforming to a closed-loop setup, which is described in the following section.

4.1.1 Augmented Cognition Closed-LoopModels

Augmented cognition aims to provide display andinformation systems that take measurements from OFSgauges, such as those described in Section 2, anduse these data to dynamically adapt human–systeminteraction. The sensor dynamics of any future OFSgauges are still to be determined, so as a starting pointsuch sensors are modeled here as simple first-orderlags with a time constant of τ = 1 second. The sensordata would be used to dynamically change inputsto a user by directing instantiation of augmentationstrategies, such as those described in Section 3. Asan example, consider an application where workloadis reduced via the task delegation augmentationstrategy (Wickens et al., 1998). In such an application,using OFS gauges to detect cognitive overload (e.g.,through a EEG-derived index of task engagement)(Prinzel et al., 2003), lower-priority tasks would beoffloaded to automated agents, with the goal ofmaintaining users working at their maximum capacity.Such a closed-loop human–system interaction modelwas implemented in the Matlab/Simulink simulationenvironment, which is illustrated in Figure 5.

Various pieces of an augmented cognition systemcan be seen in the model in Figure 5, including thehuman perceptual, cognitive, and motor processors.

Note both the OFS gauge that detects the stateof the human user (i.e., cognitive work overloadmeasurement) and the augmentation strategies (i.e.,within the PID controller) that will alter the input tothe human. The rest of the model contains task inputsto the system, displayed outputs at various points (e.g.,actual vs. measured cognitive workload), and a simplemodel of performance errors resulting from cognitiveoverload. The feedback loop being closed is nowapparent in this simulation model, which drives theneed for a systematic control theory approach.

4.2 Controller Analysis and Design

Even this simple model has already produced someimportant findings. In particular, one major findingfrom initial efforts with the model is to showhow dynamic instability can result from introducingfeedback within the system. That is to say that rapiddetection of cognitive state under high workload mightresult in input being removed, which would reduceworkload and hence information would be added,which would once again result in high workload, andthe cycle repeats. This simple illustration indicates howusers might find their display cycling rapidly throughcluttered and decluttered states as a result of changesdetected in workload. Control theory offers a means toremove such instability and optimize performance.

Figure 6 shows results from three simulations ofa task overload situation. The input to each of thesesimulations is the same: Initially, the user is fullyloaded (and making no errors), and then a step increasein workload is introduced 1 second into the simulation.This results in task overload from that point on, withsubsequent performance errors. Note that each of thesesimulations uses the same system model, so the onlydifference is how (or if) the feedback control (i.e.,augmentation strategy) is applied.

RandomTask Overload

RampTask Overload

StepTask Overload

Switch 1

Switch 2

Switch 3

0

0

1

1

Offset 1

Offset 3

Offset 2True Task Load

Task Overload

++

++ ++1

++

−

++

1 PID

1

1

0.1s + 1

1

0.7s + 1

1

s + 1

1

0.7s + 1

CompensatedTrue Task Load

CompensatedTask Overload

Perception Cognition ProcessingDelay

MotorSwitch 5

Switch 4

FeedbackControl Action

PIDController

Cognitive WorkOverload

Measurement

Actual vs. MeasuredCognitive Work True Load

Actual vs. MeasuredCognitive Work Overload

Motor Delay Dead zone Error Rate PerformanceErrors

1

0.1

Figure 5 Matlab/Simulink model of closed-loop augmented cognition human–system interaction.


0.12

0.1

0.08

0.06

0.04

0.02

00 1 2 3 4

Time in Seconds5 6 7 8 9 10

Err

or

Rate

Performance Errors-Open Loop0.12

0.1

0.08

0.06

0.04

0.02

00 1 2 3 4


Err

or R

ate

Performance Errors-Unstable Loop0.12

0.1

0.08

0.06

0.04

0.02

00 1 2 3 4


Err

or R

ate

Performance Errors-Closed Loop

Figure 6 Simulation results for an augmented cognition closed-loop dynamic model.

Starting from the left, the first plot of Figure 6shows the resulting performance errors for an open-loop simulation (i.e., with the augmented cogni-tion system disabled). As the workload of the taskincreases, the plot shows how the number of errorsquickly rises to a certain level and stays there. Thenext panel shows a poorly designed augmented cogni-tion system. This system utilizes simple proportionalcontrol; that is, the control action c(t) that reducestask workload to the user is just directly proportionalto measured overload m(t). Thus, the controller is ofthe form

c(t) = Km(t)

and the control designer simply chooses the propor-tionality constant or controller gain K , which deter-mines when an augmentation strategy (i.e., task dele-gation in this case) is to be implemented. High-gaincontrollers, with large K values, use a high-magnitudefeedback signal that tries aggressively to drive thecontrol loop to the desired point (for fast or high per-formance). If K is chosen too aggressively, however,the closed-loop system will approach (or even exceed)stability margins. In this example, the gain K is cho-sen poorly, resulting in instability of the type describedabove, with the input being reduced rapidly and thenincreased, resulting in highly fluctuating performancefrom the user. Note that the precise values of K thatdrive the system into instability depend on the specificproblem (and can be predicted accurately with con-trol theory methods), but they can certainly occur atplausible real-world values (in this example, K = 2.8).

Proportional control is what people often think ofwhen they consider feedback. A simple version is thecruise control in a car, which moves the gas pedalin a manner proportional to the difference betweenthe desired and actual speeds. However, this simplecontrol strategy can deliver only limited performanceimprovements, even when designed correctly. Forinstance, one could never get steady-state errors downto zero with this type of control. This approachis limited because it utilizes the same gain for allfrequencies (and hence all signals), so one does

not have sufficient degrees of freedom to exploitany trade-offs in the design. A very common typeof controller used in engineering applications isthe proportional–integral–derivative (PID) controller.This generates a corrective action from a measurementof the form

c(t) = KP m(t) + KI

∫ t

0m(τ) dτ + KD

dm(t)

dt

There are now three constants to be chosen (designed),KP , KI , and KD , which correspond, respectively, tothe amounts of proportional, integral, and derivativefeedback used in the closed loop. Note that theintegral action effectively includes memory and thusallows better compensation at low frequency and henceimproved steady-state performance. The derivativeaction essentially includes anticipation, which allowsfor improved high-frequency performance, resultingin better transient response and improved stabilityproperties. The overall controller has frequency-varying gain, which allows design trade-offs tobe exploited more properly. The right panel ofFigure 6 shows a functional closed-loop system usinga well-designed PID controller to deliver closed-loopstability and good performance. It is clear that evenmaximum errors never reach the level of the open-loop(automation-free) system and that they quickly dropto minimal levels (asymptotically approaching zero)without any undesirable oscillatory transient response.

4.2.1 Human Dynamics and AchievablePerformance

The first benefit of the modeling approach describedabove is that it provides some proof of concept forthe augmented cognition concept: namely, to showprecisely how an integrated system of OFS gauges,augmentation strategies, and robust controllers cancombine to augment performance. The caveat from thiswork, however, is to note that such systems need to bedesigned carefully, with a systematic control theory


approach rather than simple heuristic tuning, elseaugmented cognition may fail to fulfill its potential.

Fortunately, systematic modeling can offer assis-tance in terms of determining the nature of informationrequired and parameters necessary for driving specificOFS gauges. The types of questions that could beaddressed by this type of analysis include:

• What time constant/bandwidth is necessary fora particular OFS gauge to have a significantuseful effect (i.e., how fast)?

• What resolution is required of the OFS gauge(i.e., how accurate)?

• How much noise can reasonably be tolerated onany given measurement?

• What would additional measurements/gaugesoffer?

• What performance level could be achieved(given the above)?

These questions should be addressed in futurework in the area of modeling and analyses. Notethat both qualitative and quantitative analyses canbe carried out, and both have their uses (e.g.,qualitative analysis might steer one toward a particulartechnology, whereas quantitative analysis might allowone to design and implement it accurately). Notealso that specific scenarios can be carried out ina simulation, which would allow one to test outcertain strategies repeatedly and reliably before goingto the expense of constructing an experimental setup,including low-probability events that might not occurin an experimental setting. Furthermore, control theoryincludes powerful analysis tools that go well beyondsimple simulation to address fundamental trade-offsand limitations inherent in any feedback loop (Doyleet al., 1992).

Ultimately, the modeling strategies described inthis section would aim to predict the impact onhuman–system performance of various augmentationstrategies for changing how information is provided toa user. In addition, they have the potential to highlightareas that would receive particular benefit from suchaugmentation. Thus, overall, this work can provide thebasis for future systematic closed-loop analysis andcontroller design, bringing to bear powerful tools fromengineering control theory. The power of such analysistools is demonstrated in the next section.

4.2.2 Individual Differences and RobustnessAnalysisPreliminary robustness analysis was conducted on theclosed-loop system with PID controller modeled inFigure 5. The theory of robust control deals withsystems subject to uncertainty such as any closed-loopaugmented cognition system would be subject to due toindividual differences (among other reasons, as notedabove). Control theory provides a means of examiningwhat performance on such a system will be, ratherthan just an idealized simulation. It also allows forexamination of variation between users, since relevant

parameters in the model can be varied (e.g., speed ofthe MHP processors) to determine to what extent agiven control scheme is robust against such variations.

The theoretical tools used here to model individualdifferences were based on the structured singularvalue (SSV), or µ, and its extensions to handle realparametric uncertainty (Young, 2001). The idea is thatone first has to use linear fractional transformations(LFTs) to rearrange the problem into canonical M –�form, as illustrated in Figure 7. Here M(s) collectsall the known dynamics of the (closed-loop) system,and � is a (block) diagonal structured perturbation,which in the case of individual differences analysiswill consist of real parametric uncertainty representingvariation in the parameters of the model. Thus, thisapproach handles LFT (block diagram) perturbationsrather than handing perturbed coefficients directly in a(transfer function) model, but this apparent limitationis readily overcome, as we illustrate below.

The individual differences analysis considered vari-ations in two time constants (i.e., speed of the per-ceptual and cognitive processors). These could arisedue to variations among users, but could also be intro-duced through inaccuracies in the modeling approach.To realize this analysis, these variations were cast asa block diagram perturbation. This can be done bynoting the interconnect in Figure 8, which shows anexample of rearranging parametric uncertainty as anLFT (block diagram).

Mathematically straightforward block diagram cal-culations now reveal that the transfer function inFigure 8 is represented by

1

1 + s(τ + �τ)

M

∆

Figure 7 Canonical form for SSV analysis.

−+

−+

11s t

∆t

Figure 8 Variation in time constant as an LFTperturbation.


so that the block diagram perturbation in Figure 8actually becomes a perturbed coefficient in the transferfunction model, and to be specific it represents aperturbation in the time constant of the first-orderlag model.

This approach was applied to the closed-loopaugmented cognition system model represented inFigure 5, considering a parametric variation in thetime constants of the first-order lag models of theperceptual and cognitive processor blocks. Note thatthe motor processor block is not in the feedback loopin this scenario, so it does not affect stability andhence was not included in the robustness analysis pre-sented here. The LFT and µ analysis machinery couldthen be applied to this block diagram. The mathe-matics of this approach is quite involved and usescomputational complexity theory, complex analysis,and linear algebra among others (Young, 2001). Spaceconstraints prevent going into any kind of detailedexplanation here, but the end result of this analysiswas to give a parameter range over which (robust)stability is guaranteed. This means that no parametercombination in the allowed range can cause instabil-ity. For example, in this case one could guarantee thatno person with a combination of processing time con-stants for the perceptual and cognitive processors inthe ranges specified would cause the closed-loop aug-mented cognition system in Figure 5 to go unstable.It is important to note the power of this guaran-tee, because one cannot get such guarantees from anyamount of exhaustive simulation or testing (it is alwayspossible that a parameter combination is missed, whichcauses a problem no matter how many variations aretried).

The results of this analysis showed that both theperceptual and cognitive processor time constantscould be reduced to very small numbers (practicallyall the way down to zero), indicating that fasteruser response than predicted was no problem. Theupper limits for the perceptual and cognitive pro-cessor time constants were found to be 3.7 and 2.6seconds, respectively. Thus, it is possible for the sys-tem to go unstable with slower users. However, thedegree of robustness afforded by a PID controlleris huge. Specifically, in this example the time con-stants are a factor of more than 37 times greaterthan those nominally assumed (e.g., the MHP’s “slow-man” to “fastman” range for the perceptual proces-sor is 150 ms and for the cognitive processor is145 ms) (Card et al., 1983), meaning that tremendousvariability in the perceptual and cognitive processortime constants can be tolerated between users andtasks. These robustness analysis results were also con-firmed by a simulation model, which showed stablebehavior for all parameter variations in the rangeallowed but which could be driven unstable by param-eter combinations outside these ranges (Young et al.,2004). This individual-differences analysis serves toillustrate what could be done when an augmented cog-nition system is coupled with a systematic controltheoretic approach.

4.2.3 Robust Controller Synthesis

The control theory methods reviewed above can facil-itate the design of high-performance closed-loop sys-tems, even for systems whose dynamics are only par-tially known (Packard and Doyle, 1993; Zhou et al.,1996; Young, 2001). This is not achieved by optimiz-ing nominal performance measures as in classical opti-mal control techniques such as linear quadratic Gaus-sian/linear quadratic regulator control (Ogata, 2002).Rather, these new approaches attempt to optimizerobust performance measures utilizing techniques suchas µ-synthesis (Packard and Doyle, 1993). In this way,systems can be designed which are insensitive (orrobust) to variations in the system that are naturallyoccurring but hard to predict a priori (e.g., differencesbetween users). The mathematical machinery underly-ing such techniques is quite involved, and the asso-ciated optimization problems can be nonconvex andeven NP-hard. At first sight, such problems may appearto be intractable, and indeed, global minima usuallycannot be guaranteed. Nevertheless, practical compu-tation schemes have been developed using approxima-tion schemes such as upper and lower bounds. Theseschemes are capable of finding very good approximatesolutions in a reasonable amount of time. Moreover,there are numerically efficient implementations avail-able of the associated algorithms, usually in convenientMatlab form (Balas et al., 1991), so such designs canreadily be carried out (with the appropriate software)in a reasonable time using current computer hardware.

All this adds up to the fact that developers ofaugmented cognition systems have at their disposala number of powerful tools for robust controlleranalysis and synthesis. These theoretical techniquesoffer the potential of safely optimizing performancein an augmented cognition system while maintainingguaranteed closed-loop stability.

5 APPLICATION DOMAINS

As with any scientific discovery or technical innova-tion, there are multiple paths upon which technologycomponents will advance. Identifying specific appli-cations at the dawn of a significant advance in ourunderstanding of a field of study is problematic in thatthe assumptions on which hypothesized applicationsare based are very likely to be flawed. The assumptionsthat must be made (and that are likely to be wrong)include:

• Who will take advantage of emergent technol-ogy components?

• What components of the emergent technologywill ultimately prove most useful and robust?

• When will the various components of theemergent technology be validated sufficientlyfor incorporation into real-world systems?

• Where will the emergent technology compo-nents be found to be most useful?

• Why will the emergent technology componentsbe seen as beneficial?


• How will the emergent technology componentsbe used?

Given the challenges inherent in answering thesequestions, a good starting point is to identify potentialgeneral application domains and then extract examplesfrom these domains in hopes of describing potentialuses of emergent technology components. The gen-eral application domains likely to be affected most byaugmented cognition technology components includeoperational domains such as truck driving and powerplant operation that would benefit from real-time cog-nitive readiness and assessment capabilities; educa-tional domains, such as a scenario-based training sys-tem that can adapt in real time to trainee performance,as assessed by both overt behavior and cognitive state(as captured by OFS gauges); and clinical domainssuch as medical applications, where for example, thereal-time attention processes of children with attention-deficit hyperactivity disorder (ADHD) are monitoredand reinforcement interventions for “paying attention”are employed. The value of augmented cognition tothe operational, educational, and clinical applicationdomains should be noted; but the specific examplesreviewed below should not be assumed to be predic-tions of actual application areas.

Another way to attempt to glimpse the future ofapplications is to examine early prototypes that incor-porate the underlying science and technology of inter-est. The practice of postulating potential futures iscommon when one has demonstrated technology; for-tunately, it is not unknown at the beginnings of basicscience or the technology development process either.When he was president and CEO of Bellcore, GeorgeHeilmeier (1999), insisted that before starting off onany scientific endeavor or technology developmentproject, the following questions be addressed:

• What are you trying to do? Articulate yourobjectives using absolutely no jargon.

• How is it done today, and what are the limitsof current practice?

• What’s new in your approach, and why do youthink it will be successful?

• Who cares? If you’re successful, what differ-ence will it make?

• What are the risks and payoffs?• How much will it cost? How long will it take?• What are the midterm and final “exams” to

check for success?

Conveniently, augmented cognition as a field ofstudy was born in part through significant investmentby DARPA in the Improving Warfighter InformationIntake Under Stress Program, where advanced thoughttoward eventual application was mandatory. One canlook to the applications and prototypes generatedfrom this technology initiative to understand potentialapplications more clearly and as a guide for attemptingto postulate potential futures and to answer the two setsof questions presented at the beginning of this section.

DARPA’s Improving Warfighter Information IntakeUnder Stress Program focused initially on challengesand opportunities associated with real-time monitoringof cognitive states with psychophysiological sensors.The ultimate goal was to demonstrate the use of thisunderlying technology to increase human informationprocessing substantially in four operational militaryapplications. The applications included applying aug-mented cognition component technologies to a mili-tary driving platform, an unmanned vehicle interfaceplatform, command and control platforms, and a dis-mounted soldier platform. These operationally focusedapplications provided proof of concepts of benefits ofthe emergent component technologies; they also pro-vided testbeds for refinement and testing of new meth-ods of incorporating these technologies (the results ofmany of these efforts are available in the Proceedingsof the First International Conference on AugmentedCognition, 2005).

There are also many potential nonmilitary opera-tional applications of augmented cognition. The com-mercial sector is beginning to consider augmented cog-nition technologies for incorporation into operationalsystems. For instance, in 2002, IEEE held their sev-enth Conference on Human Factors and Power Plants,which focuses on new trends, and dedicated an entiresession to reviewing state-of-the-art augmented cogni-tion technologies and assessing the maturity level ofthese technologies. Components of interest includedmethods to detect and measure a power plant opera-tor’s workload, strategies to facilitate multitasking inmultimodal environments, and support for intelligentinterruption and context recovery. Furthermore, NASAis supporting the development of personal monitoringcapabilities to support both intelligent system automa-tion and human performance aids (Prinzel et al., 2000,2003). The demanding environment of complex mis-sions and associated dynamic information processingdemands require NASA to seek enhanced system capa-bilities and maximized human performance. Leverag-ing augmented cognition, NASA hopes to increase theability of a single human to perform numerous taskswhile maintaining the strictest margins of safety.

To date, nonoperational domains such as trainingand clinical domains have been the least affectedby augmented cognition technology. However, it isin the training area that the likelihood of successfulapplication development would seem most promising.Joseph Cohn and Amy Kruse of DARPA’s ImprovingWarfighter Information Intake Under Stress Programhave suggested that the development of an augmentedcognition system that could turn novices rapidly intoexperts would revolutionize the training community.Such a system would identify a person’s current levelof expertise and would allow the person to be guidedrapidly to heightened levels of sustained performancein a context-independent fashion. Additionally, aperson’s cognitive performance during training couldbe periodically or continuously assessed to ensurethat their training was proceeding appropriately. Cohnand Kruse’s research seeks to develop a multilevelapproach to training that capitalizes on being able to


observe patterns at both the overt behavioral leveland at a deeper structure neuro-imaging level. Theypoint to research results from the neurosciences whichindicate that activation of specific brain regions iscorrelated to novice and expert behaviors, givingevidence for a neural correlate to observed expertbehavior. Additionally, they suggest that changes inthese structures can be assessed over time to trackprogression toward “expert” neural activation. Anapplication that could characterize expert performance,identify where in the novice–expert continuum atrainee’s performance lies, and then mold the trainee’spatterns to more closely reflect an expert’s wouldrevolutionize training.

Finally, within the clinical domain one can imaginethat by leveraging augmented cognition technology,clinicians would be better able to diagnose, evaluate,and mitigate cognitive and learning decrements. Devel-opments of emergent augmented cognition technologycomponents have been least aligned with such potentialclinical applications. However, successes in the oper-ational and training domains will probably accelerateapplication developments in the clinical domain. Addi-tionally, continued investment in real-time diagnostictools by the National Institutes of Health will probablycreate a marketplace for new medical tools and asso-ciated applications that leverage augmented cognitiontechnology.

In summary, applications of augmented cognitionare in their infancy. Examples of possible applicationsof underlying technology components can readilybe imagined; however, successful instantiation andusefulness of system applications can only be guessedat. There is significant evidence to suggest thatthe technology components are ready for insertioninto mature applications, and that the operational,educational, and clinical domains have capabilitygaps that call for technology solutions offered bythe field of augmented cognition. Thus, althoughmature augmented cognition science and technologycomponents have now embarked on the path towardapplication, the only certainty along this journey isthat the applications developed will be like no othersthat have come before them.

6 CONCLUSIONS

Augmented cognition seeks to achieve Licklider’s(1960) vision of human–computer symbiosis, wherehuman brains and computing machines are tightly cou-pled, thereby achieving a partnership that surpasses theinformation-handling capacity of either entity alone.Such improvement in human–system capability isclearly a worthy goal, whether the context is clini-cal restoration of function, educational applications,market-based improvements in worker efficiency, orwarfighting superiority. Augmented cognition is anattempt to realize a revolutionary paradigm shift ininteractive computing, not by optimizing the friendli-ness of connections between human and computer butby achieving a symbiotic dyad of silicon- and carbon-based enterprises.

ACKNOWLEDGMENTSThis material is a based on work supported in partby DARPA’s Improving Warfighter Information IntakeUnder Stress Program. Any opinions, findings, andconclusions or recommendations expressed in thismaterial are those of the authors and do not necessarilyreflect the views or the endorsement of DARPA.

REFERENCESAtkinson, R. C., and Shiffrin, R. M. (1968), “Human Mem-

ory: A Proposed System and Its Control Processes,”in The Psychology of Learning and Motivation, K. W.Spence and J. T. Spence, Eds., Academic Press, NewYork.

Atkinson, R. C., and Shiffrin, R. M. (1971), “The Controlof Short Term Memory,” Scientific American, Vol. 225,No. 2, pp. 82–90.

Averbach, E., and Coriell, A. S. (1961), “Short-Term Mem-ory in Vision,” Bell System Technical Journal, Vol. 40,pp. 309–328.

Baddeley, A. (1986), Working Memory, Oxford UniversityPress, New York.

Baddeley, A. (1990), Human Memory: Theory and Practice,Allyn & Bacon, Boston.

Baddeley, A. (2000), “Short-Term and Working Memory,”in The Oxford Handbook of Memory, E. Tulving andF. Craik, Eds., Oxford University Press, New York,pp. 77–92.

Baddeley, A. (2003), “Working Memory: Looking Backand Looking Forward,” Nature Reviews: Neuroscience,Vol. 4, pp. 829–839.

Baddeley, A., and Logie, R. (1999), “Working Memory: TheMultiple Component Model,” in Models of WorkingMemory, A. Miyake and P. Shah, Eds., CambridgeUniversity Press, New York, pp. 28–61.

Balas, G., Doyle, J., Glover, K., Packard, A., and Smith, R.(1991), The µ Analysis and Synthesis Toolbox, Math-Works and MUSYN, Natick, MA.

Baron, S., Kruser, D. S., and Messick, B. (1990), Quanti-tative Modeling of Human Performance in Complex,Dynamic Systems, National Academy Press, Washing-ton, DC.

Bliss, J. C., Crane, H. D., Mansfield, P. K., and Townsend,J. T. (1966), “Information Available in Brief TactilePresentations,” Perception and Psychophysics, Vol. 1,pp. 273–283.

Boer, E. R. (2001), “Behavioral Entropy as a Measure ofDriving Performance,” keynote address delivered atDriving Assessment 2001, Aspen, CO, August 14–17;retrieved February 6, 2004, from http://www.ppc.uiowa.edu/driving-assessment/2001/Summaries/Driving%20Assessment%20Papers/44 boer edwin.pdf.

Byrne, E. A., and Parasuraman, R. (1996), “Psychophysiol-ogy and Adaptive Automation,” Biological Psychology,Vol. 42, pp. 249–268.

Card, S. K., Moran, T. P., and Newell, A. (1983), The Psy-chology of the Human–Computer Interaction, LawrenceErlbaum Associates, Mahwah, NJ.

Carlesimo, G. A., Perri, R., Turriziani, P., Tomaiuolo, F.,and Caltagirone, C. (2001), “Remembering What butNot Where: Independence of Spatial and Visual Work-ing Memory in the Human Brain,” Cortex, Vol. 37,pp. 519–537.

Casini, L., and Macar, F. (1999), “Multiple Approaches toInvestigate the Existence of an Internal Clock Using


Attentional Resources,” Behavioural Processes, Vol. 45,No. 1–3, pp. 73–85.

Cheatham, P. G., and White, C. T. (1954), “Temporal Nu-merosity, III: Auditory Perception of Number,” Journalof Experimental Psychology, Vol. 47, pp. 425–428.

Cowan, N. (2001), “The Magical Number 4 in Short-Term Memory: A Reconsideration of Mental StorageCapacity,” Behavioral and Brain Sciences, Vol. 24,pp. 87–114.

Darwin, C. J., Turvey, M. T., and Crowder, R. G. (1972),“An Auditory Analogue of the Sperling Partial ReportProcedure: Evidence for Brief Auditory Storage,” Cog-nitive Psychology, Vol. 3, pp. 255–267.

Dearden, A., Harrison, M., and Wright, P. (2000), “Alloca-tion of Function: Scenarios, Context and the Economicsof Effort,” International Journal of Human–MachineStudies, Vol. 52, pp. 289–318.

Debernard, S., Cathelain, S., Crevits, I., and Poulain, T.(2002), “AMANDA Project: Delegation of Tasks inthe Air Traffic Control Domain,” presented at the 5thInternational Conference on the Design of CooperativeSystems, COOP ’02, Saint Raphael, France, June 4–7.

Downs, T., and Downs, H. (2004), “Hairy Situations: fNIRTechnology and Novasol,” presented at the DARPA PIMeeting, Augmented Cognition: Improving WarfighterInformation Under Stress, Orlando, FL, January 6–8.

Doyle, J. C., Francis, B. A., and Tannenbaum, A. R. (1992),Feedback Control Theory, Macmillan, New York.

Engelbart, D. C. (1963), “A Conceptual Framework forthe Augmentation of Man’s Intellect,” in Vistas inInformation Handling, P. W. Howerton, Ed., SpartanBooks, Washington, DC, pp. 1–29.

ETSI (2002), Human factors (HF): Guidelines on the Mul-timodality of Icons, Symbols, and Pictograms, ReportETSI EG 202 048 v 1.1.1 (2002–08), EuropeanTelecommunications Standards Institute, Sophia Antipo-lis, France.

Farwell, L. A., and Donchin, E. (1988), “Talking off theTop of Your Head: Toward a Mental Prosthesis Utiliz-ing Event-Related Brain Potentials,” Electroencephalog-raphy and Clinical Neurophysiology, Vol. 70, No. 6,pp. 510–523.

Fernandez-Duque, D., and Johnson, M. L. (2002), “Causeand Effect Theories of Attention: The Role of Concep-tual Metaphors,” Review of General Psychology, Vol. 6,No. 2, pp. 153–165.

Feyereisen, P., and Van der Linden, M. (1997), “ImmediateMemory for Different Kinds of Gestures in Youngerand Older Adults,” Current Psychology of Cognition,Vol. 16, pp. 519–533.

Flach, J. M. (1999), “Beyond Error: The Language ofCoordination and Stability,” in Human Performance andErgonomics, P. A. Hancock, Ed., Academic Press, SanDiego, CA, pp. 109–128.

Frenchman, K. A. R., Fox, A. M., and Maybery, M. T.(2003), “The Hand Movement Test as a Tool in Neu-ropsychological Assessment: Interpretation Within aWorking Memory Theoretical Framework,” Journal ofthe International Neuropsychological Society, Vol. 9,pp. 633–641.

Gerloff, C., Richard, J., Hadley, J., Schulman, A. E., Honda,M., and Hallett, M. (1998), “Functional Coupling and

Regional Activation of Human Cortical Motor AreasDuring Simple, Internally Paced and Externally PacedFinger Movements,” Brain, Vol. 121, pp. 1513–1531.

Gevins, A., Smith, M. E., Leong, H., McEvoy, L, Whit-field, S., Du, R., and Rush, G. (1998), “MonitoringWorking Memory Load During Computer-Based Taskswith EEG Pattern Recognition Methods,” Human Fac-tors, Vol. 40, pp. 79–91.

Gratton, G., and Fabiani, M. (2001), “The Event-RelatedOptical Signal: A New Tool for Studying BrainFunction,” International Journal of Psychophysiology,Vol. 42, pp. 109–121.

Gundel, A., and Wilson, G. F. (1992), “TopographicalChanges in the Ongoing EEG Related to the Difficultyof Mental Task,” Brain Topography, Vol. 5, pp. 17–25.

Hankins, T. C., and Wilson, G. F. (1998), “A Comparisonof Heart Rate, Eye Activity, EEG and SubjectiveMeasures of Pilot Mental Workload During Flight,”Aviation, Space and Environmental Medicine, Vol. 69,pp. 360–367.

Harter, M. R. (1967), “Excitability and Cortical Scanning: AReview of Two Hypotheses of Central Intermittency inPerception,” Psychological Bulletin, Vol. 68, pp. 47–58.

Heilmeier, G. H. (1999), “1999 Woodruff Distinguished Lec-ture Transcript: From POTS to PANS.com—Transitionsin the World of Telecommunications for the Late 20thCentury and Beyond,” retrieved September 15, 2004,from http://www.me.gatech.edu/me/publicat/99trans.html.

Hoc, J. M. (2001), “Towards a Cognitive Approach toHuman–Machine Cooperation in Dynamic Situations,”International Journal of Human–Computer Studies,Vol. 54, pp. 509–540.

Hock, C., Villringer, K., Muller-Spahn, F., Wenzel, R.,Heekeren, H., Schuh-Hofer, S., et al. (1997), “Decreasein Parietal Cerebral Hemoglobin Oxygenation DuringPerformance of a Verbal Fluency Task: Inpatients withAlzheimer’s Disease Monitored by Means of Near-Infrared Spectroscopy (NIRS), Correlation with Simul-taneous rCBF-PET Measurements,” Brain Research,Vol. 755, pp. 293–303.

Horvitz, E., Kadie, C. M., Paek, T., and Hovel, D. (2003),“Models of Attention in Computing and Communica-tions: From Principles to Applications,” Communica-tions of the ACM, Vol. 46, No. 3, pp. 52–59.

Jagacinski, R. J., and Flach, J. M. (2003), Control Theoryfor Humans: Quantitative Approaches to ModelingPerformance, Lawrence Erlbaum Associates, Mahwah,NJ.

Jinks, A., and Laing, D. G. (1999), “Temporal ProcessingReveals a Mechanism for Limiting the Capacity ofHumans to Analyze Odor Mixtures,” Cognitive BrainResearch, Vol. 8, No. 3, pp. 311–325.

Keller, T. A., Cowan, N., and Saults, J. S. (1995), “CanAuditory Memory for Tone Pitch Be Rehearsed?” Jour-nal of Experimental Psychology: Learning, Memory, andCognition, Vol. 21, pp. 635–645.

King, C. C. (1991), “Fractal and Chaotic Dynamics in Ner-vous Systems,” Progress in Neurobiology, Vol. 36,pp. 279–308; retrieved February 11, 2003, from http://www.math.auckland.ac.nz/∼king/Preprints/pdf/BrChaos.pdf.

Kingsley, S. A., Sriram, S., Pollick, A., Caldwell, J., Pearce,F., and Sing, H. (2002), “Physiological Monitoring

with High-Impedance Optical Electrodes (Photrodes),presented at the 23rd Annual Army Science Conference,Orlando, FL, December 2–5.

Kirsh, D. (1996), “Adapting the Environment Instead ofOneself,” Adaptive Behavior, Vol. 4, pp. 415–452.


Klapp, S. T., and Netick, A. (1988), “Multiple Resourcesfor Processing and Storage in Short-Term WorkingMemory,” Human Factors, Vol. 30, pp. 617–632.

Kramer, A. F. (1991), “Physiological Measures of MentalWorkload: A Review of Recent Progress,” in MultipleTask Performance, D. Damos, Ed., Taylor & Francis,London, pp. 279–328.

Lachman, R., Lachman, J. L., and Butterfield, E. C. (1979),Cognitive Psychology and Information Processing: AnIntroduction, Lawrence Erlbaum Associates, Mahwah,NJ.

Laska, M., and Teubner, P. (1999), “Olfactory Discrim-ination Ability of Human Subjects for Ten Pairsof Enantiomers,” Chemical Senses, Vol. 24, No. 2,pp. 161–170.

Licklider, L. C. R. (1960), “Man–Computer Symbiosis,”IRE Transactions on Human Factors in Electronics,Vol. 1, pp. 4–11; http://www.memex.org/licklider.pdf.

Livermore, A., and Laing, D. G. (1996), “Influence of Train-ing and Experience on the Perception of Multicompo-nent Odor Mixtures,” Journal of Experimental Psychol-ogy: Human Perception and Performance, Vol. 22, No.2, pp. 267–277.

Mahrer, P., and Miles, C. (2002), “Recognition Memory forTactile Sequences,” Memory, Vol. 10, No. 1, pp. 7–20.

Matlin, M. W. (1998), Cognition, 4th ed., Hartcourt Brace,Fort Worth, TX.

Matsuda, M. (1998), “Visual Span of Detection and Recog-nition of a Kanji Character Embedded in a HorizontalRow of Random Hiragana Characters,” Japanese Psy-chological Research, Vol. 40, pp. 125–133.

Mendez, M. F. (2001), “Visuospatial Deficits with PreservedReading Ability in a Patient with Posterior CorticalAtrophy,” Cortex, Vol. 37, pp. 539–547.

Miller, G. A. (1956), “The Magical Number Seven Plus orMinus Two: Some Limits on Our Capacity for Pro-cessing Information,” Psychological Review, Vol. 63,pp. 81–97.

Morrison, J. G., Kobus, D., and St. John, M. (2003), “DARPAAugmented Cognition Technology Integration Exper-iment (TIE),” retrieved September 3, 2004, fromhttp://www.tadmus.spawar.navy.mil/AugCog Brief.pdf.

Neisser, U. (1967), Cognitive Psychology, Appleton-Century-Crofts, New York.

Norman, D. A., and Shallice, T. (1980), “Attention to Action:Willed and Automatic Control of Behaviour,” Universityof California San Diego CHIP Report 99, reprinted inM. Gazzaniga, Ed., Cognitive Neuroscience: A Reader,Blackwell, Oxford, 2000.

Ogata, K. (2002), Modern Control Engineering, 4th ed.,Prentice Hall, Upper Saddle River, NJ.

Packard, A. K., and Doyle, J. C. (1993), “The ComplexStructured Singular Value,” Automatica, Vol. 29,pp. 71–109.

Penney, C. G. (1989), “Modality Effects and the Structure ofShort-Term Verbal Memory,” Memory and Cognition,Vol. 17, pp. 398–422.

Phillips, C. L., and Parr, J. M. (1999), Signals, Systems, andTransforms, 2nd ed., Prentice Hall, Upper Saddle River,NJ.

Pickering, S. J. (2001), “Cognitive Approaches to the Frac-tionation of Visuospatial Working Memory,” Cortex,Vol. 37, pp. 457–473.

Posner, M. I., and DiGirolamo, G. J. (2000), “CognitiveNeuroscience: Origins and Promise,” PsychologicalBulletin, Vol. 126, pp. 873–889.

Posner, M. I., and Konick, A. F. (1966), “Short-Term Reten-tion of Visual and Kinesthetic Information,” Orga-nizational Behavior and Human Performance, Vol. 1,pp. 71–88.

Pravitha, R., Sreenivasan, R., and Nampoori, V. P. N. (2003),“Complexity of Brain Dynamics Inferred from theSample Entropy Analysis of Electroencephalogram,” inProceedings of the National Conference on NonlinearSystems and Dynamics, Karagpur, West Bengal, India,December 28–30.

Prinzel, L. J., Freeman, F. G., Scerbo, M. W., Mikulka, P. J.,and Pope, A. T. (2000), “A Closed-Loop System forExamining Psychophysiological Measures for AdaptiveTask Allocation,” International Journal of AviationPsychology, Vol. 10, pp. 393–410.

Prinzel, L. J., Parasuraman, R., Freeman, F. G., Scerbo,M. W., Mikulka, P. J., and Pope, A. T. (2003), Three

Experiments Examining the Use of Electroencephalo-gram, Event-Related Potentials, and Heart Rate Vari-ability for Real-Time Human-Centered Adaptive Automa-tion Design, NASA TP-2003-212442, NASA STI Pro-gram Office, Hanover, MD; retrieved February 9, 2004,from http://www.techreports.larc.nasa.gov/ltrs/PDF/2003/tp/NASA-2003-tp212442.pdf.

Rekha, M., Pravitha, R., Nampoori, V. P. N., and Sreeni-vasan, R. (2003), “Effect of Fatigue on Mental Perfor-mance: A Nonlinear Analysis,” in Proceedings of theNational Conference on Nonlinear Systems and Dynam-ics, Karagpur, West Bengal, India, December 28–30.

Samman, S. N., Stanney, K. M., Dalton, J., Ahmad,A., Bowers, C., and Sims, V. (2004), “Multimodal

Interaction: Multi-capacity Processing Beyond 7 + / −2,” in Proceedings of the 48th Annual Human Factorsand Ergonomics Society Meeting, New Orleans, LA,September 20–24.

Sarno, K. J., and Wickens, C. D. (1995), “The Role of Mul-tiple Resources in Predicting Time-Sharing Efficiency,”International Journal of Aviation Psychology, Vol. 5,pp. 107–130.

Scerbo, M. W., Freeman, F. G., Mikulka, P. J., Parasura-man, R., Di Nocero, F., and Prinzel, L. J. (2001), TheEfficacy of Psychophysiological Measures for Imple-menting Adaptive Technology, NASA TP-2001-211018,NASA Langley Research Center, Hampton, VA.

Schmorrow, D., and McBride, D. (2004), “Introduction,”Special Issue on Augmented Cognition, InternationalJournal of Human–Computer Interaction, Vol. 17, No.2, pp. 127–130.

Shallice, T. (1982), “Specific Impairments of Planning,”Philosophical Transactions of the Royal Society London,Series B, Vol. 298, pp. 199–209.

Smyth, M. M., and Pendleton, L. R. (1990), “Space andMovement in Working Memory,” Quarterly Journal ofExperimental Psychology, Vol. 42, pp. 291–304.

Sperling, G. (1960), “The Information Available in BriefVisual Presentations,” Psychological Monographs,Vol. 74, No. 11, Whole No. 498.

Sperling, G. (1963), “A Model of Visual Memory Tasks,”Human Factors, Vol. 5, No. 1, pp. 19–31.

Stanney, K., Samman, S., Reeves, L., Hale, K., Buff, W.,Bowers, C., Goldiez, B., Nicholson, D., and Lackey, S.(2004), “A Paradigm Shift in Interactive Computing:Deriving Multimodal Design Principles from Behavioraland Neurological Foundations,” International Jour-nal of Human–Computer Interaction, Vol. 17, No. 2,pp. 229–257.


Sterman, M. B., and Mann, C. A. (1995), “Concepts andApplications of EEG Analysis in Aviation Perfor-mance Evaluation,” Biological Psychology, Vol. 40,pp. 115–130.

Sullivan, E. V., and Turvey, M. T. (1974), “On the Short-Term Retention of Serial, Tactile Stimuli,” Memory andCognition, Vol. 2, pp. 600–606.

Sulzen, J. (2001), “Modality Based Working Memory,”School of Education, Stanford University, retrievedFebruary 5, 2003, from http://ldt.stanford.edu/∼jsulzen/james-sulzen-portfolio/classes/PSY205/modality-project/paper/modality-expt-paper.PDF.

Takeuchi, Y. (1999), “Change in Blood Volume in BrainDuring a Simulated Aircraft Landing Task,” Journal ofOccupational Health, Vol. 42, pp. 60–65.

Trejo, L. J., Wheeler, K. R., Jorgensen, C. C., Rosipal,R., Clanton, S. T., Matthews, B., Hibbs, A. D.,

Matthews, R., and Krupka, M. (2003), “MultimodalNeuroelectric Interface Development,” IEEE Transac-tions on Neural Systems and Rehabilitation Engineering,Vol. 11, No. 2, pp. 199–204.

Villringer, A., and Chance, B. (1997), “Non-invasive OpticalSpectroscopy and Imaging of Human Brain Function,”Trends in Neuroscience, Vol. 20, pp. 435–442.

Watkins, D. H., and Watkins, O. C. (1974), “A Tactile SuffixEffect,” Memory and Cognition, Vol. 2, pp. 176–180.

Wickens, C. D. (1984), Engineering Psychology and HumanPerformance, HarperCollins, New York.

Wickens, C. D. (1992), Engineering Psychology and HumanPerformance, 2nd ed., HarperCollins, New York.

Wickens, C. D. (2002), “Multiple Resources and Perfor-mance Prediction,” Theoretical Issues in ErgonomicsScientific, Vol. 3, No. 2, pp. 159–177.

Wickens, C. D., and Hollands, J. G. (2000), EngineeringPsychology and Human Performance, 3rd ed., PrenticeHall, Upper Saddle River, NJ.

Wickens, C. D., Mavon, A. S., Parasuraman, R., and McGee,A. P. (1998), The Future of Air Traffic Control: Human

Operators and Automation, National Academy Press,Washington, DC.

Wiener, N. (1948), Cybernetics, or Control and Communica-tion in the Animal and the Machine, MIT Press, Cam-bridge, MA.

Wilson, G. F. (2002a), “Psychophysiological Test Methodsand Procedures,” in Handbook of Human FactorsTesting and Evaluation, 2nd ed., S. G. Charlton andT. G. O’Brien, Eds., Lawrence Erlbaum Associates,Mahwah, NJ, pp. 127–156.

Wilson, G. F. (2002b), “An Analysis of Mental Workloadin Pilots During Flight Using Multiple Psychophysi-ological Measures,” International Journal of AviationPsychology, Vol. 12, pp. 3–18.

Wilson, G. F., and Eggemeier, F. T. (1991), “PhysiologicalMeasures of Workload in Multitask Environments,” inMultiple-Task Performance, D. Damos, Ed., Taylor &Francis, London, pp. 329–360.

Wilson, G. F., and Schlegel, R. E., Eds. (2003), OperatorFunctional State Assessment, Final Report, RTO-TR-HFM-104, NATO, Paris.

Woodin, M. E., and Heil, J. (1996), “Skilled Motor Perfor-mance and Working Memory in Rowers: Body Patternsand Spatial Positions,” Quarterly Journal of Experimen-tal Psychology, Vol. 49, pp. 357–378.

Young, P. M. (2001), “Structured Singular Value Approachfor Systems with Parametric Uncertainty,” InternationalJournal of Robust and Nonlinear Control, Vol. 11,pp. 653–680.

Young, P. M., Clegg, B., and Smith, C. A. P. (2004), “Dy-namic Models of Augmented Cognition,” InternationalJournal of Human–Computer Interaction, Vol. 17, No.2, pp. 259–273.

Zakay, D., and Block, R. A. (2004), “Prospective and Retro-spective Duration Judgments: An Executive Control Per-spective,” Acta Neurobiolagiae Experimentalis, Vol. 64,pp. 319–328.

Zhou, K., Doyle, J. C., and Glover, K. (1996), Robust andOptimal Control, Prentice Hall, Upper Saddle River, NJ.

CHAPTER 52 AUGMENTED COGNITION IN HUMAN–SYSTEM...

Documents

Transcript of CHAPTER 52 AUGMENTED COGNITION IN HUMAN–SYSTEM...