MetaLinks: Authoring and Affordances for Conceptual and Narrative ...

HAL Id: hal-00197317https://telearn.archives-ouvertes.fr/hal-00197317

Submitted on 14 Dec 2007

HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.

MetaLinks: Authoring and Affordances for Conceptualand Narrative Flow in Adaptive Hyperbooks

Tom Murray

To cite this version:Tom Murray. MetaLinks: Authoring and Affordances for Conceptual and Narrative Flow in AdaptiveHyperbooks. International Journal of Artificial Intelligence in Education (IJAIED), 2003, 13, pp.199-233. <hal-00197317>

https://telearn.archives-ouvertes.fr/hal-00197317

https://hal.archives-ouvertes.fr

To appear in JAIED 2003 MetaLinks

Page 1

MetaLinks: Authoring and Affordances for

Conceptual and Narrative Flow in Adaptive Hyperbooks

TOM MURRAY

Hampshire College School of Cognitive Science

[email protected], http://helios.hampshire.edu/~tjmCCS/

Abstract: MetaLinks, an authoring tool and web server for adaptivehyperbooks, is described. MetaLinks hyperbooks provide textbook-like

content in a flexible manner that supports opportunistic processes of active

reading and learning. Through adaptive methods a single on-line artifact

can address the diverse needs of a variety of user goals and backgrounds.The system is designed to support inquiry, exploratory, or curiosity-driven

learning in richly interconnected material and to ameliorate a number of

usability issues that arise in such open ended learning environments. Chiefamong the usability issues addressed are the potential for discontinuous

conceptual and narrative flow as users navigate flexibly and

idiosyncratically through hyperbook content. Our goals are addressedthrough a number of interface features and adaptive methods including

narrative smoothing, custom depth control, and thematic links. In this

paper we describe the MetaLinks system, discuss a number of hyperbook

design issues, and give the results of two formative evaluations of aMetaLinks Hyperbook.

1. INTRODUCTION

Learning and information gathering are opportunistic, dynamic processes. Because learners

have different background knowledge, learning styles, and goals, and because each learnerconstructs new knowledge in a very personal, idiosyncratic fashion, the best path through a text-

like resource may differ for each learner. Active learners and readers make moment to moment

decisions about the depth and focus of their inquiry. They decide to skim, look ahead, go backto learn prerequisite content, and go off on tangents either to investigate whether the unexplored

territory provides useful information, or just because their curiosity was sparked. Hypermedia

books (hyperbooks) have the promise of supporting this dynamic learning process more fullythan paper-based texts. The term "affordance" has been used in technology design to refer to

how visible characteristics of an artifact immediately or intuitively present themselves to the

user as having a particular purpose or function (Norman 1988). More loosely it has been used

to refer to what a particular technology or feature clearly allows the user to do. Our concern isboth with what the features of hyperbooks do afford and with what they could afford if some of

our preconceptions about books were put aside and we pushed the technology towards its full

potential.Adaptive web-based hyperbooks are a new breed of hypermedia that uses recently

available technologies to bring some of the long standing goals of hypermedia to fuller, more

dynamic realization by adding methods developed for adaptive and intelligent educationalsystems. The MetaLinks project is unique in adaptive hypermedia research in its focus on

supporting active learning, which in turn requires special support for open-ended, exploratory

reading. The transition from paper texts to hyperbooks, in addition to providing new


Page 2

capabilities, problematizes several aspects of reading and authoring. In particular, content must

be reconceptualized when it becomes dynamic, and the flow of the reading experience can be

compromised as users navigate flexibly and idiosyncratically through hyperbook content.

MetaLinks hyperbooks serve a range of user needs that differ according to backgroundknowledge, thematic and content goals, and desired level of detail, and are designed to

ameliorate the usability issues that inevitably occur in interactive books. Another goal is for the

system to be general and usable from the author's perspective, so MetaLinks includes anauthoring tool for building adaptive hyperbooks.

In the remaining sections of this paper we will describe the MetaLinks system, discuss key

design issues, and discuss the results of the two formative evaluations.1

2. BACKGROUND: HYPERMEDIA, BOOKS, AND EDUCATIONAL SOFTWARE

Because hypermedia and electronic books cover such a wide array of artifacts and research

concerns, below we situate our work within the broader context of texts, hypermedia, and

educational software.Hyperbooks vs. other hypermedia. Our working definition of hypermedia is the standard

notion of chunks of content (nodes) with hyperlinks between them (Conklin 1987). Hyperbooks

are electronic books with substantial hypermedia functionality. Hypermedia technology isblurring the distinctions between textbooks, reference books, and databases, and making

possible new forms of documents that are combinations of and extensions to these traditional

forms. Hyperbooks can contain alternative structures, content, and navigation paths that

emphasize different goals, skill levels, or perspectives (Conklin 1987, Ferguson et al. 1992;Spiro & Jeng 1990; Cleary & Bareiss 1996). The focus of this research is on hypermedia "text

books," i.e. quasi-narrative content authored for instructional or illustrative purposes. Though

our focus is not on literature, entertainment books, artistic texts, encyclopedia-like informationsources, or on unstructured multi-author collections of web pages, many of our methods and

conclusions will be relevant for these other forms of texts and hypermedia.

Hyperbooks vs. other educational software. Even though educational software can offer

the benefits of video, sound, animations, simulations, assessed activities, personalized feedback,and interactive learning environments, our focus is on more mundane yet still essential elements

of authoring and reading electronic texts. Hyperbooks can contain these other features, but

currently the MetaLinks research project deals with issues more central to the active reading,searching, and browsing activities that take place while learning with hyperbooks. While

reading and browsing are mundane activities compared to the possibilities inherent to computer-

based learning environments, there are important research issues related to these limitedactivities. Reading and browsing comprise the vast majority of the time spent in educational

multimedia and in educational uses of the WWW, and thus our work has some relevance for

using the web and educational CD-ROMs in general.

Hyperbooks and adaptive/intelligent educational software. The vast majority of web-delivered educational software is non-adaptive or "static." Though designers of static

hypermedia try to incorporate a variety of content and links to meet the needs of many user

goals, these one-size-fits-all systems can suffer under the weight of trying to address too manyneeds. Though non-adaptive hypermedia books have been designed and studied for about 30

years (Nelson 1990) it has only been recently that internet data-base technologies that support

the dynamic configuration and personalization of web pages have become common. Thisadvance has allowed technologies and approaches developed for intelligent tutoring systems to

migrate to the field of educational hypermedia resulting in adaptive hyperbooks (Brusilovsky

1998). Adaptive hypermedia documents are composed "on the fly" so that the content, style,

and/or sequencing of the pages and links is customized to the needs of the particular learner andsituation.

1 This paper combines and extends work reported in (Murray et al. 1998, 1999, 2000, and 2001; Piemonte

et a. 1999; Khan et al. 2000; Shen 2000).


Page 3

The main focus of adaptive hypermedia systems to date has been to address what we could

call the "prerequisite problem" or "content readiness problem". The orderly sequencing of

content in traditional educational materials such as textbooks attempts to account for the

prerequisite relationships between topics. Adaptive educational systems address this issues byeither warning learners that they may not have the appropriate prerequisite information to move

ahead, or by automatically sequencing content to address unknown prerequisites. If the system

bases its adaptation upon a student model which makes heuristic inferences about what astudent "knows," then the system is said to be "intelligent" as well as adaptive. InterBook

(Brusilovsky et al. 1996) does "adaptive link annotation" by labeling links according to whether

they are "ready to be learned." It bases this annotation on a prerequisite-based concept networkand a student model. AHA (De Bra and Calvi 1998) includes "adaptive content" by allowing

authors to hard-code various types of prerequisite relationships into hyperbook content using

HMTL tags. The system can hide or disable links, and hide or show sections of text in its

attempt to make the content "relevant, interesting, and understandable" as determined by astudent model. For example, the author might give a particularly detailed section of text a tag

that indicates "don't show this unless topic XX is understood." ACE (Specht & Oppermannn

1998) is based on an intelligent tutoring systems framework, and includes domain, student, andpedagogical models. In addition to incorporating adaptive content and link annotation, ACE

includes teaching strategies that adjust the teaching style based on user competence and

interests. We built a student model module for MetaLinks which allowed intelligent featuresthat addressed the prerequisite problem, but, as explained later, these features were prototyped

but not yet tested on users.

Actually, the prerequisite problem was only one of many issues that earlier researchers in

hypermedia identified as being amenable to adaptive solutions. Prior research in hypermedianominally addressed adaptivity (Conklin 1987), but solutions were limited until the advent of

the modern data-base enabled and function-enabled (i.e. with Java and JavaScript) world wide

web. Current research on adaptive hypermedia has not sufficiently gone back and revisitedimportant early issues, which can now be re-analyzed in light of realistic access to adaptive

solutions. Many basic questions concerning the effectiveness of active reading using multi-path

and dynamic hypermedia books remain unanswered. As we discuss later, the MetaLinks project

differs from other adaptive hypermedia projects in its greater emphasis on active reading andopen-ended exploratory types of learning tasks, and on its emphasis on providing good

conceptual and narrative flow as learners jump unpredictably among the pages in a hyperbook.

3. METALINKS SYSTEM DESCRIPTION

Design goals. In this section we will describe the MetaLinks software. In the next section we

will go into more detailed analysis of the issues behind the design goals and features mentioned

here. Our design goals are as follows:

1. Support active reading and learning in hypermedia texts. This involves supporting anumber of cognitive goals and tactics as learners use a book in exploratory, opportunistic, and

non-linear ways. Specifically we want to create a single artifact that has the flexibility to

support:• Diverse learner backgrounds and prerequisite knowledge.

• Multiple thematic approaches and learning goals.

• Reading at multiple levels of detail or skimming.• A spectrum of text-like to reference-like content; and a spectrum of focused to

open-ended exploratory reading and information finding strategies.

2. Ameliorate a number of usability issues that arise in the dynamic open-ended

environments created to address the goals above. These problems include disorientation,cognitive overload, poor narrative flow, and poor conceptual flow.

3. Provide a highly usable authoring tool for creating and web-serving hyperbooks. This

not only improves cost effectiveness, but tests the comprehensibility and generality of theunderlying representational framework.


Page 4

3.1 Architecture

MetaLinks content is stored in a relational database (using web-enabled FileMaker Pro) and

requires only a web browser for delivery. The system includes a sophisticated authoring toolthat makes it easy to manage content, media (graphics, applets, etc.), and hyperlinks. Figure 1

shows the software architecture of MetaLinks.2

Figure 1: MetaLinks Architecture

The browser sends HTTP requests for each hyperbook page or tool to the FileMaker server.

Included in the request is the name of the HTML (really DHTML, Dynamic HTML) templatefile and information about the user and navigation behavior that allows the new page to be

customized. The HTML template includes JavaScript code and database placeholder fields that

FileMaker fills in with the appropriate data before being served. The system has separate

databases for pages (text content), page links, glossary items, media, and user data. All usermoves and tool uses are recorded in the User database. The Media data base can include

images, movies, applets, and entire external HTML files (this last feature is used to include

tables in a hyperbook). Hyperbooks are password protected through a user log-in. The loginallows us to maintain the user's state across sessions. Upon logging in, the user can optionally

specify that some of the media is kept on a local hard drive or CD-ROM. This allows for higher

speed, as images need not be served over the Internet.

A significant amount of FileMaker and JavaScript code (over one person-year's effort)comprises the algorithms for transforming the data into adaptive web pages including main

page, glossary, table of contents, roll over and pull out menus, etc. Also, a significant effort has

gone into designing an authoring tool that supports authors in organizing and connect thevarious types of data (see below).

3.2 User InterfaceOur original grant funding was through a curriculum development grant that involved creatingweb-based Earth Science software for high school and introductory college classrooms. The

software has a full range of navigation features because of pragmatic goals, even though the

inclusion of these features was not directly relevant to our main research questions, and their

inclusion made it more difficult to analyze the data. Early observations (Khan et al. 2000)indicated that students had quite a variety of navigation styles and needs, suggesting the need

for a wide variety of navigation features.

2 The reader can try some MetaLinks hyperbooks at http://ddc.hampshire.edu/metalinks/.


Page 5

Figure 2: Tectonica Interactive page T.2.4, with a picture showing two

geologists running from a lava flow.

Figure 2 shows a typical MetaLinks hyperbook screen. From top to bottom, it contains the

navigation bar, the page title, the page text, figures, "custom depth control" navigation buttons,and a list of links to sub-topic pages. In addition to the main content window shown in the

figure, there are separate windows for the table of contents, the search tool, the glossary, and the

annotated history tool.3

Below we list the MetaLinks navigation tools, most of which are described in more detail

in the next section (and shown in Figure 5):

• Annotated Table of Contents (TOC), Pictorial TOC, Search tool

• Custom Depth Control feature (Explain More, Next Page, Return buttons)

• Go to parent, Go to next/previous sibling, Direct to page number (all in the NavBar); Go to Sub-Topic (links at the bottom of the page)

• Integrated Glossary, Glossary base page links

• Annotated History tool

• Go to Related Page ("Related Information" pop-out tab to the left of the screen)

We will briefly describe the main interface features here, and explain their function andpurpose in the next section. Green colored underlined words correspond to words in the

glossary. When the user mouses over these words the definition pops up (called "stretch text"),

as shown in the Figure. When these glossary words are clicked on, the user navigates to thehyperbook page that best explains the concept. Teal colored underlined words indicate

footnotes, which also pop up with text and (optionally) graphics. The Parent, Subtopics, and

Sibling buttons allow navigation within the hierarchical structure, based on a family tree

3 All of these are browser windows in which we hide the browser navigation buttons to force the user to

use our navigation tools.


Page 6

metaphor ("parent," "children" (subtopic), and "sibling"). The Related Information tab pulls out

to reveal a list of thematic links to other pages in the hyperbook. The Explain More button

navigates to the current page's subtopics for a deeper treatment of the topic. To the right of

"Explain More" is either a "Next" page button or a "Return" (to where I left off) button,depending on the situation.

To date, MetaLinks has been used to author four hyperbooks. Tectonica Interactive, in the

domain of introductory geology, is the largest, with approximately 400 pages, 500 graphics and750 glossary entries, its content corresponds to one third (about 200 pages) of the contents in

the original text book. The contents of the text Modern Physical Geology (Thompson & Turk

1996) were obtained in digital form from the publisher, and part of it was significantlytransformed and extended to create Tectonica. Chapters on Earthquakes, Ocean Basins, Mass

Wasting, and Geologic Resources, are entered, and chapters on basic physics and chemistry

concepts, Weathering and Soil, Mountain Ranges and Continents, Streams and Lakes, Ground

Water and Wetlands, Deserts, Glaciers, Coastlines, and Climate have not been entered. Theevaluation studies reported in this paper have all been done with Tectonica, shown in Figure 2.

The second MetaLinks hyperbook authored was the MetaLinks Users Guide. The

remaining two hyperbooks were created as part of Hampshire College service learning classes.College students in the classes used MetaLinks to build hyperbooks in collaboration with two

community based organizations. The first was Women Mathematicians which was built in

collaboration with Amherst Middle School teachers and students for use by an eight grade class(see Figure 3). The second was "Early 20th Century Children's Games," a hyperbook built in

collaboration with a group of senior citizens enrolled in a computer literacy class in Holyoke

Massachusetts. This project became somewhat of an oral history project, as seniors told us their

memories of the games they played as children, and we organized this material thematically forthe hyperbook. Both of these college-class-built hyperbooks are on the order of 30-50 pages

large. Though the results reported in this paper are based on Tectonica, by having many users

and authors involved in all four of these hyperbooks we have gained confidence in the usabilityof the system, had the benefit of user-participatory design iterations, and demonstrated its

application in several domains.

Figure 3: Women Mathematicians Hyperbook


Page 7

3.3 Authoring Interface

Authoring tools allow for cost-effective production of educational materials, and they lower the

skill or training threshold needed to author. In addition, we believe that building and testing ausable authoring tool is a good test of the generality and conceptual felicity of a knowledge

representation structure (Murray 1999). It is important that domain experts and teachers,

critical members of any design team, understand and participate in the design process. In orderto be able to do this they must understand the system's underlying conceptual framework, its

capabilities and limitations. An authoring tool reifies the underlying conceptual structure in a

system's representational framework. If an authoring tool proves to be usable andunderstandable, then this is substantial evidence that the conceptual framework is

comprehensible and general.

Figure 4: MetaLinks Authoring Tool Main Screen

MetaLinks includes a highly usable graphical interface for authoring hyperbooks. Our

representational framework has been purposefully limited to features which can be easily

portrayed in the GUI and authored by anyone with minimal training (one hour). All of thefeatures mentioned in this paper come practically "for free" when the author enters the text and

graphics, and defines links between pages. Automated features include creation of the TOC,

page numbering, layout, and glossary pop-up text. To author a MetaLinks document, onlyFileMaker and the MetaLinks code library (written in FileMaker Script, JavaScript, and

DHTML) is needed (plus any graphics programs needed to prepare figures). All of these

elements are cross platform. Figure 4 shows the main authoring screen, an application builtwith FileMaker Pro (authoring over the web is not yet supported).

Figure 4 shows data fields for a page's title and text; for its Intro-text and Question-title

(fields used in page and link adaptation); links to media items; and links to glossary items. To

create a link between two pages the author goes to one page, pushes "set FROM page," goes tothe other page, pushed "set TO page," uses the pop up list above "set TO page" to select one of

the available link types, and then presses "create link." Additional screens exist for authoring


Page 8

properties of the whole hyperbook, and for authoring the glossary. The author can include as

many figures as desired (of any media, including JPEGS, movies, Java Apps), and specify a

scaling factor for each picture. MetaLinks automates page layout while giving the author

limited control over the page layout style. The author can choose among several page stylessuch as "Front page," "chapter page," and "regular page," and can choose among several media

layouts (such as "left of text," and "below the text."). The author can also customize the

background image for the entire page.MetaLinks is one of the only hypermedia system (adaptive or not) to have a full featured

GUI authoring system. InterBook (Brusilovsky et al. 1996) addresses authoring by allowing

tagged text to be imported from word processing files. The AHA system (De Bra, P. & Calvi,L. 1998) addresses authoring by using an HTML-compatible mark-up language for specifying

prerequisite and content requirement relationships through Boolean expressions. This makes

authoring more powerful but it does not include an authoring tool, and thus does not make

authoring accessible to teachers. ACE (Specht et al. 1998) seems to be the only other adaptivehypermedia framework accompanied by a full authoring system.

4. HYPERBOOK DESIGN ISSUES, BENEFITS, AND PROBLEMS

Hyperbook design issues and innovations fall into two classes: inherent "primary" benefits andproblematized "secondary" issues (items #1 and #2 respectively in our list of design goals in

subsection 3.2). The first class of innovations takes advantage of the inherent potential of the

technology. The second class of issues attempts to deal with the problems that inevitably occur

when we create flexible media that have the features of the first class of innovations. Secondaryissues are not so named because they are less important (they are equally important) but

because their existence results from or is exacerbated by solving the primary goals. Below we

go into more depth describing the educational and cognitive issues and the underlying designrationale behind MetaLinks. First we describe several overarching design themes. Then we

look at "primary" features that support active reading and learning. Then we look at features

ameliorating the secondary issues. Finally we describe features dealing with prerequisites and

content readiness (which concerns both primary benefits and secondary issues, and is dealt within its own sub-section). Figure 5 shows how the MetaLinks features and tools address our

goals.

Secondary

Main effect

Active Reading & Learning

Primary Goals

Secondary Issues

Conceptual Flow

Authoring Tools

Mulitple Depth Levels

Multiple Themes & Persptvs

Opportunistic Exploration

Prerequisites(cont. ready)

Narrative Flow

Custom Depth Contr. & Horiz.

Reading

Thematic Links

Inquisatory Page Titles

Glossary Pop-up &

Base Page

Dual Overlay SM & Adaptive

Link Ann.

DisorientationCognitive Overload

Narrative Smoothing

Ann. TOC, Ann. History, Maps,

(& other features)

Pop-up Stretch Text

Search Tool, TOC

Focussed Search & Ref mode

Goal or Issue

Feature/Tool

KEY

.

.

Figure 5: Issues Addressed by MetaLinks Features and Tools


Page 9

General Cognitive and Usability Themes

Active reading. "Active reading" is a term used to emphasize the proactive cognitive and

metacognitive processes that, ideally, should be present in non-recreational reading. It has been

found that readers of traditional text monitor comprehension, coherence, and efficiency inrelation to their reading goals and interests, and make strategic decision about how to allocate

attention in a text (Foltz 1996). They adapt their strategy to make the text more comprehensible

in relation the existing knowledge structures. Active reading strategies include skimming (foran overview), scanning (to locate specific content), reviewing (summarizing to identifying main

points), connecting (creating meaning and relevance for new knowledge), evaluating (critiquing

and synthesizing), formulating questioning (about what is needing explanation or justification),and predicting (anticipating where the author is going) (Collins et al. 1989; Foltz 1996).

Hyperbooks can provide important tools and scaffolding for active reading behaviors. Also, by

its very nature, hypertext readers are forced to assume a more active cognitive role than readers

of traditional text because they are required to make navigational decisions as they read.MetaLinks provides organizational and cuing tools that support active reading behaviors related

to skimming, scanning, reviewing, and connecting. It does not directly scaffold metacognitive

processes (an example of which might be to remind readers to summarize or critique).4

The literature points to two types of organizational enhancements to text (Mannes &

Kintsch 1987; Foltz 1996): consistent and divergent. Outlines and advanced organizers

(Ausebel 1963) that reflect the macrostructure of the document (are "consistent" with it) appear

to help with effectiveness of remembering ideas and with efficiency of finding content.Structural cues that provide divergent, alternative, multiple, or thematic perspectives encourage

deeper processing and appear to lead to better problem solving, transfer, and better integration

of new knowledge with existing knowledge. MetaLinks provides both consistent organizationalcues (TOC and children links) and divergent organizational cues (related links). It is expected

that these and the other methods for navigation used in MetaLinks will support the diversity of

reading and information gathering strategies needed for a diversity of background knowledgeand reading goals.

Opportunistic, Exploratory, and Inquiry Reading & Learning. The vast majority of

research in adaptive hyperbooks assumes an instructional goal to cover a specific set of topics

as in traditional classrooms and academic courses. However, much of the earlier work on (non-adaptive) hyperbooks assumed or allowed a more open-ended experience. As alluded to above

in the distinction between consistent and divergent navigational cues, learner's goals in

navigating through hypermedia material vary along a spectrum from convergent or "finding"goals through divergent or "exploratory" goals (McAleese 1989; Heller 1990). MetaLinks

contains features that support both types of user goals, but our work differs from other adaptive

hypermedia projects in its greater emphasis on the support of behavior called inquiry-based,discovery, or exploratory (or divergent). Exploratory navigation is appropriate for open-ended

questions and/or learning in ill-structured domains in which the richness of the content suggests

multiple themes, perspectives, or learning paths (Spiro & Jehng 1990; Jacobson & Spiro 1995).

Also, inquiry-based learning methods involve initial stages of articulating and refining thedriving question and then exploring potential sources of information before focusing an

information search (Wallace et al. 2000). Finally, as mentioned above, all truly active reading

involves some degree of opportunistic, non-linear use of the text, even for relatively focusedlearning goals. A single MetaLinks hyperbook can serve multiple purposes: as a textbook

which is read in quasi-prerequisite order; as a reference book for focused investigations; and as

a hyperbook that can be explored in an open-ended way.

MetaLinks facilitates exploratory and inquiry navigation behavior in several ways. First,the related links feature facilitates exploring related but tangential topics. Second, custom depth

control (described later) allows the user to read the material at their chosen depth level, and

4 Actually, there is some question in the literature as to whether providing scaffolding supports or inhibits

learning metacognitive skills because the system exposes structures "for freee" that might otherwise have

to be actively inferred (Mannes & Kintsch 1987).


Page 10

easily explore any topic more deeply. Third, the inquisitory page titles (explained later) express

page relationships in terms of questions, to support a inquisitory Q&A navigation style. Finally,

to the degree that the design of the subtopic links, search, and TOC features alleviate the four

navigation "side effects" described later, the software supports maintaining orientation and flowas the user in branches out from the default navigation.

Three Epistemic Forms. Most hyperbooks are structured according to three ubiquitous

forms, simultaneously: narrative, network, and hierarchy. In addition to being common generalorganizing structures, these forms are seen throughout the literature in cognition and software

design as organizing forms for memory and thought, and as organizing forms for the design of

content and software artifacts. Collins and Ferguson (1993) call such recurring forms"epistemic forms" because they act as templates (and their associated "epistemic games" act as

procedures) that can guide and scaffold the construction of new knowledge. In our context

these forms guide both the design of educational material and the learning that subsequently

takes place. All books have aspects of these three forms (as in the table of contents, the text,and the index) but hyperbooks accentuate the network (associative) aspect and problematize the

narrative (linear) aspect. Each form has an important function. Reading is a linear activity, as

is the flow of experience and conscious thought. Thus learning has traditionally been mediatedby narrative (or episodic) structures such as spoken words and books, and we have deep

cognitive expectations about narrative continuity and structure in the learning process. Lucklin

et al. (1998, page 1) observes that "the very nature of [hypermedia can result in] adeconstruction of the narrative which is normally present in [traditional] media…The narrative

can be suspended or altered and may thwart or confuse our expectations." In addition to its

narrative and linear aspects, cognition also has non-linear network-like aspects. Many aspects

of memory, learning, and thinking are "random access," and associative, as in semanticnetworks (Woods 1975; Anderson 1983). And finally, cognition also has hierarchical aspects.

For example, Ausebel's subsumption theory of meaningful learning includes content sequencing

principles that capitalize on the specialization and generalization relationships that naturallyexist between concepts (Ausebel 1963). The interplay between there three epistemic forms

provides an essential tension from both the designer's and the user's perspective.

Our approach is as follows. We believe that completely unstructured or spaghetti-linked

content has poor usability, and that the link structure of a hyperbook's content should reflect andreify the knowledge structure of the domain (Jonassen & Reeves 1997; Ross 1993; Eklund

1997).5 MetaLinks hyperbooks have a primary hierarchical structure defined with

"parent/child" (i.e. parent/sub-topic) links, and an additional network-like structure definedusing "thematic links" (i.e. "related information links"). The limitations of imposing a single

primary hierarchical structure are outweighed by the usability and navigation benefits of having

each page have a natural home with parent, child, and sibling relationships that specify keysubordinate/super-ordinate content relationships. Later we will describe the thematic links that

specify the non-hierarchical relationships among content. Non-linear (hierarchical and

network) structures are common in hypermedia designs, but our project is among the few that

focus on trying to maintain the narrative structures that are compromised when non-linearnavigation options are introduced. Narrative is supported with features called "narrative

smoothing," "horizontal reading," and "custom depth control," described later.

Hypermedia side effects. The distinguishing characteristic of hypermedia, i.e. the abilityto navigate easily from one location to another, in addition to being responsible for its benefits,

leads unavoidably to a set of problems: disorientation, cognitive overload, poor narrative flow,

and poor conceptual flow (Conklin 1987; Lucklin et al. 1998; Beasley & Waugh 1995;Plowman et al. 1998; Stanton & Baber 1994). These problems can also be seen as a result of

the tension between the linear/narrative and non-linear/associative natures of hyperbooks.

5 Some argue that the user should be able to impose her own structure on domain knowledge, in

accordance with the constructivist pedagogical principle that learners construct knowledge in personal

and idiosyncratic ways. Though some user linking or re-structuring features may be beneficial, we

believe that the structure and internal relationships (links) that a domain expert defines are "content" and

are just as important as the expertise encoded in the text.


Page 11

Disorientation refers to users not knowing where they are, where they have been, or how to get

to where they want to go in hypermedia space (the "lost in hyperspace" problem). Cognitive

overload refers to users being overwhelmed or confused by the options available to them in

multi-path, multi-tool environments such as hyperbooks. Narrative flow refers to the didacticor dialogical flow of the text itself. Conceptual flow refers to the flow of ideas or concepts,

and might also be called overall "comprehensibility".6 Good conceptual flow includes "content

readiness," which is the traditional intelligent tutoring systems goal of tailoring content so thatthe learner is neither bored because it is too easy, nor overwhelmed because it is too difficult

(i.e. remaining within the learner's "zone of proximal development" -- see (Wertsch 1984) and

(Murrary & Arroyo 2002)). Our goal is to support learners' exploration of hypermedia spaceswhile avoiding or ameliorating these problems.

7 Both convergent and exploratory navigation

behaviors can result in disorientation and other issues, but these problems are more severe for

exploratory learning behaviors because they tend to be less systematic and direct by nature.

Adding adaptivity to hypermedia has the potential to exacerbate disorientation and cognitiveoverload. A user who visits the same page twice may see different things, adding to confusion.

Figure 5 illustrates how MetaLinks features relate to these hypermedia "side effects," or

navigation issues. Most of these features will be described below.

Features Supporting Active Reading/learning and Exploration

To summarize the discussion thus far: our goal is to support active reading and learning using

hypermedia texts. This means we want to support certain cognitive processes and informationgathering activities, including flexibly choosing the level of depth, and allowing questions,

themes, and topics of interest to arise and be followed in an opportunistic way. The extensive

hyperlinking implied in such flexible reading inevitably leads to a number of usability issuesthat should be addressed in the software design. Finally, related to all of these goals is the goal

to provide prerequisite information as needed.

Modular Pages and Rhetorical Structure. Before going on to describe how MetaLinksaddresses the issues above with its Narrative Smoothing, Custom Depth Control, Thematic

Links, and other features, we have to explain some basic representational issues.8 First, unlike

textbooks, in which page breaks are of little significance and topics are organized around

sections, MetaLinks hyperbooks are composted of quasi-modular "pages" of relatively smallsize. The pages are not completely independent, but usability concerns compel us to organize

the material into small grain-sized thematic units. Some hypermedia documents have long

pages with lots of content that must be scrolled down. But to support navigation for multipleuses and perspectives, pages should be broken into constituent parts based on their main ideas.

Second, most MetaLinks links go from page to page. In standard HMTL documents a link goes

from a phrase (the underlined text) to another page (or location within a page). The majority ofnavigation links in MetaLinks hyperbooks, including the subtopic links and related information

links, support a semantic (and hierarchical) network content structure with pages as nodes.9

Though current HTML hypermedia tends not to, many early hypermedia projects used the page

to page metaphor (including NoteCards (Marshall & Irish 1989) and Project Xanadu (Neslon1980)).

6 To clarify the difference, one could imagine a text with a good conceptual flow which was poorly

written and choppy, thus having poor narrative flow. Similarly one could imagine text that seemed to

read very smoothly but did not make rational sense, or in which prerequisite concepts were not introduced

sufficiently for understanding the text, and thus the text has poor conceptual flow.7 The means to achieve all of these goals conflicts to some degree, such that designing a hypermedia

learning environment will always involve tradeoffs.8 This discussion also corresponds to the issues that arose in transforming the geology text book into

hyperbook form (of course, MetaLinks can be used to author a hyperbook "from scratch" as well as to

"port" and transform a traditional text book).9 The authoring tool does allow the author to create "regular" hyperlinks from text phrases to other pages

in the hyperbook or to external WWW pages when needed.


Page 12

Third, traditional books are structured around a chapter/section/subsection decomposition

model. MetaLinks hyperbooks include a hierarchical decomposition, but we don't call parts

chapters or sections because the hierarchy can go arbitrarily deep, and different users may have

"top level" entry points that vary in how deep they are in the hyperbook. For example, what is a"section" to one person, reading an entire book to a moderate level of depth, may be treated like

a "chapter" to another, whose reading is deep and focused on a small part of the hyperbook.

The rhetorical structure of most books includes transitional elements before or after primarycontent that introduce, conclude, or relate sections. In MetaLinks hyperbooks a parent page is

considered a summary, overview, and/or introduction to all of its children pages (unless the

page has no children). Think of everything in the family tree "below" a page as containingfurther details or elaborations of that page. As explained below, after reading a page, the user

decides whether they want more depth on that topic (Explain More). If they navigate to more

depth, eventually they will "pop back up" to the original page to maintain a sense of orientation.

They are given a visual indication that they have returned to a page they branched off of (see thered dot to the left of the page title in Figure 2). According to this temporal flow of events, the

original page becomes both an introduction and a conclusion for its children. Thus, the author

is encouraged to compose a page so that it works as both a summary and a refresher to itschildren's contents.

Horizontal Reading and Custom Depth Control. The default narrative flow (a linear

navigation path for which the reading or organization of the content is most natural orperspicuous) in MetaLinks hyperbooks differs from text books and other hyperbooks -- it is

breadth-first rather than depth-first, and organized for "horizontal reading." The default "next"

page is the sibling page. Thus the default is to continue reading at the same level of generality.

As mentioned, the subtopics of any page cover the material at greater depth. Horizontal readingaddresses the narrative flow issue by providing a framework in which the author can add more

depth to any topic without having to rewrite the narrative. In contrast, in normal books adding

more depth requires, in effect, inserting pages into the narrative.

Figure 6: Custom Depth Control Illustration

Horizontal reading sets the stage for an innovation called "custom depth control" which

allows easy reading or skimming at any level of depth. It has been shown that some users ofhyperbooks are overwhelmed by the navigation options and are happy to navigate by limiting

their navigation to pressing the Next and Back buttons (Brusilovsky & Eklund 1998). Custom

depth control is a simple technique which gives such users greater control while not

significantly increasing the complexity of the interaction (it still involves only two buttons, andthus lessens the potential for cognitive overload). Traditional hyperbooks have Next and Back

buttons. MetaLinks hyperbooks have Explain More, Next, and Return buttons. "Next" goes to

the next page which, as explained above, continues at the same level of generality. "ExplainMore" begins a path across the children of the current page. When the last child in a sibling

sequence is reached the Next button is replaced with a Return button, and the user is returned to

the parent page where they originally pressed the Explain More button. Thus, the user hascontinuous control over whether they want to continue at the same level or delve into more


Page 13

detail on the current topic. Figure 6 illustrates a navigation path using custom depth control.

The user is reading the page at "1" and finds she is particularly interested in the second

subtopic, and clicks on the subtopic link to go directly there (step 2). Upon reading this

subtopic she finds she wants even more depth or detail and presses Explain More and is taken to3. She presses Next twice to continue at that level, and then is particularly interested in the third

child (at 5). She presses Explain More again to go to 6, and continues on at this level with

Next, Next. At 8 she presses Return to return to "9" (same node as 5) and continues from therewhere she left off by pressing Next to go to 10. After 10 she is presses Return, then Return

again to be brought back to the beginning (12, the same node as 1).

This implements a stacking or sub-goal navigation paradigm. Explain More pushes thesubtopics of the current page onto an agenda mechanism (the Return Stack) and the Next button

pops the top item from the agenda. When the user navigates using any method other than

Custom Depth Traversal (Next and Explain More) then this is treated as a type of navigational

tangent, and the originating page is pushed onto the Return Stack. In these situations Returnacts like the Back button in traditional hypermedia, and returns to the most recent page. The

agenda mechanism allows the user to go off on tangential paths and return to where they left

off.Readers with little background knowledge do not have preexisting schema that allows them

to easily assimilate new content or understand how the macrostructure of a text relates to the

concepts being conveyed (Foltz 1996). The MetaLinks horizontal reading structure allowsreaders to get a narrative overview before going deeper into the content. This should allow

them to build high level knowledge structures which will help with the assimilation of

subsequent levels of detail. Foltz's analogies to "guided tours" or "training wheels" for subject

matter novices is relevant here.Thematic "Related" Links. As mentioned, the primary organizational structure for

MetaLinks hyperbooks is the hierarchy, as reified by the TOC and facilitated by custom depth

control. However, hierarchies do not capture the conceptual richness of most domains. Eachconcept is related to others in numerous ways. There are multiple perspectives on the material,

suggesting multiple learning paths. From a given topic or page the most useful or interesting

next topic will differ for different learners. MetaLinks includes thematic (non-hierarchical,

associative, or tangential) links labeled Related Information on the user interface. Each pagehas a set of Related Links to other pages. These links are "typed" or categorized to indicate the

type of relationship they represent. The authoring tool provides a default list of link types, but

the author can create her own types for each hyperbook or domain. Related Links address theissue of Conceptual Flow. They allow the learner to maintain a path through the material that

matches their curiosity and inquiry goals. Figure 1 shows the Related Links, grouped by link

type, that appear in the pop-out menu from Tectonica Interactive's page T.2.4. The pop-outonly shows those links and link types which have been created for that page. Here are some of

the approximately 20 link types we defined for the geology domain:

Where in the world? Are scientists sure? Extreme cases and famous catastrophes

Related Phenomena I want to try it! Geologists used to think…

How is it measured? Please explain deeper What do I need to know first?

The authoring tool allows authors to define a set of link types for each book (or domain).

For example, the Women Mathematicians hyperbook shown in Figure 2 uses the following link

types: Who influenced her? Where did she get her degree? Where did she live? Where did shework? What about the time period? How did she influence our lives?

In normal HTML hypermedia, links are not typed. One sees an underlined phrase and

knows that clicking it will lead somewhere, but does not have information about why they

should go there or the relationship of the linked page to the linking page. Using typed links thatstructure the information space according to a fixed set of common issues, questions, or themes,

similar to that used in ASK systems (Cleary & Bareiss 96, Ferguson et al. 1992). Such

conceptual structuring of the information space aides both user and author in creating a mentalmodel of the semantic space with appropriate expectations for how things can be related.


Page 14

Thematic links provide more textual "coherence" (Foltz 1996). Coherence (the amount of

interrelatedness of the text's propositions) has been shown to improve comprehension in reading

traditional texts.

Authors can use the link types to organize their content creation. Link types can act asprompts that remind or inspire the creation of related content or the creation of more links

among existing content. For example, the existence of a link type called "alternate theories" not

only helps the author organize the data base of knowledge, but may inspire her to authorsomething about alternate theories to the current page, which she may not have thought to do

otherwise.

Inquisitory Page Titles. To support our goal of inquiry based and exploratory navigation,MetaLinks pages have an inquisitory title in addition to the regular title (see the corresponding

field in the authoring tool Figure). For example the page T.2.1.1.1, titled "Earth's Layers: The

Crust," has a question title "What are the properties of the earth's crust?" The inquisitory page

title appears in small font just above the page title. The main purpose of inquisitory titles is intheir use in page links. Another page with a related link to T.2.1.1.1 will have "What are the

properties of the crust?" as the link text, rather than "Earth's Layers: The Crust." Using

inquisitory titles gives the navigational interaction a conversational feeling, adding to thenarrative flow of the experience. Upon navigating to a page that addresses one question, new

links answering new questions are available, giving the interaction a feeling of a question and

answer dialog. The use of inquisitory page titles for adding a conversational feeling tonavigation is similar to the inquisitory link types used in ASK systems.

10

Non-exploratory navigation. Though we highlight MetaLinks features for active reading

and exploratory navigation, we have not compromised the ability to do less-exploratory reading

and more focused navigation. MetaLinks has features such as a search engine, table of contents,and glossary that allow focused use and reference-like use, common in encyclopedia-style

multimedia. Our tests have given no indication that the additional exploratory features get in

the way of more focused use.

Dealing with Negative "Side Effects"

MetaLinks has several features that address the secondary issues, i.e. minimize the inevitable

negative side effects of flexible navigation. Below we mention those dealing with narrativeflow and disorientation. In the next subsection we focus on content readiness (conceptual flow).

Narrative Smoothing. We have a simple but elegant partial solution to the narrative flow

problem. Each page has associated with it an "intro text" paragraph (see the associated field inthe authoring tool Figure). This paragraph eases the reader into the subject of the page, giving a

little background or introduction. If the user jumps to that page in a non-standard way, the

intro-text is pre-pended to the main text of the page. If the user navigates to a page using theNext button, i.e. if they are engaged in horizontal reading, no intro text is shown, nor needed.

There are local and global (structural) aspects to narrative. Previous work (Lucklin et al. 1998,

Plowman et al 1998, & Weller 2000) focuses on maintaining a global narrative structure in

hypermedia. This is to maintain a coherent thread of meaning from beginning to end, where alldetails can be seen in relation to the main ideas of the document. We want our hyperbooks to

be usable for diverse learning goals and thematic threads, so we do not try to maintain a single

global narrative structure. Rather, we focus on the local flow of narrative from page to page. Asmoother local flow of text, where the context of sentences and their relationship to other

material is made explicit, enhances comprehension (Foltz 1996).11

Features for Orientation. MetaLinks has a number of features to help with disorientation

and lessen the chance for cognitive overload. Since these features are not particularly

10 The author can also define the text/question for each specific link, overriding the link's target page

inquisitory page title.11 The author can also create a transition paragraph from one specific page to another, but in general this

would be too labor intensive. There are an exponential number of possible transitions between pages in

contrast to the one intro text paragraph with each page.


Page 15

innovative, we will describe them only briefly. Orientation can take several forms: local,

global, and temporal. The Table of Contents (TOC) provides global orientation, i.e. locating

ones self within the context of the whole text. It is annotated with colors and markings to show

pages that have been visited, and the "you are here" current page. We have also implemented a"pictorial table of contents" feature that allows the use of "image maps" with hot spots to show

a graphical portrayal of a section of the hyperbook. Several studies have shown that visual

representations of concept relationships and hierarchies are beneficial to diminish disorientation(Beasley & Waugh's 1995; Edwards & Hardman 1989; Vanhoeven & Waarendorf 1999). Local

orientation is provided through the page number and the list of subtopic links. Pages are

numbered to show their place in the hierarchy, for example 4.5.6.2 is clearly a sibling of 4.5.6.3.The typed links allow us to provide an "annotated navigation history" that helps with the

"where have I been?" (temporal) disorientation issue. The list of visited links is annotated with

the type of link, i.e. the "reason" they went there. This makes it easier for users to go back to

where they left off after going down one or a series of tangents to their main exploration goal.They can visually see the difference between going to a "next" page, vs. diving down for more

information, vs. jumping off on a tangential link. When they want to go "back to where I was"

they may want to retrace to the last tangent or perhaps recover their path back at an earliertangent, and the annotated navigation history provides information that allows them to make

such decisions. As with the "return stack" used in custom depth control, the annotated history

tool implements a sub-goal navigation structure. These illustrate our belief that navigation has ahierarchical rather than linear structure. Thus Return can mean "end this tangential excursion"

as well as "go back to the previous page."

Pop-up Text. As mentioned earlier, the flexible use of hypertext is facilitated by

organizing the content into quasi-modular pages that represent small grain-sized thematic units.Non-essential text and graphics such as examples and footnotes can be hidden inside pop-up

"stretch text" (really "stretch media" as MetaLinks allows for graphics to be included in the

stretch text). Stretch text is colored, and when the user passes the mouse over stretch text thehidden text (or graphics) pops up adjacent to the colored text, and disappears when the mouse is

moved away from the green text. Using this method the user can choose whether to see

additional detail. It serves the same function as adaptive content (Brusilovsky 1998; Stern &

Woolf 1998) but lets the learner decide what will appear. Stretch text helps with the cognitiveoverload issue by reducing the amount of content visible on the page.

Dealing with Prerequisite Concepts

Glossary pop-up and base page. Users have access to a glossary tool that lists all the words in

the glossary with their definitions. Associated with each glossary word is a link to its "base

page," which is the page in the book that best explains the term. As a first line of defense forusers who don't understand a concept being used on a page, the glossary definition pops up

when a glossary term is moused over (see Figure 2). The terse definitions that pop up may not

be enough to alleviate the learner's ignorance about a concept. If the user clicks on a glossary

term they navigate to its base page for more information. This feature addresses the contentreadiness and conceptual flow issues by allowing learners to easily learn about prerequisite

concepts. Base pages also address the cognitive overload issue by anchoring starting points for

exploration. Insufficient background knowledge leads to poorer comprehension, as learnersmust use additional cognitive resources to make connections between new information and

existing knowledge structures, or they must expend resources in sub-goaling to fill in needed

knowledge (Foltz 1996). Pop-up definitions (and to a lesser extent base-page links) minimize

the effort needed to acquire prerequisite information. The glossary features also supportopportunistic learning and diverse user backgrounds, as learners can feel free to explore without

worrying that prerequisites have been missed.

Student Modeling. User modeling techniques have been used in intelligent tutoring andadaptive hypermedia systems to infer what the student understands and does not understand.

This information can be used for many purposes (for example to select or generate problems or

hints), but its most common use is in dealing with prerequisite concepts. In its most common


Page 16

form, a semantic network representation of the important concepts of the domain is encoded

containing prerequisite relationships between topics (or concepts). As students use a system it

accumulates evidence about the level of understanding of these topics (the system may also

begin with some proficiency assumptions, based on pre-testing or user profiling). The systemcan then try to ensure that unknown prerequisites of a topic are given prior to that topic. In

some intelligent tutors topics are sequenced automatically to deal with prerequisites. In

adaptive hypermedia systems, the approach has been to use "link annotation" (Brusilovsky1998) to inform the user about whether the links shown are "already learned," "ready to be

learned" (with prerequisites satisfied), or "not ready to be learned" (prerequisites not satisfied).

We have implemented a prototype of what we call a Dual Space Overlay Model thatrepresents relationships in both the page space and the concept space. The terms in the

MetaLinks Glossary serve as the concepts for the concept map (as in InterBook). Though this

feature was implemented (as part of a graduate student Masters thesis (Shen, 2000)), it has not

found its way into actual use in MetaLinks testing. This is because in the end we decided thatwith only page visitation data available, we did not have enough information to make confident

inferences about mastery (until a quizzing component is added). In the formative evaluations

described below, done without the user modeling feature, we looked for evidence for the needto add more intelligent prerequisite-based help. As we will show, at least for the Tectonica

Interactive hyperbook tested, the glossary pop-up definitions and base page links seems to

suffice to allow users to access prerequisite concepts.

5. FORMATIVE EVALUATIONS

Adaptive hyperbooks, are relatively new types of artifacts with numerous novel features. There

have been very few attempts at providing design guidelines based on either theoretical concerns

or empirical studies. The vast majority of design recommendations are ad-hoc, there is nothingapproaching consensus on appropriate theoretical bases, and the results of empirical studies are

ambiguous (Dillon & Gabard 1998; Eklund & Brusilovsky 1998). Therefore it is important that

hyperbook frameworks undergo user-participatory design and iterative formative evaluation

trials to determine overall usability and usefulness. Additional more controlled studies areneeded to help determine which of the many hyperbook features contribute to various effects.

Tectonica Interactive has undergone a pilot study evaluation with five subjects in 1998 (see

Khan et al. 2000), an "exploratory" (Murray 1993) formative evaluation in 1999 (n=19), acontrolled formative evaluation in 2000 (n=24), and three field-based trials. In the following

sections we describe the two formative evaluations, both using the Tectonica Interactive

hyperbook.MetaLinks includes a number of innovative features which needed to be field tested. Our

first formative study was therefore an exploratory study in which we hoped answer the general

questions "are the current features useful and usable?" (individually and together) and "what

evidence can we find to support the need for the more sophisticated features and intelligence?"We were particularly interested in the use and usability of narrative smoothing, custom depth

control, and thematic links. We were also interested in observing which of the many navigation

options subjects used, and why. We also wanted to discover whether subjects had problemsrelated to disorientation, cognitive overload, discontinuous flow, and content readiness, as

mentioned above. And finally, we wanted to gather information about overall perceived

usefulness and enjoyment of the system.The second formative evaluation focused on particular issues identified in the first

evaluation: feature salience and the effectiveness of the narrative smoothing feature. As is

apparent from the above design discussions, there are many issues that could be investigated

more closely under controlled conditions. Our work thus far establishes baseline data for theusability and usefulness of the system as a whole, and of each of its major features, and will

allow us to delve deeper in future studies. For example, we hope to study more closely the

cognitive processed involved in active reading in hyperbooks, and to compare hyperbook use indifferent domains.


Page 17

Method for Study #1 & Study #2

Both Formative evaluations shared the following methodological characteristics (with

insignificant variations). Subjects received one hour of training in a practice session that took

place two to seven days before the main trial. In this practice session they were giveninstruction in the tools and navigation concepts of Tectonica, and were given a check-off list to

indicate that they used and understood each of the features in the MetaLinks Navigation and

Orientation Feature List in Section 3.2 (which they all completed). Subjects were familiarizedwith the family tree metaphor ("parent," "children" (subtopic), and "sibling") used to organize

content. Despite the many modes of navigating in MetaLinks hyperbooks, the tutorial session

seemed adequate to make users comfortable using all of the features. After the practice session

subjects filled out a pre-questionnaire to collect personal data.Subjects were randomly assigned to one of three questions: 1) Use the program to learn

what you can about volcanoes, their causes, effects, how they exist and specific examples. 2)

Use the program to learn what you can about alpine glaciers, their causes, effects, how theyexist and specific examples; or 3) Use the program to learn what you can about the Mid

Oceanic Ridge, their causes, effects, how they exist and specific examples. These questions

were chosen based on sections of the hyperbook for which main and related topics wererelatively completed. Along with each question was the instruction "Feel free to explore any

material in which you become curious. Be prepared to informally explain (to a peer) what you

learned about your question, and anything else you explored" Subjects were given 45 minutes

to use MetaLinks to learn about their topic. They were then given 20 minutes to explain in pairswhat they had learned (we did not gather data during this phase--we gave them the explanation

task so they would take the study task seriously). After the session subjects were given 20

minutes to take an on-line questionnaire, and then they were gathered into a focus group for 20minutes for a general discussion about their experiences. The volunteers, college students who

came from a variety of backgrounds and from two different colleges, were compensated $15 for

participating in the study. For Study #1 most of the subjects were enrolled in an undergraduate

introductory Oceanography course. Study #1 had 19 subjects 12

and Study #2 had 24 subjects.13

Collecting several types of data allowed us to gather both quantitative and qualitative

information, and to "triangulate" our measurements and look for concordances. Much of the

analysis went into comparing the various features and their frequency of use.14

The data typeswere:

• A subject profile questionnaire was given prior to the sessions, in which informationwas gathered about age, sex, and previous exposure to geology and computers.

• Program traces of navigation and tool use. The trace included the pages visited,durations, the "reason" the page was visited, i.e. the type of link or tool used to get

there, and uses of all buttons and tools.

• The on-line questionnaire was comprised of about 60 questions (a few more for Study

#1). Most questions were Likert scale agreement questions with choices "stronglyagree" "agree" "neutral" "disagree" "strongly disagree" (some used a similar

Often/Rarely Likert scale and a few were fill-in questions). All questions had a space

for typing in additional comments (which our subjects made heavy use of). Thequestions in the on-line questionnaire are described in the Appendix.

• The focus-group sessions were video taped, and notes were taken during them. Thefocus-group discussions were semi-structured, with the facilitator trying to touch upon a

prioritized set of 28 questions, while allowing the participants' comments to direct the

flow of the discussion.

12 This number does not include the five subjects who's sessions or data were incomplete and rejected.13 This does not include two subjects whose data was discarded because they did not complete the task.14 In addition, the program traces were analyzed to produce concept map style visualizations of the

navigation paths for several sessions to be analyzed for patterns (see Khan 2000).


Page 18

• Random short think-aloud interviews were done as students used the system during

Study #1. The interviewer observed the subject using Tectonica for 5 to 10 minutes,asking him to verbalize his thoughts and the reasons for decisions. Data was recorded

on a paper form. Think-aloud interview data was gathered from 10 of the 19 subjects in

Study #1.Though the above method was similar for both studies, there were two major differences

between Study #1 and Study #2. In Study #2 we used four different versions of the software to

test specific features, as explained later. Also in Study #2 we performed more in-depth

statistical analysis of the data.

5.1 Study #1: Exploratory Formative Evaluation

Due to the largely positive results and the concurrence of data from multiple sources, we onlyneed include a summary of the results here, and have put more details on the data in the

Appendix.

Overall Navigation Statistics. To give a general idea of navigation activity, there were anaverage of 81 moves per session, or an average of 1.1 moves per minute. "Moves" includes

going to and from TOC, search, and glossary pages, as well as navigating to content pages.

Disorientation & Cognitive Overload. When users needed to "reorient" themselves they used a

variety of tools, including TOC, search, and history pages. We found no substantial evidence ofdisorientation or cognitive overload. In general subjects did "explore" the material rather than

go directly to an "answer." They were focused on the content itself rather than trying to figure

out how to use the system.Narrative Flow, Conceptual Flow and Content Readiness. We found no evidence of

choppiness or unevenness in narrative flow or conceptual flow. We also found no evidence of

frustration over the lack of prerequisite knowledge (despite the fact that the majority had never

taken a geology course).Overall Enjoyment and Perceived Learning. Despite the fact that most subjects

indicated that they were not particularly interested in geology, they indicated overwhelmingly

that using Tectonica was enjoyable, easy, and productive. This was slightly surprising, since wesuspected that with so many features available to them that some would find it frustrating to use

(however, below we comment on features that were not used much). Significantly, the two

questions that compared Tectonica with textbook learning favored the hyperbook (2 of the 19subjects would prefer a text book).

Navigation Tool Use. The major disappointment of the trial was that subjects did not use

either the Custom Navigation or the Related Links as much as we had hoped. Since these are

potentially the most complex features to understand, the low use may have contributed to thepositive findings noted above. Of the 18 types of navigation moves tracked, the most common

over all subjects were: Go To Child (19% of the total moves), Return (17%), Search (13%),

Custom Depth feature (10%), TOC (9%), and History (6%). Navigation using Related Linksoccurred in just 1% of the moves (it was used a total of 10 times).

Subjects did navigate through the hierarchical content space, maintaining or

deepening/expanding topics of interest, but usually they selected particular subtopics rather thanselecting Explain More to visit all of the subtopics of a page. Also, they did exhibit exploratory,

non-hierarchical navigation, but chose to use the TOC and search engine as jumping off points

for their tangential explorations rather than Related Links. Use of the Custom Depth feature

was, though less then hoped for, respectable, while use of Related links was almost non-existent, in sharp contrast with use of the direct-to-child links. We strongly suspected that the

visual salience of these two features affected their frequency of use. In the next study we

switched the location of these features to measure the effect of salience on use.

5.2 Study #2: Semi-controlled Formative Evaluation

We came away from the first formative evaluation with a number of questions, two of which led

to a follow-up experiment which is the subject of the second evaluation. First, although we


Page 19

assumed that our narrative smoothing feature was effective because subjects did not indicate

any problems in the area of narrative flow, we needed a controlled experiment to determine

whether the narrative smoothing feature actually had its intended effect. Second, although users

indicated that the tools were useful for navigating through a hyperbook, and they did not givesubstantial evidence of problems in any of the navigation issues, they did not make as much use

of one of the more powerful (and cognitively complex) navigation features: Related Links. Our

hypothesis was that this was because these links were less salient than other modes ofnavigation. Related links are accessed by clicking on a "Related Information" tab on the left

side of the page, to pop out the list of related links. The most frequently used feature was the

direct links to subtopics pages, which are listed in plain view at the bottom of each page's text(see Figure 2).

To address some of the questions raised, we modified the software so that we could

manipulate and test two separate aspects of the hyperbook's interface in a controlled study.

First, we added a feature allowing us to toggle the intro text (narrative smoothing) feature offfor selected subjects. Second, we added a feature to swap the location of the subtopic links and

related links for selected subjects. This would in effect increase the salience of one of the least

used features (related links) and decrease the salience of the most used feature (child links).Finally, we were interested in further characterizing the use of the custom depth control feature,

though we did not include variant conditions for this feature in the software or experimental

design.The experiment used a 2x2 factorial design based on the related links (RL) and intro-text

(IT) software settings. Six subjects were randomly assigned to each of four experimental

conditions: intro-text on and related links shown (IT+RL+), intro-text on and related links

hidden (IT+RL-), intro-text off and related links shown (IT-RL+), and intro-text off and relatedlinks hidden (IT-RL-). Remember that when the related links are hidden within the pull-out

menu the subtopic links are shown at the bottom of the page, and vice versa. It was determined

that 5 was the minimum number we needed in each of the 4 conditions to allow statisticalsignificance. We originally had 7 in each condition, but in the end had usable data for 6

subjects in each condition (total of 24 usable subjects).

In addition to the RL and IT software variables, we also identified the following variables

(based on questionnaire response) that might have to be controlled for in the analysis: computeruse, geology experience, task number, and trial date (there were several trial dates spanning a

two week period, to accommodate subject availability). Computer use and geology experience

were Likert scale questions. Task number is a number from 1 to 3 indicating which of the threegeology questions the subject was given.

We intended to test the following hypotheses:

H1. Text Smoothing. Subjects in the IT+ condition experience better narrative flow than

subjects in the IT- condition.

H2. Feature Salience. Subjects in the RL+ condition have higher use of related links, and less

use of subtopic links (as indicated by navigation tracking); and they understandand like the related links feature more; vs. subjects in the RL- condition. We

expected the converse to be true for children links.

H3. Exploratory navigation. The related links and custom depth features lead to betterexploratory/inquiry navigation behavior, as indicated by the Exploration

measure, in the following ways:

A. The RL+ condition leads to more positive scores on the Exploration measurethan the RL- condition.

B. There is a positive correlation between the use frequency of the related links the

Exploration measure.

C. There is a positive correlation between the use frequency of custom depth and

the Exploration measure.

Since we were also interested in validating the results from the first study, we gave subjects analmost identical post-use questionnaire. To allow a more complete analysis of data in study #2,


Page 20

we constructed a number of composite measures by grouping questionnaire items. The overall

"Satisfaction" measurement aggregates six Likert scale questions including: "I found Tectonica

easy to use," "I would have preferred reading a textbook," and "I learned the subject matter

well."15

The "Exploration" measurement is a composite of three items related toexploratory/inquiry navigation behavior. Composites consisting of from two to four questions

were created for questions measuring each of the four navigation issues (cognitive overload,

disorientation, conceptual flow, and narrative flow). Also, we analyzed navigation and tool useand analyzed the characteristics of the hyperbook pages that were visited, to determine, for

example, how many pages that subjects visited had related links available on them.

Study #2 Results

As expected, we found no evidence of interactions between the RL and IT variables, which

allows us to report on the main effects of the variables separately. Analysis of the dataindicated that there was an uneven distribution of computer experience among the RL

conditions. Our subjects ranged from average to high levels of computer experience (with only

one subject indicating "low" experience) and there were significantly more subjects with highcomputer experience in the RL+ condition. As a consequence, in analyzing the effects of other

variables, we controlled for computer experience (by treating it as a covariate) when its effects

were found to be statistically significant. We also checked whether task number, trial date, orgeology experience should be treated as covariates, but none of these variables was found to

have significant effects on the measures we examined. Below we summarize the results and

their implications (data details are in the Appendix).

Overall Navigation Statistics. Overall, in the course of answering the three task questions,the 24 subjects viewed 180 of the 478 pages in the hyperbook. These correspond to the areas

dealing with the task questions they were given and tangential but related topics. There were

1086 navigations to pages, for an average of 45 per subject. This duplicates the finding ofapproximately one page per minute found in Study #1.

Concurrence with Study #1. As in Study #1 we did not find appreciable evidence of

problems with narrative flow, conceptual flow, disorientation, or cognitive overload. Usability,

usefulness, enjoyment, and perceived learning were all quite high, as in Study #1. There was astrong positive correlation between good ratings on the navigation questions and the overall

Satisfaction score. There was no evidence of navigation issues being affected by the

experimental conditions or the relative use of navigation features.H1. Text Smoothing. We found no significant effect of the IT variable on questionnaire

items or navigation patterns. Therefore, we can not claim that the positive findings related to

narrative flow are due to the text smoothing feature. A post-hoc analysis of the pages visited bysubjects during the study indicated that the results are limited by the relatively small number of

times subjects saw intro-text.

H2. Feature Salience. The related links feature accounted for a total of 1.5 percent of all

navigation moves (a 50% use increase over Study #1). While this is still a low use of thefeature, there was a significant effect of the RL variable on related links use, confirming

hypothesis H2. The related-links feature was used by significantly more subjects in the RL+

condition than in the RL- conditions (9 vs 2 out of 12 subjects). A similar effect is found withthe mirror feature, children links, which was used much more often, 9.3 percent of the total

moves overall. Child links were used 2.8 times more often when visible (RL- condition) than

when hidden (RL+). When related links are shown, the related links feature is generally ratedmore highly and the subtopics links feature is rated lower.

H3. Exploratory Navigation. In general subjects indicated that they did explore topics

unrelated to their initial question, and that the software facilitated exploring according to

curiosity or interest. There was suggestive but not conclusive evidence for attributing this to the

15 The aggregate measure, which was adjusted to account for questions with negative polarity, was a valid

aggregate, as was indicated by the very high degree of pair-wise correlation among answers to the

individual questions.


Page 21

related links and custom depth features. The frequency of use of both the custom depth feature

and the related pages feature had a significant positive correlation with the Satisfaction measure

(p=.01).

We will call the questions used in the Exploration composite measurement E1 ("Did youvisit and learn about topics that were not directly related to you main question?"), E2 ("Related

information [feature] compelled me to visit subject material unrelated to my initial inquiry"),

and E3 ("The software allowed me to navigate though the material according to my curiosity").There was no significant correlation between the Exploration measurement and the RL

condition (H3A). There was weak evidence (p=.07) of a positive correlation between the

Exploration measurement and the number of times subjects used the related links feature (H3B).The frequency of use of the related links feature was positively correlated (p=.01) with E1 but

not with E2 or E3. That the related links feature was not used very often (1.5% of the total

moves) contributed to the lack of statistically significant findings.16

The custom depth feature was the most often used feature (overall it accounted for 33% ofthe moves). It was used equally whether related links were shown or hidden. This result differs

from Study #1 in that in the earlier study subtopic links were used most often, and Custom

Depth only 10%. We suspect that this may be because the verbal tutorial description of thecustom depth feature was clearer in Study #2. The frequency of use of the custom depth feature

had a weakly significant positive correlation (p=.07) with the Exploration questions (H3C); a

weakly positive significant correlation (p=.055) with E3.

Limitations to Results of Studies #1 and #2

Generalizations from the results of Studies #1 and #2 are limited in several ways. First, ofcourse, is the small sample size (19 for the first study and 24 for the second), which made it

difficult to find statistically significant patterns. Second, we can not be sure how these results

transfer to other domains. Introductory geology has many concepts that are densely connected.Starting with almost any topic one could create a short path of connections to almost any other

topic. Compared to some other domains, the key concepts can be understood without extensive

prerequisites. The concept space is broad (i.e. not very deep, or has a shallow prerequisite

structure). We hypothesize that our results should transfer to domains that have a similarbreadth characteristic, and which have multiple perspectives and multiple pedagogical paths.

Third, the nature of the tasks we gave to subjects limits the generalizability of results.

Subjects were asked to investigate a topic to the extent that they could explain it to anotherstudent. They were encouraged to navigate according to their curiosity. This contrasts with, for

example, the task given in Brusilovsky & Eklund's (1998) study of a hyperbook developed with

the InterBook system that taught how to use database and spreadsheet functions in ClarisWorks.In this type of task the learner is intent on learning a particular skill. Learner needs and

behavior may differ for such tasks. Fourth, our results were also limited by ceiling and floor

effects. The vast majority of positive ratings may have caused a ceiling effect, and in the future

we may want to repeat the experiment using a more demanding task or domain. The relativelylow use of the intro-text and related links features limited our conclusions about their effects,

and in future tests we will have to insure that subjects will be more likely to encounter these

feature as they engage in their tasks.

6. CONCLUSIONS AND FUTURE DIRECTIONS

In this paper we have described the MetaLinks authoring tool and web server for adaptive

hyperbooks. MetaLinks encourages active reading and learning in hypermedia texts by

supporting exploratory reading and curiosity-driven information gathering behaviors. Itsupports flexibly choosing the level of depth and allowing questions, themes, and topics of

16 Of the 180 unique pages visited by users, 169 had related links available, so scarcity in encountering

the feature was not an issue as it was with the intro-text feature.


Page 22

interest to arise and be followed in an opportunistic way. The extensive hyperlinking used in

opportunistic reading inevitably leads to a number of usability issues. As hierarchical and

associative relationships are accentuated, narrative flow can be compromised. Increased

disorientation and cognitive overload are also possible. Conceptual flow (including prerequisitereadiness) can also be compromised. MetaLinks aims to minimizing the negative side effects of

opportunistic and active reading strategies. MetaLinks includes many navigational tools and

several adaptive features, but we have focused on those that are more innovative: narrativesmoothing, custom depth control, and thematic linking.

Two formative evaluations of Tectonica Interactive, a geology hyperbook built with

MetaLinks, showed that it did support exploratory and opportunistic navigation behaviors.Overall satisfaction, usability, and perceived learning effectiveness were high. Significant

opportunistic and exploratory navigation did occur, and it was positively correlated with overall

satisfaction. We also noted that there was a wide range of navigation styles and tool use among

subjects. For example, to answer the task questions some subjects relied heavily on the searchengine, and some preferred to use the global perspective offered by the table of contents. The

range of navigation styles indicates that our range of navigation tools is appropriate (with the

possible exception of the Related Links feature, which did not get used often). There was onlysuggestive evidence for attributing exploratory behavior to the related links and custom depth

features. Subjects did not report problems with narrative flow, but the data was inconclusive

regarding whether the narrative smoothing feature contributed to this.Feature salience. In our second evaluation we found that the visible salience and

accessibility of a feature had a significant effect on its use. A corollary to this notion, which we

did not test but has been proven out in numerous projects, is that no matter how intelligent or

sophisticated educational software is, inadequacies in the interface or usability of the softwarewill surely overshadow the benefits of the intelligence.

17 Hiding a feature behind a tab or

making it one click away makes it harder to use, less likely to be used, and rated as less useful.

This may not seem surprising in retrospect, but it is an important result. Most sophisticatedsoftware has numerous features, and the screen would be too cluttered if all features were

shown at all times. Difficult design decisions must be made about which features to hide

behind in pull-down menus, pop-out menus, tab sheets, tool bars, etc. Of course more advanced

users "learn the ropes" and often define short cuts or key-strokes for their favorite power-tools.But if the goal is to facilitate the use of certain features it may be better to err in the direction of

a cluttered screen than to risk the feature not getting used, especially during software evaluation

or when introducing it to new users. However, cluttered screens can lead to cognitive overload,so the correct balance is an empirical question for each system.

Conservative design strategy. It takes significant effort to include adaptive or intelligent

features in educational software. In addition, the more complex a system is, and the more of itsworkings are opaque from the perspective of teachers and authors, the more difficult it may be

to author content. Also, as indicated above, basic interface design concerns (such as salience)

can overshadow the effects of sophisticated and potentially powerful features. Therefore we

recommend a conservative iterative design approach where less sophisticated versions ofsoftware are tested before adding much intelligence or adaptivity. For example, our data

indicated that the glossary pop up and glossary base page, two non-adaptive features, seemed to

satisfy user's needs to learn prerequisite topics as they navigated opportunistically. We may nothave discovered this if our more intelligent adaptive link annotation and dual overlay student

model were turned on for the studies. The current system leaves the locus of control and

intelligence solidly with the student. Good interface design can sometimes provide the benefitsthat are ascribed to more sophisticated or intelligent features (which can be more presumptive,

controlling, or intrusive than passive features). Similarly, AHA allows authors to adapt the

content of each page to include or not include information based on the users knowledge level.

In MetaLinks we decided to use pop-up stretch text to give the user a choice to see additional or

17 In a similar way, educational technology researchers are finding that the pragmatic issues of getting

software working in classrooms often overshadows the particular technological innovation being

introduced.


Page 23

parenthetical information, rather than have the system make that choice. Next we describe

intelligent features we have considered implementing.

Directions for Future Design and Research

We have mapped out several directions for future development for the MetaLinks project. In

order to make the software more useful to school systems, a number of mundane but importantfeatures may be needed, including: 1) a bibliography database, 2) quiz or test items, 3) allowing

user annotations and bookmarks, and 4) more administrative features such as rostering and

grading. Also, though MetaLinks currently focuses on learning by (active) reading, we wouldlike to add components seen in other educational software that address other styles of learning:

learning by doing (simulations); learning by explaining, reflection, and/or critique (via quizzes

or collaborative work), and learning by creating and synthesizing (students using the authoring

tool to create, modify, critique, and annotate their own work and peer's work).Though our empirical testing of the Tectonica hyperbook showed no serious usability

problems, we predict that additional usability or effectiveness issues will surface when we

create hyperbooks containing more complex structure, and test them using more complexlearning tasks. We would like to test MetaLinks hyperbooks for different instructional methods

such as spiral teaching and case-based instruction using "landscape crisscrossing" (Cognitive

Flexibility Theory, Furgeson et al. 1992).In addition to developing an adaptive link annotation module using the dual space overlay

model, we have considered adding a guided inquiry module that coaches users in maintaining

important learning goals. We believe that analyzing the structure of learner navigation paths

(similar to methods used in early versions of Belvedere (Suthers & Weiner 1995)) can allow usto scaffold navigation and metacognition. The ACE system (Specht et al. 1998) includes

procedures that address goals for convenience, unit completion, student interest, and novelty.

We have defined rules for identifying needs and providing guidance for the following types ofstudent and teacher goals: knowledge broadness, breadth, convenience, completeness,

efficiency, novelty, and content readiness. The method presumes that the learner specifies her

learning goals or style, or the instructor sets these goals for the student.

Finally, we plan to implement an automated link generation facility. Authoring thematiclinks relating pages is one of the more labor intensive tasks. Text analysis tools could be used

to partly automate this prcess by identifying common or synonymous terms among pages.

ACKNOWLEDGEMENTS

Most of the work reported here took place at the University of Massachusetts Computer Science

Department in, Amherst, MA. The author would like to thank Janette Piemonte Gartner for managing

evaluation trials, data collection and analysis, authoring Tectonica Interactive, and creating the User's

Guide hyperbook; Professor Chris Condit for acting as domain specialist in Geology, and inspiring some

of the initial MetaLinks design ideas; Tina Shen, Jason Thibedeau, and Erik Haugsjaa for FileMaker,

HTML, and JavaScript development; Dr. Phil Kelleher for extensive statistics consulting on trial #2; and

Samia Khan for extensive help with data collection and analysis of the pilot study and trial #1. We would

like to thank Saunders College Publishing for the use of content from their introductory geology text

book (Thompson & Turk 1997). This material is based upon work supported by the National Science

foundation's under Grant No. NSF DUE-9652993, and by the Office of Naval Research AASERT grant

no. ONR/N00014-97-1-0815. Disclaimer: Any opinions, findings, and conclusions or recommendationsexpressed in this material are those of the author(s) and do not necessarily reflect the views of the Nation

Science Foundation or the Office of Naval Research.

REFERENCES

Anderson, J. (1983). The Architecture of Cognition. Harvard Univ. Press: Cambridge, MA.

Ausubel, D. P. (1963). The Psychology of Meaningful Verbal Learning. New York: Grune & Stratton.


Page 24

Beasley, R.E. & Waugh, M.L (1995). Cognitive mapping architectures and hypermedia disorientation:

An empirical study. J. of Educational Multimedia and Hypermedia 4(2/3), 239-255.

Brusilovsky, P. & Eklund, J. (1998). A study of user model based link annotation in educational

hypermedia. J. of Universal Computer Science, Vol. 4, No. 4, 429-448.

Brusilovsky, P. (1998). Methods and Techniques of Adaptive Hypermedia. In P. Brusilovsky, A. Kobsa,

and J. Vassileva (Eds), Adaptive Hypertext and Hypermedia, Chapter 1, pages 1-44, Kluwer

Academic Publishers: The Netherlands.

Brusilovsky, P., Schwartz, E., & Weber, G. (1996). A Tool for Developing Adaptive Electronic

Textbooks on the WWW. Proc. of WebNet-96, AACE.

Cleary, C. & Bareiss, R. (1996). Practical Methods for Automatically Generating Typed Links. Proc. of

the Seventh ACM Conference on Hypertext, Washington, DC, March 16-20, 1996.Collins, A., Brown, J.S., Newman, S. (1989). Cognitive Apprenticeship: Teaching the Craft of Reading,

Writing, and Mathematics. In Resnick (Ed.) Cognition and Instruction: Issues and Agenda.

Hillsdale, NJ: Lawrence Erlbaum.

Collins, A. & Ferguson, W. (1993). Epistemic Forms and Epistemic Games: Structures and Strategies to

Guide Inquiry. Educational Psychologist, 28(1), 25-42.

Conklin, J. (1987). Hypertext: An Introduction and Survey. IEEE Computer, September 1987, 17-41.

De Bra, P. & Calvi, L. (1998). AHA: a generic adaptive hypermedia system. Proceedings of the 2nd

Workshop on Adaptive Hypertext and Hypermedia: Hypertext '98, Pittsburgh, June, 1998.

http://www.contrib.andrew.cmu.edu/~plb/HT98_workshop/.

Dillon, A. & Gabbard, R. 1998. Hypermedia as an educational technology: a review of the quantitative

research literature on learner comprehension, control, and style. Review of Educational Research,Vol. 68 No. 3, 322-349.

Edwards, D.M. & Heardman, L. (1989). Lost in Hyperspace: Cognitive Mapping and Navigation in a

Hypertext Environment. Chapter 7 in R. McAleese (Ed.) Hypertext: Theory Into Action. Norwood

NJ: Ablex Publ.

Eklund, J. (1997). Cognitive models for structuring hypermedia and implications for learning from the

world wide web. Proceedings of AusWeb-97 (www.scu.edu.au/sponsored/auswev/).

Eklund, J. & Brusulovsky, P. (1998). The Value of Adaptivity in Hypermedia Leaning Environment: A

short review of empirical evidence. Proceedings of the 2nd Workshop on Adaptive Hypertext and

Hypermedia: Hypertext '98, Pittsburgh, June, 1998.

Ferguson, W., Bareiss, R., Birnbaum, L, & Osgood, R. (1992). ASK Systems: an approach to the

realization of story-based teachers. J. of the Learning Sciences 2(1), 95-134.

Foltz, P.W. (1996). Comprehension, coherence, and strategies in hypertext and linear text. In Rouet,Levonen, Dillon & Spiro (Eds) Hypertext and Cognition. Hillsdale, NJ: Lawrence Erlbaum.

Heller, R. (1990). The Role of Hypermedia in Education: A look at the research Issues. J. of Research

on Computing in Education, Vol. 22, 431-441.

Khan, S., Murray, T., & Piemonte, J. (2000). Qualitative Hypermedia Evaluation Methods. A paper

presented at the Eastern Educational Research Association, Clearwater, FL.

Jacobson, M.J., & Spiro, R.J. (1995). Hypertext Learning Environments, Cognitive Flexibility, and the

Transfer of Complex Knowledge: An Empirical Investigation. J. Educational Computing

Research, 12(4), 301-333.

Jonassen, D, Reeves, T. Hong, H., Harvey, D., & Peters, K. (1997). Concept Mapping as Cognitive

Learning and Assessment Tools. J. of Interactive Learning Research, Vol. 8 No. 3/4, 289-308.

Laurillard, D., Stratfold, M., Luckin, R., Plowman, L, Taylor, J. (2000). "Affordances for learning in anon-linear narrative medium." J. of Interactive Media in Education. [www.jime_open.ac.uk/00/2]

Luckin, R., Plowman, L., Laurillard, D. Straforld, M., & Taylor, J. (1998). Scaffolding Learners'

Constructions of Narrative. Proceedings of Int. Conf. of the Learning Sciences, 1998, 181-187.

Mannes, S. & Kintsch, W. (1987). Knowledge Organization and Text Organization. Cognition and

Instruction Vol. 4 No. 2, 91-115.

Marshall, C. & Iris, P. (1989). Guided tours and on-line presentations: How authors make existing

hypertext intelligible for readers. Hypertext-89 Proceedings, 15-26.

McAleese, R. (1989). Navigation and Browsing in Hypertext. Chapter 2 in R. McAleese (Ed.) Hypertext:

Theory Into Action. Norwood NJ: Ablex Publ.

Murray, T. (1993). Formative Qualitative Evaluation for "Exploratory" ITS Research. J. of AI and

Education, Vol. 4. No. 2, 179-207.

Murray, T (1999). Authoring Intelligent Tutoring Systems: Analysis of the state of the art. Int. J. of AI

and Education, Vol. 10 No. 1, 98-129.


Page 25

Murray, T. & Arroyo, I. (2002). Toward Measuring and Maintaining the Zone of Proximal Development

in Adaptive Instructional Systems. Cerri, Gouarderes & Paraguacu (Eds.) Intelligent Tutoring

Systems: 6th International Conference, ITS 200. Berlin: Springer Verlag, 749-758.

Murray, T., Condit, C. & Haugsjaa, E. (1998). MetaLinks: A Preliminary Framework for Concept-Based

Adaptive Hypermedia. Proceedings for ITS-98 Workshop on WWW-Based Tutoring. San Antonio,

August, 1998.

Murray, T., Condit, C., Piemonte, J., Shen, T., & Khan, S. (1999). MetaLinks—A Framework and

Authoring Tool for Adaptive Hypermedia. Proceedings of AIED-99, 744-746.

Murray, T., Condit, C., Piemonte, J., Shen, T.,& Khan, S. (2000). Evaluating the Need for Intelligence in

an Adaptive Hypermedia System. In Frasson & Gautheir (Eds.), Proceedings of Intelligent

Tutoring Systems 2000, Springer-Verlag:New York, 373-382.Murray, T., Gartner-Piemonte, J., & Kelleher, P. (2001). Evaluating Features for Conceptual and

Narrative Flow in an Adaptive Hyperbook. Proceedings of the International Conference on

Artificial Intelligence and Education: AIED-2001, 21-32.

Nelson, T. H. (1980). Literary Machines. Mindful Press: Sausalito, CA.

Norman, D. (1988). The Design of Everyday Things. Doubleday: NY.

Piemonte, J., Condit, C., Murray, T., & Khan, S. (1999). Tectonica Interactive: Methods and techniques

for presenting a web-based multimedia geological tutorial. Presented at the American Conference

on Geoscience Education. Available from the authors.

Plowman, L., Luckin, R., Laurillard, D., Stratfold, M., & Taylor, J. (1999). Designing Multimedia for

Learning: Narrative Guidance and Narrative Construction, In the Proceedings of CHI 99 (pp. 310-

317). May 15-20, 1999, Pittsburgh, PA USA.: ACM.Ross, T.W. (1993). Bloom and Hypertext: Parallel Taxonomies? ED-TECH Review. Autumn/Winter,

1993, 11-16.

Shen, T. (2000). Concept Mapping for Adaptive Hypermedia. Unpublished UMass Masters Thesis.

Specht, M. & Oppermannn, R. (1998). ACE - Adaptive Courseware Environment. New Review of

Hypermedia and Multimedia, Vol. 4, 1988, 141-162.

Spiro, R.J. & Jehng, J.C. (1990). Cognitive Flexibility and Hypertext: Theory and Technology for the

Nonlinear and Multidimensional Traversal of Complex Subject Matter. In D. Nix & R. Sprio

(Eds.) Cognition, Education, and Multimedia. Hillsdale, NJ: Lawrence Erlbaum.

Stanton, N.A. & Baber, C (1994). The Myth of navigating in hypertext: How a "bandwagon" has lost its

course! J. of Educational Multimedia and Hypermedia, Vol. 3 No. 3/4, 235-249.

Stern, M., & Woolf, B.P. (1998). Curriculum Sequencing in a Web-Based Tutor. Proceedings of

Intelligent Tutoring Systems-98.Suthers, D. & Weiner, A. (1995). Groupware for developing critical discussion skills. Proceedings of

CSCL '95, Computer Supported Collaborative Learning, Bloomington, Indiana, October 1995.

Thompson, G.R. & Turk, J.T. (1997). Modern Physical Geology, Second Edition. Saunders Publishing

Co: Philadelphia, PA.

Vassileva, J. (1994). A Practical Architecture for User Modeling in a Hypermedia-Based Information

System. In 4th International Conference on User Modeling, pages 115-120, Hyannis, MA, 1994.

Verhoeven, A. & Waarendorf, K. (1999). External Navigation Control and Guidance for Learning with

Spatial Hypermedia. J. of Interactive Media in Education Vol. 99 No. 1.

Wallace, R.M., Kuperman, J., Krajcik, J. & Soloway, E. (2000). Science on the Web: Students Online in

a Sixth Grade Classroom. Journal of the Learning Sciences, Vol. 9 No. 1, 75-104.

Weller, M. J. (2000). The Use of Narrative to Provide a Cohesive Structure for a Web Based ComputingCourse. To appear in the Journal of Interactive Media in Education. On line at www-

jime.open.ac.uk.

Wertsch, J. (1984). "The zone of proximal development: Some conceptual issues." In B. Rogoff, J.

Wertsch (Eds.) Children's Learning in the "Zone of Proximal Development," 7-18. San Francisco:

Jossey-Bass Inc.

Woods, W.A. (1975). What's in aLink: Foundations for Semantic Networks," in D. G. Borrow & A.M.

Collins (Eds.) Representation and Understanding: Studies in Cognitive Science, 35-82. New

York: Academic Press.

APPENDIX

Questionnaire items


Page 26

The post-use questionnaire used in Studies #1 and #2 included questions related to overall satisfaction, to

the navigation and orientation issues mentioned above, to inquiry/exploratory navigation behavior, and to

the importance and ease of use of several of the navigation features. Below we list or describe the most

important items. The questionnaire used in Study 2 was slightly different than that used in Study 1, but

the differences are not relevant to the level of analysis in this paper. All questions were Likert Scale

agreement questions (unless otherwise noted) with choices "strongly agree" "agree" "neutral" "disagree"

"strongly disagree."

ENJOYMENT AND LEARNING QUESTIONS

• I was able to answer the questions thoroughly.• I found Tectonica easy to use.

• I did NOT find Tectonica enjoyable to use.

• I would recommend Tectonica to others interested in learning more about geology and plate

tectonics.

• I learned more using Tectonica than I would have using the textbook.

• I would have preferred reading a textbook.

• I feel that I have learned the subject matter well.

• I feel that learning with Tectonica was a waste of time.

• I would recommend Tectonica to others interested in learning more about geology.

NAVIGATION AND FLOW QUESTIONS

• I visited and learned about topics that were not directly related to the main question.

• There was NOT an appropriate amount of material on each page.

• The time needed to use Tectonica was adequate for the material learned.

• If I diverged from a topic/page of interest, it was easy for me to return to that information.

• I felt lost as I navigated through Tectonica.

• The text flowed nicely from one page to another.

• The presence of introductory text facilitated my understanding of the material on the page.

• Which of these questions did you ask yourself most frequently? Where am I? What should I look

for next? How do I go there? Where have I been? (Fill-in question)

QUESTIONS RELATED TO SPECIFIC FEATURES

There were about 25 such questions and we show only a small sample here:

• I found page numbers useful in knowing where I was in the table of contents.

• The questions presented in the ‘Related Information" links compelled me to visit subject

material unrelated to my initial inquiry.

• I noted my position within the table of contents.

• The ‘Return’ button worked as expected.

• Having different categories of links (e.g. "famous catastrophes" and "how do scientists measure

this?") within the ‘Related Information’ Menu was useful.

Study #1 Results Details

Disorientation. Below are the questionnaire questions related to disorientation, along with the

average and standard deviations of the answers:

• Did you visit and learn about topics that were not directly related to your main question? 15 of the 19subjects answered "yes." This indicates that subjects did in general "explore" the material rather than

go directly to an "answer."

• If I diverged from a topic/page of interest, it was easy for me to return to that information. AGREE(ave 2.2 SD .9).

• I felt lost as I navigated through Tectonica. RARELY (ave 3.2 SD .8)

• Which question did you ask yourself most frequently: "Where am I? What should I look for next?

How do I go there? Where have I been?" 12 answered "what should I look for next?" and 3indicated they did not ask themselves any of these questions (the 4 remaining were evenly split


Page 27

among the other answers). This indicates that subjects were predominantly focused on the content

rather than trying to figure out the system or the task.

In the focus groups subjects were also explicitly asked questions about disorientation. Interviews and

focus-group data confirm the finding that the great majority of users felt that they knew where they were,

where to go next, and how to get there. As mentioned below, we found that when students needed to

"reorient" themselves they used a variety of tools, including TOC, search, and history pages.

Cognitive Overload. Below are the questionnaire questions related to cognitive overload, along

with the average and standard deviations of the answers:

• I was able to answer the questions thoroughly. AGREE (ave. 1.8 SD .9)

• There was NOT an appropriate amount of material on each page. DISAGREE (ave. 3.8 SD .9)

• The time needed to use Tectonica was adequate for the material learned. AGREE (ave. 1.9 SD .6)

These answers mostly relate to overload in terms of the content and task, and indicate that students werenot overwhelmed or confused with the content or task. Focus-group and interview data did not show

significant evidence of cognitive overload regarding use of the MetaLinks features. Answers to the 4th

Disorientation question above ("Which question did you ask yourself most frequently") also indicate that

users were not overly confused by the software.

Narrative Flow. Below are the questionnaire questions related to narrative flow:

• The text flowed nicely from one page to another. AGREE (ave 1.9 SD .7)

• The presence of introductory text facilitated my understanding of the material on the page. The

average was OFTEN (1.9).Narrative flow was explicitly addressed in the focus-groups and, in agreement with the questionnaire data

above, there was no evidence of narrative choppiness or unevenness.

Conceptual Flow and Content Readiness. In focus groups we asked whether students ever felt

that they had navigated to a page for which they were lacking the prerequisite knowledge and therefore

could not understand it. We found no evidence of frustration over the lack of prerequisite knowledge

(despite the fact that the majority had never taken a geology course).

Overall Enjoyment and Perceived Learning. There were 10 questions related to overall

enjoyment and perceived learning effectiveness. We will not list them all, but their general intentions had

significant overlap. Examples are: " I found Tectonica easy to use," " I did NOT find Tectonica

enjoyable to use," "Would you recommend Tectonica to others interested in learning more about geology

and plate tectonics?" and " I learned more using Tectonica than I would have using the textbook." Despitethe fact that most subjects indicated that they were not particularly interested in geology, the answers to

these questions indicated overwhelmingly that subjects found Tectonica enjoyable, easy to use, and

productive. This was slightly surprising, since we suspected that with so many features available to them

that some would find it frustrating to use (however, later we comment on features that were not used

much). Significantly, the two questions that compared Tectonica with book learning favored the

hyperbook (two of the 19 subjects would prefer a text book).

Areas where 3 to 5 subjects expressed frustration included (from focus-group data): not finding

content they wanted because the book was incomplete; the TOC, showing all of the 400 pages listed with

hierarchical nesting, was overwhelming; subjects were sometimes surprised to find themselves back at a

page they had previously been; and several subjects were confused with the parent/child/sibling metaphor

used for the hierarchical structure of the hyperbook.

Study #2 Results Details

Text Smoothing. We found no significant effect of the IT variable on questionnaire items or

navigation patterns. As in our previous study, subjects overall did not indicate problems with text flow.

For Q24 "the text flowed nicely" the average was 2.4 AGREE (SD .82), and for Q32 page "transitions

did not make sense to me" the average was 3.6 RARELY (SD 1.1). We can not tell whether our findings

from this and the previous study indicating good narrative flow was due to the inclusion of the intro-text

feature. A post-hoc analysis of the pages visited by subjects during the study indicated that the results are

limited by the relatively small number of times subjects saw intro-text.

Feature Salience. As in the previous (1999) trial the related links feature was used relatively

infrequently, a total 1.5 percent of all navigation moves. But there was a clear effect of the RL variable

on related links use. The related-links feature was used by significantly more subjects in the RL+condition (9 out of 12 subjects) than in the RL- conditions (2 out of 12 subjects).18 A similar effect is

18 The use of this feature is distributed relatively evenly among the three task questions, indicating that

the availability of related links was not a function of what topic subjects were exploring.


Page 28

found with the mirror feature, children links, which was used much more often, 9.3 percent of the total

moves overall. In the RL- condition child links (which were visible in this condition) were used 2.8 times

more often. A supporting pattern was found with the questionnaire items. When related links are shown,

the related links feature is generally rated more highly and the subtopics links feature is rated lower. The

converse is true when related links are hidden. The effect of RL on Q3 the "related information (feature)

was useful" was significant (p=.001), and the effect of RL on Q59 "related links are (not) difficult or

confusing" was significant ( p=.04, with computer knowledge treated as a covariate).

Exploratory navigation and Custom Depth Control. As mentioned above, exploratory

navigation is supported by both the related links and custom depth features. We attempted to measure

exploratory/inquiry navigation behavior in several ways. The Exploration Questions were: Q8 "did you

visit and learn about topics that were not directly related to you main question?", Q15 "Relatedinformation (feature) compelled me to visit subject material unrelated to my initial inquiry," and Q31 "the

software allowed me to navigate though the material according to my curiosity." We are interested in the

relationship between these questions and the RL condition and the usage of the related links and custom

depth features. Overall we found the following descriptive statistics: Q8 Ave 1.29 YES (SD .46), Q15

Ave 2.46 AGREE (SD .9), and Q31 Ave 2.0 AGREE (SD .7). Related links accounted for 1.5 percent of

total moves (compared with 1% in Study #1). Of the 180 unique pages visited by users, 169 had related

links available (so scarcity in encountering the feature was not an issue as it was with the intro-text

feature). The custom depth feature was the most often used feature (overall it accounted for 33% of the

moves). It was used equally whether related links were shown or hidden.19 There was a wide variation

in the use of this feature among subjects, with range using it from 6% to 78%.

The frequency of use of the related links feature was positively correlated (p=.04) with theSatisfaction measure, and positively correlated (p=.01) with Q8. However there was no significant

correlation between Q15 or Q31 and the RL condition or related links page frequency. The frequency of

use of the custom depth feature had a positive correlation with the Satisfaction measure (p=.004), a

weakly positive significant correlation (p=.055) with Q31 "navigated according to my curiosity," a

weakly significant positive correlation (p=.07) with the Exploration questions, and positive correlation

with Q14 "if I diverged from a topic it was easy for me to return." There was no significant correlation

between custom depth use and the RL condition.

Other navigation issues. We have already discussed exploratory and inquiry navigation behaviors.

We are also interested in how subjects responded to the questions about cognitive overload,

disorientation, conceptual flow, and narrative flow. Questions 8, 14, 23, 32, and 26 were related to

disorientation; 4, 12, and 35 to cognitive overload, 23 and 32 to narrative flow, and 31 and 33 to

conceptual flow. The results were very similar to those from Study #1. The vast majority of subjectsindicated that they did not have problems with the four types of navigation problems. For the 12

questions relevant to navigation issues, depending on the question, between 1 and 7 of the 24 subjects

indicated that they did experience a navigation problem. There were statistically significant positive

correlations between the navigation questions and the Satisfaction measure. We found no significant

correlations between the navigation questions and the RL condition or use frequencies of the navigation

tools. We found that higher use of the more advance navigation features (related links and custom depth)

did not have a significant effect on the Navigation Issues questions.

19 This result differs from Study #1 in that in the earlier study subtopic links were used most often, and

Custom Depth only 10%. We have no explanation for this difference.

MetaLinks: Authoring and Affordances for Conceptual and Narrative ...

Documents

Transcript of MetaLinks: Authoring and Affordances for Conceptual and Narrative ...