C R E S S T / U C L A

Paper presented at AERANew Orleans—April 2000

Using Technology to Assess Students’ Web Expertise

Davina C. D. KleinCRESST/UCLA

Louise YarnallCenter for Technology in Learning, SRI International

Christina Glaubke CRESST/UCLA

C R E S S T / U C L A

C R E S S T / U C L A

2

Technology Focus: WWW

Over 1.5 billion Web pages currently available, increasing at rate of 1.9 million pages per day

At least 40% of U.S. classrooms linked to Internet With 9 million children using Internet at school

Most common classroom use for WWW is research 3rd most common use of computers at school

Has surpassed drill-and-practice software

Nearly 90% of teachers perceive classroom WWW access as valuable or essential for their teaching

C R E S S T / U C L A

3

Research Approach

Study experienced Web users

Identify set of measures to be used to assess students’ fluency with the World Wide Web

Web Expertise Assessment (WEA) Web-based, authentic performance assessment

Tasks require students to navigate through large information space searching for relevant information and bookmarking relevant findings

All measures logged and coded

C R E S S T / U C L A

4

WEA Search Task

Imagine you are learning about the U.S. presidents in your history class. Your teacher has asked you to write a report about what presidents said during their speeches when first elected to office. She has asked you to find out which presidents spoke of the importance of an educational system available to all without charge.

Use WEA to find this information for your report.

Find as many useful pages as you can.

Bookmark pages by clicking on the Add Bookmark button near the top of your screen.

You may bookmark as many useful pages as you think necessary.

*

C R E S S T / U C L A

11

WEA Background Questionnaire

Used to evaluate students’ WWW background knowledge Paper-and-pencil survey

Students rated statements on scale of 1 (“I really don’t agree”) to 5 (“I really agree”) e.g., “The information on the World Wide Web is not

very useful”

C R E S S T / U C L A

12

Participants

120 middle and high school students Students had strong technology

background

Students had access to WWW in class

Students were familiar with navigating the World Wide Web

C R E S S T / U C L A

13

Methods

Students completed WWW background questionnaire

Students trained on WEA

Students given 20 minutes to complete WEA search task Reminded to bookmark relevant pages

Data coded with high interrater reliabilities (ranging from .97-.99)

C R E S S T / U C L A

14

Evaluating WEA PerformanceItem Factor loadings

F 1 F 2 F 3 F 4Number of times back used .97Number of steps in search .92Number of revisited info pages .76Average bookmark score .92Quality of bookmark set .87Efficiency of search .85Web helpful with search .81Web info useful .80Web info correct .73Web info detailed .69Number of good searches .91Quality of keyword search set .77Number of redirected searches .66Eigenvalues 2.54 2.37 2.34 2.00Percent variance explained 19.5% 18.2% 18.0% 15.4%Note. Only factor loadings with absolute values of .30 and higher shown.

navigationalstrategiesα = .88

findingabilityα = .86

backgroundWeb

knowledgeα = .76

searchingexpertiseα = .71

C R E S S T / U C L A

15

Creating WEA Scales

Navigational strategies (α = .88) Number of times back button used Number of steps in search Number of revisited information pages

In general, students navigated well Students used back often (M = 17) for orientation Students revisited over one third of info pages

visited, orienting themselves in Web space (M = 4) Students completed many steps (M = 93)

C R E S S T / U C L A

16

Creating WEA Scales (con’t.)

Finding ability (α = .86) Average bookmark score Quality of bookmark set Efficiency of search

In general, students able to find information Average bookmark peripherally relevant to task

(M = 1.8 on 0-3 scale) Quality of bookmark response set good

(M = 1.8 on 0-3 scale) About one fifth of pages bookmarked

appropriately (efficiency M = .21)

C R E S S T / U C L A

17

Creating WEA Scales (con’t.)

Background Web knowledge (α = .76) WWW helpful in finding information Information on WWW not very useful (r) Information on WWW is accurate/correct Not a lot of detailed information on WWW (r)

In general, students familiar with Web Students agreed that WWW is helpful in finding information (4.2) Students disagreed that information on WWW is not useful (2.1) Students were neutral/agreed that information on WWW is

accurate (3.1) Students disagreed that there is not a lot of detailed or in-depth

information on WWW (2.3)

C R E S S T / U C L A

18

Creating WEA Scales (con’t.)

Searching expertise (α = .71) Number of good searches Quality of keyword searching set Number of redirected searches

In general, students had difficulty searching (consistent with literature) Number of good searches low (M = 2.0) Quality of keyword searching set rather poor

(M = 1.7 on 0-3 scale) Students redirected searches, browsing search

output before selection (M = 2.3)

C R E S S T / U C L A

19

Results

Work explored constructs underlying students’ Web fluency

Students demonstrated their abilities to search, navigate, and find information

Analyses identified important individual measures and we coded these reliably

Composite indicators from factor analysis make sense theoretically

C R E S S T / U C L A

20

Next Steps

Future research will focus on validity of assessment Can we distinguish between expert and novice

Web users with WEA?

Is WEA sensitive to instruction?

Once validity established, use WEA to examine effects of Internet usage

Link assessment and instruction Create guidelines for teaching the Web effectively

C R E S S T / U C L A

21

For More Information

Visit our Web site at: http://www.cse.ucla.edu/CRESST/pages/aera00.htm

Available: Overheads of this presentation

Full paper

And much, much more...