IPS Usability Testing Report - Report

8/7/2019 IPS Usability Testing Report - Report

1/56

USABILITY TESTINGREPORT

INDIANAPOLIS PUBLIC SCHOOLS

WWW.IPS.K12.IN.US

I N F O I 5 4 3 U S A B I L I T Y A N D E V A L U A T I V E M E T H O D S

I N D I A N A U N I V E R S I T Y S C H O O L O F I N F O R M A T I C S

F A L L 2 0 1 0

D R . D A V I D E B O L C H I N I

I N S P E C T E D B Y T E A M 2 :

S T E V E N E N T E Z A R I

H A I D A N H U A N G

J A Y W H E E L E R


2/56

2

Table of Contents

I. Executive Summary ................................................................................................. 4

II. Instruments and Methods ...................................................................................... 7Participants .................................................................................................................................................... 7Data Collection............................................................................................................................................. 7Environment & Timeframe....................................................................................................................... 8Tools................................................................................................................................................................ 9Tasks................................................................................................................................................................ 9

III. Results ................................................................................................................ 11Task 1 School Board Meetings .......................................................................................................... 15Task 2 Finding the Technology Plan ................................................................................................ 20Task 3 College Resources .................................................................................................................... 24Task 4 Find the Closest School ......................................................................................................... 27

Task 5 Magnet School Application ................................................................................................... 31Task 6 Payroll Contact Information (Teachers Only).................................................................. 34Task 7 Teacher Amendment (Teachers Only) ............................................................................... 34

IV. Synthesis of Results from Inspection ................................................................... 36Content........................................................................................................................................................ 36Information Architecture........................................................................................................................ 37Navigation................................................................................................................................................... 40Presentation................................................................................................................................................ 41

V. Overall Recommendations for Improvement ........................................................ 43Content........................................................................................................................................................ 43

Navigation................................................................................................................................................... 44Information Architecture........................................................................................................................ 45Presentation................................................................................................................................................ 46

VI. Appendixes ......................................................................................................... 48Appendix A: Pre-test Questionnaire .................................................................................................... 48Appendix B: Usability Test Script......................................................................................................... 50Appendix C: Post-Task Questionnaire ................................................................................................ 54Appendix D: Post-Test Questionnaire................................................................................................ 55Appendix E: Usability Testing Videos................................................................................................. 56


3/56

3

Table of Figures

Figure 1 - Average times it took each participant to finish each task. .......................................... 11

Figure 2 - Full-Success/Partial-Success/Give-Up rates for each participant per task.......... .... 12Figure 3 - Summary of quantitative data showing average time on task, lostness, success rates

and ease of use for each task......................................................................................................... 13Figure 4- Correlation of measured items ............................................................................................. 14

Figure 5 - IPS Homepage ..................................................................................................................... 16

Figure 6 - Main Navigation for IPS Site .......................................................................................... 17

Figure 7 - Task 2 Quantitative Summary ...................................................................................... 20

Figure 8 - Divisions/Departments page with no Technology section .......... .......... ........... 21

Figure 9 - Task 3 Quantitative Summary ...................................................................................... 24

Figure 10 - Long list of state colleges and universities ........................................................... 25

Figure 11 - Task 4 Quantitative Summary ................................................................................... 27




4/56

4

I. Executive Summary

The Indianapolis Public School System (IPS) Corporation website is an informational

website about the Indianapolis Public School Corporation. The purpose of the site is to

communicate news, events and educational related information to primary groups of users

who include faculty, parents and the community.

This report identifies key findings from the usability study conducted by our team on

the IPS Corporation website. The usability study included 11 participants comprising 5

community members, 4 parents and 2 teachers. All participants performed 5 tasks except for

teachers who performed 7. Each task was recorded to captured data, which was later

analyzed and included in this reported. Tasks were selected for their saliency in providing

information to perspective users based on activities that were performed both routinely such

as finding school board meetings to those that were performed less frequently such as finding

cost associated to a technology plan budget. Teachers performed two additional task which

were designed to help the team identify how recognizable the layout and content was

between the general users understanding of the site and teachers understanding of the site. In

general, the tasks selected had to include at least 3 steps to be considered for the study. In

addition, an earlier usability inspection was performed which helped establish the direction

and objectives for the final usability study.

The following are the key set of findings captured from both of the usability

inspection and usability study, which have been identified as having the most impact to the

usability of the website:

1. Content: Many participants became confused due to vague, non-dated or missinginformation.


5/56

5

2. Navigation: Many problems were due to inconsistency in the location and style oflinks. Additionally, the page banner located on every page due to its size and

distracting messages causes all the users unnecessary scanning of pages to locate the

appropriate navigation and/or content identified in the set of tasks.

3. Information Architecture: The website needs to align to how the target usersunderstanding a traditional website. Many problems surrounded the organization of

topics within a subject area. Topics that belonged to different subjects were included

into other unrelated subjects, which caused vague or inconsistent relationships

between information that confused participants.

4. Presentation: A lack of consistency in style and organization of page elements (i.e.links, headers, banners) caused additional overhead in the participants ability to

navigate and use the website effectively.

After careful review of both the usability inspection and evaluation study given the top

four impacted areas mentioned above our team recommends the following top seven critical

areas for improvement. Additional recommendations can viewed in the Overall

Recommendations For Improvement section later in this report.

Key Recommendations:

1. Establish a checklist and a content review board to ensure information is up-to-date and correct

2. Ensure dates are captured and identified on items that have relevance around adate or time3. Use proper formatting ensuring consistency between text and links, as well as

banner picture and links.

4. Keep the most necessary navigation and get rid of the ones irrelevant to the page.Make sure the appearance and layout of the navigation are salient and consistent.

5. Provide a sitemap of the website to aid the user gain an overview of the logicalarchitecture of the website, so that they can find the information they need.


6/56

6

6. Design should be include visual cues in the navigation that highlights their currentlocation. Ensure each page has a title matching the pages content.

7. Make the navigation bar around the banner area more salient. Avoiding usingcolors similar to the banner for the background of navigation bar. Make theappearance of the navigation bar more touchable like a button.


7/56

7

II. Instruments and Methods

This study employed several tools and techniques for collecting information

pertaining to the usability study. In the following sections an account of participants,

methods and instruments are explained. A complete list of artifacts including the Pre & Post-

Test Questionnaire, Post-Task Questionnaire, Usability Study Script among others used in

the study can be found in Appendix.

Participants

The IPS district website caters to three primary groups of users: Teachers, Parents

and the Community. For our study we focused on these key user group which we solicited

11 participants (5 community members, 4 parents and 2 teachers). All 11 participants did

have at least some computer experience with 7 self-identifying as very experienced. All 11

participants had over 20 hours of Internet usage per week. Additionally, 9 the users had

never visited the IPS website, however, 4 parents and both teachers visited their own school

corporations website.

Data Collection

Due to the complex nature of collecting both quantitative and qualitative data our

team developed and employed a well-crafted process and artifacts, which were used in the

study (see Appendix B). The evaluator first started by traveling to the participants location

with a laptop, which was loaded with recording and capturing tools (see tools below). Once

the evaluator was setup they began by giving the participant a pre-test questionnaire, which

solicited demographic information along with computer and Internet usage information.

Once the participant completed the pre-test questionnaire they were then read a script that

provided an over view of the study which included information regarding the purpose of the


8/56

8

study along with the intention of the information collected, privacy & confidentiality and

their rights as a participant in the study. After the participant agreed to the terms the

evaluator then gave the participant an introduction to the computer system and to the

website allowing them to complete a nominal task before the official test was initiated.

Once the participant was familiar with the setup of the laptop, browser and site the

evaluator began the test. The evaluator began recording the session along with reading each

task out load. The evaluator recorded both the start and end times along with the success,

partial success or failure for each of the task. In addition the participants we encouraged to

think out load as they worked through each task. This helped provided additional insight and

rich qualitative data, which was noted and later analyzed by the evaluator and team. Also

after each task the participant was given a post-task closed question questionnaire, which

solicited their feedback on how satisfied with the ease of completing the task they were. The

questionnaire ranged from 0 (strongly disagree) to 5 (strong agree).

After all the tasks were completed the evaluator then moved to the last section of the

script, which included the post-test questionnaire and wrap-up. The post-test evaluation

included several open-ended questions to give the participant a chance to give their overall

impressions of the website and to reflect on the site as a whole. Once completed with the

post-test questionnaire the participants were again thanked for their participant and given

contact information of the evaluator in case additional thoughts or concerns arose about the

website or the study in general.

Environment & Timeframe

Due to the short timeframe for this study and the need to emulate an environment

that best represented our key user groups most of the testing was conducted in either the


9/56

9

participants home or place of work. The testing was conducted between November 1st and

November 8th, 2010 at times that were most convenient for the participants.

Tools

A variety of tools were used to help record and capture data from the participants.

These tools included Clearlefts Silverback 2.0, Camstudio and Syniums Screenium for local

participants and Ciscos Webex for remote participants. These tools allowed our team to

record the participants actions on the screen, the participants verbal comments along with

the participants face, which allowed us to capture both verbal and nonverbal cues.

Tasks

Below are the following tasks used in the study for participants to complete. Tasks 6 and

7 were included in the test if the participant was a teacher. Tasks were selected for their

saliency in providing information to perspective users based on activities that were performed

both routinely such as finding school board meetings and to those that were performed less

frequently such as finding cost associated to a technology plan budget. The usability

inspection was also used to identify steps used within the tasks that would expose

problematic areas found by our experts. The additional tasks for teachers were designed to

help the team identify how recognizable the layout and content was between the general users

understanding of the site and teachers understanding of the site. In general, the candidate

tasks had to include at least 3 steps (or navigation points) to be considered for the study.

1. Find the next school board briefing meeting so that you can attend.As a parent you are growing concerned about the direction the IPS School board ismoving towards concerning year round school. Another parent mentioned that thereare school board meetings where you can go and voice your concerns. Assume todayis December 1st and you want to go to the next briefing session. Please identify thedate for this meeting.


10/56

10

2. Find the dollar amount IPS has budgeted towards technology in itsTechnology Plan.

As a parent you feel that your childs ability to learn and use technology is very

important. In the midst of growing concerns about budget cuts you are interested inknowing how much money IPS has allocated to implementing new technology. Findthe budgeted amount IPS has allocated in its Technology Plan.

3. Find contact information for Sawyer College in Merryville.As a parent of a high school junior, another parent told you that IPS has resources foryou and your child to start exploring colleges. Find contact information about SawyerCollege in Merrillville.

4. Find the middle school closest to your house.As a parent you want to identify the schools that are nearest to your home where yourchild would attend middle school. Your house is located at 816 N. Audubon Road,Indianapolis, IN 46219. Please name the closest high school to your house and findthe enrollment information for that school.

5. Find the 2011-2012 English Student Application for the Magnet Schools.You noticed that a school located near your house was a magnet school. Pleaselocate the 2011-2012 English student application for this magnet school.

If you are a teacher, please complete the additional two tasks:

6. Find the telephone number and location of IPS payroll.As a new teacher you had an issue with you pay check this week. You contacted thefront office at your school, however, because they do not directly deal with payrollthey directed you to the corporation website to find the appropriate contactinformation and the location were you needed to go to sort everything out. Identifythe telephone number and location of the payroll office

7. Find amendment 5233 of Teacher Contract.As a teacher you wanted to review updates to the 2010 teacher salary informationspecifically Amendment 5233. Locate information about Amendment 5233.


11/56

11

III. Results

After careful analysis during both live and recorded user-testing review, we identified

trends common to many of our participants. In our analysis, we looked at quantitative data;

specifically with regard to time-on-task, success/partial-success/fail rates per task, post-task

questionnaire answers, and a calculation on the lostness of the participant. Qualitative data

that we focused on include our team observations, questionnaires, demographic data, and

other such items that gave a sense of the users state of mind, the sites pleasurability, and the

overall acceptance of the site by the user. These results were analyzed first individually, then

in comparison with each team-member for inter-rater reliability. They were then cross-

referenced to the heuristic and scenario-based evaluations conducted previously for this site.

Here is a summary of the tasks, mentioned before:

Task 1 Find the next school board briefing meeting so that you can attend.Task 2 Find the dollar amount IPS has budgeted towards technology in its Technology Plan.Task 3 Find contact information for Sawyer College in Merryvile.Task 4 Find the middle school closest to your house.Task 5 Find the 2011-2012 English Student Application for the Magnet SchoolsTask 6 Find the telephone number and location of IPS payroll

Task 7 Find amendment 5233 of Teacher Contract

Figure 1 - Average times it took each participant to finish each task.

0

50

100

150

200

250

300

350

400

1 2 3 4 5 6 7

Time(sec)to

completetask

Time on Task Analysis

Time-On-Task

Perceived Ease


12/56

12

We measured the time it took each participant to accomplish the tasks. Above you

see the average time on task for the participants per task. Notice the times for tasks two and

four are highest. These two tasks were proven to be the most compelling for our participants;

as you will see in later parts of this paper. This graph also compares the time on task with the

perceived ease for the participant. Notice, on task 2, that the perceived ease is significantly

lower than the other 6 tasks. This indicates that the user found it most difficult andspent the

most time attempting to complete it. One thing that should be mentioned with regards to the

time on task calculations is that the time on task takes into account the attempts by

participants that were unsuccessful and incorrect. Next you will see success rates which will

correlate to these findings.

Figure 2 - Full-Success/Partial-Success/Give-Up rates for each participant per task

For every participant and for every task the participant started, we kept a tally on the

number of successful completions, partial successes, and complete withdraws. A successful

completion occurs when the user completes the task with no input (or input that would not

alter the intended course of the participant) from the proctor. Partial success occurs when the

participant has successfully completed the task, but was, at some point during the scenario,

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7

Analyzing Levels of Success

"Give up/ wrong/ use

outer tool"

Partial success

Complete success


13/56

13

assisted by the proctor. Give up or withdraw simply means that the participant never

completed the task; even after given help. While task 2 has the lowest success rate of all the

tasks, notice, from before, that it still maintained the highest time on task of all tasks. This is

indicative of the user attempting, by-all-means, to complete the task yet failing to do so.

Figure 3 - Summary of quantitative data showing average time on task, lostness, success ratesand ease of use for each task.

After combining the two items listed before, time on task and success rate, with a

measure of lostness and ease of use, we can see that the most problematic tasks were Task 2

and Task 4. Both of these tasks displayed the highest time-on-task and lostness levels. They

also displayed the lowest success rate and ease-of-use averages. This not only indicates that

these two tasks were quantifiably more difficult than the rest, but it also indicated that the

user could feel this while completing the task; based on the post-task questionnaire regarding

the users ease-of-use.

Another thing to note is that tasks 6 and 7 were performed only when the participant

was a teacher. There were a total of two teachers, who both were parents. All 11 participants

completed tasks 1 through 5. The data shows that these two tasks (task 6 and task 7) were

both completed with complete success, relatively no lostness, and a comparatively low time

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

0.00

50.00

100.00

150.00

200.00

250.00

300.00

350.00

Task

1

Task

2

Task

3

Task

4

Task

5

Task

6

Task

7

Quantitative Summary

Time on Task

Lostness

Success Rate

Ease of Use


14/56

14

on task average. This indicates that the teacher-participants may have been experts with the

system as they need to use it for daily-work. While this may imply that increased exposure to

the site generates higher satisfaction with the usability, it is important to note that the user

may become habituated to a poor usability experience and learn the wrong ways efficiently.

After examining tasks 1 through 5, we noticed an interesting phenomenon regarding

task 3. Out of the 5 main tasks, task 3 had the lowest time on task of all of them. The ease of

use was relatively high in comparison to the other tasks. Success rate tended to be higher than

average and the lostness of the task completion seemed to be extremely low. These facts may

imply a conflict between our usability evaluation based on heuristics and scenarios compared

to the usability testing done by our participants. This disconnect will be explored later when

looking at each task individually.

LostnessTime onTask

SuccessRate Ease

Lostness 0.92 -0.93 -0.39

Time onTask -0.95 -0.68

Success Rate 0.66Ease

Figure 4- Correlation of measured items

After running a correlation on the measured items, we found three interesting, and

very strong, correlations. First, for the average participant there was a +0.92 correlation for

time on task and lostness. This indicates that as the participant would spend more time on

the page, they would get more and more lost. This also implies that time was not spent solely

on a few pages for purposes of our time-on-task. Lostness factors in page counts, therefore,

the user spent a great deal of time searching many pages to complete the task. We also found

a -0.93 correlation regarding success rates and lostness. This is very interesting as it indicates

that as the user got more and more lost in the system, they would have a much less chance of


15/56

15

succeeding. It also implies that the more the users would succeed, the less lost they would be

in the system; meaning they needed less clicks to get to their final destination. The final and

strongest interesting correlation deals with the participants time on task and their success rate.

Interestingly, as the success rate of an individual would go down, their success rate would go

up. This implies that when the participant did succeed, they did so quickly and the items that

the participant was unsuccessful on took more time.

Task 1 School Board Meetings

Task description

Average Time on Task/Average of All

3:49 / 3:54

Success Rates

Full-Success 7Partial-Success 2Fail 2

Lostness0.53

Participants completed task 1 after completing a warm-up task to help avoid the

learning curve of the site. While this does hellp with avoiding some skewed statistics, it does

not take away from the fact that users of any site will be more proficient with general

navigation, logical constructs, and overal usability of any site; good or bad design.

As a parent you are growing concerned about the direction the IPS School boardis moving towards concerning year round school. Another parent mentioned that

there are school board meetings where you can go and voice your concerns. Please

identify the date for the next briefing session.

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

0.00

50.00

100.00

150.00

200.00

250.00

300.00

350.00


Time on Task

Lostness

Success Rate

Ease of Use


16/56

16

Given that, however, the time on task for task 1 was right on average for the first five

tasks (the tasks intended for parents). Seven of the participants completed this task without

any help from the proctors while 2 needed assistance to finish successfully and 2 failed or

gave up. The users had a lostness of 0.53. This number does indicate a relatively high lostness

for this task.

Usability Issues encountered by participants:

# Problem 1: Missed Navigation

* Problem Description:

We noticed that participants failed to

recognize the main navigation area as

clickable links; a phenomena Don Norman

calls Banner Blindness. This is especially

harmful on the homepage of IPS site due to

the dashboard-like style. We observed that users tended to spend a large amount of time

specifically looking for their course-of-action

in the homepage content structure. This was

emphasized in our usability testing in a section denoted Homeless Homepage where we

displayed the problems with a homepage containing only links. We saw that this dashboard-

like style lead the participants down a path that reinforced them to think the homepages main

content had allt he answers for their task. This information architecture set up is one of the

reaons contributing towards a high lostness level. It was assumed that if it wasnt in the

homepage area, it was a task that could not be completed.

* Severity | 1 (Show Stopper)

Figure 5 - IPS Homepage


17/56

17

* Design defect | Navigation

#Problem 2: Inconsistent Labeling

*Problem Description:The users continuously wanted to find

school board calendars via the calendar links on

both the homepage and the individual school sites.

They were surprised and a little frustrated when

browsing the page that appeared as a result of

clicking the calendar link to find multiple different

types of calendar, but none pertaining to the

school board meetings. The content simply didnt

match the participants expectations. It was

interesting to see that when users ended up on the

IPS school board calendar page, they did so after

passing the link, on average, 4 times before. This accounts for the high lostness displayed by a

majority of the participants.

* Severity | 2 Medium

* Design Defect | Navigation

#Problem 3: Logical Constructs Not Adhered To


Participants were fixated on the fact that when looking for school board meetings,

you are looking

from the

The access to the school boardmeeting should also be in the Parents

part

The task is about parents school

board meetings, so I think I can find

the meeting time in parents

I dont think it would be in general

information because the title seemed

to be about static information, but the

meeting times seem more dynamic.

Balanced Calendars seems more

relevant.

I wouldprefer it showed general

information once I am on the school

board page.

The members list in school boards isdistracting. It makes me think the

School Board [link] is about personal

introductions with no information

about events

What Is DOE?

Figure 6 - Main Navigation for IPS Site


18/56

18

perspective of a parent. While logically, school board meetings may seem to belong under the

category of School Board, our participants still felt that it was an activity that Parents attend.

Given this, for the participants, it seemed only logical to navigate to the main parent link and

search the information in that subdirectory. It was only after much confusion in the parent

directory, that many of ths users finally decided to reevaluate their approach and searh other

main links or simply give up at this point. This emphasizes the idea that the participants had

enormous expectations regarding the location of school board meetings. The majority of

participants expected to find school board meetings within the parents link as opposed to the

school boardlink.


* Design Defect | Information Architecture

#Problem 4: Banner Data Inaccessible

* Problem DescriptionOne item that was not prevelent in our usability evaluation before, but became very

evident during our user testing, was the problems with the banner. For a few participants,

when the rotating banner rotated to a particular ad dealing with school borads (although it

was an ad that was not relevent to specific task), the users felt the need to wait for the ad to

come full circle to proceed. This rotation of the banner caused a higher time on task because

of the wait time for it to come back around.



#Problem 5: User Calculation



19/56

19

Once the users navigated to the correct page, there were some characteristics of the

interaction that were quite interesting. Users noted that when navigating through the list of

school board meetings, it was hard to identify the next meeting because of the order of the

list. They expected the next meeting to be displayed first and didnt want to hunt and search

down the page for the every day to commpare it to today. They needed some kind of

distributed cognition to assist with their calculations of today, the displayed date for the

meeting, and the difference between those two dates.




20/56

20

Task 2 Finding the Technology Plan

Task Description


5:28 / 3:54

Success RatesFull-Success 3

Partial-Success 4Fail 4

Lostness0.60

As evident from the graph above, the time on task for task 2 was higher than any of

the other tasks in this report. At 328 seconds, participants seemed to take as more time to

complete this than a majority of the other tasks, even though the ideal path for a majority of

these tasks consisted of only 4 pages. Also, there were fewer successes on this task than for

any of the other main tasks (tasks 1 through 5). Both the time on task and success rates

mentioned before alludes to a high lostness level. As shown in the graph above, the lostness

level for this task was higher than any other task in this report.

Usability Issues encountered by participants

#Problem 1: Skip Navigation

* Problem Description

0.000.501.001.502.00

2.503.003.50

0.0050.00

100.00150.00200.00

250.00300.00350.00


Time on Task

Lostness

Success Rate

Ease of Use

As a parent you feel that your childs ability to learn and use technology is very important. In

the midst of growing concerns about budget cuts you are interested in knowing how much

money IPS has allocated to implementing new technology. Find the budgeted amount IPS has

allocated in its Technology Plan.

Figure 7 - Task 2 Quantitative Summary


21/56

21

Participants consistently skipped the navigation of the page and skimmed through

the pages content in hopes of finding the answer. This contributes a great deal to the average

time on task for this task. We believe the participants felt the need to skim every page for the

answer due to not only the banner blindness explained earlier, but also the mislabeling of

pages. Page titles within the About IPS section are all the same. The participant had trouble

differentiating between which About IPS section they were in. Also, related to this, many

participants were unsure of their current location on the page due to page title and navigation

not retaining their visited or current pages. This issue was addressed before in our evaluation

in a section under the Information Architecture section labeled Where am I.

* Severity | 1 Show Strpper

* Design Defect | Information Architecutre

#Problem 2: Language Disparities

*Problem Description:

There were some semiotic related

problems regarding the language used on

some of the pages. For an example of this,

consider this tasks language of Technology

Department. When attempting to find the

technology department in the

Divisions/Departments page (which was

navigated to quite efficiently by most

participants), they all immediately scrolled

to the Ts to locate Technology, when in

Figure 8 - Divisions/Departments page with no

Technology section


22/56

22

actuality, the correct page was Information Technology. Four users simply left the page,

assuming they were in the wrong area. A few of them were redirected back to the page and

given the hint of thinking of synonyms of Technology while others found it, simply by

chance. This is a prime example of ensuring that the language written matches the language

understood by users. In this case, the users are parents who may or may be familiar with

technology, but usually not. Ensuring both methods of language are covered will allow users

to find what they are looking for, rather than simply leaving the page as our participants so

eagerly did.


* Design Defect | Semiotics

#Problem 3: Where to Begin

*Problem DescriptionA couple of our participants navigated to the

superintendents blog or some area of news such as recent

performance metrics or school district goals. Some even expected

it to be a form of general information which, they assumed,

would be located in an About Us type of page. Regardless of

where the users thought the budget, or technology plan, would be,

they didnt expect it to be where it is initially. They all got to the

intended page quickly, but only after some hesitation at the

beginning of the encounter.



I dont even know where

I am

If Im looking for

something like report or

annual then its notgoing to be connected

with a department; its

for the whole

organization!

The technology budget

plan should be under

About IPS

I think the tech-budget

belongs to the IPS budget

in general.

Honestly, I wouldprobably just end up

calling them at this


23/56

23

#Problem 4: Navigation of PDF

*Problem Description:When the users found the technology plan, time on task was taken. The goal here was

to see how long it would take our users to loacte and access the technology plan. After the

task was deemed copmlete, we allowed the users to continue without telling them we werent

timing them any more. This proved to be rather revealing about another issue we didnt even

think about. Once the user has accessed the technology plan, finding the actual information

proved to be rather difficult. The navigation and information architecture of the pdf

document was not easily navigable and elicited some frustration in our users. The users, even

when in the document, didnt know where they were or how to get to where they wanted to

go. We recommend some form of hypertextual linkage on either the pdf document or an

html page with the same information as the pdf document. As of now, the table of content

listed an approximate page that the user had to scroll to; if they used the table of contents at

all.

* Severity | 1 Show Stop

* Design Defect | Navigation & Information Architecture


24/56

24

Task 3 College Resources

Task Description


2:23 / 3:43



Lostness0.31

Time on task for task 3 was lower than any of the other 5 main tasks. 143 seconds

surpassed all of our expectations and gave some insight on what we thought was a clear

usability problem. Task 3 had more Full and Partial successes than any other task. Lostness,

while still high, was the lowest of all five tasks as well. The one participant that failed to

complete this task or gave up on it does not account for a major shift in the data due to the

fact that the average for the other 10 successful participants average approximately the same

time on task.

As a parent of a high school junior, another parent told you that IPS has resources for you and

your child to start exploring colleges. Find contact information about Sawyer College in

Merrillville.

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

0.00

50.00

100.00

150.00

200.00

250.00

300.00

350.00


Time on

TaskLostness

Success Rate



25/56

25

The pages on this site

contained a lot of information in one

location and were a major focus of our

usability testing. Users searching for

the school began to get frustrated at

the number of scrolls and page clicks

they had to go through just to find

information on a state university.

Some didnt think that resources was

the appropriate place for state universities.

Others, similar to the problems outlined in

task 1, believed that state university searches were activities the parents would complete and,

therefore, would be located in the parents section.

Some areas of focus that were identified during this task by

our team dealt mostly with general elements needing correction as

opposed to task-specific items. Some such items include some

elements discussed earlier. Some examples of these include

navigation blindness due to the banner, mislabeled pages, and

inconsistent sub-navigations.

As discussed earlier, this task had a lower time on task than

any of the other tasks in the 5 main tasks for parents. Some other

important discoveries are made regarding the quantitative data as well.

The success rate, on average, was ultimately higher than any other 5

Figure 10 - Long list of state colleges and

universities

Can I just Googleit?

Does thenavigation change

every time I go to

a new page?

Why are there

two home links?

(While browsingwithin the

schools) How can I

get back to that

other page?

I would have just

Googled this


26/56

26

main tasks and the lostness level was lowest of all of these. Our usability evaluation indicated

that the resources section would be a very usability-error-prone section of the website due to

the information architecture and navigation presented. Our assumptions were proven

false with regards to quantitative metrics. The participants were able to get to where they

needed to go, relatively efficiently. However, getting there is only half of the battle with

regards to usability. The pleasure attained by the user in accomplishing the task is as equally

important as the accomplishment of the task itself.

Usability Issues encountered by participants

#Problem 1: Mislabeling of Pages

* Problem Description:Participants expressed distaste with their experience of the site based upon psycho-

pleasures related to page semiotics. Some mislabeling of pages and multiple links meaning

the same thing caused grief with the participants. The information architecture that the

users experienced to develop their mental map of the site was truly tested at this point. The

users have developed a sense of how things are set up with the first two tasks and now can

use that construct developed to accomplish this task. However, given the two-task prep with

this task, some users still felt like they could not navigate the site appropriately based upon

some of the structure of the sub-pages.


* Design Defect | Semiotics & Information Architecture


27/56

27

Task 4 Find the Closest School Task Description:

Average Time on Task /Average of All

4:26 / 3:43



Lostness0.59

The time on task for finding the school closest to your home was the second highest

time of all tasks. There were a total of 5 complete successes, 1 partial success (which involved

redirection by the proctor), and 5 complete fails or give ups. The level of lostness is almost

equivalent to the highest lostness (only one hundredth off).

It is important to note that, even though

time on task for this scenario is the second

highest of all tasks, it could be worse than it looks.

Due to the fact that 5 of the participants

completely failed the task, the numbers for this

task may be slightly skewed. If these participants

As a parent you want to identify the schools that are nearest to your home where your child

would attend middle school. Your house is located at 816 N. Audubon Road, Indianapolis, IN

46219. Please name the closest high school to your house and find the enrollment informationfor that school.

0.00

0.50

1.001.50

2.00

2.50

3.00

3.50

0.00

50.00

100.00150.00

200.00

250.00

300.00

350.00


Time on Task

LostnessSuccess Rate

Ease of Use



28/56

28

gave up too quickly, these numbers could have factored into the time on task we saw by

dragging it lower than the average would actually have been. By having 5 participants fail or

give up, and a lostness score of 0.59, it is evident that the participants, in general, were not

pleased with what it took to accomplish the task. On top of this, the average ease-of-use

index was the second lowest of all of the tasks.


#Problem 1: Banner Inconsistencies

* Problem Description:Participants noticed that the banner was static for some of the pages

and dynamic for others. This caused much confusion when users

were attempting to figure out how they ended up in certain points of

their navigation. Upon navigating over a banner, the user was unable

to determine whether it was a link or not. This enhances the previous

concerns regarding banner blindness; if the banner act like text, the

user will be more likely to avoid text and consider it as part of the

banner than ever before.



#Problem 2: Logical Path Undefined

A logical path was selected by a few of our participants

dealing with enrollment. When participants went to the enrollment

section of the site, they expected to be able to locate a school there.

Locating a school and enrolling in it seemed to be common place to

The Boundary Map

doesnt help. Where

can I give my

address?

The Map is too

small to see the

information

Im not familiarwith the school

boundary map

I think I left the

website

My first reaction isto use Google

maps

Can I use GoogleMaps?


29/56

29

be together for most of our participants. When accessing the enrollment page and not finding

a school-locator, the users either remained confused and randomly browsed, or simply gave

up. Some users even felt the need to search the schools-specific sites to see if there was

something there that could assist them. This lead to some information architecture issues

we explained earlier.



#Problem 3: Impossible Task- Does not exist

*Problem Description:

Another logical area the users expected to find the schools location was in the schools section

of the page. Some participants went immediately here and assumed that it was a trick

question because they couldnt believe the developers would not implement something like

this here. The area they were looking for was the boundary map, which was not located in the

school sub-pages. The semiotics used for the boundary map and expectation of it being in

enrollment or schools are understandable expectations and links to a boundary map should

be included in both of these sections.


* Design Defect | Semiotics

#Problem 4: Syntax Rules not Obvious

* Problem Description:The actual boundary map itself had some issues with technology and functionality.

We found that when the user types in their address to locate the nearest school, they need to

match the syntax rules of the system, displayed under the map and input field. When they


30/56

30

enter an address that doesnt match the syntax rules, they are presented with an alternate

address as a Did you mean type of suggestion. However, upon clicking this, the

subsequent page is another page notifying us that our input is incorrect. The suggestion itself

is as incorrect as the users input. Another higher level issue deals with the users

understanding of the semiotics of the page. Many of our participants did not like the idea of

Boundary Map as the area they navigate to in attempts to locate a school close to them.


* Design Defect | Semioics


31/56

31

Task 5 Magnet School Application

Task Description


3:32 / 3:43



Lostness0.50

Task 5 could be denoted as the Average task for our participants. The time on task

was 212 seconds with seven of our participants completely finishing, 2 finishing with

assistance, and 2 failing or giving up. The lostness scale is right at 0.50 which is about average

for this site, but completely unreasonable for usage in general.


#Problem 1: Mismatched User Expectations


This task summarized many of the points previously made in other tasks. One point referred

to before for other sections, and now this section, is the labeling of pages and how those

You noticed that a school located near your house was a magnet school. Please locate the

2011-2012 English student application for this magnet school.

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

0.00

50.00

100.00

150.00

200.00

250.00

300.00

350.00


Time on Task

Lostness

Success Rate

Ease of Use



32/56

32

semiotics match the users expectations. Users expected the magnet school applications to be

in the general information area; especially an area denoted as Application.

Some participants felt the need to go to the schools list and find a magnet school to

hone in on while searching for the application for that particular school. At this point in the

scenario, some participants felt confused as to some of the semiotics used to describe the

schools. Magnet schools were both located under high school and other schools. This

confused the user into thinking magnet schools had to be one or the other. However, even

when the user found a magnet school, it was to no avail as the individual school page did not

have what the user expected to be there; an application to that school. This poor set up in

information architecture could cause the participant to feel like there is no place to go from

here.

* Severity | 1 Show Stopper

* Design Defect | Information Architecture & Semiotics

#Problem 2: Ignore Navigation


Users tended to want to read all of the material before clicking on the magnet

link. This has to do with an area we discussed in our evaluation testing focusing

on the side navigation being ignored by the user. Most times, especially for this task (which is

odd because it was the last task for most people) the side navigation was not referenced as

often as we expected. While time on task was not horrible, it was high enough to elicit

concerns that items like this navigation instance could cause some major pleasure losses for

the site.

Is other magne


33/56

33




34/56

34

Task 6 Payroll Contact Information (Teachers Only) Task Description:

Task 7 Teacher Amendment (Teachers Only) Task Description:

Average Time on TaskTask 6 - 60 secondsTask 7 - 120 seconds



Lostness

0.00

In both task 6 and task 7, there were only two participants due to the nature of the

task. The tasks involved items only teachers would need to do, so they were conducted only

by our two teacher participants. Time on task for both of them was 60 and 120, respectively

for tasks 6 and 7. They both succeeded fully within the ideal number of clicks.

Users both found what they needed to efficiently. This may not be due to great user

experience designing, yet, adaptation by the participants to this poor user design. Since the

users were both teachers within the IPS school district, they were our expert users and thus,

As a new teacher you had an issue with you pay check this week. You contacted the front

office at your school, however, because they do not directly deal with payroll they directed you

to the corporation website to find the appropriate contact information and the location wereyou needed to go to sort everything out. Identify the telephone number and location of the

payroll office

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

0.00

50.00

100.00

150.00

200.00

250.00

300.00

350.00


Time on Task

Lostness

Success Rate

Ease of Use

As a teacher you wanted to review updates to the 2010 teacher salary information specifically

Amendment 5233. Locate information about Amendment 5233.



35/56

35

have had much experience with this design interface. Being accustom to a poor user design

experience does not mean a poor user design experience doesnt exist.


36/56

36

IV. Synthesis of Results from Inspection

The team analyzed the qualitative data discovered during the usability testing and

categorized it according to the five design dimensions: Content, Information Architecture,

Navigation, Presentation, and Semiotics. The team then compared the testing results with

the problems discovered during the inspection to reaffirm the initial findings and to uncover

any new findings.

Content

New Problems

Within the Content section, participants encounter three new problems during the

usability testing which was not discovered in the inspection (ID: 5, 6, 7). They included

search problems caused by Unclear input requirement, Unclear boundary map, and the

inconsistency between participants expectations and the Vague enrollment information.

Inspection Problems

Of the four problems the team found in the inspection, the participants encountered

none of them during the usability testing. The problem of Missing and inconsistent

information (ID: 1) was already updated on the website by the time our participants began

testing. The other three (ID: 2,3,4) were not confirmed as they were included in the tasks,

however, participants who did visit those pages during the test did not uncover any problems.

The table below shows the comparison of problem findings in the Inspection and Usability

testing.


37/56

37

ID Inspection Results Usability Results Conclusion1 Missing and inconsistent

informationThis problem is repaired bythe website. Not a problem

2 Target User Misidentified Participants didnt encounterthis problem. SeverityDecrease.

3 Consistency of informationand representation

Participants didnt encounterthis problem. SeverityDecrease.

4 When was the news new? Participants didnt encounterthis problem. SeverityDecrease.

5 Unclear inputrequirement

Major new problem

6 Unclear boundary map Major new problem7 Vague enrollment

informationMinor new problem

Information Architecture

In the Information Architecture dimension, participants encountered 7 problems,

including 4 problems predicted by the inspection and 3 new ones.

New Problems:

Mismatch information categorization between user and the website (ID: 1) is

a new problem which was found in usability testing and has a very high severity. About 8

participants encountered this problem and it is a main reason causing the highest time-on-

task and lostness, and lowest success rate of Task 2. Users categorize information according

to the semiotic closeness between target information and category, while the website is


38/56

38

grouped according to location and structure of the physical IPS departments. The users were

not familiar with the organizational layout of the IPS departments. Problem finding items

in a long list (ID: 10) mainly occurred when users were looking for schools from a page

that contained lists. When the users found finally found the school they were looking for the

expected it to be according to the school name sort alphabetically, however the schools were

listed in the order of IPS school number which did not provide much help.

Overlapping Problems:

The Overlapping Problems (ID: 2,4,5,8) were predicted in the inspection and proved

by the usability testing. Users encountered Problem of find school according to distance

(ID: 2). Their first reaction so solve the problem was by using the boundary map, but the

static and separate boundary map of each school provide little help for measuring distance

from home to school or location comparison. Three of them used Google Map to finish the

task. Regarding to the problem ofNo way go back (ID: 4), users were confused on how

to go back to the previous page, but they overcame it by clicking the back button multiple

times. The problem ofWhere am I? (ID: 5) was caused by both mislabeled pages and a

lack of cues for current location. The weak awareness of the current location makes it

difficult for the user to find the page again as well as understand the website structure.

Participants encountered the problem of Homepage without main content (ID: 8).

Most of them wandered on the homepage for a long time, especially in Task 2, seeking the

entrance to the information they were looking for. They found little help with the

dashboard-like homepage with its many links.

Inspection Problems:

The three Inspection Problems (ID: 3, 7, 9) were not discovered by participants

during the usability testing (Too much information accommodating too many people


39/56

39

(ID: 3) and Archiving (ID9)). Due to the lack of evidence in testing the severity for these

three problems was decreased.

An Interesting phenomena appeared regarding the problem Unrelated

Navigation. When users were finding information within the page, they naturally ignore

the unrelated navigation. The severity for this problem was also decreased.

ID Inspection Results Usability Results Conclusion1 Mismatch information

categorization between userand the website.

New problemencountered by mostusers with high severity

2 Visualization of schoolproximity

Problem of finding schoolaccording to distance

Increase Severity

3 Too much informationaccommodating toomany people

Not discovered byparticipants. Severitydecrease.

4 No way back Users are confused and havedifficulty back from the schoolwebsite

Increase Severity

5 Where am I? No clues of the currentlocation

Increase Severity

6 Unrelated Navigation Ignore by users. Severitydecrease.

7 Homepage without main

content (Homelesshomepage)

Homepage provide weak

navigation assistant into thewebsite.(user fixate on thehomepage, they are sure theinformation can be foundthere)

Increase Severity

8 Archiving Severity Decreased9 Problem of finding items from

a long listNew minor problem


40/56

40

Navigation


In Navigation, all problems predicted in Inspection were proved in Usability Testing.

Undistinguishable Links (ID: 1) was a reoccurring problem where users were confused

by the static text, images (including banners) and links. Users also encountered the problem

of Too much navigation (ID: 2) as predicted by the inspection. The users reaction to this

problem was by ignoring the links (even the useful ones). An example would be instead of

clicking the in-page quick link at the top of the page they instead scrolled down to find the

section they were looking for. No indication when leaving the website (#ID: 3), was yet

another problem that confused the users. Many expected to find the information within the

IPS website, however, were surprised when they had to navigate or find the window back to

it. Again, as proved by the inspection, Provide choices without decision information

(ID: 4), was encountered in the testing and continues to be a problem.

New Problems:

The Usability Testing also uncovered a new problem Navigation bar blocks

navigating within the topic (ID: 5). The navigation bar does not consistently including all

links within the topic.


41/56

41

ID Inspection Results Usability Results Comments1 Undistinguishable links Undistinguishable links

(text, & banner)Increase Severity

2 Too much navigation User missed navigationlinks

Severity Remain

3 No indication when leaving thewebsite Lack external websitealerts lead confusion Severity Remain

4 Provide choices without decisioninformation

School link with vaguedescription

Severity Increase

5 Technique problematicnavigation bar blocknavigating within thetopic

New major problem

Presentation


Users encountered the problem of Error Recovery (ID: 7). In the search feature,

they have difficulty in recovering from the error when the information they input cannot

meet the required format because there is no instruction telling them how to do. Regarding to

the problem of Mislabeled Pages (ID:8), users are confused by the phenomena that

several pages have the same label.

New Problems:

The Problem of Banner Blindness (ID: 1) was not predicted in the inspection,

however, due to issues uncovered in the usability testing the team classified this as a high

severity. Several users overlooked the navigation bar on the top, especially on the homepage,

which makes it difficult to gain access to the topics within the website. In contrast, the users

can notice the so call deemphasized menus on the right. The other problem The Search

Bar (ID: 10) is too small to be noticed. The third problem, Un-salient link (ID: 11), does

not show links in a consistent format (i.e. no visual cue the text is a link).

Inspection Problems:


42/56

42

The problem Users dont know where they are (ID: 2) and Mixed

representation of information and linked files (ID: 3) are another aspect of the

overlapping problem, Where am I and Undistinguishable link. Thus, the severity of

above problems should remain. Other problems, ID: 3, 5, 6, 9 were not discovered by users

in the usability testing, thus their severity was decreased.

ID Inspection Results Usability Results Conclusion1 Banner Blindness. High severity2 Users dont know where they are Severity remain3 Inconsistent font sizes, weights &

decoration (Hyperlinks)Severity Decreased

4 Mixed representation ofinformation and linked files

Severity Remain

5 Inconsistent design of sub-domains Severity Decreased

6 Deemphasizing position ofimportant menus

Severity Decreased

7 Error recovery Searching featureprovide not enoughinformation indicatinghow to recover wheninput informationdoesnt meet therequired format

Severity Remain

8 Mislabeled Pages Different pages have asame label

Severity Remain

9 Insufficient Information Severity Decreased10 The Search Bar is too

small to be noticedNew major problem

11 Un-salient links New minor problem


43/56

43

V. Overall Recommendations for Improvement

The teams recommendations span over four design dimensions; Content, Navigation,

Information Architecture and Presentation. Each dimension, ordered by severity as

discovered in our Usability Inspection, contains our recommendations ordered by severity

based upon our inspection and user feedback. Items highlighted in yellow were identified in

both our inspection and during user testing.

Content

In the content dimension, there are no problems overlapping Inspection and

Usability Testing. Regarding to the severity of the problem, we recommend the website

should supply a section proving the schools enrollment information (ID: 6) as users expected.

More over, they should also improve the Searching feature (ID: 4) by providing clear input

format requirement and error recovery help. Also, the Boundary map (ID: 5) should also be

improved to provide useful information, which is clear enough to read.

The problems with lower severity can also be improved if develop team has enough

time. The relevant recommendation and their reference please refer to table below (ID: 1, 2,

3).

ID Recommendation Severity Reference1 Establish a checklist and a content review board to

ensure information are updated and correct1 Inspection:Missing and

inconsistent information2 Notify the user of missing/ incomplete information

and when it will be updated1 Inspection:Missing and

inconsistent information3 Ensure dates are captured and identified on items

that have relevance around a date/ time1 Inspection:When was the

news new?4 Search feature: Provide a hint of the requirement to

input keyword for the search feature; for example,provide an example of the address in the formatrequired.

2 Testing:Unclear inputrequirement

5 Boundary map: Make sure the boundary mapprovides information that the users need.

2 Testing:Unclear BoundaryMap: The informationprovide on boundary map


44/56

44

doesn't meet the user'sneed. The boundary maponly provides a static mapof area around the schoolwithout any explanations,while the users want to

know the distance betweenschool and their home.

6 Enrollment Information: Clarify enrollment processby including the link in various subject areas toprovide enrollment information, including how toenroll, current enrollment status, resource of formsand etc.

3 Testing:Vague information:The users expect to knowthe information about howto enroll and the situationof current enrollment, whilethe current enrollmentinformation is a number ofhow many students enrollin this school

Navigation

The priority recommendation for Navigation (ID: 1, 3, 2, 4, 5) as they have a high

severity in inspection and caused great difficulty in users finishing their tasks in the test.

Regarding navigation links, make sure the overall navigation include links to each pages

within the topic. The website should also keep the links consistent and salient in appearance

and layout to make them easy to be notice and distinguishable from the text or static pictures.

Moreover, strengthen users awareness of boundary of the website to improve the confidence

of information by adding an external navigation alert.

It is also necessary to strengthen the users ability to control the navigation, by adding

a controller to the flash banner. (ID: 6)

ID Recommendation Severity Reference

1 The navigation bar to make sure the links cover allthe pages within the topic

3 Inspection, Testing:Technically problematicnavigation bar blocknavigating within the topic

2 Use proper formatting ensuring consistency betweentext and links, as well as banner picture and links

3 Inspection, Testing:Undistinguishable Links(text & banner)

3 Keep the most necessary navigation and get rid of 3 Inspection, Testing:many


45/56

45

the ones irrelevant to the page. Make sure theappearance and layout of the navigation are salientand consistent for easy noticed. Distinguish thenavigation of different level by differentpresentation.

navigations that usersignore one or two

4 Provide clear awareness of the boundary of the

website, by alerts when the user is navigated to theexternal website while keeping the graphical andnavigation consistency within the main website.

2 Inspection, Testing:No

indication of leaving thewebsite leads to the user'sconfusion as to where IPSended and new site started.

5 In the School List, provide more strategies forschool searching, such as filtering or arranging theschools according to specific school attribute.Provide more information or attributes to theschools (i.e. address, email fax, etc.) A possible wayto keep the list clear with rich information is toredesign the school list as a folding list like GoogleReader. Users can tab the school name to unfold

and show more information about it. A show allbutton is needed to easily unfold all the items.

2 Inspection, Testing:Providechoices without decisioninformation, and users haveproblem with the schoollink with vague description

6 Provide controller for navigating through theflashed banner: There should be some kind of wayfor the user to go back to a previously flashedbanner quickly. Links to each banner should bedisplayed below the banner, allowing the user tonavigate these asynchronously.

2 Testing:The uncontrollablerotation of the flashedbanner is difficult for usersto browse and make use of.

Information Architecture

Regarding that this is a deep and wide website with the severe problem of mismatch

information categorized between users and the website (ID: 1), however, it takes a great

effort to re-architecture the website, we recommend proving a sitemap as solution. The link

to sitemap should be in a salient position for easy access at any time, which will show the

overall website structure to the user to help them understand the conceptual model of the

website and easy locate the information they are seeking. The website should also provide

visual cues indicating users current location (ID: 2), breadcrumbs and highlighted labels are

possible solutions. The Homepage should also be improved to provide clear and overall

introduction to the website and helpful navigation assistance to the inner of the website (ID:

3).


46/56

46

ID Recommendation Severity Reference1 Provide a sitemap of the website to aid the user gain

an overview of the logical architecture of the website,so that they can find the information they need.

3 Testing:Mismatchinformation categorizationbetween users and thewebsite. (Usability Testing)

Users categorizeinformation according tothe closeness of their name,but the website groupaccording the departmentwhile the users are notfamiliar with theorganization ofdepartments.

2 Design should be include visual cues in thenavigation that highlights their current location.

Possible solutions are breadcrumb providing thetrack and navigate within the hierarchy. Ensure eachpage has a title matching the pages content.

3 Inspection, Testing:Where amI? Users are lost because

there are no clues of thecurrent location

3 Develop an area for main content to the displayed tothe user. This should summarize the site and whatthe site represents.

2 Inspection, Testing:Homepage without maincontent provides weaknavigation assistant into thewebsite.

4 Provide an access to the School Board meetingtimetable in the Parent Section.

1 Testing:Mismatchinformation categorizationbetween user and thewebsite.

Presentation

One of the most critical problems the team feels is important to fix is the problem of

Banner Blindness, especially on the Homepage (ID: 1). Possible solution is making the

navigation bar more salient and keeps it a distance away from the banner. Also, need to keep

the useful widget, including links and search button more salient in visual and layout aspect.

(ID: 2, 4). Clear error recovery instruction should also be provided (ID: 3).

ID Recommendation Severity Reference1 Make the navigation bar around the banner area

more salient. Possible solution is avoiding using the3 Testing:Banner Blindness


47/56

47

color similar to the banner for the background ofnavigation bar. Make the appearance of thenavigation bar more touchable like a button.

2 Make the links more visually salient. On the sametime, put them in a more outstanding position, suchas on the top of the page and avoid mixing within

the body of the content.

2 Inspection:Mixedrepresentation ofinformation and linked

files, Un-salient links3 Provide clear recovery instruction when the error

appears. Especially to the search feature, providemore options for user to choose to let themrecognize rather than recall. Provide formatrequirement and example near the input area.

2 Inspection, Testing:Errorrecovery & Users dontknow how to recover fromthe search error

4 Make the search button in the banner area moresalient in order to be noticed. Provide an input areato make it more affordable.

3 Testing:The search bar istoo small to be noticed.


48/56

48

VI. Appendixes

Appendix A: Pre-test Questionnaire

IPS Pre-Test Questionnaire

Name:

Date:

Age:

Gender:

Type of Computer:

Browser Used:

Location:

Race:

Type of User (Parent/Teacher/Community/Staff/etc):

Computers

How familiar are you with computers?

Very Somewhat Not At All

How often do you use a computer?

Very Somewhat Not At All

What do you mostly use the computer for?

Email SocialNetworks

Work School Browsing Other

School and Computers

Have you ever visited a school corporations website?

Yes No


49/56

49

If yes, was it IPSs?

Yes No N/A

If yes, how often in the past month?

1 3 Times 4 9Times

10-20Times

20-30Times

More than 30

Would you ever access the IPS site if you were/are not a parent, student, or staff member?

Yes No


50/56

50

Appendix B: Usability Test Script

Introduction

Hi, Im [Name]. Welcome and thank you for coming. How are you? Today we will be conducting anevaluation on the Indianapolis Public School (IPS) Corporation website.Im helping IPS understand how well their website works for the people who use the website. I willbe observing what you are doing today. In the evaluation you will have a chance to use the website tosee what you think of their product: what seems to work for you, what doesnt, and so on.

This evaluation should take about half an hour.

Were going to be videotaping the screen as a record what happens here today. This video is for ouranalysis only. Our conversation will be recorded as soundtrack to capture full information foranalysis. Its primarily so I dont have to scribble notes and can concentrate on talking to you. Themembers of our evaluation team will also review the video and audio. The information obtainedtoday is strictly for the purposes of this study and your information will be kept confidential.

Like I said, wed like you to help us with a product were evaluating. Its designed for people like you,

so wed really like to know what do you think about it and what works and doesnt work for you.You may run into features or functions that will not work right. Please feel free to tell us about yourexperiences as you go through the evaluation.

Procedure:The procedure were going to do today goes like this: were going to start out and talk for a fewminutes about how you use the web, what you like, what kinds of problems you run into, that sort ofthing.

Then Im going to show you a website of Indianapolis Public Schools which we are evaluating andhave you try out a couple of things with it. Then well wrap up, you will talk about your experience ofusing it and Ill ask you a few more questions about it, and were done.

Any questions?

Now I'd like to read you what's called a statement of informed consent. It's a standard thing I read toeveryone I interview. It sets out your rights as a person who is participating in this kind of research.

As a participant in this research

You may stop at any time.

You may ask questions at any time.

You may leave at any time.There is no deception involved.

Your answers are kept confidential.

Any questions before we begin?

Let's start!


51/56

51

Preliminary Interview

[Soundtrack recorder on]

-Online experience-1. How much time does you normally spent on the web in a given week (or within a day?)2. How much of that is for work use, and how much of that is for personal use?3. Other than email, is there any one thing you do the most online?4. Do you ever use educational system website? What do you usually visit that website for?

How often do you use it?5. Do you ever do research online for schools information?

-Offline Habits-1. (For parents) What kind ofinformation do you want to know about your childs education?

From where do you usually get this information?2. What kind of information do you want to know about the local schools or educational

system? How do you usually get it?

Evaluation Instructions

In a few minutes, I will ask you to begin your evaluation of the IPS website. However, before we getstarted, let's talk a little about what will be happening.

Please try to feel as comfortable as possible while using the interface. We are studying how thewebsite elicits your actions. There is nothing that you can do wrong as a user of this website. Wewant to observe your thoughts and reactions on the site's interface. With that, it is extremely helpfulto us if you could narrate your experience while using the site. We would love for you to have a nicestream of consciousness while interacting with the site. If you don't like something, let us know. Ifyou really, really like something, let us know. If anything stands out, doesn't stand out enough, or ifyou have an idea as to something that should be that isn't, your suggestions will help future users'experiences.

So, here is just a quick summary of what we discussed. We are testing the site; not you. Make sure tolet us know what you are thinking as you are thinking it. Finally, and most of all, be comfortable. Ifthere is anything we can do throughout the process to make your experience as natural as it would beat home, let us know.

Do you have any questions on the process? Does it all make good sense?

Great! Let's go to our Internet browser and go towww.ips.k12.in.us. As the site loads, please feel freeto move the chair, mouse, monitor, and keyboard to a comfortable position.

Tasks

8. As a parent you are growing concerned about the direction the IPS School board is movingtowards concerning year round school. Another parent mentioned that there are schoolboard meetings where you can go and voice your concerns. Assume today is December 1stand you want to go to the next briefing session. Please identify the date for this meeting.
http://www.ips.k12.in.us/http://www.ips.k12.in.us/http://www.ips.k12.in.us/http://www.ips.k12.in.us/


52/56

52

9. As a parent you feel that your childs ability to learn and use technology is very important. Inthe midst of growing concerns about budget cuts you are interested in knowing how muchmoney IPS has allocated to implementing new technology. Find the budgeted amount IPShas allocated in its Technology Plan.

10. As a parent of a high school junior, another parent told you that IPS has resources for youand your child to start exploring colleges. Find contact information about Sawyer College inMerrillville.

11. As a parent you want to identify the schools that are nearest to your home where your childwould attend middle school. Your house is located at 816 N. Audubon Road, Indianapolis,IN 46219. Please name the closest high school to your house and find the enrollmentinformation for that school.

12. You noticed that a school located near your house was a magnet school. Please locate the2011-2012 English student application for this magnet school.

If you are a teacher, please complete the additional two tasks:

13. As a new teacher you had an issue with you pay check this week. You contacted the frontoffice at your school, however, because they do not directly deal with payroll they directedyou to the corporation website to find the appropriate contact information and the locationwere you needed to go to sort everything out. Identify the telephone number and locationof the payroll office

14. As a teacher you wanted to review updates to the 2010 teacher salary information specificallyAmendment 5233. Locate information about Amendment 5233.

Evaluation Wrap-up

Wonderful. Now, if you would, please, simply exit your Internet browser and we'll come together fora quick wrap-up session.

If you could describe this site in one or two sentences to a colleague of friend of yours, how wouldyou do so? What adjectives would you use?

Would you say, by the end of the process, that you were satisfied by the service provided by IPS?

When you first started the process, did you expect the site to contain what it did? More than it did?

Based on your experience with the website, can you give three general pros and three general consabout the site?

Is there anything that you wish this site would provide to people in your same shoes with regards tostake in the system (IPS)?

Thank you so much for your time and participation. Many times, after completing evaluations,people tend to have amazing ideas when they are at home, browsing the Internet and utilizingservices like these in their home environment. If you have anything you could add to this evaluationtoday, tomorrow, or even as far out as next week, please don't hesitate to contact us at


53/56

53

[email_address].

Thanks again and have a wonderful day!


54/56

54

Appendix C: Post-Task Questionnaire

IPS Post-Task Questionnaire

Evaluator: ________________ Participant ID: ________________ Task ID: ___________

I was satisfied with the ease of completing this task?

StronglyDisagree

Strongly Agree


55/56

55

Appendix D: Post-Test Questionnaire

IPS Post-Test Questionnaire

Evaluator: ___________________ Participant ID:___________________

1. How would you explain the set up of the site; Architecture?

2. How would you explain how you are feeling after completing the tasks?

3. Do you feel like you were In Control of navigating around the site?

4. What would you change about the site?

5. Could you understand where you were in relation to the sites structure?

6. What would you change about the site?

7. Is there anything youd like to add?


56/56

Appendix E: Usability Testing Videos

Please reference the attached CD File submitted with this project for a complete list of videos.

IPS Usability Testing Report - Report

Documents

Transcript of IPS Usability Testing Report - Report