Usability Final Paper

iakm-dev2.slis.kent.edu/Usability Testing

Cathy CarrHolly C. MurphyMatthew PahlsDecember 8, 2009

Table of Contents

Statement of Work . . . . . . . . . . . . . 3Characterizations of Tasks . . . . . . . . . .4-5Data Collection . . . . . . . . . . . . . . 6Characterization of Participants . . . . . . . . 7Conclusions. . . . . . . . . . . . . . . . 8Recommendations for Redesign . . . . . . . . 9Raw Data . . . . . . . . . . . . . . . .10-22

Statement of WorkThe document provided will describe our research group’s findings pertaining to our usability test of a newly created iteration of the IAKM public facing web site. The testing completed was performed utilizing the ClearView eye tracking software to accurately monitor the user’s interaction on multiple dimensions. The facility allowed us to work directly with the user, and monitor interaction via click tracking, eye movement, audio and video. Combining those attributes, based on a time scale, our research team was able to more accurately identify where the site changes were successful and where they were not.

The Information Architecture and Knowledge Management web site, like many other programs in a large university’s environment, utilizes its web presence to interact with a worldly audience. Commonly the web site is first and foremost, a first impression for a prospective student. So having a strong web presence is paramount to sustaining positive enrollment and retention. The IAKM program at Kent State University roots itself in a rapidly evolving field that has strong ties to the web. As the field establishes itself, IAKM must be able to accurately portray the field in a manor such that people with little preexisting knowledge could identify with it.

The site recently underwent some structural changes, and our team was tasked with conducting a full usability study on the product. Since the changes are not dramatic, we needed to know if these slight alterations would have a positive or negative impact on the overall experience. The hierarchical changes were based around the main navigation. The intent of the changes was to improve the overall findability of information within the site. Our group’s investigation would prove or disprove the effectiveness of the changes.

The IAKM office tasked our team with focusing our efforts on two particular audience segments; the prospective student and the current student. We used a short screener developed in Qualtrics to identify our four user participants that fit either of the two audience segments. From there, we had each of the users come in to the usability lab individually and we conducted a series of task based analysis tests. These tests were targeted at particular tasks that the audience segments should have the capabilities of easily performing. We quickly discovered that the tasks we perceived as relatively simple turned out to be rather difficult, and the users were having issues completing the tasks.

Characterization of Tasks Two sets of tasks were developed to test the site with two sets of users. The first set of tasks tested the site for prospective students. These tasks were standard site activities that a user researching or preparing for enrollment would conduct. The second set of tasks tested the site for current students. These tasks were intended for students already in the program. The activities supported navigating the site for content that the student would need during the program cycle.

Prospective IAKM student

Task 1: You are a prospective KM certificate student. What are the core-required courses?

Rationale: This is a basic task that a student would perform to locate information vital to proceeding within the program.

Task 2: You are a prospective UXD master’s student. How would you apply to the program?

Rationale: For many students this may be their first experience with the site. It is imperative that this task be easy to find and perform.

Task 3: You are interested in a graduate assistantship. Find information to apply for this position.

Rationale: All prospective students may not attempt this task. But, it is an important item for graduate schools that should be easily located on their sites.

Current IAKM student

Task 1: You are interested in a graduate assistantship. Find information to apply for this position.

Rationale: Again, this may not be a task attempted by all current students. But, it is possible for current students to become graduate assistants.

Task 2: It is time to schedule for the Spring 2010 semester. Use your current student standing and find offered classes using the IAKM site.

Rationale: The IAKM site lists classes offered for every semester. This is very helpful for students when preparing for registration.

Task 3: You are approaching graduation and beginning to think about the master’s project. Find information that will help you.

Rationale: Every student will need to locate this information, not only to begin the project, but also to submit the appropriate forms.

Task 4: You wish to take a workshop next semester. Find the form you will need to ensure you receive credit for the workshop.

Rationale: This specific task may not be performed by every student. The ease of locating this particular information assures that the user is able to navigate to the forms page of the site, where other vital program information is located.

Task Administration The testing took place in the Kent State University Usability Lab. The above tasks were typed, printed, and read by the facilitator to each participant. The facilitator either sat or stood to the side of the participant while the session was in progress. Each task began from the home page. The participant was able to ask for clarification or restatement of the task. Some guidance was given if the participant strayed off track. The participant was asked to think-aloud during the tasks. ClearView was used to capture audio, video, and eye tracking. These recordings were analyzed at a later time for user comments, difficulties, and successes.

Data CollectionInformation was collected from our selected participants on the dimensions of auditory, visually, and click tracking. Each axis is powerful in its own right, but combining them and being able to analyze them collaboratively allows us to produce truly accurate analysis of the participant. While tasks were in progress, we elicited verbal feedback so we could record what the user was thinking. As they were talking out the tasks we were able to understand why the user was doing the task and seeing why they had their own understanding on how the site should function. The lab is also equipped with a computer monitor that has the capability to track eye movements (“gaze”). These recordings are accomplished by having the monitor use an infrared signal projected onto the back on the participants eye and the signal then reflects off the eye and recorded at that point. As the user interacts with the site, the monitor tracks the movement and combines that information with the current page of interaction. This allows us to see exactly what the user us looking at in real time. The recorded instances of the visual attribute can be exported to two rather interesting and rather valuable outputs. Gaze plots and heat maps allow us to take the visual recording and map them to the site so we can visually see where the user focused and how long they focused for.

Now we add the fourth and final component to our recording attributes and this is time. Combined with the auditory, visual, and click actions, we can accurately track all user interaction in retrospect to the tasks assigned to the user throughout the entire test. These elements empower the research team to effectively capture the data necessary for proper data analysis, and more importantly see how effective the site alterations are on our target user segments.

Characterization of Participants Not all of the study participants fit the user profiles established in the requirements analysis. The established profiles are as follows:

Prospective IAKM student• Age: 26 – 40• Sex: Female• State: Ohio• Visual impairment: None• Is not involved with the program• Has not used the current IAKM site• Is looking for more information about the IAKM program• Computer experience: average to above average• Uses a PC every day

Current IAKM student• Age: 26 – 40• Sex: Male• State: Ohio• Visual impairment: None• Is involved in the program• Learned of the program from a former or current student• Computer experience: above average to expert• Uses a PC every day

For the study, both a male and female prospective student were selected to participate. The female wore glasses. One of the current student participants also wore glasses. The other current student uses a Mac.

Conclusions • Vocabulary confusion. Two of the participants were confused with the site terminology. More

interesting is that these were the current student participants that have used the previous IAKM site.

• Student standing. The participants that were chosen as prospective IAKM students consider themselves as current students because they are already enrolled in KSU.

• GA task. All four participants were given the task of locating information to apply for a graduate assistantship. Each individual chose a different path to locate this information, and verbalized a different rationale for their path.

• Approach. The two current students performed very different approaches to locate the task information. One performed the hunt and peck method, while the other took a few seconds to plot out a course of action. The prospective students conducted trial and error approaches.

• Searching Issues. When the users tried to perform a search the results were somewhat inaccurate.

• Using a Crutch. When users are confused they turned to the FAQ as a crutch for locating the information. This proves the user is becoming quickly frustrated with the site and tries to find the quickest path to the proper knowledge. All users tested were personally vested in finding the information because they were being tested on it. The general user might not have that investment in finding the information so they will tend to use the path of least resistance.

Recommendations for Redesign• Evaluate the navigation labels and organization. This was a stumbling block for the participants.

The user had to decide where he or she thought the information was located and was incorrect several times. We recommend performing a card sorting exercise on the labels that are already defined, because the information that is present in the site is accurate, it just lacks proper organization.

• Relocate the graduate assistantship information. This task was very insightful from both the current and prospective student perspective. This information may warrant multiple locations within the site.

• More prominent display for end of study requirements. The Master’s Project information was located by two different methods. When it was located, it was not where the prospective student would have looked, under forms.

• Improve the accuracy of the search. Improving the search will elicit a quick fix. The new search will allow the sites user population to find information while they become accommodated to any heuristic alterations.

Raw Data:Each task and user is broken down into significant actions. The actions are then grouped into navigation paths, with each path illustrated below the table.

User 1: Current IAKM student

User 1/Task 1:

Illustration User 1/Task 1 TimeBegin Task :35Looks at side menu 1:03

1.1 Clicks on Programs1.2 Clicks on Current Students 1:22

Clicks on Employment Opportunities 1:461.3 Clicks on FAQs Advising1.4 Tries Search 2:57

Found target 3:27Total task time 2:53

Side Menu

Target

User 1/Task 1

1.1 Programs

Employment Opportunities

1.2 Current Students

Advising

1.3 FAQs

IAKM Home Page

1.4 Search

User 1/Task 2:Illustration User 1/Task 2 4:30none Clicked on Courses and found target

:05

User 1/Task 3:

Illustration User 1/Task 3

3.1 Looks at side menu 5:053.2 Looks at concentrations in center of

page and clicks5:22

Found target 5:30:25

3.1 Side Menu

Target

User 1/Task 3 IAKM Home Page

3,2 UXD Concentration Master’s Program link

top center of home page

User 1/Task4:Illustration User 1/Task 4 Time

Begin Task 6:204.1 Clicks on Courses 6:30

Looks at side menu. Subject seems lost and says, “why don’t they have a button labeled forms?”

6:50

4.2 Goes to search 7:20 Clicks on forms 7:35Found target 7:50

1:30

Side Menu

Target

User 1/Task 4

4.1 Courses

IAKM Home Page

4.2 Search

User 2: Current IAKM student

User 2/Task 1:Illustration User 2/Task 1 Time

Begin Task :171.1 Looked at side menu

Clicked on Prospective Students :29Looked at sub-menu :36Clicked on Financial Aid :38Found target :41Total time :24

1.1 Side Menu

Target

User 2/Task 1

Prospective Students

IAKM Home Page

Financial Aid

User 2/Task 2:

Illustration User 2/Task 2 TimeBegin Task 1:26

2.1 Looked at side menuClicked on Courses 1:30Clicked on SchedulesFound target 1:43Total time :17

2.1 Side Menu

Target

User 2/Task 2

Courses

IAKM Home Page

Schedules

User 2/ Task 3

Illustration User 2/Task 3 TimeBegin Task 2:30Looked at side menu

3.1 Clicked on Programs 2:353.2 Clicked on Courses 2:373.3 Clicked on Current Student 2:46

Clicked on Forms 3:01Found Target 3:18Total time :48

User 2/Task 3

Side Menu

Target

3.1 Programs

3.2 Courses

Forms


IAKM Home Page

User 2/ Task 4:Illustration User 2/Task 4 Time

Begin Task 4:10Looked at side menu

3.1 Clicked on Current Students 4:123.2 Clicked on Forms

Found Target 4:17Total time :07

1.1 Side Menu

Target

User 2/Task 4

Current Students

IAKM Home Page

Forms

User 3: Prospective IAKM student

User 3/Task 1:

Illustration User 3/Task 1 TimeBegin Task 5:04Looked at center of page

3.1 Clicked on KM Certificate 5:07Found Target 5:15Total time :11

1.1 KM Concentration Certificate Program link

top center of home page

Target

User 3/Task 1 IAKM Home Page

User 3/Task 2:

Illustration User 3/Task 2 TimeBegin Task 5:45Looked at center of page

2.1 Clicked on UXD Master’s link

5:48

2.2 Looked at side menuClicked on Apply Now 5:55Found target 6:00Total time :15

User 2/Task 3

2.1 UXD Certificate Program link top center of home page Target

2.2 Apply Now

IAKM Home Page

User 3/Task 3:

Illustration User 3/Task 3 TimeBegin Task 7:02Looked at side menu

3.1 Clicked on Current Students 7:15

Clicked on EmploymentOpportunities

7:34

Looked at Side Menu3.2 Clicked FAQs 7:49

Clicked on Financial AidFound target 7:52Total time :50

User 2/Task 3

Side Menu

Target



Financial Aid

3.2 FAQs

IAKM Home Page

User 4: Prospective IAKM student

User 4/Task 1:

Illustration User 4/Task 1 TimeBegin Task 4:25Looked at side menu 4:29

1.1 Clicked on Courses 4:40

Clicked on Catalog 4:501.2 Clicked on Prospective Students1.3 Programs1.4 Looks at concentrations in center of

page and clicks on KM5:05

Found targetTotal time :40

Target

User 4/Task 1

Side Menu

1.1 Courses

Catalog

1.3 KM Concntration Certificate Program link top center of

home page

IAKM Home Page

1.2 Prospective Students


User 4/Task 2:

Illustration User 4/Task 2 TimeBegin Task

2.1 In Courses sub menu 5:55Clicked on Curriculum in page 6:03

Found targetTotal time :08

Target

User 4/Task 2

Side Menu

IAKM Home Page

Clicked on KM curriculum on page

2.1 Courses

User 4/Task 3:

Illustration User 4/Task 3 TimeBegin TaskLooked at side menu 7:50

3.1 Clicked Alumni and Professionals 7:58

3.2 Clicked on Current Students 8:17Clicked on Employment Opportunities

8:19

3.3 Clicked on Prospective Students 9:14Clicked on Financial Aid 9:23Found target 9:32Total time 1:42

User 4/Task 3

Side Menu

Target

3.1 Alumni and Professionals

3.2 Current Students Forms


IAKM Home Page


Usability Final Paper

Education

Transcript of Usability Final Paper