CityofBoston

HF750 Team 2 - CityOfBoston.gov

CityofBoston.gov

Team 2Testing & Assessment

HFID-750

City of Boston.gov

Team 2HF750 Testing & Assessment

12.08.08

Usability Testing & Analysis


Table of Contents

• Overview• Executive Summary• Goals & Objectives• Methodology• Findings & Recommendations• Conclusion• Next Steps• Lessons Learned• Q&A


Overview

Engagement:

• We engaged with the development and management teams for the cityofboston.gov website to conduct usability testing.

Website: CityofBoston.gov

• General Audience: Residents, Business interests, Visitors, Students

• Purpose: To facilitate the dispensing and communication of information about the City of Boston…

– To anyone who might have a question about how to perform common tasks such as paying fees, finding a job, getting permits, licenses, etc.

– To provide contact information for city officials to facilitate communication between the city government and concerned citizens

– To find out when and where public events will occur

– To physically get around the city and obtain information on local communities


Executive Summary

• Participants found the website very informative. – “It gives you basically the whole…whatever you’re looking for in Boston. It is informative.”– “Seems to have a lot of information that people would want to use, like most requested pages.”

• Participants liked the visual design (colors, graphics etc.) of the portal and commented that it aided in navigation – “The color scheme really made it easy. You could just follow the colors.”

• Organization of information was not intuitive and users relied on‘Search’ function to find what they were looking for.– “Certain things were not where I expected them to be.”

• Participant’s expectations of online services on the website werenot met.– “I don’t see an online application…I thought there would be an online form.”

• Participants reported that their overall experience with thesite was positive and gave it an average grade of “B.”

P Grade1 C

2 B3 B

4 B5 A6 B

AVG: B


Goals & Objectives


Goals & Objectives - Site Goals

• Increase efficiency of completing commonly performed tasks• Provide overall ease of use• Provide an easy method to locate desired information• Establish a forum of communication for site users• Deliver useful and informative content that incentives users to

return to the site


Goals & Objectives – Business and User Goals

• Business Goals– Increase Site Traffic– Create opportunities for partnerships within the community– Provide comprehensive information for users to access desired information

to alleviate support calls to government offices– Ensure that all user/audience goals are being met– Ensure that Business and User Goals are accurately aligned– To be a best-in-class city destination website

• User Goals– To learn more about Boston and what the city has to offer– To obtain desired information with respect to each audiences’ needs– To complete online transactions with ease– To improve community interaction with residents, business interests, and visitors


Goals & Objectives - Test Objectives

• Gather user feedback on the City of Boston online portal through observations and questions asked during the interview session

• Identify usability issues especially those related to information architecture, and navigation

• Understand users’ overall subjective satisfaction with the website through their responses to a post-test questionnaire

• Uncover opportunities for improvement and/or enhancements• Compile and present findings to the CityofBoston.gov stakeholders


Methodology


Methodology - Testing Protocol

Where: Tests were conducted remotely as well as in person at Bentley’s DUC.

When: October 30th and November 5th, 2008

Who: Participants recruited from Bentley student population as well as our friends. They were tested in one-on-one usability sessions. 6 participants were recruited to represent 4 types of user groups, Businesses, Residents, Students, and Visitors ranging in age from 18-55+

What: Participants signed a consent form and responded to a pre-test questionnaire that identified their level of experience with using the internet and government related web sites

How:

• Participants were asked to use think-aloud protocol while they completed each task.

• Scenarios were constructed for the participants to replicate normal tasks on the CityofBoston.gov.

• Operating within a given scenario, users were asked to perform independent tasks with clearly defined measures of success or failure.


Participant Demographics

Date P Time User Group Occupation Gender Age Group Local/Remote

Thur 10/30

1 6:30-7:30 Visitor Software/QA M 55+ Local

2 8:00-9:00 Visitor Help Desk F 18-25 Local

3 9:00-10:00 Resident Management F 26-40 Local

Wed 11/5

4 6:30-7:30 Visitor Developer M 18-25 Remote

5 7:30-8:30 Student Mechanical Engineer M 26-40 Remote

6 9:00-10:00 Business Self-Employed M 41-55 Local


Methodology - Conducting the Test

• During the usability evaluation, 6 participants, matching the user profile(s), were asked to spend one hour with the site. During this hour, participants:

– Completed a user background questionnaire– Answered questions about initial site impressions– Performed real-world tasks on the site while thinking aloud – Answered questions about their overall satisfaction


Methodology - Analysis and Reporting

During the sessions we focused on capturing the following qualitative metrics:• Task initiation and user expectations• Points of confusion (points where user is stalled, isn’t sure what to do)• User’s perceived ease of use

Capturing Information through Video: • The computer screen that the user sees as he/she is performing tasks

• The participant him/herself.

– Locating parts of the tasks where participants are either satisfied or frustrated

– Locating assists and failures

Severity Ratings:• High, Medium, Low ratings• Based on our views of importance and the amount of difficulty seen over the course of testing


Methodology - Pre-Test Interview

At the beginning of each session, we asked participants five questions, including:

Questions Sample Responses (related to City of Boston site)

How do you feel you fit into the “Boston” Group? Live, work, study?

“…visit Boston frequently for pleasure”“I live in Waltham and go in to see the sights”“I live in Boston”

Have you ever used a town or city information web site before? Which one?

“Yes, the Reading Website”“Usually to get business information, they usually have business listings. Town hall, police station information”

What was your first impression of the site? “Font is small…I like the pictures, they coordinate with the text of what the stories are about…”“I think it is pretty basic.”

Just from looking at this site, what kinds of information do you think you could get from this site? Please be specific.

“Services, bill paying online, business or events going on in the town. Hopefully standard information.”“Calendar of events, blogs, pictures”


Methodology - Post-Test Interview

At the end of each session, we asked participants 9 questions:

Questions Sample Responses

How would you rate your experience with this site? (Letter Grade and Why)

The average rating was a B. •“Would have given it a D but there were a few easier tasks”•“It has a nice layout, felt comfortable using but certain things were not where I expected them to be.”

How Informative was the site? •“A lot of information, so I’d say it was very informative. You just had to be persistent and patient.”•“Most business sites are better. Chances are you won’t find broken links like this site.”•“Got plenty of information if you know where to find it, can always use search, seems to have a lot of information that people would want to use, like most requested pages”

If you could change one thing about the site what would it be?

•“Broken links”•“Larger font”•“Less clutter”

Would you like to make any suggestions about thecityofboston.gov?

•“Pick a bar”•“Take up the whole page”

What adjectives would you use to describe the site? adequate, organized, wordy, busy, informative,


Findings & Recommendations


Findings & Recommendations - Task List

During the usability evaluation, participants were asked to complete eight (8) scenarios or Real-life tasks on the site. The tasks were presented in the same order for each participant.

1. You have just learned from a friend that you can pay your personal property tax online through the CityOfBoston.gov web site. How would you go about finding where to do this?

2. You just bought a new puppy from a store in Boston and have to get him licensed, but you aren’t sure how. How would you go about finding this information on the City of Boston web site?

3. You’re planning a trip to Boston in a few weeks, and heard about an attraction called the “Freedom Trail.” How would you go about finding more information?

4. You want to find out what special city events are planned for the month of November. How would you go about finding this information?

5. You’re researching the quality of Boston’s public school system, and need to find a list of all of the public schools in the city. How would you go about finding this information?

6. The upcoming election is only a few weeks away, and you’re going to be out of town on Election Day. Using the City of Boston web site, find out how to apply for an absentee ballot.

7. You’d like to look at the minutes from the last Boston City Council meeting. How would you go about finding this information?

8. You have been injured on the job and will be unable to continue working for a few months. Find out how to

apply for worker’s compensation.


Findings & Recommendations - Task One: Payment of Personal Property TaxSeverity: HighFindings:• Participants did not encounter difficulty reaching the Personal Property Tax page, but

were confused by the page, as it asked for a Business ID. 33% of participants did not know they were on the correct page, and failed to complete the task.

“This isn’t for residents because you need a business ID. Hm. And you need a bill number. It says personal property tax, not business.” –P6

“This brought me to a business ID…this is confusing, because I’m not a business.” –P2

Recommendations:• Because the Business ID is labeled

as Identification Number (note the instructions in the right column) on the paper form required to make a payment, change the Business ID label in both columns to Identification Number.

• This will help reduce confusion as to whether individuals are on the correct page to pay their property tax.


Findings & Recommendations - Task Two: Licensing a New PuppySeverities: MediumFindings:• 83% of participants completed this task without assistance.

• The location of the general dog licensing form is at the bottom of the page, under a specialty pit bull license. Participants commented on this:

“All other dogs should have been at the top, pit bulls towards the bottom.” –P3“I would have given it a 1 [easiest] but general license was below pit-bull license application.” –P1

• Participants expected an online form or application instead of a PDF that needed to be printed and mailed:

“I clicked on license application and expected a form.” –P1“I don’t see an online application…I thought there would be an online form.” –P4

Recommendations:• Move the general dog license application above the pit bull license application so that

no scrolling is required to get to the form link at a resolution of 1024x768.• Feature Request: Consider allowing dog license registrations to be submitted with

payment online


Findings & Recommendations - Task Three: Information on the Freedom TrailPositive FeedbackFindings:• All participants completed this task without difficulty.

“It was pretty easy to navigate where I wanted to go.” –P2“Didn’t have to go anywhere else…was right there, pretty easy” –P5“It was right on the [Visitors] home page.” –P6


Findings & Recommendations - Task Four: November Special EventsSeverities: HighFindings:• 83% participants completed this task, but all expressed concerns with the date range

displayed in the calendar and the general presentation of the list of events.• Participants were confused as to what the checkboxes on the left-hand side of the

calendar were for:“Curious as to what check box does? Not sure why it’s here or what it does.” –P1“Easy to get to, but once I was there I didn’t like the check boxes, I didn’t feel like I got any

feedback, loading in the background. Hard to find specific event.” –P4• Participants noticed that the calendar, by default, only displays events for a two week

period. They expected a full month’s worth of events to be shown:“I was unsure if I had everything on that one page (all of the events).” –P2“This looks like…this is only one week...” –P6

• Participants expected to see an actual calendar instead of a list of events:“I would have expected a calendar I could click on, day, week month for whatever view you want

and then you could see all the events.” –P4“Easy to get to, but with a calendar she would expect a sign. The page opens to a list, she would

like to see this in a different format. “ –P2


Findings & Recommendations - Task Four: November Special Events

Recommendations:• Display a full month’s worth of events by

default; participants expected the calendar to default to a “month” view.

• Consider presenting events in an actual calendar format. This makes it easier to determine when events start and end.

• Add end times (when available) to event listings

• Add a heading under the Search box on the left-hand navigation titled “Filter Calendar Events” in larger type (to draw attention).

• Consider displaying different categories (e.g. Arts, Sports, business, community event) in different colors to allow users to select and filter on a higher level of genre


Findings & Recommendations - Task Five: List of Boston Public SchoolsSeverities: HighFindings:• Participants remarked that it wasn’t clear if this information should be located under the

Residents or Students section.“If you’re looking for schools…would you expect to find it under students? I was a little bit

confused.” –P5“I went to residents as residents would be interested in this info.” –P3“There were a number of things that seemed like they were going to give me a list of schools. They

were really for finding which schools to register for. I guess choose a school might somehow suggest registering for a school? It wasn’t clear to me that it wasn’t gonna be a list of schools.” –P6

• 33% of participants (because of the path they took initially) noticed that the link to the list of schools on the Schools, Colleges, and Universities page was broken.


Findings & Recommendations - Task Five: List of Boston Public Schools

Recommendations:• Consider placing links to this information in both the Residents and Students sections of

the site. In the Residents section, place the list link on the front of the Education and Schools page (83% of participants took this path).

• Fix the broken link on the Schools, Colleges, and Universities page to point to the new list on the Boston Public Schools web site.

• Task completion required leaving the CityofBoston.gov website and moving to the bostonpublicschools.org site. This was a transparent change. Consider a message letting the user know that they are leaving the current site.


Findings & Recommendations - Task Six: Absentee Ballot

Severity: HighFindings:• 83% of participants completed this task successfully• Participants expressed concerns about placing the Absentee Ballot form under the

Register to Vote section:“Register to vote seems logical, but that isn’t what I really want to do.” –P2“I might have expected it to have a link pointing to it at the same level as the voter registration.You don’t have to register to vote with an absentee ballot.” –P6“These links were not intuitive.” –P1

Recommendations:• Place the Absentee Ballot form link on the front of the Elections and Voting section of

the Residents area of the site; not all absentee voters are registering for the first time.• The text informing the user that they could apply for an absentee ballot was always

listed after, and in the same sentence with registering to vote. Registering to vote for the first time and requesting an absentee ballot are separate tasks.


Findings & Recommendations - Task Seven: Meeting MinutesSeverity: LowFindings:• 83% of participants completed this task successfully• Participants who completed this task remarked that it was “pretty easy to get to” and

“very straightforward.”• One of the older participants noted that the small font size that the Meeting Minutes link

was in could cause issues:“The link was small and down at the bottom, difficult to find.” –P4

Recommendations:• Consider moving the text size adjustment tool (in the lower

left corner of every page) to the title bar about 10% of the way down from the top of the screen.

• This will allow users to adjust the default text size on the site to suit their needs. In its current location, the tool is not visible (no participants commented on its presence).


Findings & Recommendations - Task Eight: Worker’s CompensationSeverity: HighFindings• Only 16% of participants completed this task successfully. All participants who failed said

that they would use the Search tool; no participants looked in the Services drop-down.• Participants expected to find the application in the Business or Forms section of the site.

“Didn’t exactly complete task, would have expected it to be within forms permits and applications page, not sure where else it would be.” –P4

“Would think it would be under applications, but must be wrong because it is not there.” –P2“Looking for a form or something like that ” –P5

Recommendations• Place a link for Worker’s Compensation in both the Forms section of the site and on the

Business section (under Forms and Applications).


Task Success / Failure

P Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8

1 Fail Pass Pass Pass Pass Pass Pass Fail

2 Fail Pass Pass Pass Pass Pass Pass Fail

3 Pass Fail Pass Pass Pass Pass Fail Pass

4 Pass Pass Pass Pass Fail Pass Pass Fail

5 Pass Pass Fail Pass Pass Pass Pass Fail

6 Fail Pass Pass Pass Pass Fail Pass Fail

Pass = User was able to accomplish task without an assistFail = User was unable to complete task, needed an assist or relied on search


Task ratings

P Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8

1 5 2 4 2 1 4 1 4

2 4 1 2 2 5 3 1 5

3 1 2 1 2 3 2 2 1

4 1 1 1 3 2 3 4 5

5 1 3 1 1 2 2 1 5

6 1 3 1 4 3 4 1 5

Avg 2.17 2 1.67 2.33 2.67 3 1.67 4.17

1 = Very Easy2 = Moderately Easy3 = Neutral4 = Moderately Difficult5 = Very Difficult


Next Steps

• Review the list of findings in detail and consider refining the website based on the recommendations provided.

• Consider expanding the user test group to include business audiences• Include differently-abled groups for next usability study

(to check for Section 508 compliance)• Work with development team to retest redesigned website


Lessons Learned

• Rotating roles of moderator, observers and note-takers helped us to experience the roles first-hand

• Critical to check equipment prior to test (i.e. Phone for remote testing)• Important to adequately space out participants during testing sessions

(esp. Remote Testing)• Having a test script prepared was very helpful and helped ensure continuity to our

testing approach• Explaining testing logistics to participants up front was helpful• Pre-planning with group members helped streamline testing process• Giving $10 Starbucks gift cards as honorarium was appreciated by participants• Keeping some snacks and candies handy were good. It helped with blood sugar level

and ability to concentrate• Don’t put the most difficult task at the end of the list of tasks; participants should leave

on a ‘high note’• Make it more obvious that participants should return to the home page before starting

each new task


Q&A


• Compiled Session Notes• Test script• DVD recordings of participants• Consent forms

Appendices

CityofBoston

Documents

Transcript of CityofBoston