Post on 28-Nov-2014
description
NYPL Mobile Hotspot Lending100 Unit Test
Preliminary MetricsJuly 2014
2
We have developed a logic model to identify how we can measure impact against the programs 4 goals.
Goal 1Increase students/household access to at home broadband internet
Goal 2Raise digital exposure and confidence
Goal 3Increase engagement with library-provided and/or recommended learning resources
Goal 4Establish a scalable public library model for tackling the national digital divide
3
TALLY DATA
• A total of 106 households showed interest in the program by completing a user interest form
• Of those who communicated interest, 84% were eligible
• A total of 91 households are participating in the pilot (98% of those eligible)
• A total of 91 wireless routers and 50 Chromebooks were loaned Chromebooks were only made available to Out of School Time (OST) students
• 34 households checked out both devices; 47 checked out only the wireless router; and 6 checked out only the Chromebook
• 47 participants are ESOL students at Bronx Library Center (BLC); the remaining participants are parents of OST students at Mott Haven (19), Highbridge (13), and Port Richmond (12)
Lent
0 5 10 15 20 25 30 35 40 45 50
Wifi & ChromebookWiFiChrome Book
Spanish
English
4
Goal 1Increase students/household access to at home broadband internet
NOTE: • 37% of participating households primarily speak
Spanish• TechConnect not channel not tested
5
ISP DATA
6
There are three program goals measured through ISP and survey data
Goal 1Increase student/household access to at home broadband internet
Goal 2Raise digital exposure and confidence
Goal 3Increase engagement with library-provided and/or recommended learning resources
Activity measurement limitations and related assumptions
7
Method: Activity measured by connections on a particular port.We are not given full destination website addresses due to privacy regulations of the ISP. We were provided the port labels of the destination networks. While not perfect, it does provide a hint as to the nature of activity because certain activity (such as email, streaming video, social media, gaming, chat and messaging) which occurs on uniquely labeled ports.
Assumption: Chromebook ActivityThe port traffic label SPDY we assume to be Google Chromebook traffic. Because we lent Chromebooks to 50% of our participants that traffic may potentially skew the results but only by 2%. “Google” activity for this analysis is essentially Chromebook/Google Apps activity.
Assumption: Anonymous TrafficWe assume that the "Anonymous" traffic (traffic on ports commonly labeled "HTTP") consists of Google traffic as well. Google owns roughly 68% of all web traffic so we assume 68% of the "Anonymous" traffic is Google traffic as well.
NOTE: For the 10K pilot, if we chooses Sprint, we will be able to route all traffic through a monitoring service to intercept CIPA restricted sites (black listed web site categories.) We will therefore, have greater visibility as to the category of traffic based on the full address. This will be anonymous and in aggregate so we preserve privacy.
Users are spending an average of 3.19 hours online per day, and most often at times when the library is closed (6PM – 12AM local)
12:00 AM
2:00 AM
4:00 AM
6:00 AM
8:00 AM
10:00 AM
12:00 PM
2:00 PM
4:00 PM
6:00 PM
8:00 PM
10:00 PM0
500000
1000000
1500000
2000000
2500000
3000000
3500000
Time of Day (UTC)
Conn
ectio
ns
• Average Time on-line: 3.19 hours/day
• This equates to 191 minutes vs a single 40 minute session at library during opening hours
8
Goal 1Increase students/household access to at home broadband internet
Note: TIME of Day on chart is UTC so subtract 3 hours for local time.
9
YouT
ube
Video
App S
tore
Mus
ic
Onlin
e File
Sto
rage
Phot
o sh
arin
g
Inst
ant M
essa
ging
VOIP
Comm
erce
News
Map
s
Onlin
e App
s
Utility
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
% Activity vs % Data Volume
% Traffic % VolumeData
384.7 GB/mo173.2 GB/mo102.7 GB/mo51.3 GB/mo
HH Usage GB/mo
4.23 GB/HH/mo1.90 GB/HH/mo1.13 GB/HH/mo0.56 GB/HH/mo
Activity
Anonymous YouTubeEncryptedVideo
Goal 1Increase students/household access to at home broadband internet
Goal 2Raise digital exposure and confidence
We see healthy traffic levels around >9GB per household per month, most of which is anonymous
File Sharing, Social Media, YouTube, and Google Web Properties seem to represent bulk of traceable activity online.
Anonymous68%
File Sharing14%
En-crypted6%
So-cial Media6%
YouTube3%
Google2%
Entertainment1%
95% of Traffic
AnonymousFile SharingEncryptedSocial MediaYouTubeGoogleEntertainment
Anonymous48%
YouTube22%
Encrypted13%
Video
6%
So-cial Media4%
App Store3%
Entertainment1% Music
1%
95% of Volume
AnonymousYouTubeEncryptedVideoSocial MediaApp StoreEntertainmentMusic
10
Goal 2Raise digital exposure and confidence
11
Usage of educational content is not measurable through ISP data for the 100 unit test, so we will rely on exit surveys to measure this goal
Anonymous48%
YouTube22%
Encrypted13%
Video6%
Social Media4%
App Store3%
Entertainment1%
Music1%
95% of Volume
AnonymousYouTubeEncryptedVideoSocial MediaApp StoreEntertainmentMusic
For the 10K test, we will have the ability to capture URL traffic programmatically.
We could assume Khan Academy is part of the YouTube activity for our OST students. Other resources like Mango for ESOL or Math Whiz or NYPL are most likely masked due to port labeling and lack of URL logging.
Goal 3Increase engagement with library-provided and/or recommended learning resources
12
SURVEY DATA – PRELIMINARY
Survey data includes 23 responses from parentswith children participating in the library’s OST program.
Participants report an average of 2.2 adults and 1.6 children per household, or an average of 3.8 total household users
1 2 30%
10%
20%
30%
40%
50%
60%52%
35%
13%
Child Users (ages 5+) per Household
# of child users
% o
f house
hold
s
1 2 3 4 50%
10%
20%
30%
40%
50%
60%
22%
57%
9% 9%4%
Adult Users per Household
# of adult users
% o
f house
hold
s
13
Goal 1Increase students/household access to at home broadband internet
preliminary survey n = 23
$50K - 75K
$25K - 50K
Under $25K
0% 20% 40% 60%
11%
50%
39%
Total Household Income
% of households
Employed full-time
Employed part-time
Not employed, looking for work
Student
0% 20% 40% 60%
27%
55%
14%
5%
Employment Status – Head of Household
% of households
14
Goal 1Increase students/household access to at home broadband internet
preliminary survey n = 23
Nearly all households report incomes under $50K, and less than one third of all heads of household are employed full-time
15
The majority of respondents report that they have gained confidence in both their own and their children’s ability to use the Internet
Own Ability (Adult)
Child's Ability0%
10%20%30%40%50%60%70%80%90%
100%
26% 19%
43% 57%
30% 24%
Much more confident than before
More confident than before
About as confident as before
Since participating in the Internet Lending pilothow confident are you in your own / your child’s ability to use the
Internet?
Goal 2Raise digital exposure and confidence
preliminary survey n = 23
**“Much less confident” and “less confident” were also answer options
Finding or applying for a job
Connecting with friends / family (e.g. Skype, Facebook)
Learning job-related skills
Keeping informed of current events
Checking information about child's school
Helping child with schoolwork
0% 20% 40% 60% 80% 100%
39%
39%
43%
57%
70%
91%
% of respondents
Nearly all respondents used the Internet to help their children with schoolwork, and many used it for job-related purposes
In the past two months, have you used the Internet
for any of the following reasons?
16
Goal 2Raise digital exposure and confidence
preliminary survey n = 23
If offered Internet service for $10 / month, the majority of respondents feel it is somewhat or very likely they would purchase it in the next six months
How likely is it that you will purchase a computer or Internet service in the next six months?
0%10%20%30%40%50%60%70%80%90%
100%
30%10%
13%
19%
13%
13%
38%
30% 33%Very likely
Somewhat likely
Neither likely nor unlikely
Somewhat unlikely
Very unlikely
17
Goal 2Raise digital exposure and confidence
preliminary survey n = 23
18
The vast majority of respondents were promoters of both the library’s wireless Internet and Chromebook services
90%
10%
NPS – Internet Service
PromoterDetrac-torPassive
Net Promoter Score = 80
96%
4%
NPS – Chromebook
PromoterDetrac-torPassive
Net Promoter Score = 96
Goal 2Raise digital exposure and confidence
preliminary survey n = 23
Net Promoter measures the likelihood of patrons recommending our services to a friend or coworker by asking, “on a scale of 0 to 10 (10 being the highest and 0 being the lowest), how likely are you to recommend (X) to a friend or colleague?” Promoters are people who select 9 or 10, passives 8 or 7, and detractors those who picked 0 through 6. The Net promoter score is then calculated by subtracting the % of promoters by the % of detractors.
Nearly all households intend to renew their devices
19
Goal 2Raise digital exposure and confidence
preliminary survey n = 23
How likely is it that you will renew the deviceafter the current loan period ends?
Wireless Router
Chromebook0%
10%20%30%40%50%60%70%80%90%
100%
5% 5%5%5% 10%
85% 86%
Very likely
Somewhat likely
Neither likely nor unlikely
Somewhat un-likely
Very unlikely
Khan Academy
Math Whiz Power My Learning
0%
20%
40%
60%
80%
100%
18% 14% 23%9% 9%
9%
27% 32%
41%
36% 32%14%
9% 14% 14%
In the past two months, how often did your child visit the following websites?
Daily or more
A few times a week
Once a week
20
Goal 3Increase engagement with library-provided and/or recommended learning resources
preliminary survey n = 23
Respondents indicated frequent usage of all three library-recommended learning resources included in the survey