[@IndeedEng] Data-Driven Product Design
-
Upload
indeedeng -
Category
Technology
-
view
14 -
download
4
description
Transcript of [@IndeedEng] Data-Driven Product Design
Data-DrivenProduct Design
J Christopher GarciaChris Hyams
DesignJ Christopher Garcia
User Experience Lead, Indeed
2008
2013
114 tests244 variations
84,000 combinations
Direct observation can be inefficient
Before and afterdoesn't work
Traffic doesn't look like this
Hourly traffic looks like this...
...and daily traffic looks like this
...and monthly traffic looks like this
Local maximum
What we've learnedUI Principles
Clickable items should look clickable
Users read
a word is worth a thousand icons
+14%more search results pages
BEWARE MULTIPLE CHANGES
Searches: -2.0%Job Clicks: -1.7%Saved jobs: -7.6%
Searches: -1.8%Job Clicks: -0.8%Saved jobs: 0%
Really?
Really.
ProductChris Hyams
SVP, Product & Engineering
I help people get jobs.
What's best for the job seeker?
How do we know what's best for job seekers?
Test and measure EVERYTHING
Measurement
1. Log everything
jobsearch indexabredistimeacmetimeaddltimeadscadsdelayadsibadscbadsiboostojcboostojibsjcbsjcwiabsjibsjindappliesbsjindappviewsbsjrevbsjwiackcntckszcountsctkagectkagedaysdayofweekdcpingtimedomTotalTimeds-mpo
dsmissdstimefeatempfjfreekwacfreekwarevfreesjcfreesjrevfrmtimegalatdelayiplatiplongjslatdelayjsvdelaykwackwacdelaykwaikwarevkwcntlacinsizelacsgsizelmstimempotimemprtimenavTotTimendxtime
ojcojclongojcshortojcwiaojiojindappliesojindappviewsojwiaoocscpageprcvdlatencyprimfollowcntprvwojiprvwojlatprvwojopentimeprvwojreqradscradsirecidlookupbudgetrectimeredirCountredirTimerelfollowcntrespTimereturnvisitrojc
rojirqcntrqlcntrqqcntrrsjcrrsjirrsjrevrsavailrsjcrsjirsusedrsviableserpsizesjcsjcdelaysjclongsjcntsjcshortsjcwiasjisjindappliessjindappviewssjrevsjwiasllatsllong
sqcsqisugtimesvjsvjnostarsvjstartadsctadsitimetimeofdaytotcnttotfollowcnttotrevtottimetsjctsjcwiatsjitsjindappliestsjindappviewstsjrevtsjwiaunqcntvpwacinsizewacsgsize
acmepageacmereviewmodacmeserviceacmesessionadclickadcrequestadcrevadschanneladsclickadsenseclickadveadvtagghttpaggjiraaggjobaggjob_waldorfaggsherlockaggsourcehealthagstimingapiapijsvapisearcharchiveindexarchiveindex_shingled_testbincarclicksclickclickanalyticscobranddctmismatchdrawdupepairsdupepairs_minidupepairs_olddupepairsalldupepairsall_miniejcheckeremilyops
feedbridgeglobalnavgooglebot_organichomepageimpressionindeedapplyjhstjobalertjobalertorganicjobalertsearchjobalertsponsoredjobexpirationjobexpiration2jobexpiration3jobprocessedjobqueueblockjobsearchjssquerykeywordAdlocsvclucyindexermainmechanicalturkmindyopsmobhomepagemobilmobilemobileorganicmobilesponsoredmobrecjobsmobsearchmobviewjobmyindeedmyindfunnelmyindpagemyindrezcreatemyindsessionoldopsesjasx
organicorgmodelorgmodelsubsetorgmodelsubset90passportaccountpassportpagepassportsigninramsaccessrecjobsrecommendserviceresumedataresumesearchrexcontactsrexfunnelreximpressionrexsearchrezSrchSearchrezalertrezalertfunnelrezfunnelrezjserrrezsrchrequestrezviewsearchablejobsseosessionsjmodelsponsoredsysadappinfosysadapptimingtestndxtestndx1testndx2tmpusrsvccacheusrsvcrequestviewjobwebusersignin
16 million jobs100 million UV
3 billion searches
1TB per day
Everything you ever wanted to know about logging but were afraid to ask
Logrepo: Enabling Data-Driven Decisions
2. Index data
3. Build tools toquery indexes
[another tech talk]
Tools answer questions
Mobile searches per hour inJP vs. UK?
Resume creation by country?
Impressions in job alerts byemail domain?
Percent of app downloads fromiOS, Android, Windows?
How quickly does a datacenter take on traffic after a failover?
Test EVERYTHING
TextVisual designLayoutControl flowNew featuresAlgorithms [also another tech talk]
Everything
5% 5%grp:bluebtntst0 grp:bluebtntst1
split:
log:
Measure
statistical relevance
p-value
Big changes require less data
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
source: http://go.indeed.com/jyusg
Small changes require more data
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
10% +1% 0.042 750,000
source: http://go.indeed.com/ommgd
Product Principles
There's no such thing as a no-brainer
-30%
Corollary: if a test fails,it means the test failed
-30%
+25%
Little things can makea big difference
organic
sponsored
+34%
Corollary: be skepticalof big changes
Does color really matter that much?
a A
a B
a C +34%
+10%
Are people just boredof blue?
+29% +22%
Will the effect wear off?
Get it right,then do it right
Location backfill: job clicks
+8%
Location backfill: search time
+1,500%
Location backfill: performance fix
+1,500%
Location backfill: performance fix
+1,500% +0%
Location backfill: job clicks
+8%
Location backfill: clicks after fix
+8% +9%
Measure everything(aka, Beware of unintended consequences)
April Fools' 2010
Organic clicks
+1.8%
Job alert signups
-17%
Revenue
+15.5%
Revenue
+15.5%
Job alert signups: top v. bottom
-17% +6%-21%
Corollary: attention is a zero-sum game
(usually)
SearchClickRevenueEmail signupResume create...
Relative value of actions
Different for everyone
Location auto-complete
Zero-results page
Zero-results pages
-2.7%
Organic clicks
+0%
Clicks w/ auto-complete
+8%
Keyword ad revenue
+1,410%
Keyword ad revenue
+1,410%
keyword ads
Keyword ad revenue after fix
Measure everything,every time
Getting Started
Fewer tests
Focus on big wins
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
10% +1% 0.042 750,000
source: http://go.indeed.com/ommgd
Straw man
Determine # of trials in a weekDetermine minimum improvementOne new test each weekIf success, roll out; if not, kill & move on
What We Get
Quality
Risk tolerance
Culture of measurement
(Empirically)better products
fin