Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.

©2013 LinkedIn Corporation. All Rights Reserved.

Hacking Data SciencePatrick PhilipsVitaly Gordon

©2013 LinkedIn Corporation. All Rights Reserved. 2

Overview of ML pipeline

Gather data

Feature engineering

Model fitting

Evaluation


Understanding Seniority


Companies are not standard


Titles are not enough


Things change


Learning to target better


Classifying names to genders


Let’s look at Monica again


Not so fast …


Even slower …


Sometime the answer is just under your nose


Comment Spam on Influencer content


Challenge 1: Binary tasks are too guessable


Challenge 2: Context matters


Spam Comment Annotation Task


Quality: Gold distributions and skewed datasets


Using results to evaluate new features

Model ΔP ΔR ΔPRC

Baseline - - -

Variation 1 + - +

Variation 2 - + +

Variation 3 - ++ - -

Variation 4 - +++ ++



Variation 7 - ++++ +++



Variation 10 - ++++ +++


“As simple as possible, but not simpler”


Linkedin Channels


Labels aren’t free


Suggest likely candidates for topics then expand


Evaluate suggested article-topic pairs

Using results to evaluate new implementations of spam classifier– Improve Prec without drop in Rec

18k comments labeled in 54 hrs for $180


Quality: Not by Gold alone


Using results to evaluate existing classification framework


“Help your helpers”


Search is a major portal to information


LI Search is personalized


Evaluation is still possible


Search Evaluation – WTF@1


Quality: Behavioral metrics are good too!


“Pick a solvable problem”


Standardizing titles


Which question is easier

1. Find a better name for the title “account executive”?

2. How similar are “account executive” and “sales executive”?


Notable Experts


First attempt


Second attempt


Third attempt


What makes the best data mining expert?

Education?

Industry experience?

Amount of publications?

Communication skills?

Hacking skills?

Knowledge of statistics?

Number of endorsements?


“More bad data != better data”


Summary

1. Use the data you already have

2. Keep it simple, but not too simple

3. Pick a solvable problem

4. Help your helpers

5. Sample intelligently

6. More (bad) data != better data


Questions?

Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.

Technology

Transcript of Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.