![[Appendix] Continual Learning on Noisy Data Streams via ...](https://static.fdocuments.net/doc/165x107/6294f3daf6befa5a433d865e/appendix-continual-learning-on-noisy-data-streams-via-.jpg)
Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers
32
Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers New York University Stern School Victor Sheng Foster Provost Panos Ipeirotis
description
New York University Stern School. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. Victor Sheng Foster Provost Panos Ipeirotis. Outsourcing KDD preprocessing. - PowerPoint PPT Presentation
Transcript of Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers
Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers
New York University Stern School
Victor ShengFoster ProvostPanos Ipeirotis
2
Outsourcing KDD preprocessing
Traditionally, data mining teams have invested substantial internal resources in data formulation, information extraction, cleaning, and other preprocessing
– Raghu from his Innovation Lecture“the best you can expect are noisy labels”
Now, we can outsource preprocessing tasks, such as labeling, feature extraction, verifying information extraction, etc.
– using Mechanical Turk, Rent-a-Coder, etc.– quality may be lower than expert labeling (much?) – but low costs can allow massive scale
The ideas may apply also to focusing user-generated tagging, crowdsourcing, etc.
ESP Game (by Luis von Ahn)
3
Other “free” labeling schemes
Open Mind initiative (www.openmind.org) Other gwap games
– Tag a Tune– Verbosity (tag words)– Matchin (image ranking)
Web 2.0 systems?– Can/should tagging be directed?
5
Noisy labels can be problematic
Many tasks rely on high-quality labels for objects:– learning predictive models– searching for relevant information – finding duplicate database records – image recognition/labeling– song categorization
Noisy labels can lead to degraded task performance
6
40
50
60
70
80
90
100
1 20 40 60 80 100
120
140
160
180
200
220
240
260
280
300
Number of examples (Mushroom)
Accu
racy
Quality and Classification Performance
Labeling quality increases classification quality increases
P = 0.5
P = 0.6
P = 0.8
P = 1.0
Here, labels are values for target variable
Summary of results
Repeated labeling can improve data quality and model quality (but not always)
When labels are noisy, repeated labeling can be preferable to single labeling even when labels aren’t particularly cheap
When labels are relatively cheap, repeated labeling can do much better (omitted)
Round-robin repeated labeling does well Selective repeated labeling improves
substantially
9
Majority Voting and Label Quality
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 3 5 7 9 11 13
Number of labelers
Inte
grat
ed q
ualit
y
P=0.4
P=0.5
P=0.6
P=0.7
P=0.8
P=0.9
P=1.0
Ask multiple labelers, keep majority label as “true” label Quality is probability of being correct
P is probabilityof individual labelerbeing correct
10
Tradeoffs for Modeling
Get more labels Improve label quality Improve classification Get more examples Improve classification
40
50
60
70
80
90
100
1 20 40 60 80 100
120
140
160
180
200
220
240
260
280
300
Number of examples (Mushroom)
Accu
racy
P = 0.5
P = 0.6
P = 0.8
P = 1.0
11
Basic Labeling Strategies
Single Labeling– Get as many data points as possible– one label each
Round-robin Repeated Labeling– Fixed Round Robin (FRR)
keep labeling the same set of points
– Generalized Round Robin (GRR) repeatedly-label data points, giving next label to point with fewest so far
12
Fixed Round Robin vs. Single Labeling
p= 0.6, labeling quality#examples =100
FRR(100 examples)
SL
With high noise, repeated labeling better than single labeling
13
Fixed Round Robin vs. Single Labeling
p= 0.8, labeling quality#examples =50
FRR (50 examples)
Single
With low noise, more (single labeled) examples better
50
60
70
80
90
100
80 1680 3280 4880 6480Data acquisition cost (mushroom, p=0.6)
Acc
urac
yGen. Round Robin vs. Single Labeling
P=0.6, k=5
Repeated labeling is better than single labeling
P: labeling qualityk: #labels
GRR
SL
15
Tradeoffs for Modeling
Get more labels Improve label quality Improve classification Get more examples Improve classification
40
50
60
70
80
90
100
1 20 40 60 80 100
120
140
160
180
200
220
240
260
280
300
Number of examples (Mushroom)
Accu
racy
P = 0.5
P = 0.6
P = 0.8
P = 1.0
16
Selective Repeated-Labeling
We have seen: – With enough examples and noisy labels, getting multiple
labels is better than single-labeling– When we consider costly preprocessing, the benefit is
magnified (omitted -- see paper)
Can we do better than the basic strategies? Key observation: we have additional information to
guide selection of data for repeated labeling– the current multiset of labels
Example: {+,-,+,+,-,+} vs. {+,+,+,+}
17
Natural Candidate: Entropy
Entropy is a natural measure of label uncertainty:
E({+,+,+,+,+,+})=0 E({+,-, +,-, +,- })=1
Strategy: Get more labels for examples with high-entropy label multisets
||||log
||||
||||log
||||)( 22 S
SSS
SS
SSSE
negativeSpositiveS |:||:|
18
What Not to Do: Use Entropy
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
0 400 800 1200 1600 2000
Number of labels (waveform, p=0.6)
Labe
ling
qual
ity
ENTROPYGRR
Improves at first, hurts in long run
Why not Entropy
In the presence of noise, entropy will be high even with many labels
Entropy is scale invariant (3+ , 2-) has same entropy as (600+ , 400-)
19
20
Estimating Label Uncertainty (LU)
Observe +’s and –’s and compute Pr{+|obs} and Pr{-|obs} Label uncertainty = tail of beta distribution
SLU
0.50.0 1.0
Beta probability density function
Label Uncertainty
p=0.7 5 labels
(3+, 2-) Entropy ~ 0.97 CDF=0.34
21
Label Uncertainty
p=0.7 10 labels
(7+, 3-) Entropy ~ 0.88 CDF=0.11
22
Label Uncertainty
p=0.7 20 labels
(14+, 6-) Entropy ~ 0.88 CDF=0.04
23
Label Uncertainty vs. Round Robin
24
0.6
0.7
0.8
0.9
1
0 400 800 1200 1600 2000
Number of labels (waveform, p=0.6)
Labe
ling
qual
ity
GRRLU
similar results across a dozen data sets
50
60
70
80
90
100
80 1680 3280 4880 6480Data acquisition cost (mushroom, p=0.6)
Acc
urac
y
Recall:Gen. Round Robin vs. Single Labeling
P=0.6, k=5
Multi-labeling is better than single labeling
P: labeling qualityk: #labels
GRR
SL
Label Uncertainty vs. Round Robin
26
0.6
0.7
0.8
0.9
1
0 400 800 1200 1600 2000
Number of labels (waveform, p=0.6)
Labe
ling
qual
ity
GRRLU
similar results across a dozen data sets
27
Another strategy:Model Uncertainty (MU)
Learning a model of the data provides an alternative source of information about label certainty
Model uncertainty: get more labels for instances that cannot be modeled well
Intuition?– for data quality, low-certainty “regions”
may be due to incorrect labeling of corresponding instances
– for modeling: why improve training data quality if model already is certain there?
+ ++
++ + +
+
+ ++
+
+ ++
+ + +
+ +
- - - -- - - -- -
- -
- - - -
- - - -- - - -- -
- -
- - - -
?
?
28
Yet another strategy:Label & Model Uncertainty (LMU)
Label and model uncertainty (LMU): avoid examples where either strategy is certain
MULULMU SSS
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
0 400 800 1200 1600 2000Number of labels (waveform, p=0.6)
Labe
ling
qual
ity
GRRMULULMU
Comparison
29
Label Uncertainty
GRR
Label & Model Uncertainty
Model Uncertainty alone also improves
quality
30
Comparison: Model Quality
60
65
70
75
80
85
90
0 400 800 1200 1600 2000
Number of labels (spambase, p=0.6)
Acc
urac
y
GRRMULULMU
Label & Model Uncertainty
Across 12 domains, LMU is always better than GRR. LMU is statistically significantlybetter than LU and MU.
Summary of results
Micro-task outsourcing (e.g., MTurk, RentaCoder ESP game) has changed the landscape for data formulation
Repeated labeling can improve data quality and model quality (but not always)
When labels are noisy, repeated labeling can be preferable to single labeling even when labels aren’t particularly cheap
When labels are relatively cheap, repeated labeling can do much better (omitted)
Round-robin repeated labeling can do well Selective repeated labeling improves substantially
32
Opens up many new directions…
Strategies using “learning-curve gradient” Estimating the quality of each labeler Example-conditional quality Increased compensation vs. labeler quality Multiple “real” labels Truly “soft” labels Selective repeated tagging
Thanks!
Q & A?
