GloVe: Global Vectors for Word Representation · Theatre artist Patrice Pavis de nes theatricality,...
Transcript of GloVe: Global Vectors for Word Representation · Theatre artist Patrice Pavis de nes theatricality,...
GloVe: Global Vectors for Word Representation
Jeffrey Pennington, Richard Socher, Christopher D. Manning
Presented by Chris Kedzie
March 25, 2015
Chris Kedzie GloVe March 25, 2015 1 / 30
Overview
1 Introduction
2 Problem
3 GloVe Model
4 Experiments
Chris Kedzie GloVe March 25, 2015 2 / 30
GloVe
1 Introduction
2 Problem
3 GloVe Model
4 Experiments
Chris Kedzie GloVe March 25, 2015 3 / 30
Word Representations: A history
Chris Kedzie GloVe March 25, 2015 4 / 30
Neural Language Models – Recurrent NNLM
ht−1
wt
...
...
ht
ot+1
Chris Kedzie GloVe March 25, 2015 5 / 30
Neural Language Models – Recurrent NNLM
ht−1
wt
...
...
ht
ot+1
Chris Kedzie GloVe March 25, 2015 5 / 30
Neural Language Models – Recurrent NNLM
ht−1
wt
...
...
ht
ot+1
wt
Chris Kedzie GloVe March 25, 2015 5 / 30
Neural Language Models – Continuous BOW
wt−2
wt−1
wt+1
wt+2
wavg
...
...
ot
Chris Kedzie GloVe March 25, 2015 6 / 30
Neural Language Models – Continuous BOW
wt−2
wt−1
wt+1
wt+2
wavg
...
...
ot
Chris Kedzie GloVe March 25, 2015 6 / 30
Linear Relationships
Semanticwking − wman + wwoman ≈ wqueen
Syntacticweasy − weasiest + wluckiest ≈ wlucky
Chris Kedzie GloVe March 25, 2015 7 / 30
Scalable Embedding Learning
Noise Contrastive Estimation
–no more normalization required!
wt−2
wt−1
wt+1
wt+2
wavg
...
...
ot
Chris Kedzie GloVe March 25, 2015 8 / 30
Scalable Embedding Learning
Noise Contrastive Estimation –no more normalization required!
wt−2
wt−1
wt+1
wt+2
wavg
...
...
ot
Chris Kedzie GloVe March 25, 2015 8 / 30
Scalable Embedding Learning
Noise Contrastive Estimation –no more normalization required!
wt−2
wt−1
wt+1
wt+2
wavg
...
...
ot
wavgw
Chris Kedzie GloVe March 25, 2015 8 / 30
GloVe
1 Introduction
2 Problem
3 GloVe Model
4 Experiments
Chris Kedzie GloVe March 25, 2015 9 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)
Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Local Online Optimization
Lots of time spent scanning context windows to learn a distribution for
P (w|the)Theatre or theater is a collaborative form of fine art that uses live performers to present the experience of a real or imaginedevent before a live audience in a specific place. The performers may communicate this experience to the audience throughcombinations of gesture, speech, song, music, and dance. Elements of art and stagecraft are used to enhance the physicality,presence and immediacy of the experience. The specific place of the performance is also named by the word ”theatre” as derivedfrom the Ancient Greek (thatron, ”a place for viewing”), itself from (theomai, ”to see”, ”to watch”, ”to observe”).Modern Western theatre comes from large measure from ancient Greek drama, from which it borrows technical terminology,classification into genres, and many of its themes, stock characters, and plot elements. Theatre artist Patrice Pavis definestheatricality, theatrical language, stage writing, and the specificity of theatre as synonymous expressions that differentiatetheatre from the other performing arts, literature, and the arts in general.Theatre today, broadly defined, includes performances of plays and musicals, ballets, operas and various other forms.
There’s got to be a better way!
Chris Kedzie GloVe March 25, 2015 10 / 30
Matrix Factorization Methods
e.g. SVD, COALS, etc. directly on co-occurrence matrix.
Main drawback: frequent words like the and a have an outsized effect onthe representation learning.
Chris Kedzie GloVe March 25, 2015 11 / 30
Matrix Factorization Methods
e.g. SVD, COALS, etc. directly on co-occurrence matrix.
Main drawback: frequent words like the and a have an outsized effect onthe representation learning.
Chris Kedzie GloVe March 25, 2015 11 / 30
GloVe
1 Introduction
2 Problem
3 GloVe Model
4 Experiments
Chris Kedzie GloVe March 25, 2015 12 / 30
GloVe Model
J =
V∑i,j=1
f (Xij)(wTi w̃j + bi + b̃j − logXij
)2
Chris Kedzie GloVe March 25, 2015 13 / 30
Notation!
X ∈ RV×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
Xi =∑V
k Xik total number of occurrences of word i in corpus
Pij = P (j|i) = Xij
Xia.k.a. probability of word j occurring within the
context of word i
w ∈ Rd a word embedding of dimension d
w̃ ∈ Rd a context word embedding of dimension d
Chris Kedzie GloVe March 25, 2015 14 / 30
Notation!
X ∈ RV×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
Xi =∑V
k Xik total number of occurrences of word i in corpus
Pij = P (j|i) = Xij
Xia.k.a. probability of word j occurring within the
context of word i
w ∈ Rd a word embedding of dimension d
w̃ ∈ Rd a context word embedding of dimension d
Chris Kedzie GloVe March 25, 2015 14 / 30
Notation!
X ∈ RV×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
Xi =∑V
k Xik total number of occurrences of word i in corpus
Pij = P (j|i) = Xij
Xia.k.a. probability of word j occurring within the
context of word i
w ∈ Rd a word embedding of dimension d
w̃ ∈ Rd a context word embedding of dimension d
Chris Kedzie GloVe March 25, 2015 14 / 30
Notation!
X ∈ RV×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
Xi =∑V
k Xik total number of occurrences of word i in corpus
Pij = P (j|i) = Xij
Xia.k.a. probability of word j occurring within the
context of word i
w ∈ Rd a word embedding of dimension d
w̃ ∈ Rd a context word embedding of dimension d
Chris Kedzie GloVe March 25, 2015 14 / 30
Notation!
X ∈ RV×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
Xi =∑V
k Xik total number of occurrences of word i in corpus
Pij = P (j|i) = Xij
Xia.k.a. probability of word j occurring within the
context of word i
w ∈ Rd a word embedding of dimension d
w̃ ∈ Rd a context word embedding of dimension d
Chris Kedzie GloVe March 25, 2015 14 / 30
Notation!
X ∈ RV×V word co-occurrence matrix
Xij frequency of word i co-occurring with word j
Xi =∑V
k Xik total number of occurrences of word i in corpus
Pij = P (j|i) = Xij
Xia.k.a. probability of word j occurring within the
context of word i
w ∈ Rd a word embedding of dimension d
w̃ ∈ Rd a context word embedding of dimension d
Chris Kedzie GloVe March 25, 2015 14 / 30
Motivation
Prob. and Ratio k = solid k = gas k = water k = fashion
P (k|ice) 1.9× 10−4 6.6× 10−5 3.0× 10−3 1.7× 10−5
P (k|steam) 2.2× 10−5 7.8× 10−4 2.2× 10−3 1.8× 10−5
P (k|ice)P (k|steam) 8.9 8.5× 10−2 1.36 0.96
Chris Kedzie GloVe March 25, 2015 15 / 30
Motivation
Prob. and Ratio k = solid k = gas k = water k = fashion
P (k|ice) 1.9× 10−4 6.6× 10−5 3.0× 10−3 1.7× 10−5
P (k|steam) 2.2× 10−5 7.8× 10−4 2.2× 10−3 1.8× 10−5
P (k|ice)P (k|steam) 8.9 8.5× 10−2 1.36 0.96
Chris Kedzie GloVe March 25, 2015 15 / 30
Motivation
Prob. and Ratio k = solid k = gas k = water k = fashion
P (k|ice) 1.9× 10−4 6.6× 10−5 3.0× 10−3 1.7× 10−5
P (k|steam) 2.2× 10−5 7.8× 10−4 2.2× 10−3 1.8× 10−5
P (k|ice)P (k|steam) 8.9 8.5× 10−2 1.36 0.96
Chris Kedzie GloVe March 25, 2015 15 / 30
Motivation
Prob. and Ratio k = solid k = gas k = water k = fashion
P (k|ice) 1.9× 10−4 6.6× 10−5 3.0× 10−3 1.7× 10−5
P (k|steam) 2.2× 10−5 7.8× 10−4 2.2× 10−3 1.8× 10−5
P (k|ice)P (k|steam) 8.9 8.5× 10−2 1.36 0.96
Chris Kedzie GloVe March 25, 2015 15 / 30
Motivation
Prob. and Ratio k = solid k = gas k = water k = fashion
P (k|ice) 1.9× 10−4 6.6× 10−5 3.0× 10−3 1.7× 10−5
P (k|steam) 2.2× 10−5 7.8× 10−4 2.2× 10−3 1.8× 10−5
P (k|ice)P (k|steam) 8.9 8.5× 10−2 1.36 0.96
Chris Kedzie GloVe March 25, 2015 15 / 30
Derivation
F (wi, wj , w̃k) =PikPjk
F should encode information in the ratio PikPjk
.
Chris Kedzie GloVe March 25, 2015 16 / 30
Derivation
F (wi, wj , w̃k) =PikPjk
F should encode information in the ratio PikPjk
.
Chris Kedzie GloVe March 25, 2015 16 / 30
Derivation
F (wi − wj , w̃k) =PikPjk
Chris Kedzie GloVe March 25, 2015 17 / 30
Derivation
F((wi − wj)T w̃k
)=PikPjk
Some more desiderata:
F should be unchanged by exchanging w → w̃ and X → XT
This requires that
F((wi − wj)T w̃k
)=
F(wTi w̃k
)F(wTj w̃k
)⇒ F (wTi w̃k) = Pik
Chris Kedzie GloVe March 25, 2015 18 / 30
Derivation
F((wi − wj)T w̃k
)=PikPjk
Some more desiderata:
F should be unchanged by exchanging w → w̃ and X → XT
This requires that
F((wi − wj)T w̃k
)=
F(wTi w̃k
)F(wTj w̃k
)⇒ F (wTi w̃k) = Pik
Chris Kedzie GloVe March 25, 2015 18 / 30
Derivation
F((wi − wj)T w̃k
)=PikPjk
Some more desiderata:
F should be unchanged by exchanging w → w̃ and X → XT
This requires that
F((wi − wj)T w̃k
)=
F(wTi w̃k
)F(wTj w̃k
)⇒ F (wTi w̃k) = Pik
Chris Kedzie GloVe March 25, 2015 18 / 30
Derivation
F((wi − wj)T w̃k
)=PikPjk
Some more desiderata:
F should be unchanged by exchanging w → w̃ and X → XT
This requires that
F((wi − wj)T w̃k
)=
F(wTi w̃k
)F(wTj w̃k
)⇒ F (wTi w̃k) = Pik
F(wTi w̃k − wTj w̃k
)=
F(wTi w̃k
)F(wTj w̃k
)Chris Kedzie GloVe March 25, 2015 18 / 30
Derivation
F((wi − wj)T w̃k
)=PikPjk
Some more desiderata:
F should be unchanged by exchanging w → w̃ and X → XT
This requires that
F((wi − wj)T w̃k
)=
F(wTi w̃k
)F(wTj w̃k
)⇒ F (wTi w̃k) = Pik
exp(wTi w̃k − wTj w̃k
)=
exp(wTi w̃k
)exp
(wTj w̃k
)Chris Kedzie GloVe March 25, 2015 18 / 30
Derivation
exp(wTi w̃k − wTj w̃k
)=
exp(wTi w̃k
)exp
(wTj w̃k
)
wTi w̃k = logPik = logXik − logXi
wTi w̃k = logXik − bi − b̃kwTi w̃k + bi + b̃k = logXik
Chris Kedzie GloVe March 25, 2015 19 / 30
Derivation
exp(wTi w̃k − wTj w̃k
)=
exp(wTi w̃k
)exp
(wTj w̃k
)wTi w̃k = logPik = logXik − logXi
wTi w̃k = logXik − bi − b̃kwTi w̃k + bi + b̃k = logXik
Chris Kedzie GloVe March 25, 2015 19 / 30
Derivation
exp(wTi w̃k − wTj w̃k
)=
exp(wTi w̃k
)exp
(wTj w̃k
)wTi w̃k = logPik = logXik − logXi
wTi w̃k = logXik − bi − b̃kwTi w̃k + bi + b̃k = logXik
Chris Kedzie GloVe March 25, 2015 19 / 30
Derivation
exp(wTi w̃k − wTj w̃k
)=
exp(wTi w̃k
)exp
(wTj w̃k
)wTi w̃k = logPik = logXik − logXi
wTi w̃k = logXik − bi − b̃k
wTi w̃k + bi + b̃k = logXik
Chris Kedzie GloVe March 25, 2015 19 / 30
Derivation
exp(wTi w̃k − wTj w̃k
)=
exp(wTi w̃k
)exp
(wTj w̃k
)wTi w̃k = logPik = logXik − logXi
wTi w̃k = logXik − bi − b̃kwTi w̃k + bi + b̃k = logXik
Chris Kedzie GloVe March 25, 2015 19 / 30
Derivation
This suggests a least-squares objective function,
but...
J =
V∑i,j=1
(wTi w̃j + bi + b̃j − logXij
)2
⇒ J =V∑
i,j=1
f (Xij)(wTi w̃j + bi + b̃j − logXij
)2
Chris Kedzie GloVe March 25, 2015 20 / 30
Derivation
This suggests a least-squares objective function, but...
J =
V∑i,j=1
(wTi w̃j + bi + b̃j − logXij
)2
⇒ J =V∑
i,j=1
f (Xij)(wTi w̃j + bi + b̃j − logXij
)2
Chris Kedzie GloVe March 25, 2015 20 / 30
Derivation
This suggests a least-squares objective function, but...
J =
V∑i,j=1
(wTi w̃j + bi + b̃j − logXij
)2
⇒ J =V∑
i,j=1
f (Xij)(wTi w̃j + bi + b̃j − logXij
)2
Chris Kedzie GloVe March 25, 2015 20 / 30
Derivation
This suggests a least-squares objective function, but...
J =
V∑i,j=1
(wTi w̃j + bi + b̃j − logXij
)2
⇒ J =V∑
i,j=1
f (Xij)(wTi w̃j + bi + b̃j − logXij
)2
Chris Kedzie GloVe March 25, 2015 20 / 30
Derivation
This suggests a least-squares objective function, but...
J =
V∑i,j=1
(wTi w̃j + bi + b̃j − logXij
)2⇒ J =
V∑i,j=1
f (Xij)(wTi w̃j + bi + b̃j − logXij
)2where f has the following desiderata:
1 f(0) = 0
2 f(x) should be non-decreasing so that rare co-occurrences are notoverweighted.
3 f(x) should be relatively small for large values of x, so that frequentco-occurrences are not overweighted.
Chris Kedzie GloVe March 25, 2015 20 / 30
Derivation
This suggests a least-squares objective function, but...
J =
V∑i,j=1
(wTi w̃j + bi + b̃j − logXij
)2⇒ J =
V∑i,j=1
f (Xij)(wTi w̃j + bi + b̃j − logXij
)2
where f(x) =
{(x
xmax
)αif x < xmax
1 otherwise
Chris Kedzie GloVe March 25, 2015 20 / 30
Weighting Function
Chris Kedzie GloVe March 25, 2015 21 / 30
Optimization
J =
V∑i,j=1
f (Xij)(wTi w̃j + bi + b̃j − logXij
)2
where f(x) =
{(x
xmax
)αif x < xmax
1 otherwise
In this paper: α = 34 and xmax = 100.
The model is trained using AdaGrad and stochastically sampling non-zeroelements from X. An initial learning rate of .05 is used.
Chris Kedzie GloVe March 25, 2015 22 / 30
GloVe
1 Introduction
2 Problem
3 GloVe Model
4 Experiments
Chris Kedzie GloVe March 25, 2015 23 / 30
Word Analogies
a is to b as c to ?
Paris is to France as Tokyo is to ?
arg maxw′ cosine-sim(
wb − wa + wc
, w′)
Chris Kedzie GloVe March 25, 2015 24 / 30
Word Analogies
a is to b as c to ?Paris is to France as Tokyo is to ?
arg maxw′ cosine-sim(
wb − wa + wc
, w′)
Chris Kedzie GloVe March 25, 2015 24 / 30
Word Analogies
a is to b as c to ?Paris is to France as Tokyo is to ?
arg maxw′ cosine-sim(wb − wa + wc, w′)
Chris Kedzie GloVe March 25, 2015 24 / 30
Word Analogies – Results
Chris Kedzie GloVe March 25, 2015 25 / 30
Word Similarities
Humans scored similarity of word pairs.
word 1 word 2 human score (mean) (1-10) cosine-similarity (-1, 1)
king cabbage 0.23 0.11king queen 8.58 0.78king rook 5.92 0.25
Embeddings are evaluated by Spearman rank correlation of human scoresto cosine similarity.
Chris Kedzie GloVe March 25, 2015 26 / 30
Word Similarities
Humans scored similarity of word pairs.
word 1 word 2 human score (mean) (1-10) cosine-similarity (-1, 1)
king cabbage 0.23 0.11king queen 8.58 0.78king rook 5.92 0.25
Embeddings are evaluated by Spearman rank correlation of human scoresto cosine similarity.
Chris Kedzie GloVe March 25, 2015 26 / 30
Word Similarities
Humans scored similarity of word pairs.
word 1 word 2 human score (mean) (1-10) cosine-similarity (-1, 1)
king cabbage 0.23 0.11king queen 8.58 0.78king rook 5.92 0.25
Embeddings are evaluated by Spearman rank correlation of human scoresto cosine similarity.
Chris Kedzie GloVe March 25, 2015 26 / 30
Word Similarities – Results
Chris Kedzie GloVe March 25, 2015 27 / 30
Named Entity Recognition
NER is a sequence tagging task where the goal is to identify namedentities:
Jim bought 300 shares of Acme Corp . in 2006 .B-PER O O O O B-ORG I-ORG I-ORG O B-TIME O
Combined discrete features of existing system (Stanford NER).
Word embeddings were treated as additional features in a linear-chain CRFmodel.
Chris Kedzie GloVe March 25, 2015 28 / 30
Named Entity Recognition
NER is a sequence tagging task where the goal is to identify namedentities:
Jim bought 300 shares of Acme Corp . in 2006 .B-PER O O O O B-ORG I-ORG I-ORG O B-TIME O
Combined discrete features of existing system (Stanford NER).
Word embeddings were treated as additional features in a linear-chain CRFmodel.
Chris Kedzie GloVe March 25, 2015 28 / 30
Named Entity Recognition – Results
Chris Kedzie GloVe March 25, 2015 29 / 30
The end! Thanks!
Chris Kedzie GloVe March 25, 2015 30 / 30