Language Model Methods and Metrics
-
Upload
dara-burke -
Category
Documents
-
view
33 -
download
1
description
Transcript of Language Model Methods and Metrics
Language Model Methods and Metrics
Gary LuuRyan Fortune
Skip N-grams
• Interpolated with Bigram• Get Influence of words further away without
increasing dimensionality• Learning Curve
Skip N-gram Learning Curve
Content Word Language Model
• Help predict next word using last uncommon word, try to capture context
• Found list of 250 most common words• Tried different sizes for common words• Interpolated with language models, since this
wouldn’t maintain grammar• P(w|C)
Content Word Model
Bag Generation Metrics
• Bag Generation – NP-Hard• Random Restart Greedy Hill-Climbing• Stability Metric
• Give model correct sentence, does it maintain it as an optima?
• A percentage of sentences that remain stable
• Reconstruction Metric• Needs to be compared against lucky/random
Bag Generation Metrics
Clustering -IBMFullPredict
• Clustering overview• Perplexity down to 107 with million sentence
corpus
• Pibmfullpredict(wi|wi-2wi-1) = [λP(W|wi-2wi-1) + (1-λ)P(W|Wi-1Wi-2)] * [μP(w|wi-1wi-2,W) + (1-μ)P(w|Wi-2,Wi-1,W)]
Learning Curve for IBMFullPredict