Outline: Where have we been and were are we going?

• We’re making consistent progress, or• We’re running around in circles, or

– 1950s: Empiricism (Information Theory, Behaviorism)– 1970s: Rationalism (AI, Cognitive Psychology)– 1990s: Empiricism (Data Mining, Statistical NLP, Speech)– 2010s: Rationalism (TBD)

• We’re going off a cliff…– Don’t worry; be happy

0%20%40%60%80%

100%

19

85

19

90

19

95

20

00

20

05

ACL Meeting

% S

tati

sti

cal

P

ap

ers

Bob Moore Fred Jelinek

No matter what happens, it’s goin’

be great!

Rising tide of data lifts all boats

Rising Tide of Data Lifts All BoatsIf you have a lot of data, then you don’t need a lot of methodology

• 1985: “There is no data like more data”– Fighting words uttered by radical fringe elements

(Mercer at Arden House)• 1995: The Web changes everything• All you need is data (magic sauce)

– No linguistics– No artificial intelligence (representation)– No machine learning– No statistics– No error analysis– No data mining– No text mining

“It never pays to think until you’ve run out of data” – Eric Brill

Banko & Brill: Mitigating the Paucity-of-Data Problem (HLT 2001)

Fire everybody and spend the money on data

More data is better data!

No consistentlybest learner

Quo

ted

out

of c

onte

xt

Moore’s Law Constant:Data Collection Rates Improvement Rates

The rising tide of data will lift all boats!TREC Question Answering & Google:

What is the highest point on Earth?

The rising tide of data will lift all boats!Acquiring Lexical Resources from Data:

Dictionaries, Ontologies, WordNets, Language Models, etc.http://labs1.google.com/sets

England Japan Cat cat

France China Dog more

Germany India Horse ls

Italy Indonesia Fish rm

Ireland Malaysia Bird mv

Spain Korea Rabbit cd

Scotland Taiwan Cattle cp

Belgium Thailand Rat mkdir

Canada Singapore Livestock man

Austria Australia Mouse tail

Australia Bangladesh Human pwd

Applications• What good is word sense disambiguation (WSD)?

– Information Retrieval (IR)• Salton: Tried hard to find ways to use NLP to help IR

– but failed to find much (if anything)• Croft: WSD doesn’t help because IR is already using those methods• Sanderson (next two slides)

– Machine Translation (MT)• Original motivation for much of the work on WSD• But IR arguments may apply just as well to MT

• What good is POS tagging? Parsing? NLP? Speech?• Commercial Applications of Natural Language Processing,

CACM 1995– $100M opportunity (worthy of government/industry’s attention)

1. Search (Lexis-Nexis)2. Word Processing (Microsoft)

• Warning: premature commercialization is risky

Don’t worry;Be happy

Sanderson (SIGIR-94)http://dis.shef.ac.uk/mark/cv/publications/papers/my_papers/SIGIR94.pdf

Not much?

• Could WSD help IR?• Answer: no

– Introducing ambiguity by pseudo-words doesn’t hurt (much)

Short queries matter most, but hardest for WSD

F

Query Length (Words)

5 Ia

n A

nder

sons

Sanderson (SIGIR-94)http://dis.shef.ac.uk/mark/cv/publications/papers/my_papers/SIGIR94.pdf

• Resolving ambiguity badly is worse than not resolving at all– 75% accurate WSD

degrades performance– 90% accurate WSD:

breakeven point

Soft WSD?

Query Length (Words)

F

Some Promising Suggestions(Generate lots of conference papers, but may not support the field)

• Two Languages are Better than One– For many classic hard NLP

problems• Word Sense

Disambiguation (WSD)• PP-attachment• Conjunction• Predicate-argument

relationships• Japanese and Chinese

Word breaking– Parallel corpora plenty

of annotated (labeled) testing and training data

– Don’t need unsupervised magic (data >> magic)

• Demonstrate that NLP is good for something– Statistical methods (IR & WSD)

focus on bags of nouns,• Ignoring verbs, adjectives,

predicates, intensifiers, etc.– Hypothesis: Ignored because

perceptrons can’t model XOR– Task: classify “comments” into

“good,” “bad” and “neutral”• Lots of terms associated with just

one category• Some associated with two

– Depending on argument• Good & Bad, but not neutral:

Mickey Mouse, Rinky Dink– Bad: Mickey Mouse(us)– Good: Mickey Mouse(them)

– Current IR/WSD methods don’t capture predicate-argument relationships

Web Apps: Document Language Model ≠ Query Language Model

• Documents– Function Words– Adjectives– Verbs– Predicates

• Queries– Typos– Brand Names– Celebrities– Named Entities– Slower Vocab Growth

Technical Op: Reduce IR to Translation

Promising Apps: Web Spam, Frame Problem

Speech Data Mining & Call Centers:

An Intelligence Bonanza • Some companies are collecting

information with technology designed to monitor incoming calls for service quality.

• Last summer, Continental Airlines Inc. installed software from Witness Systems Inc. to monitor the 5,200 agents in its four reservation centers.

• But the Houston airline quickly realized that the system, which records customer phone calls and information on the responding agent's computer screen, also was an intelligence bonanza, says André Harris, reservations training and quality-assurance director.

Speech Data Mining• Label calls as success or failure based on

some subsequent outcome (sale/no sale)

• Extract features from speech

• Find patterns of features that can be used to predict outcomes

• Hypotheses:– Customer: “I’m not interested” no sale– Agent: “I just want to tell you…” no sale

Inter-ocular effect (hits you between the eyes);Don’t need a statistician to know which way the wind is blowing

Outline

• We’re making consistent progress, or

• We’re running around in circles, or– Don’t worry; be happy

• We’re going off a cliff…

According to unnamed sources:Speech Winter Language Winter

Dot Boom Dot Bust

Sample of 20 Survey Questions(Strong Emphasis on Applications)

• When will– More than 50% of new PCs have dictation on them, either at

purchase or shortly after.– Most telephone Interactive Voice Response (IVR) systems

accept speech input.– Automatic airline reservation by voice over the telephone is the

norm.– TV closed-captioning (subtitling) is automatic and pervasive.– Telephones are answered by an intelligent answering machine

that converses with the calling party to determine the nature and priority of the call.

– Public proceedings (e.g., courts, public inquiries, parliament, etc.) are transcribed automatically.

• Two surveys of ASRU attendees: 1997 & 2003

Hockey StickBusiness Case

2003 2004 2005

t

$

LastYear

ThisYear Next

Year

2003 Responses ≈ 1997 Responses + 6 Years(6 years of hard work No progress)

Wrong Apps?

• New Priorities– Increase demand for

space >> Data entry• New Killer Apps

– Search >> Dictation• Speech Google!

– Data mining

• Old Priorities– Dictation app dates back to

days of dictation machines– Speech recognition has not

displaced typing• Speech recognition has

improved• But typing skills have

improved even more– My son will learn typing in

1st grade– Sec rarely take dictation

– Dictation machines are history• My son may never see one• Museums have slide rulers

and steam trains– But dictation machines?

Great Challenge: Annotating Data

• Produce annotated data with minimal supervision

• Active learning– Identify reliable labels– Identify best candidates for annotation

• Co-training• Bootstrap (project) resources from one

application to another

Borrowed Slide: Jelinek (LREC)

Self-organizing “Magic” ≠ Error Analysis

Great Strategy Success

Grand Challengesftp://ftp.cordis.lu/pub/ist/docs/istag040319-draftnotesofthemeeting.pdf

Roadmaps: Structure of a Strategy(not the union of what we are all doing)

• Goals– Example: Replace keyboard with

microphone– Exciting (memorable) sound bite– Broad grand challenge that we can work

toward but never solve• Metrics

– Examples: • WER: word error rate• Time to perform task

– Easy to measure• Milestones

– Should be no question if it has been accomplished

– Example: reduce WER on task x by y% by time t

• Accomplishments v. Activities– Accomplishments are good– Activity is not a substitute for

accomplishments– Milestones look forward whereas

accomplishments look backward• Serendipity is good!

• Small is beautiful– Quantity is not a good thing– Awareness– 1-slide version

• if successful, you get maybe 3 more slides

• Size of container– Goal: 1-3– Metrics: 3– Milestones: a dozen

• Mostly for next year: Q1-4• Plus some for years 2, 5, 10 & 20

– Accomplishments: a dozen• Broad applicability & illustrative

– Don’t cover everything– Highlight stuff that

• Applies to multiple groups• Forward-Looking / Exciting

€ € €

ResourcesApps & Techniques

Grand Challenges

Goal: Reduce barriers to entry

Goals:1. The multilingual companion2. Life log

Goal: Produce NLP apps that improve the way people communicate

with one another

Evaluation

Summary: What Workedand What Didn’t?

• Data– Stay on msg: It is the data, stupid!It is the data, stupid!

• WVLC (Very Large) >> EMNLP (Empirical Methods)• If you have a lot of data,

– Then you don’t need a lot of methodology

• Rising Tide of Data Lifts All Boats

• Methodology– Empiricism means different things to different people

1. Machine Learning (Self-organizing Methods)2. Exploratory Data Analysis (EDA)3. Corpus-Based Lexicography

– Lots of papers on 1• EMNLP-2004 theme (error analysis) 2• Senseval grew out of 3

Substance: Recommended if…

Magic: Recommended if…

Promise: Recommended if…

Short term ≠ Long term

Lonely

What’s the right answer?

There’ll be a quiz at the end of the decade…