Bederson and resnik june 2010
39
Translation as a Collaborative Activity Benjamin B. Bederson & Philip Resnik Computer Science Department Human-Computer Interaction Lab Department of Linguistics Computational Linguistics and Information Processing Lab University of Maryland
-
Upload
university-of-maryland -
Category
Technology
-
view
871 -
download
1
description
On June 10-11, 2010, the University of Maryland hosted a Workshop on Crowdsourcing and Translation. Originally put together in connection with a project on collaborative translation, the goal was to bring together a set of people whose work is helping to define this new and exciting area, and to create an opportunity for discussions that will help define its future directions.
Transcript of Bederson and resnik june 2010
Translation as a Collaborative Activity
Benjamin B. Bederson & Philip Resnik
Computer Science DepartmentHuman-Computer Interaction Lab
Department of LinguisticsComputational Linguistics and Information Processing Lab
University of Maryland
Source: Global Reach, Internet World Stats
Languages on Internet by Population
English28%
Chinese23%
Spanish8%
Japanese5%
the rest37%
2009
English32%
Chinese21%
Spanish8%
Japanese8%
the rest31%
2005
English52%
Chinese5%
Spanish5%
Japanese9%
the rest29%
2000
International Children’s Digital Library
www.childrenslibrary.org
A real-world problem: ICDL
Now– 4,386 books– 54 languages– Some translations in a few
languages– 49,420 adults, 29,102 children
registered– ~100K unique visitors/month
Goal– 10,000 books– 100 languages– Every book in every
language!
Machine Translation (MT)
Large volume, cheap, fast Unreliable quality
(餐厅 = restaurant, dining hall)
Professional Translators
High quality, but slow and expensive(even for common language pairs)
Amateur Translators
The key idea
Translation with the Crowd
vs. 75,000 contributors Wikipedia: 800 translators
Translate with the Monolingual Crowd
Quality
Affor
dabi
lity
MachineTranslationMachineTranslation
Professional Bilingual Human ParticipationProfessional Bilingual Human Participation
Amateur Bilingual Human ParticipationAmateur Bilingual Human Participation
MonolingualHumanParticipation
Monolingual translation protocol
F0
Original source sentence E0
MTNoisy target hypothesis
E1Fluent target hypothesisMTF1
Noisy back translation
F2Fluent, accurate E2
MT Et cetera…
Monolingual post-editing
HTER editing
Enrich Translations
Increase redundancy and shared context …
… to help make detectable errors correctable… to help make undetectable errors detectable
PierreSays: En général, on s'entend bien, tous les deux. (lit. In general, we get along together, the two of us.)
Mary
Sees: In general, it means well, both.MT
PierreSays: En général, on s'entend bien, tous les deux. (lit. In general, we get along together, the two of us.)
Sees: En général, Il est à la fois de nous.
Mary
Sees: In general, it means well, both.
Edits into: In general, it is about both of us.
MT
MT
PierreSays: En général, on s'entend bien, tous les deux. (lit. In general, we get along together, the two of us.)
Sees: En général, Il est à la fois de nous.
Edits into: En général, nous nous entendons bien.
(lit. In general, we get along well.)
Mary
Sees: In general, it means well, both.
Edits into: In general, it is about both of us.
Sees: In general, we get along fine.
MT
MT
MT
enrichment
PierreSays: En général, on s'entend bien, tous les deux. (lit. In general, we get along together, the two of us.)
Sees: En général, Il est à la fois de nous.
Edits into: En général, nous nous entendons bien.
(lit. In general, we get along well.)
Sees: En général, nous sommes de bons amis.(lit. In general, we are good friends.)
Mary
Sees: In general, it means well, both.
Edits into: In general, it is about both of us.
Sees: In general, we get along fine.
Edits into: In general, we are good friends.
MT
MT
MT
MT
enrichment
PierreSays: En général, on s'entend bien, tous les deux. (lit. In general, we get along together, the two of us.)
Sees: En général, Il est à la fois de nous.
Edits into: En général, nous nous entendons bien.
(lit. In general, we get along well.)
Sees: En général, nous sommes de bons amis.(lit. In general, we are good friends.)
Proposes to stop with current translation
Mary
Sees: In general, it means well, both.
Edits into: In general, it is about both of us.
Sees: In general, we get along fine.
Edits into: In general, we are good friends.
Agrees to stop with current translation
MT
MT
MT
MT
enrichment
Web link
Image
Mark ok
Mark problematic
Preliminary validation of the protocol
• Language pair: Russian to Chinese– Hard case: no orthographic cues– Easy to find local volunteers
• Two Russian speakers and four Chinese speakers– Four Russian-Chinese translation pairs (Russians twice)– One hour per pair
• Worked on 44 sentences (6 pages), finished 28– = ~8.5 minutes per sentence (~1 word per minute) – N.B. average translators: ~2500 words per 8hr day
(= ~5 words per minute)
Google Translate …
none little much most all0
2
4
6
8
10
12
14
6 6
9
43
Google TranslateMonotrans
Original meaning translated
# of
sen
tenc
es
… Monotrans
None Little Much Most All0
2
4
6
8
10
12
14
6 6
9
43
0
45
12
7
Google TranslateMonotrans
Original meaning translated
# of
sen
tenc
es
Ishida, Lin and colleagues at Kyoto University Department of Social Informatics have independently developed a very similar back-and-forth protocol.
In their protocol, there is no enrichment to increase redundancy: if the target participant cannot make sense of the whole sentence, he or she requests that the entire original sentence be rephrased.
Global Internet User Population
Source: http://www.internetworldstats.com/stats7.htm
لتحرير الشعبية الحركة اعلنتفي مرشحها سحب السودانفي اجراؤها المقرر الرئاسية االنتخاباتتاكيدات/ حسب الجاري، ابريل نيسان
الحركة في .لقياديين
Announced a Popular Movement for the Liberation of Sudan to withdraw its candidate in the presidential elections scheduled in April this as confirmation of the leaders in the movement.
http://news.bbc.co.uk/2/hi/africa/8597996.stm
Announced SPLM withdraw its candidate in the presidential elections in April by assurances to leaders in the movement.
Do I really need to worry about this?
With some things, it’s very hard to say no.
Monolingual post-editing
Callison-Burch (2005)
Enriching context
• Ppl r rlly gd t gttng mst f th mnng rght, vn frm dfcnt txt.
{minus,weak,deficient}
One more observation
The original source sentence is not the only way the intended meaning could have been expressed.
Perhaps one of these alternatives can be more successful
Suppose this phrasing is difficult to translate correctly
a restaurant close by
Polls indicate Brown, a state senator, and Coakley, Massachusetts’ Attorney General, are locked in a virtual tie to fill the late Sen. Ted Kennedy’s Senate seat
Les sondages indiquent Brown, un s ´enateur d’ ´ etat, et Coakley, Massachusetts’ Procureur g´en´eral, sont enferm´ es dans une cravate virtuel `a remplir le regrett´e s ´enateur Ted Kennedy’s si`ege au S´enat.
Les sondages indiquent Brown, un s´enateur d’ ´ etat, et Coakley, Massachusetts’ Procureur g´en´eral, sont enferm´es dans une cravate virtuel `a remplir le regrett´e s´enateur Ted Kennedy’s si`ege au S´enat.
Polls indicate Brown, a state senator, and Coakley, Massachusetts’ Attorney General, are locked in a virtual tie to fill the late Sen. Ted Kennedy’s Senate seat
Polls indicate that Brown, a state senator, and Coakley, the Attorney General of Massachusetts, are locked in a virtual tie to fill the Senate seat of the Sen. Ted Kennedy, who died recently.
Les sondages indiquent que Brown, un s ´ enateur d’ ´ etat, et Coakley, le procureur g ´en´eral du Massachusetts, sont enferm´ es dans une cravate virtuel pourvoir le sige au S´enat de Sen. Ted Kennedy, qui est d´ ec´ed´e r´ecemment
Automatically determining where the errors are
F
MT
E
MT
F’
the most recent probe to visit jupiter was the pluto-bound new horizons spacecraft
NP PP
NP
The most recent
probe
to
Mismatches?
D S
visit Jupiter
was the Pluto-bound new horizons spacecraft
The latest research
visit Jupiter
was the Pluto-bound new horizons spacecraft
• the press trust of india quoted • the government minister for relief and rehabilitation kadam
– kadam, the government’s relief and rehabilitation minister (2/3)– the government minister concerned with relief and rehabiliation kadam (1/3)
• as revealing today that in the last week, the monsoon has started in • all of india’s states one
– every one of india’s state, one (3/3)– each of India’s states one (2/3)– all states of india one (1/3)
• after another, and that the financial losses and casualties have been serious in all areas. just in maharashtra, the state which includes
• mumbai, india’s largest city, – india's largest city, mumbai (3/3)– the largest city in India, Mumbai, (3/3)– mumbai, the largest city of india, (3/3)
• the number of people • known to have died
– who died (3/3) – identified to have died (2/3) – known to have passed away (2/3)
• has now reached 358.
For 31% of the sentences in an English-to-Chinese experiment, at least one new version of the sentence leads to better translation.
Often the gains are quite substantial.
Polls indicate Brown, a state senator, and Coakley, Massachusetts’ Attorney General, are locked in a virtual tie to fill the late Sen. Ted Kennedy’s Senate seat
Les sondages indiquent Brown, un s ´enateur d’ ´ etat, et Coakley, Massachusetts’ Procureur g´en´eral, sont enferm´ es dans une cravate virtuel `a remplir le regrett´e s ´enateur Ted Kennedy’s si`ege au S´enat.
Les sondages indiquent Brown, un s´enateur d’ ´ etat, et Coakley, Massachusetts’ Procureur g´en´eral, sont enferm´es dans une cravate virtuel `a remplir le regrett´e s´enateur Ted Kennedy’s si`ege au S´enat.
Polls indicate Brown, a state senator, and Coakley, Massachusetts’ Attorney General, are locked in a virtual tie to fill the late Sen. Ted Kennedy’s Senate seat
Polls indicate that Brown, a state senator, and Coakley, the Attorney General of Massachusetts, are locked in a virtual tie to fill the Senate seat of the Sen. Ted Kennedy, who died recently.
Les sondages indiquent que Brown, un s ´ enateur d’ ´ etat, et Coakley, le procureur g ´en´eral du Massachusetts, sont enferm´ es dans une cravate virtuel pourvoir le sige au S´enat de Sen. Ted Kennedy, qui est d´ ec´ed´e r´ecemment
Where to from here?
• Larger and more formal validation of the protocol• Exploring the space of richer annotations• Reconsidering UI for:
– Ease of use– Throughput– Parallel, multi-person contribution
• Exploring the space of automatic and human error detection and paraphrase
Collaborators and Sponsors
Chang HuCS Ph.D. student
Olivia BuzekCS/Linguistics undergrad
Alex QuinnCS Ph.D. student
Yakov KronrodLinguistics Ph.D. student
