A quantitative uncertainty metric controls error in neural ...
Translation Error Rate Metric
-
Upload
westbrook-sterling -
Category
Documents
-
view
15 -
download
0
description
Transcript of Translation Error Rate Metric
March 10, 2005 METEOR 1
Translation Error Rate Metric
• Reference: “the Iraqi weapons are to be handed over to the army within two weeks”
• MT output: “in two weeks Iraq’s weapons will give army”
March 10, 2005 METEOR 2
Step-1: Remove Stop Words
• the Iraqi weapons are to be handed over to the army within two weeks
• in two weeks Iraq’s weapons will give army
March 10, 2005 METEOR 3
Step-2: Word Align with Matcher
• the Iraqi weapons are to be handed over to the army within two weeks
• in two weeks Iraq’s weapons will give army
March 10, 2005 METEOR 4
Step-3: Count Flips (Reorder)
• the Iraqi weapons are to be handed over to the army within two weeks
• in two weeks Iraq’s weapons will give army
• Reordered:• in Iraq’s weapons will give army two weeks• Flip count = 4
March 10, 2005 METEOR 5
Step-4: Count Insertions/Deletions/Substitutions
• the Iraqi weapons are to be handed over to the army within two weeks
• in two weeks Iraq’s weapons will give army
• Reordered:• in Iraq’s weapons will give army two weeks• Flip count = 4• Substitutions: 2• Insertions (from Ref): 5• Deletions (from MT): 1
March 10, 2005 METEOR 6
Step-5: Score
• the Iraqi weapons are to be handed over to the army within two weeks
• in two weeks Iraq’s weapons will give army
• Reordered:• in Iraq’s weapons will give army two weeks• Flip count = 4• Substitutions: 2• Insertions (from Ref): 5• Deletions (from MT): 1• TER = flips + subst + ins + del / ref length =
4+2+5+1/12 = 12/12 = 1.0
March 10, 2005 METEOR 7
Another Example
• Ref: north korea is prepared to allow washington to verify that it does not manufacture nuclear weapons
• MT: North Korea prepared to allow Washington to the verification of that is to manufacture nuclear weapons
• Flips: 0• Substitutions: 3• Insertions: 1• Deletions: 0• TER Score = 0+3+1+0/16 = 4/16 = 0.25