Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco...
-
Upload
xavier-harper -
Category
Documents
-
view
220 -
download
2
Transcript of Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco...
![Page 1: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/1.jpg)
Geneva, 25 November 2011
The NER model to assess accuracyin respeaking
Pablo Romero-Fresco (Roehampton University, CAIAC research centre)
Juan Martínez (Respeaking consultant)
ITU-T Workshop on“Telecommunications relay services for persons
with disabilities ”
(Geneva, 25 November 2011)
![Page 2: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/2.jpg)
Accuracy in Respeaking
Quality in respeaking
Delay
Accuracy
![Page 3: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/3.jpg)
Accuracy in Respeaking
97-98% accuracy
![Page 4: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/4.jpg)
Basic requirements for a model
1) Functional and easy to apply
2) Include the basic principles of WER calculations in SR
3) Different programmes, different editing
4) Possibility of edited and yet accurate respeaking
5) Compare subtitles with original spoken text
6) Include other relevant info (delay, position, speed)
7) Provide both percentage and food for thought in training
![Page 5: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/5.jpg)
Traditional WER methods
US National Institute of Standards and Technology
N - ErrorsAccuracy Rate ------------------------ × 100 = %
NBut...
Well, you know, you have to try
and put out a good performance,
I mean, yeah, it’s kind of a
stepping stone, isn’t it, really?
You have to try to put out a good
performance. It’s a stepping stone.
![Page 6: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/6.jpg)
Traditional WER methods
US National Institute of Standards and Technology
N - ErrorsAccuracy Rate ------------------------ × 100 = 16%
NBut...
Well, you know, you have to try
and put out a good performance,
I mean, yeah, it’s kind of a
stepping stone, isn’t it, really?
You have to try to put out a good
performance. It’s a stepping stone.
![Page 7: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/7.jpg)
Spain = SDH guidelines
Different European countries
UN Accessibility Focus Group
N – E – R Accuracy ------------------------ ×
100 = % N
Correct editions:Serious errors:Assessment:
![Page 8: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/8.jpg)
NER Model
205 – 3 – 2 Accuracy ------------------------ × 100 = 98.6%
205
![Page 9: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/9.jpg)
NER Model
226 – 13 – 1 Accuracy ------------------------ × 100 = 93.8%
226
Assessment: poor editing (not quantity, but quality)
![Page 10: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/10.jpg)
NER Model
257 – 1 – 13Accuracy ------------------------ × 100 = 94.3%
257
Assessment: poor recognition (including serious mistakes)
![Page 11: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/11.jpg)
WGBH: “There is a wide range of error types in real time captioning and they are not all equal in their impact to caption viewers”.
“Treating all errors the same does not provide a true picture of caption accuracy”.
![Page 12: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/12.jpg)
Types of errors (feedback from DTV4ALL project)
1) “There are errors, yes, but you can easily figure out what the correct form was meant to be. Now I’m bilingual –I can speak English and teletext”
2) “Live subtitles? - Sound like gobbledygook to me”
3) “As far as I’m concerned they are not errors, but lies”
![Page 13: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/13.jpg)
Types of errors (feedback from DTV4ALL project)
1) Minor edition or recognition errors (0.25)
2) Normal edition or recognition errors (0.5)
3) Serious errors (1)
![Page 14: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/14.jpg)
Minor Errors
What a great goal by a Ryan Giggs!
Simon brown has been appointed new chairman of Rolls Royce.
For people are still missing following Sunday’s tornado.
![Page 15: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/15.jpg)
Standard Errors
He’s a buy you a bull asset.
Is it really attend Tatian?
![Page 16: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/16.jpg)
Serious errors
Public funding for universities has been cut by 15% this year.
He never talks dirty.
![Page 17: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/17.jpg)
Serious errors
Public funding for universities has been cut by 15% this year.
He never talks dirty.
He never talks to Rudy.
![Page 18: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/18.jpg)
![Page 19: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/19.jpg)
NER MODEL
N – E – RAccuracy ------------------------ ×
100 = % N
Correct editions:Comments:
Target = 98%
![Page 20: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/20.jpg)
NER Orange with apples
![Page 22: Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco (Roehampton University, CAIAC research centre) Juan Martínez.](https://reader036.fdocuments.net/reader036/viewer/2022062511/551542195503465e608b60f5/html5/thumbnails/22.jpg)
Geneva, 25 November 2011
The NER model to assess accuracyin respeaking
Pablo Romero-Fresco (Roehampton University, CAIAC research centre)([email protected])
Juan Martínez (Respeaking consultant)
ITU-T Workshop on“Telecommunications relay services for persons
with disabilities ”
(Geneva, 25 November 2011)