P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization
Transcript of P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization
![Page 1: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/1.jpg)
MANDIAC: A Web-based Annotation System For Manual
Arabic Diacritization
Collaborators: Houda Bouamor, Wajdi Zaghouani, Mahmoud Ghoneim, Abdelati Hawwari, Mona Diab and Kemal Oflazer
Ossama ObeidCarnegie Mellon University in Qatar
![Page 2: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/2.jpg)
Introduction• Arabic text is composed of consonants, long vowels, and short
vowels (diacritics).• Absence of diacritics:
oAdds lexical and morphological ambiguity.oConfusing to beginners.o Impacts performance of Arabic NLP tasks.
• Very few texts are diacritized.
![Page 3: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/3.jpg)
Introduction
Possible pronunciation and meanings of the undiacritized Arabic word ذكر.
![Page 4: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/4.jpg)
Introduction• Most automatic diacritization systems trained on Arabic
Treebanks.• Different genre and dialects need new datasets:
oTime consuming.oMust insure data quality and consistency.
![Page 5: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/5.jpg)
Currently Available Annotation Tools• Very basic text-editor-like interfaces.• Can’t handle a large number of documents and annotators.• Not easily customizable.
![Page 6: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/6.jpg)
MANDIAC• Web-based.• Intuitive and easy to use.• Easily manages thousands of documents.• Distributes tasks (including IAA evaluation tasks) to tens of
annotators .• Doubles annotation speed!• Based on QAWI.• Provides Annotation and Annotation Management interfaces.
![Page 7: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/7.jpg)
Annotation Interface• Token-based annotation system similar to QAWI.• Annotators can choose pre-computed diacritizations (derived
using MADAMIRA) and/or manually edit diacritics.• Additional features to increase annotator productivity.
![Page 8: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/8.jpg)
Annotation InterfaceExtra Features:• Undo/Redo buttons• Edits restricted to diacritics only• Timer• Counter indicating number of words left to annotate• Link to annotation guidelines• Token highlighting:
o Annotated wordso Tokens that should not be edited (eg digits, non-Arabic words, punctuation)
• Flag documents• Mark tokens as ambiguous
![Page 9: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/9.jpg)
Annotation Interface
Annotation Interface at a glance
![Page 10: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/10.jpg)
Annotation Interface
Dropdown showing top 3 automatically diacritized
candidates.
Manual token editor
![Page 11: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/11.jpg)
Management InterfaceUser Management• Add/remove users.• Add users to annotation groups.• Display user activity log and statistics.
![Page 12: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/12.jpg)
Management InterfaceAnnotation Workflow Management:• Upload files in various formats.• Organize files into groups.• Assign files to individuals or to a group (for IAA).• Highlight tasks as untouched, edited, or completed.
![Page 13: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/13.jpg)
Management InterfaceEvaluation and Monitoring:• Evaluate IAA.• Compare annotations to gold reference.• Use WER and DER as metrics.• 10% of assigned documents are randomly assigned for IAA.
![Page 14: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/14.jpg)
Management Interface
User management view
![Page 15: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/15.jpg)
Management Interface
Task assignment popup
Tasklistview
![Page 16: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/16.jpg)
System Design and Architecture• Four main components:
oAnnotation interfaceoManagement interfaceoBack-end serveroMADAMIRA
Component interaction diagram
![Page 17: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/17.jpg)
System Design and ArchitectureData storage:• Relational database (SQL):
o Fast data search and retrieval.o Almost any SQL database can be used.
• Annotation data stored as JSON blobs:o Flexible data format.o Quickly add new functionality and annotation modes with little back-end
modification.
![Page 18: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/18.jpg)
EvaluationExperimental setup:• Around 1,500 words were extracted from Penn Arabic Treebank.• Five annotators were asked to fully diacritize the extracted words:
o First half of the text using a text editor.o Second half of the text with MANDIAC:
−Use automatically diacritized candidate if possible.−Manually edit otherwise.
![Page 19: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/19.jpg)
Evaluation
• Experimental results:oUsing a text editor: 302 words/houroUsing MANDIAC: 618 words/hour
• Using the text editor introduced typos.
![Page 20: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/20.jpg)
Acknowledgements• This project has been funded by the Qatar National Research
Fund (grant NPRP 6-1020-1-199).• We also thank the annotators for their feedback on MANDIAC.
![Page 21: P03- MANDIAC: A Web-based Annotation System For Manual Arabic Diacritization](https://reader033.fdocuments.net/reader033/viewer/2022042722/58a3af891a28ab9e6a8b6633/html5/thumbnails/21.jpg)
Thank You!