QUALITIES SPEECH SAMPLE ANALYSIS TEST … Meetings Seminars and... · Flight Safety Foundation,...
19
12/14/2010 1 QUALITIES OF GOOD TESTING PRACTICE Sergey MELNICHENKO, CompLang Aviation Training Centre (Moscow), Deputy Director International Civil Aviation English Association, Member of the Board Flight Safety Foundation, Member QUALITIES Qualities of Good Testing Practice SPEECH SAMPLE ANALYSIS TEST ADMINISTRATION ISSUES
Transcript of QUALITIES SPEECH SAMPLE ANALYSIS TEST … Meetings Seminars and... · Flight Safety Foundation,...
12/14/2010
1
QUALITIES OF GOOD TESTING PRACTICE
Sergey MELNICHENKO,
CompLang Aviation Training Centre (Moscow),
Deputy Director p y
International Civil Aviation English Association, Member of the Board
Flight Safety Foundation,
Member
QUALITIES
Qualities of Good Testing Practice
SPEECH SAMPLE ANALYSIS
TEST ADMINISTRATION ISSUES
12/14/2010
2
QUALITIES
Qualities of Good Testing Practice
RELIABILITYCONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITY
RELIABILITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITYPRACTICALITY
12/14/2010
3
Qualities of Good Testing Practice
Same results every time we administer the test
Objective rating
Qualities of Good Testing Practice
‐ inter‐rater reliability
‐ intra‐rater reliabilityy
12/14/2010
4
RELIABILITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITY
Qualities of Good Testing Practice
Meaningfulness and appropriateness
Are we measuring what we h d i t d d t ?had intended to measure?
12/14/2010
5
RELIABILITYCONSTRUCT VALIDITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITY
AUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITY
Qualities of Good Testing Practice
Rich in topic, theme or context
Degree to which test task characteristics match those of th TLU t kthe TLU tasks
12/14/2010
6
Qualities of Good Testing Practice
RELIABILITYCONSTRUCT VALIDITYAUTHENTICITY
INTERACTIVENESSIMPACTIMPACTPRACTICALITY
Qualities of Good Testing Practice
Are there test tasks to evaluate:
‐Comprehension of various accents?‐Ability to paraphrase?‐Language behavior in linguistic complications?Ad d i t d t f‐Adequacy during unexpected turn of events?‐Familiarity with cultural subtleties? ‐etc…
12/14/2010
7
RELIABILITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESS
IMPACTIMPACTPRACTICALITY
Qualities of Good Testing Practice
Aviation English test results are felt around the globe
12/14/2010
8
Qualities of Good Testing Practice
RELIABILITYCONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACT
PRACTICALITY
Qualities of Good Testing Practice
A
DesigningDevelopingAdministeringRating
VS Available resources
12/14/2010
9
RELIABILITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITY
SPEECH SAMPLE ANALYSIS
Qualities of Good Testing Practice
SpeakingListening ComprehensionReadingWriting (Typing)g ( yp g)
12/14/2010
10
SPEECH SAMPLE ANALYSIS
Qualities of Good Testing Practice
PronunciationStructuresVocabularyFluencyComprehensionInteractions
‐Accents
Qualities of Good Testing Practice
PRONUNCIATION
12/14/2010
11
Qualities of Good Testing Practice
‐Pronunciation errors• Mispronunciation of meaning‐di i i hi d (b b )distinguishing sounds (burn – born)
•No difference between short and long vowels (feet – fit, lead – lid)•Devocalization of sonorous•Devocalization of sonorous consonants in the end of words (surge – search, bag – back)
Qualities of Good Testing Practice
‐Pronunciation errors•Wrong stress in “international”
d ( d h l)words (academy, hotel)• Intonation typical for the native language• Mispronunciation of interdentalconsonants (think sink)consonants (think – sink)
12/14/2010
12
STRUCTURES
Qualities of Good Testing Practice
• Basic and complex structures• Errors do not affect meaning• Errors affect meaning• Grammatically accurate structures
d i it ti ff tused in a wrong situation affect meaning
VOCABULARY
Qualities of Good Testing Practice
•General English lexis• Semi‐technical lexis • Terminologies• Ability to paraphrase• Switching to native language
12/14/2010
13
Qualities of Good Testing Practice
FLUENCY •Smoothness of speechp• Rate of speech• Absence of excessive pausing• Absence of disturbing hesitation markers
Qualities of Good Testing Practice
FLUENCY •Length of utterancesg• Speech connectedness
12/14/2010
14
Qualities of Good Testing Practice
COMPREHENSION•Phonological form of wordsg• Morphological form of words• Syntactical and semantic connections
Qualities of Good Testing Practice
COMPREHENSION•Ability to recognize y gcommunicative function of utterances• Ability to deduce meaning of unfamiliar lexical items from context
12/14/2010
15
Qualities of Good Testing Practice
INTERACTIONS•Immediate?• Informative?•Appropriate?
Qualities of Good Testing Practice
QUANTITATIVE MEASURING• consistentlyy
• almost never• rarely• sometimes• frequently• usually• not always• often• occasionally• mostly• generally
12/14/2010
16
Qualities of Good Testing Practice
Adequate test+
Collection of evidence for both negative andCollection of evidence for both negative and positive examples of language use
+Quantitative measuring
+Correlation of resultsCorrelation of results with the ICAO Scale
=Adequate evaluation
of language proficiency levels
TEST ADMINISTRATION
Qualities of Good Testing Practice
ISSUES
12/14/2010
17
Qualities of Good Testing Practice
≠
‐Division between testing and training services
T R A I N I N G
Qualities of Good Testing Practice
Teachers shall not be interlocutors or‐Teachers shall not be interlocutors or raters for the group they have taught
12/14/2010
18
Qualities of Good Testing Practice
‐Test service providers shall be responsible for their domestic andresponsible for their domestic and
foreign affiliates
Qualities of Good Testing Practice
Level 4
Publishing a “black list” of TSP‐ Publishing a black list of TSP
12/14/2010
19
Bibliography1. ICAO Doc 9835 –Manual on the Implementation of ICAO Language
Proficiency Requirements, Second Edition – 2010
2. ICAO Circ 318 – Language Testing Criteria for Global Harmonization
3. L.F. Bachman, A.S. Palmer – Language Testing in Practice, Oxford University Press, 1996
4. J. Charles Alderson, Caroline Clapham, Dianne Wall – Language Test Construction and Evaluation, Cambridge University Press, 1995
5. Dan Douglas – Assessing Languages for Specific Purposes, Cambridge University Press, 2000
6. James E. Purpura – Assessing Grammar, Cambridge University Press, 2004
7. John Reed – Assessing Vocabulary, Cambridge University Press, 2000
8. Sari Luoma – Assessing Speaking, Cambridge University Press, 2004
9. Gary Buck – Assessing Listening, Cambridge University Press, 2001
10. Lyle F. Bachman – Statistical Analysis for Language Assessment, Cambridge University Press, 2004
11. Test of English Language Level for Controllers and Pilots (TELLCAP®) ‐Statute and testing materials, 2005‐2010
CompLang Aviation Training Centre
Molodogvardeyskaya Str., 61MoscowMoscow,
121351, [email protected]
Tel (+7495) 589 1356(+7495) 991 3703
Thank You
Fax (+7495) 589 1356www.complang.ru
