Language Issues

Constructs, Theories, and Scales

Construct = The trait or traits that a test intends to measure. A construct can be defined as an ability or a set of abilities that will be reflected in test performance. A construct is generally defined in terms of a theory; in the case of language, a theory of language. A test, then, represents an operationalization of the theory.

Davies, A. et. al. (1999). Dictionary of language testing. Cambridge University Presss.

Bachman & Palmer (p. 61)

“[O]ur approach to the design and development of language tests is based on the premise that if we want to use the scores from a language test to make decisions about individuals or inferences about their language ability, we must be able to demonstrate how performance on that language test is related to language use in specific settings other than the language test itself.”

Organizational knowledge – how utterances or sentences and texts are organized

Grammatical knowledge – how individual utterances or sentences are organized

Textual knowledge – how utterances are organized to form texts

Pragmatic knowledge – how utterances or sentences and texts are related to the communicative goals of the language user and to the features of the language use setting

Functional knowledge – how utterances or sentences and texts are related to the communicative goals of language users

Sociolinguistic knowledge – how utterances or sentences and texts are related to features of the language use setting

Areas of language knowledge

Goal setting – deciding what one is going to do

Assessment – taking stock of what is needed, what one has to work with, and how well one has done

Planning – deciding how to use what one has

Strategic Competence

Language “Skills”

“It is this conception of language use as the performance of specific situated language use tasks that provides, we believe, a much more useful means for characterizing what have traditionally been called language skills. We would thus not consider language skills to be part of language ability at all, but to be the contextualized realization of the ability to use language in the performance of specific language use tasks…Thus, rather than attempting to define ‘speaking’ as an abstract skill, we believe that it is more useful to identify a specific language use task that involves the activity of speaking, and describe it in terms of its task characteristics and the areas of language ability it engages.” (Bachman & Palmer, pp. 75—76)

Language Scales

Table 1: Sample of Bachman and Palmer decontextualized scale

Pragmatic competence

Rating Vocabulary Cohesion

0 Extremely limited vocabulary(A few words and formulaic phrases. Not possible to discuss any topic, due to limited vocabulary)

No cohesion(Utterances completely disjointed, or discourse too short to judge.)

1 Small vocabulary(Difficulty in talking with examinee because of vocabulary limitations.)

Very little cohesion(Relationships between utterances not adequately marked; frequent confusing relationships among ideas.)

2 Vocabulary of moderate size(Frequently misses or searches for words.)

Moderate cohesion(Relationships between utterances generally marked; sometimes confusing relationships among ideas.)

3 Large vocabulary(Seldom misses or searches for words.)

Good cohesion(Relationships between utterances well-marked.)

4 Extensive vocabulary(Rarely, if ever, misses or searchers for words. Almost always uses appropriate word.)

Excellent cohesion(Uses a variety of appropriate devices; hardly ever confusing relationships among ideas.)

Adapted from Bachman (1990).

Table 2: Interagency Language Roundtable Levels and selected contexts - SPEAKING___________________________________________________________________________________Speaking 0 No ProficiencySpeaking 0+ Memorized Proficiency Dealing with requestsSpeaking 1 Elementary Proficiency Getting facts over telephoneSpeaking 1+ Elementary Proficiency, Plus Making arrangementsSpeaking 2 Limited Working Proficiency Eliciting and informed opinionSpeaking 2+ Limited Working Proficiency, Plus Defending a point of viewSpeaking 3 General Professional Proficiency Clarifying, Answering objectionsSpeaking 3+ General Professional Proficiency, PlusSpeaking 4 Advanced Professional ProficiencySpeaking 4+ Advanced Professional Proficiency, PlusSpeaking 5 Functionally Native Proficiency

Table 3: The ILR to ACTFL concordanceILR ACTFL

Levels 0 –0+ Novice-LowNovice-MidNovice-High

Level 1 Intermediate-lowIntermediate-mid

Levels 1+ Intermediate-highLevel 2 AdvancedLevel 2+ Advanced PlusLevels 3-5 Superior

Table 6: Organization of Canadian Language Benchmark components

AN OVERVIEW

Benchmark Proficiency Level Speaking and Listening Competencies

Reading Competencies

Writing Competencies

STAGE I: BASIC PROFICIENCY

1 Initial 

Creating/interpreting oral discourse in routine non-demanding contexts of language use in:Social interactionInstructionsSuasion (getting things done)Information

Interpreting simple texts: Social interaction textsInstructionsBusiness/service textsInformation texts

Creating simple texts:•Social interaction•Recording information•Business/service messages•Presenting information

2 Developing 

3 Adequate 

4 Fluent 

STAGE II: INTERMEDIATE PROFICIENCY

5 Initial 

Creating/interpreting oral discourse in moderately demanding contexts of language use in: Social interactionInstructionsSuasion (getting things done)Information

Interpreting moderately complex texts Social interaction textsInstructionsBusiness/service textsInformation texts

Creating moderately complex texts:•Social interaction•Recording information•Business/service messages•Presenting information/ ideas 

6 Developing 

7 Adequate 

8 Fluent 

STAGE III: ADVANCED PROFICIENCY

9 Initial 

Creating/interpreting oral discourse in very demanding contexts of language use in: Social interactionInstructionsSuasion (getting things done)Information

Interpreting complex and very complex texts Social interaction textsInstructionsBusiness/service textsInformation texts

Creating complex and very complex texts:•Social interaction•Recording information•Business/service messages•Presenting information/ ideas 

10 Developing 

11 Adequate 

12 Fluent 

Figure 2: Sample tasks (Norris, et al., 1998)ITEM 1Situation: Your friend John has broken a bone in his hand. He cannot write (see photo of John). You told him that you would help him with writing. Now, he wants you to fill out a change of address form for him. Study the form provided. Be prepared to listen for the information requested on the form. John said he would leave the information on your answering machine.Task: Play the message from John. Listen for the information from the change of address form. Fill in the form for John. You may listen to the message as many times as you need to get the correct information.Time: You have 10 minutes to complete this task.Product: Completed change of address form. 

 Figure 4: Example Task-Dependent Rating Scale for Task F05

Item 1 inadequate   able   adept

  descrIptors

 

Examinee incorrectly fills out change of address form such that any essential elements (listed in the able descriptor) are not processable by the post office (this might include illegibility, incorrect placement of information, absence of information, etc.

Examinee performance contains some elements from the inadequate descriptor and some elements from the able descriptor.

Examinee fills out change of address form according to information given by John, minimally including with correct spelling and correct locations (see form for details)--name--new address--old address--starting date--signature and printed name(either John Harris or examinee’s own name).

Examinee performance contains some elements from the able descriptor and some elements from

the adept descriptor.

Examinee correctly fills out change of address form with ALL applicable information given by John on the answering machine message (see form for details).

Rating 1 2 3 4 5

 Figure 5: Example Task-Independent Rating Scale across all task performances.---------------------------------------------------------------------------------------------------------------------------------•Inadequate: A rating of insufficient indicates that the student seems generally incapable of coming to terms with the particular processing component (code, cognitive, communicative) on task like those found on the (test).•Student performance contains some elements from the inadequate descriptor and some elements from the able descriptor.•Able: A rating of able indicates that the student seems generally capable of coming to terms with the particular processing component on tasks like those found on the (test).•Student performance contains some elements from the able description and some elements from the adept descriptor.•Adept: A rating of adept indicates that the student seems quite capable of coming to terms with the particular processing component on tasks like those found on the (test); additionally, the student seems to have little or no difficulty in accomplishing such tasks in terms of the processing component.