Tim McNamara

Plenary 2

Tim McNamaraSchool of Languages and Linguistics, University of Melbourne

What language assessment

theorists and practitioners have

learned about Occupation-

specific Language Assessments

Tim McNamara

School of Languages and Linguistics &

Language Testing Research Centre

University of Melbourne

Working with language across boundaries

What is the role of language in the competence

of health professionals?

Do language professionals and health

professionals (‘domain experts’) share an

understanding of this role?

How can language professionals come to

understand the thinking of domain experts?

How can language specialists understand the

the specific purpose domain?

The problem of reconciling the different

perspectives of language and non-

language professionals is a perennial

concern for Languages for Specific

Purposes practitioners.

Basturkmen & Elder (2004, p. 677)

Occupational English Test (OET)

Specific-purpose English language and

communication (ESP) test for migrant health

professionals in 12 professions whose training has

not been in English

4 skills test designed to assess the readiness of candidates

to manage the communicative tasks of the health workplace

– Speaking, writing – communicative tasks specific to each

profession

– Listening, reading – communicative tasks common to all health

professions

– Recognized for professional registration of migrant health

professionals in Australia, New Zealand and Singapore

The policy question…

Need to ‘get it right’ about who to admit, who to

exclude

Involves balancing

patient safety standards vs

not creating unreasonable restrictions on IMGs’

access to clinical experience

Evidence that the decisions are not always right –

aggrieved failing candidates

complaints from clinical supervisors

By what criteria should candidates be judged?

2 studies: on speaking criteria, on writing criteria

Australian Research Council Linkage grant (LP0991153)

Ethics approval for study given by University of Melbourne

Languages & Linguistics Human Ethics Advisory Group

7

Towards improved healthcare communication:

Development and validation of language

proficiency standards for non-native English

speaking health professionals

Catherine Elder, Robyn Woodward-Kron, Tim McNamara,

Elizabeth Manias, Geoff McColl, Gillian Webb

John Pill, Sally O’Hagan, Diana van Die

OET Speaking sub-test

Test task: role-plays between health professional and patient

– profession-specific

Test content: common situations encountered in the workplace

– accuracy and currency confirmed by health professional advisors

Assessment: linguistic criteria applied by language professionals to

audio recorded performances of interaction

Overall Communicative Effectiveness

Intelligibility

Fluency

Appropriateness of Language

Resources of Grammar and

Expression

SAMPLE MATERIALS - SPEAKING SUB-TEST

(PHYSIOTHERAPISTS): Candidate’s card

SETTING: Hospital clinic

PATIENT: An elderly person who is recovering from a stroke (CVA). The patient is making slow progress in learning to walk again.

TASK: Talk to the patient about the following pieces of equipment:

• a wheelchair

• a walking frame

• a walking stick.

– Explain the advantages and disadvantages of each one.

– You would like the patient to be as independent in his/her movements as possible. You feel the frame is not appropriate.

– You want the patient to have a stick. You do not want the patient to have a wheelchair at this stage.

SAMPLE MATERIALS - SPEAKING SUB-TEST

(PHYSIOTHERAPISTS): Role-player’s card

SETTING: Hospital clinic

PATIENT: You are an elderly person who is recovering from a stroke. You feel you are making painfully slow progress, and don't really expect to be able to walk again.

You feel you should be allowed to have a wheelchair.

TASK: Ask the physiotherapist when you will be given a

wheelchair.

Insist on your need for this equipment. Explain that you feel that

the painful exercises you are doing at the moment are pointless,

and that you are pessimistic about your chances of making real

progress.

Be difficult!

3 yr study of ‘indigenous’ assessment criteria for

judging communication in medical workplace

Investigation of feedback by clinical educators on

clinical spoken interaction between trainee health

professionals (HPs) and patients:

to investigate what qualities are valued by health

professionals in spoken clinical communication

and

to see if these should be and can be included in

a test of English for spoken clinical

communication

Towards improved healthcare communication

(Australian Research Council grant, 2009-2013)

Jacoby (1998): Indigenous assessment

Indigenous assessment – naturally

occurring, routine instances where

communication is judged

Indigenous assessment criteria – criteria

which emerge in these contexts

Investigating views of content experts

Jacoby 1998:

Studied feedback from expert physicist to junior

physicists rehearsing conference papers

Presenters both native and non-native speakers

of English S and NNS (some with limited

language proficiency)

No distinction between these groups in

feedback – occasional remarks on language

Organization, timing etc more important

Towards improved healthcare communication

(Australian Research Council grant, 2009-2013)

Phase 1

Identifying criteria indigenous to workplace context

Translating them into assessment criteria

Phase 2

Implementing new criteria

Investigating how they function

Phase 3

Setting cut-scores (e.g., pass/fail)

3 Professions: Medicine, Nursing and Physiotherapy

WHAT

HOW

HOW

MUCH

Phase 1

Phase 1

Identifying criteria indigenous to the workplace context

for judging the adequacy of clinical communication skills

Eliciting and analysing health professionals HP

judgements of actual clinical communication

What do (HPs) themselves value in communication with

patients?

Result: model of what HPs value (Pill 2013) – shows

strong differences from linguistic criteria

Translating this into assessment criteria to be used in

judging performances on the Occupational English Test

Model of what health professionals value in

patient communication (Pill 2012)

OET Speaking assessment criteria

Current

language-related

criteria

Proposed

interaction-related

criteria

IntelligibilityClinician

EngagementFluency

Appropriateness of

LanguageManagement of

InteractionResources of

Grammar and

Expression

Proposed new criteria

Clinician Engagement

This criterion refers to the ability to use language to demonstrate a professional

manner towards and awareness of the patient.

You should consider whether the candidate demonstrates:

• a suitably positive manner towards the patient (i.e., attentive, tolerant, respectful,

non-judgemental)

• awareness of the patient’s situation, narrative, opinions and expectations,

acknowledging the patient’s concerns, needs and emotions.

(4)• Professional manner generally strong and positive.

• Generally high level of awareness of the patient.

(3)• Professional manner sufficiently positive.

• Generally adequate level of awareness of the patient.

(etc.) …

18

Clinician Engagement

Management of Interaction

Phase 2: Can this model inform OET criteria?

What is at stake in the statistical analysisCan assessment of these two new

aspects be handled successfully by

language professionals?

How do the new criteria relate to the

existing criteria?

Are the new criteria measuring a new

aspect of the same construct or

introducing a different construct?

New criteria represent related but somewhat different

dimension of competence

The addition of two new criteria in a revised test would

represent more of what health professionals value in

clinical spoken communication with patients/relatives

Existing OET (language-focused) assessors can be

trained to apply these new criteria consistently (though

further training necessary)

Two options to report the speaking sub-test results:

• As two separate grades for the new and existing criteria;

• As a single grade for existing criteria supplemented by a

diagnostic report relating to the scores given on the new

criteria.

Summary of findings – Phases 1 & 2

Findings of Phase 3

Fewer test takers would pass using new criteria

But more of those would achieve an A (“strong”) grade

Cut-scores for nursing were a bit lower than for medicine

Moderate to good correlation between

– Health professionals’ (HPs’) holistic judgement

– OET assessors’ grading using six criteria

(2 interactional and 4 language-related)

The new criteria appear to capture what matters to HPs

Phase 2:

The two additional criteria allow claims that a revised

test would represent more of what health professionals

value

OET (language-focused) assessors can be trained to

apply these criteria consistently

Phase 3:

Implementation of new criteria would change current

outcomes for some candidates

Promises better decisions about who to admit, who not

to admit, to clinical practice

Summary and conclusion

Australian Research Council Linkage grant (LP130100171)

Ethics approval for study given by University of Melbourne

Languages & Linguistics Human Ethics Advisory Group

23

Toward improved quality of written patient

records: Language proficiency standards for

non-native English speaking health

professionals

Tim McNamara, Ute Knoch, Robyn Woodward-Kron, Catherine

Elder, Elizabeth Manias, Eleanor Flynn, Barbara Zhang

Sharon Yahalom, Simon Davidson, Annemiek Huisman

OET Writing sub-test

Test task: Letter of referral/discharge

Test content: Based on case notes

Assessment: Linguistic criteria applied by language professionals

Overall Task Fulfilment

Appropriateness of Language

Comprehension of Stimulus

Linguistic Features (Grammar & Cohesion)

Presentation Features (Spelling, Punctuation & Layout)

Overview of the project

Phase 1

• Interviews with stakeholders who contribute to or read medical records

Phase 2

• Establishing healthcare professional relevant criteria

Phase 3

• Applying the professionally relevant criteria to the OET writing task

Phase 4

• Setting minimum standards on the OET writing task

Phase 2

Conducted workshops in which doctors were asked to

comment on the strengths and weaknesses of a range of

handover documents

Participants: 61 health professionals (doctors, nurses,

health information managers)

18 experienced doctors from a range of sub-disciplines and

contexts

Average years of experience: 25 years

Average years as educators: 17 years

Stimulus material: 10 handover documents (referral letters

and discharge summaries) selected from 200 patient

medical records

Referral letter

Discharge summary

Sample medical record

Extraction of all referral letters and discharge summaries

Preselection of documents

Researcher meeting for final selection

Methodology

Data collection procedures

Workshops conducted in small groups or individually

Audio recorded

Data analysis

Audio recordings transcribed

Coded for salient themes and sub-themes

Methodology

Results

1. Appropriateness

Writer Text Reader

1. Appropriateness

Appropriateness of the writing to the text

type/ style of the document

Any clinician would be able to understand it but it

is not the sort of language you want in a

discharge summary (DW1, M2, D1,2,3)

It is almost like somebody has had a private

investigator following him but it doesn’t scream

of medical assessment

(DW1, M22, D4,5)

1. Appropriateness

Appropriateness of writing to audience

I mean give it a medical term because it is going to the

GP (DW1, M12, D1,2,3)

P: I also find it quite difficult sometimes when you are

referring to a consultant, I don’t like to be saying ‘Well

this is what I would like’ because sometimes

consultants don’t like being told what to do

P: Yeah, don’t tell me what to do, I will decide what to

do, so therefore often I will say, ‘I would be pleased

for your expert management’ or something like that.

(DW2, R13, D11,12)

1. Appropriateness

Appropriateness of the writing to the

sub-discipline of the writer

… when I went to psych that is what I was doing, I

was doing bullet points and they said, “No, you have

to write novels”. So then I started writing novels and

then coming back here, I am like I don’t have to write

novels anymore. (DW2, M25, D8,9,10)

2. Conciseness

• Conciseness:

So it is succinct which is also highly valued and as you get

more experienced you get more succinct.

(DW1, R66, D4,5)

2. Conciseness

This discharge summary falls into the trap of trying to be

succinct with bullet points but at the same time being very

verbose (DW2, M12, D8,9,10)

3. Sufficiency

Sufficiency:

You have got the information mostly that you need

here and so he/she has done that well (DW2, M20,

D11, 12)

P: Um, is there too much information sometimes?

P: No, I think as an initial letter because this is the first

time they have seen him in clinic, I think that is perfect.

P: and this would give me all the information I need

about his background (DW4, M25, D14)

4. Accuracy of content

P: when you get particular

things like antibiotics that

they would have prescribed

five years ago and it is still

listed on their medication

list

P: so I would find that a

little bit embarrassing to

send it. It reflects poorly on

you if you send off a letter

which is incorrect.

5. Clarity

Clarity of content

– Being explicit

– Making purpose of letter clear (in case of referrals)

– Making request or follow up plans clear (both referrals

and discharge summaries)

– Prioritizing information

It is not clear who is doing what or even what has

really happened. It is just a jumble of thoughts

really (DW1, R66, D1,2,3)

Towards the end I think when you really want to lay

it out, clarity is really crucial (DW1, M25, D1,2,3)

5. Clarity

Clarity of organisation & presentation

– Logical organisation of content

– Prioritization of information

– Use of paragraphing, dot points, spacing when

appropriate

– Legibility & general ‘tidiness’

The information is there and the information is good

but actually that still doesn’t do the job because of

how it is presented (DW2, R13, D8,9,10)

So I think it is very well structured in terms of you

could pick this up and get a good picture of the

patient and what they look like (DW4, M22, D14)

5. Clarity

Clarity of language

– Grammatical accuracy, spelling, sentence

structure were all mentioned

So the grammar and spelling and English is not 100%

but that is not an issue in terms of understanding what is

there (DW4, R36, D14)

You could ignore the sentence structure, provided what

was written down was comprehensive and useful (DW2,

R36, D8, 9, 10)

5. Clarity

Clarity of language

– Use of acronyms

– Use of technical language

P: I don’t like acronyms, I don’t know what BRAF is?

P: no

P: there is a reference to BRAF, which is obviously an abbreviation

but within the oncology field (DW2_M25_D11, D12)

P: But it doesn’t read very easily because there is a lot of acronyms,

there is a lot of shortcuts (DW2_R13_D8, 9, 10)

Discussion – what doctors value in handover

documentation

AppropriatenessWriter

Text

Reader


documentation

Conciseness

&

Sufficiency

Appropriateness

All key information

present

No irrelevant

information


documentation

ClarityConciseness

&

Sufficiency

Appropriateness

content

organisation

presentation

language


documentation

ClarityConciseness

&

Sufficiency

Appropriateness

Accu

racy o

f conte

nt

Discussion

Values of domain experts cannot always be directly

translated do not all transfer into the (more limited)

language testing domain

Many of the aspects mentioned by the domain experts are

represented in the current criteria

Some aspects not present

Grouping might have to be re-examined

Weighting will have to be re-examined

Next steps:

Examining how well descriptors are reflected in our empirical data

Develop checklist indicators

Apply checklist indicators to OET writing data

Strong and weak performance tests

Two types of test:

‘in the strong sense, tasks will represent real-world tasks, and

performance will primarily be judged on real-world criteria, that

is, the fulfilment of the task set. Aspects of language ability as

such may or may not be assessed at all’

‘in the weak sense, the focus is on language performance.

Task may resemble or simulate real-world tasks … however,

the capacity to perform the task is not actually the focus of

assessment. Rather, the purpose … is to elicit a language

sample so that second language proficiency … may be

assessed.’

McNamara, T. F. (1996). Measuring second language performance. Harlow: Longman. (pp. 43-44)

[The body] responsible for the OET is required by law to

assess the English language proficiency of overseas-trained

health professionals, but is not permitted to make

assessments of their professional competence (McNamara,

1996, p. 40)

(Impossible) separation of communicative skill and

professional competence

Assessment of communicative skill in hands of language

professionals (language teachers, language testing experts)

But are language assessors and clinicians oriented to the

same qualities in communication in interactions between

clinicians and patients?

Construct issue: Divided responsibility for

assessment

Plenary 2

Tim McNamaraSchool of Languages and Linguistics, University of Melbourne

Tim McNamara

Education

Transcript of Tim McNamara