Standardisation of In – Car Assessments

Draft final report to Forum

May 2007

JOHN HUNTER

Background

• Forum has talked about standardisation for many years

• There has been progress as a result of meetings and the Greenwich course, but no agreement on how best to achieve this goal

• TRL Report confirmed lack of uniform approach

AIMS

• Establish the range of methodologies used to record Details and Outcome of in – car assessments throughout Forum

• Establish the basis / rationale for the approach followed in each Centre

• Analyse the methodologies objectively

• Make recommendations

METHOD

• Questionnaire sent to all centres seeking 1 Recording Sheets 2 Their basis? 3 Factors recorded / scored on sheets 4 Relationship between records and decision on IN CAR outcome 5 Influence of other factors on final outcome of DRIVING Assessment 6 Any other observations / points?

Measuring and Recording

• Qualitative

• Semi-quantitative –Ordinal Scales - Adjacent scores either better or worse - Intervals between points variable

• Resistance to use – Why?

Why Measure?

• Aide memoire

• Evidence ‘Bad records, poor defence; No records, no defence’ (MDDUS)

• Audit / Research

What to Measure?

FACTORS should be• Observable • Recordable• RelevantSCALES should be• Sensitive• Specific• (Acceptable)TERMINOLOGY should be Accurate and Valid

RESULTS – 1Scales used in Recording sheets

Interval In-Car Clinic Overall after Return Outcome5-point 1 14-point 7 5 13-point 3 42-point 1 None / 10 7 11 not specified

RESULTS – 2Basis for sheets used

• Developed from DSA Test

• ‘Cognitive’ Assessment based on Mavis workshop 1999 and Level 3

• ADI Experience

• Progressive development of original Forum sheet (1980’s)

• Other

• Mixture of above

VALIDITY

• Does the scale measure what it is supposed to?

• Elements - Face - Construct (theoretical basis) - Content (convergent, discriminant, concurrent, predictive)

Content Validity

• Do the components cover all aspects of the factor being assessed?

• Legal basis for driving assessment is the European Driving Directive 2006/126/EC - Specified Skills - Higher Level Road Safety Considerations

• Provides criteria against which recording sheets can be evaluated

EDD 2006 / 126 / ECSpecified Skills

Clutch Road position –Junctions Gears and Roundabouts Accelerator Lane Discipline Brake Traffic Lights Steering Road Signs Signals Car Control Observation Distance from other cars Anticipation Priority (X-roads & Speed Junctions) Mirrors

‘Defensive & Social Driving Behaviour’

Evidence that Specified Skills have been Assessed

Factors Recorded (n) No. of Centres

20 2 19 3 18 1 12 – 16 5 7 – 11 4 None specified 2

2006 / 126 / ECHigher Driving Skills

• Recognise Traffic Dangers

• Have Command of Vehicles so as - NOT to create dangerous situations, and - To react appropriately if they occur

• Comply with Traffic Regulations

• Retain full use of Faculties needed for Safety

• Ensure Safety of All Road Users

RESULTS – 3Higher Level Driving Skills (A)

• Take Appropriate Action 10 • Multi-tasking / Divided Attention 10• Lane Selection / Discipline 9• Emergency Stop 9• Retention of Information / Instructions 7 • ‘Spatial Awareness’ / Clearance of

Stationary Vehicles 7• Route planning / Navigation using

Road Signs 6

RESULTS – 4Higher Level Driving Skills (B)

• Decision Making 5• Speed Regulation 5• Road Sign Recognition / Recall 5• Note of Assessors Safety Actions 5• Forward Planning 3• Merge with traffic from Right or

Dual Carriageway 3• Others (including ‘Figure of 8’ Manoeuvre)

Basis for Decisions - Results

• Patterns of Problems 11 (trends, consistency)

• Attribution to clinical / cognitive factors 7• Critical / Dangerous Incident 6• Overall Impression 6• Potential for Improvement 4• Mixture of Above 9• Scores Totalled 2

Influence of Other Factors on Final Decision of Driving Assessment

Med Prog Cog SAU Fam Almost Never 1 3 10 Seldom 1 1 5 3 Often 10 6 4 6 4 Almost Always 5 11 12 2

(Others? Yes 9, No 8)

Med – Medical Prog – Prognosis Cog – Cognitive

SAU – Static Assessment Unit Fam – Family Views

CONCLUSIONS

• Study has confirmed, and given an idea of the extent of, the lack of standardisation of the In - Car part of Driving Assessment

• Some concepts have been misapplied • Forum needs to address these shortcomings soon• Many of the deficiencies can be remedied quickly and

easily, but some will require changes in attitude and approach

• There are examples of good practice which can be built upon (and adapted if necessary) so they can be adopted and applied in all Centres

How Best To Go Forward?

• Consider having a NUMBER of Proformas for the different stages of the In – Car Ax

• A method of assessing and recording the Specified Skills should be easy to standardise and accept

• The elements of a systematic approach to assessing and recording Higher Level Skills are known but need to be agreed

• Outcome categories should be agreed

Recording In – Car Performance

ASSESS

• CONTROL SKILLS and

• DRIVING BEHAVIOUR

Using 3,4 or 5 Point scales

• The Specified Skills can be recorded directly, but

• The Higher Skills require agreement on terminology and method of evaluation

TRIP Methodology

• 4 – Point scale used to record performance, sometimes broken down into different elements, at specific points on the routes e.g. safe

probably safe

probably unsafe

unsafe

Comments on TRIP - 1

• Already used in 5 Centres for On – Route recording

• Each Centre has its favoured scale which it applies to everything being assessed BUT

• Is this appropriate for all parameters?• TRIP forces decisions to be made on borderline

performance BUT• Is this always the right thing to do?

Comments on TRIP - 2

• Some cases are genuinely in middle • Observer bias (hard or easy markers) (particularly important when patterns are being

used as basis for decisions)

• Unexpected occurrences are not scored, even if highly relevant or decisive

• TRIP was developed as a tool for research on groups of people, but in DA, we need to be get ‘correct’ decision in individual cases

Comments on TRIP - 3

• Can we consider adapting TRIP to take account of these concerns?

• RECOMMENDATIONS - Use 5 – point Scale - Use different terminology where necessary for different parameters - Have section on forms for recording unexpected incidents and assessor interventions

In Clinic Summary of TRIP Findings

• Profile of Parameters used on this form should be the same as on In – Car Form

• The Profile of scores should be transposed to this Summary Sheet PLUS a space for describing Critical Incidents

Content of On – Route MeasuresSuggestions for Higher Level Skills

Parameter Extremes of Scales

Vehicle Control Unsatis-------------Satis

Orientation in Traffic Unsure ------------Sure

Actions in Traffic Obstructs-------Courteous

Reactions in Traffic Delayed---------Normal

Accuracy of Reaction Poor------------Correct

Speed Behaviour (1) Too fast-------------Normal

Speed Behaviour (2) Too slow-----------Normal

Content of On – Route MeasuresSuggestions for Higher Level SkillsParameter Extremes of Scales

Distance (forward) Too close----------Satis

Distance (side) Too close----------Satis

Lane Discipline Unsatis-------------Satis

Merge with Traffic® Unsafe-------------Safe

Emergency Stop Delayed--------Normal

Risk behaviour Aggressive--Defensive

Content of On – Route MeasuresSuggestions for Higher Level SkillsParameter Extremes of Scales

Attention Wanders----------Normal

Divided Attention Distractable------Normal

Anticipation Limited-------Foresighted

Forward Planning Unsatis---------------Satis

New Learning Poor-------------Normal

Memory (tasks) Poor--------------Normal

Content of On – Route MeasuresSuggestions for Higher Level SkillsParameter Extremes of Scales

Roundabouts Unsafe---------------Safe

Decision making Indecisive------Decisive

Route Navigation Unable-----------Correct

Critical Events Dealt with---------Reacted

(specify) by Assessor Appropriately

Others?

Outcome

• The OUTCOME should be described as1.Satisfactory2.Minor problems, probably satisfactory3.Review4.Probably unsatisfactory5.Unsatisfactory

• Need for adaptations / tuition etc should be separate from Outcome

Final Thoughts

Summary of Recommendations for standardisation of content and

process of Recording Sheets