Understanding and improving Level 3 Value Added Performance in the Further Education Sector

RCU, 3 Tustin Court, Port Way, Preston, PR2 2YQTel: 01772 734855 email: [email protected]

AoC Teaching, Learning and Assessment Conference

Understanding and improving Level 3 value added performance

19thth January 2016

Presented by Richard Boniface (RCU Ltd)

Martin Laverick (Harlow College)


Session overview

1. Introduction

2. Feedback on analysis of national level data

3. Value added at Harlow College – making it work in practice

4. Activity and discussion


Two new progress measures:

Value added measure for Academic and Applied General qualifications –based on L3VA

Completion and Attainment measure for Tech Levels and Technical Certificates (at Level 2)

Both measures will show the progress of students compared to others taking the same subject nationally

In addition:

the value added measure will compare students with the same starting points

The completion and attainment measure will treat non completion as a fail.

These measures will form the basis for new progress based minimum standards

New Progress Measures (2016)


Analysis of National Level 3 Value Added Data

Richard BonifaceRCU Ltd.


• Level 3 value added score for each qualification taken by a learner is:

Actual Level 3 performance MINUS Predicted Level 3 performance

• The predicted score is derived from a statistical model based on the performance of other learners who took the same qualification and had the same Key Stage 4 points score;

• The published data aggregates learner level data to give scores by qualification, qualification type and institution (academic and vocational);

• The baseline Key Stage 4 score was calculated at age 15 for 16,17 year old completers but age 16 for 18 year old completers.

Analysis of Level 3 Value Added Data


Academic Value Added Scores

Provider TypeNumber of

EntriesValue Added Score

Lower

Confidence

Limit

Upper

Confidence

Limit

Academy 358,046 0.03 0.03 0.03

Sixth form college 213,704 0.00 0.00 0.01

Independent school 124,777 0.12 0.12 0.12

Community school 91,557 -0.02 -0.02 -0.01

Voluntary aided school 59,208 0.01 0.00 0.01

General further education college 45,489 -0.13 -0.14 -0.13

Foundation school 43,728 -0.01 -0.02 -0.01

Sponsored academy 37,164 -0.09 -0.10 -0.09

Tertiary college 32,349 0.01 0.00 0.01

Voluntary controlled school 9,149 -0.01 -0.02 0.00

City technology college 1,316 -0.04 -0.06 -0.01

Free school - mainstream 772 -0.09 -0.11 -0.06

Agriculture and Horticulture College 701 -0.10 -0.13 -0.07

Free school - 16-19 628 0.30 0.26 0.34

Free school - UTC (university technology college) 202 -0.26 -0.33 -0.19

Art, design and performing arts college 183 -0.05 -0.12 0.02

Free school - studio school 102 -0.24 -0.31 -0.17


Academic Value Added Scores

0%

10%

20%

30%

40%

50%

60%

>-1

and

<=-0

.8

>-0.

8 an

d <=

-0.6

>-0.

6 an

d <=

-0.4

>-0.

4 an

d <=

-0.2

>-0.

2 an

d <=

0

>0 a

nd <

=0.2

>0.2

and

<=0

.4

>0.4

and

<=0

.6

>0.6

and

<=0

.8

>0.8

and

<=1

Perc

enta

ge o

f Pro

vide

rs in

Val

ue A

dded

Ban

d

Value Added Score

All

Academy

Sixth Form College

Community School

GFE/Tertiary


Vocational Value Added Scores

Provider TypeNumber of


Lower

Confidence

Limit

Upper

Confidence

Limit

General further education college 73,750 -0.14 -0.14 -0.14

Academy 24,571 0.16 0.15 0.16

Sixth form college 22,017 0.14 0.13 0.14

Tertiary college 16,685 0.04 0.03 0.05

Sponsored academy 14,480 0.08 0.07 0.09

Community school 11,606 0.14 0.13 0.15

Foundation school 6,129 0.14 0.12 0.15

Voluntary aided school 4,837 0.20 0.19 0.21

Agriculture and Horticulture College 3,207 0.03 0.01 0.04

Voluntary controlled school 694 0.11 0.08 0.14

Independent school 611 0.22 0.19 0.26

Art, design and performing arts college 561 0.37 0.33 0.40

General further education college (special) 536 0.03 -0.02 0.07

City technology college 532 0.38 0.35 0.42

Free school - UTC (university technology college) 168 0.31 0.25 0.38

Free school - studio school 104 0.27 0.19 0.36

Free school - mainstream 13 0.32 0.08 0.56


0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

>-1

.6 a

nd

<=

-1.4

>-1

.4 a

nd

<=

-1.2

>-1

.2 a

nd

<=

-1

>-1

an

d <

=-0

.8

>-0

.8 a

nd

<=

-0.6

>-0

.6 a

nd

<=

-0.4

>-0

.4 a

nd

<=

-0.2

>-0

.2 a

nd

<=

0

>0

an

d <

=0

.2

>0

.2 a

nd

<=0

.4

>0

.4 a

nd

<=0

.6

>0

.6 a

nd

<=0

.8

>0

.8 a

nd

<=1

>1

an

d <

=1

.2

>1

.2 a

nd

<=1

.5

Pe

rce

nta

ge o

f P

rovi

de

rs in

Val

ue

Ad

de

d B

and

Value Added Score

All

Academy

Sixth Form College

Community School

GFE/Tertiary

Vocational Value Added Scores


VA Scores by Qualification Type (GFE)GFE/Tertiary

Qualification Type

Number of


Lower

Confidence

Limit

Upper

Confidence

Limit

A Level 53,197 -0.06 -0.06 -0.05

EDEXCEL BTEC Level 3 Extended Diploma 47,842 -0.05 -0.06 -0.05

AS Level (Not Continued to A2) 21,698 -0.03 -0.03 -0.02

Edexcel BTEC Level 3 90-credit Diploma 15,107 -0.05 -0.06 -0.04

EDEXCEL BTEC Level 3 Subsidiary Diploma 10,726 -0.29 -0.30 -0.28

EDEXCEL BTEC Level 3 Diploma 6,764 -0.33 -0.34 -0.32

CACHE Level 3 Diploma in Child Care and Education 1,809 -0.06 -0.08 -0.03

BTEC National Diploma 1,700 -0.13 -0.15 -0.10

Extended Project (Diploma) 1,433 -0.01 -0.04 0.02

EDEXCEL BTEC Level 3 Certificate 1,046 -0.13 -0.15 -0.10

Level 3 Diploma (ABC or UAL) 615 0.12 0.07 0.16

Applied GCE Single Award 610 -0.08 -0.11 -0.04

CACHE Level 3 Extended Diploma 498 -0.00 -0.03 0.02

UAL Level 3 Extended Diploma 483 -0.15 -0.20 -0.09

L3 Foundation Diploma in Art and Design 413 -0.05 -0.11 -0.00

UAL Level 3 Diploma 315 0.00 -0.06 0.06

City & Guilds Level 3 Extended Diploma 291 0.04 0.01 0.08

IFS Level 3 Diploma in Financial Studies 258 -0.28 -0.33 -0.23

IFS Level 3 Certificate 236 -0.05 -0.10 0.01

City & Guilds Level 3 Diploma 196 0.08 0.01 0.14

CACHE Level 3 Certificate in Child Care and Education 185 0.04 -0.02 0.10

Pearson BTEC Level 3 Diploma in Beauty Therapy (QCF) 173 -0.05 -0.11 0.01

Applied GCE Double Award 172 -0.37 -0.43 -0.31

Applied GCE AS level 166 -0.04 -0.10 0.02

Level 3 Subsidiary Diploma (RSL or City & Guilds) 155 0.16 0.10 0.22

International Baccalaureate 139 0.00 -0.05 0.06

IFS Level 3 Certificate in Financial Studies 96 0.24 0.14 0.34

ABC Level 3 Foundation Diploma 74 0.05 -0.08 0.18

BTEC National Award 73 -1.31 -1.41 -1.20

OCR Level 3 Cambridge Technical Extended Diploma 70 -0.49 -0.60 -0.38

BTEC National Certificate 64 0.54 0.41 0.67

City & Guilds Level 3 Subsidiary Diploma 54 -0.02 -0.10 0.05

OCR Level 3 Cambridge Technical Introductory Diploma 30 -0.09 -0.25 0.08

Pre-U Principal Subject 16 -0.05 -0.21 0.11

Pearson BTEC Level 3 National Award 15 0.28 0.09 0.48

OCR Level 3 Cambridge Technical Diploma 13 0.19 -0.02 0.41

Free Standing Maths 8 0.24 -0.15 0.64

OCR National Certificate 7 -0.97 -1.27 -0.67


VA Scores by Qualification Type (SFC)

Qualification TypeNumber of


Lower

Confidence

Limit

Upper

Confidence

Limit

A Level 139,358 0.02 0.01 0.02

AS Level (Not Continued to A2) 58,852 -0.01 -0.01 -0.00

Extended Project (Diploma) 8,799 0.10 0.07 0.13

EDEXCEL BTEC Level 3 Subsidiary Diploma 7,583 0.17 0.16 0.17

EDEXCEL BTEC Level 3 Extended Diploma 7,148 0.15 0.14 0.15

Applied GCE Single Award 4,330 0.13 0.09 0.16

EDEXCEL BTEC Level 3 Diploma 3,639 0.20 0.19 0.20

Applied GCE AS level 1,205 0.11 0.05 0.17

EDEXCEL BTEC Level 3 Certificate 1,078 0.18 0.16 0.21

Applied GCE Double Award 882 0.09 0.04 0.15

Edexcel BTEC Level 3 90-credit Diploma 792 0.17 0.16 0.17

IFS Level 3 Diploma in Financial Studies 377 0.18 0.13 0.23

BTEC National Diploma 283 0.36 0.34 0.39

CACHE Level 3 Diploma in Child Care and Education 183 0.16 0.14 0.19

International Baccalaureate 153 0.01 -0.04 0.07

OCR Level 3 Cambridge Technical Introductory Diploma 149 0.03 -0.14 0.19

OCR National Diploma 146 -0.12 -0.12 -0.12

OCR Level 3 Cambridge Technical Diploma 104 -0.11 -0.32 0.10

City & Guilds Level 3 Diploma 43 -0.06 -0.12 0.01

IFS Level 3 Certificate 36 0.34 0.28 0.39

BTEC National Award 33 0.71 0.61 0.82

UAL Level 3 Extended Diploma 33 0.24 0.18 0.29

IFS Level 3 Certificate in Financial Studies 31 0.66 0.56 0.76

L3 Foundation Diploma in Art and Design 26 0.23 0.18 0.28

Pearson BTEC Level 3 National Award 25 0.20 0.01 0.40

Level 3 Diploma (ABC or UAL) 16 0.02 -0.02 0.07

Free Standing Maths 12 -1.27 -1.66 -0.87

CACHE Level 3 Extended Diploma 10 0.41 0.39 0.44

BTEC National Certificate 7 -0.72 -0.85 -0.59

City & Guilds Level 3 Subsidiary Diploma 7 0.17 0.10 0.24

ABC Level 3 Foundation Diploma 7 0.31 0.19 0.44


Academic Vocational

College Value Added Scores by Region

-0.30 -0.20 -0.10 0.00 0.10 0.20 0.30

North East

West Midlands

East Midlands

London

Yorkshire and The Humber

South East

East of England

North West

South West

Value Added Score

-0.30 -0.20 -0.10 0.00 0.10 0.20 0.30

London

Yorkshire and The Humber

East Midlands

West Midlands

East of England

North East

South West

South East

North West

Value Added Score


Value Added Scores by Deprivation

Academic Vocational

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0 10 20 30 40 50 60

Val

ue

Ad

ded

Sco

re

IMD 2010 16-18 Classbased L3 Deprivation Score

GFE / Tertiary College

Sixth form college

Land-based/Art andDesign Colleges

Least Deprived Most Deprived

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0 10 20 30 40 50 60

Val

ue

Ad

ded

Sco

re

IMD 2010 16-18 Classbased L3 Deprivation Score

GFE / Tertiary College

Sixth form college

Land-based/Art andDesign Colleges

Least Deprived Most Deprived


Value Added Scores by OFSTED Grades

Academic Vocational

-0.20 -0.10 0.00 0.10 0.20

Outstanding

Good

Satisfactory/Requires Improvement

Inadequate

Value Added Score

-0.20 -0.10 0.00 0.10 0.20

Outstanding

Good

Satisfactory/Requires Improvement

Inadequate

Value Added Score


• Schools were withdrawing students who they expected to do badly, improving their headline Value Added scores;

• Level 3 Value Added measure does not take into account the impact of deprivation;

• The lack of a common Key Stage 4 baseline year disadvantages GFEs with 18 year olds and does not show the true value that colleges add;

• Statistical model may not be accurate for learners with relatively low KS4 scores;

• Small sixth forms benefit from a ‘shrinkage factor;’

• Value Added scores included qualifications taken at other institutions.

Issues Raised from Analysis


• Common baseline introduced - only specific qualifications achieved during Key Stage 4 will be included;

• New methodology for predicting scores to be used which should be more accurate for students with low prior attainment (20 bands rather than multi-level modelling);

• Shrinkage factor removed;

• Introduction of a separate retention measure;

• Progress measure only applies to Academic and Applied General qualifications;

• Level 3 value added scores will be reported separately for each provider if a learner changes institution.

Changes for 2016


Harlow College

Martin LaverickDirector of Quality Standards and Assessment


• Values around

– A Levels +0.30

– Vocational +0.30

• A brief history

– Starting Point – Sport

Harlow College Experience


• Discuss and write down three things that are stopping your organisation from achieving high value added

– Caveat – these must be internal factors

• By 9 minutes each table should list these factors on separate post-it notes (min of 3)

• At 10 minutes stick them on the wall.

Activity


• Set a challenging target grade for the student based on KS4/GCSE results

– Predicted plus one?

– Target based on data from DfE, ALIS, ALPS, etc.

– Does the accuracy of the target matter?• Don’t get hung up on the correlation at individual student level.

– New Assessment Rules QCF and NQF Edexcel

• Negotiate a student target

Our Process


• Are your tutors/lecturers and managers held accountable?

– How?

• Do you have a central tracking system?

– Advice - have something simple that works

• In year progress– Which groups underperformed last year.

– Where are they are now?

– Can you predict the influence of other changes?

Monitor Student Progress


What evidence?


At student level?

Note: the headings for the

third and fourth columns

should be:

Personal Target Points

Harlow College Target Points


• Staff

– Teaching to the higher grade

– Not accepting the “just pass”

– Other reasons

• Students

– The basics• Attendance (>85% will cost you a grade)

– For every negative VA.

Changing Mindsets


• Daily targets that build on prior knowledge, are personalised/differentiated, and include support and intervention, as well as, stretch and challenge.

• Unit Days - students go home when s/he has met the differentiated SMART target(s).

One Approach


• Data/Data Capture

• Data Arrives too late

• Teachers don’t understand

• Not understanding what a distinction profile like

• Lack of Stretch and Challenge/personalised targets

• Vocational teachers understanding of calculated

• Value Added high achievers

• Too much focus on success – lack of aspirations

• Feedback on work doesn’t help learners improve

• Not collecting quals on entry

Summary and Questions

Understanding and improving Level 3 Value Added Performance in the Further Education Sector

Education

Transcript of Understanding and improving Level 3 Value Added Performance in the Further Education Sector