Measuring important learning outcomes in the context of two linked UK (mathematics education) projects: From

instrument development, to measure validation and statistical modeling

TLRP: “Keeping open the door to mathematically demanding F&HE programmes” (2006 – 2008)

TransMaths: “Mathematics learning, identity and educational practice: the transition into Higher Education” (2008-2010)

Maria Pampaka (The University of Manchester) Trondheim, February 2011

Outline of Presentation

•Some background to the projects•Instrument Development•Measure construction and validation

A measure of pedagogyA measure of Mathematics Self Efficacy

•A modeling approach to respond to the research questions

• Funded by ESRC (Economic and Social Research Council)• The University of Manchester, School of Education• The team:

Lead Principal Investigator: Prof Julian WilliamsOther PIs: – Laura Black– Pauline Davis – Graeme Hutcheson – Brigit Pepin – Geoff Wake

Researchers: Paul Hernandez-MartinezMaria Pampaka

TLRP: “Keeping open the door to mathematically demanding F&HE programmes” (2006 – 2008)

TransMaths: “Mathematics learning, identity and educational practice: the transition into Higher Education” (2008-2010)

Educational System in UK (England)

Aims of the Projects

TLRP: To understand how cultures of learning and teaching can support learners in ways that help them widen and extend participation in mathematically demanding courses in Further and Higher Education (F&HE)• AS Mathematics Vs AS Use of Mathematics

TransMaths: To understand how 6th Form and Further Education (pre-university) students can acquire a mathematical disposition and identity that supports their engagement with mathematics in 6fFE and in Higher Education (HE)• Focus on Mathematically demanding courses in HE (‘control’ : non mathematically demanding, e.g. Medicine and Education)

•Mixed methodology: longitudinal case studies, interviews and surveys

March 06

Sept 06

Programme effectiveness

Classroom practices

Learner identities

Questionnaire design Pilot case studies

June 07

Sept 07

Dec 07

(i) initial questionnaire

(ii) post test

(iii) delayed post test

Case studies in UoM and traditional AS

Follow up case studies

(i) initial interviews

(ii) interviews round 2(iii) follow-up interviews

DP1Sept-Nov 2006

DP2Apr-June 2007

Teachers’Survey

TLRP Research Design: The Survey in the general framework

DP3Sept-Dec 2007

RQ1: How do different mathematics educational practices found in pre-university/university transition interact with social, cultural and historical factors to influence students’ (a) learning outcomes, and (c) decisions in relation to learning and using mathematics?

RQ2: How are these practices mediated by different educational systems (their pedagogies, policies, technologies, assessment frameworks, institutional conditions and initiatives)?

Survey

Case Studies

April 2008

Instruments

Interviews

Start of 2nd

year HE

Sept-Oct 2009

Early 1st year

HE

End of A2

The Surveys within the TransMaths Project and the Relevant Research Questions

Analytical Framework

Instrument Development

Measures’ Construction and Validation(Rasch Model)

Model Building(Multiple Regression, GLM)

• Section A: Background and class information

The teacher’s Survey [TLRP]

The TLRP Instrument (Student Questionnaire)

• Section A: Background Information

• Section B: [Disposition to enter HE and study mathematically demanding subjects]

• Section C: Mathematics Self-Efficacy Instrument

Section A: Background Information

• Name, college, date, date of birth• Address and telephone number (for follow up

survey/interview)• Gender• Course (UoM or AS Maths)• Previous math qualification (GCSE grade and tier)• University attended by close family• Language of first choice• Education Maintenance Allowance (EMA)

Section B: Dispositions…

Disposition to go to HE Intention to go to University?• Expectations: family, friends, teachers

Disposition to continue with mathematically demanding courses in HE

• Intention to study more maths after this course?• Amount of mathematics in preferred option• Importance of amount of mathematics of course in decision• Feelings about future study involving maths• Preferred type of maths (familiar, new)

University: what course?

Using Mathematics: Self Efficacy

A “pure” item:You are asked to rate how confident you are that you will be able to

solve each problem, without actually doing the problem, using a scale from 1(=not confident at all) to 4(= very confident)

Using Mathematics: Self Efficacy

An “applied” item:

The TransMaths Student Questionnaire

• Section A: Background Information– University, Course/Programme – Previous math qualifications– Ethnicity, gender, country of origin, language– Proxies of socio economic background– Special educational needs

• A series of instruments about different aspects of the transition to HE…

Items / Measures DP4 DP5 DP6Reasons for choosing University and course Experiences that influence choice of Uni Programme Disposition to complete chosen course Preparedness and Usefulness of ways of studying Transitional Experiences Mathematics Dispositions Perceived Pedagogic Practices at Pre-uni (maths) experience Perceived Pedagogic Practices at Uni (maths) Mathematics Self Efficacy Confidence with Mathematics Usefulness of Mathematics Perceived Mathematical support at Uni Relevance of mathematics

The TransMaths ‘instruments’

Analytical Framework

Instrument Development

Measures’ Construction and Validation(Rasch Model)

Model Building(Multiple Regression, GLM)

Constructing the measures: Measurement methodology

• ‘Theoretically’: Rasch Analysis– Partial Credit Model– Rating Scale Model

• ‘In practice’ – the tools:– FACETS and Quest Software [Winsteps more user

friendly]

• Interpreting Results:– Fit Statistics (to ensure unidimensional measures)– Differential Item Functioning for ‘subject’ groups– Person-Item maps for hierarchy

Example 1:Measuring Mathematics Self Efficacy…some background

• Self-efficacy (SE) beliefs “involve peoples’ capabilities to organise and execute courses of action required to produce given attainments” and perceived self-efficacy “is a judgment of one’s ability to organise and execute given types of performances…” (Bandura 1997, p. 3)

• "a situational or problem-specific assessment of an individual's confidence in her or his ability to successfully perform or accomplish a particular maths task or problem" (Hackett & Betz, 1989, p. 262)

Background – Why Mathematics Self Efficacy?

• ‘Important in students’ decision making (sometimes more than actual test scores)

• Positive influence on students’ academic choices, effort and persistence, and choices in careers related to maths and science.

• How to measure?

Contextualised questions • TLRP project: a 30 item instrument for pre-university

students

Instrument measuring students’ confidence in different mathematical areas ( 10 items):• Calculating/estimating• Using ration and proportion• Manipulating algebraic expressions• Proofs/proving• Problem solving• Modelling real situations• Using basic calculus (differentiation/integration)• Using complex calculus (differential equations / multiple

integrals)• Using statistics• Using complex numbers

Example 1: Measuring Mathematics Self Efficacy [at the transition to university]

An example Item …An example Item …Measuring Mathematics Self EfficacyAn example “applied” item

1. Problem solving: The table below shows how the circumference of a glass varies with height. Find, using mathematics, the height to which you would need to fill the glass so that you have half a glass of

beer.

Not confident at all Not very confident Fairly confident Very confident

Methods and Sample

•10 items• 4 point Likert Scale (for frequency)• Sample:1630 students

•Rasch Rating Scale Model

Table 2: Measures and fit statistics for the items of the scale -------------------------------------------------------------------------------------------------- | Obsvd Obsvd Obsvd Fair-M| Model | Infit Outfit | | | | Score Count Average Avrage|Measure S.E. |MnSq ZStd MnSq ZStd | PtBis | Nu Items | -------------------------------------------------------------------------------------------------- | 3182 1328 2.4 2.41| 1.28 .04 | 1.0 0 1.0 0 | .53 | 1 modeling | | 4225 1331 3.2 3.25| -.56 .05 | 0.9 -3 0.9 -3 | .56 | 2 calculating | | 4055 1323 3.1 3.13| -.26 .04 | 1.0 -1 1.0 -1 | .54 | 3 ratio_proportion | | 4568 1326 3.4 3.54| -1.39 .05 | 1.0 0 0.9 -2 | .69 | 4 algebra | | 3770 1320 2.9 2.91| .25 .04 | 1.0 0 1.0 0 | .65 | 5 proof | | 3691 1318 2.8 2.85| .38 .04 | 0.9 -2 0.9 -2 | .55 | 6 problem_solving | | 4239 1309 3.2 3.31| -.74 .05 | 1.0 1 0.9 -1 | .71 | 7 Basic_calculus | | 3633 1313 2.8 2.82| .46 .04 | 0.9 -2 0.9 -2 | .65 | 8 complex_calc | | 3683 1309 2.8 2.86| .36 .04 | 1.3 7 1.4 8 | .39 | 9 statistics | | 3764 1308 2.9 2.93| .20 .04 | 1.2 4 1.1 3 | .63 | 10 complex_num | -------------------------------------------------------------------------------------------------- | 3881.0 1318.5 2.9 3.00| .00 .04 | 1.0 0.3 1.0 -0.1| .59 | Mean (Count: 10) | | 374.2 8.0 0.3 0.30| .71 .00 | 0.1 3.2 0.1 3.3| .09 | S.D. | -------------------------------------------------------------------------------------------------- RMSE (Model) .04 Adj S.D. .71 Separation 16.15 Reliability 1.00 Fixed (all same) chi-square: 2476.3 d.f.: 9 significance: .00 Random (normal) chi-square: 9.0 d.f.: 8 significance: .34 --------------------------------------------------------------------------------------------------

Item Fit Statistics t check for the assumption of unidimensionality

Results [1] – Checking Validity One measure?

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

-2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 Students of Math-Demanding Courses

Non

-Mat

hs S

tude

nts

SE1: Modelling

SE9: Statistics

SE2: Caculating/Estimating

SE7: Calculus

SE4: Algebra

SE10:Complex Numbers

SE6: Problem Solving

SE3: Ratio/Proportion

Items more relevant to AS/A2 Maths contextMore difficult for non maths students

Differential Item Functioning

Results [2] – Checking validity Differences among student groups

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

-2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 Students of Math-Demanding Courses

Non

-Mat

hs S

tude

nts

Calculus

Proof

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

-1.5 -1.0 -0.5 0.0 0.5 1.0 1.5

Students of Math-Demanding Courses

Non

-Mat

hs S

tude

nts

SE1: Modelling

SE9: StatisticsSE3: Ratio/Proportion

2a. AS-related MSE@Uni 2b. Applied MSE@Uni

Multidimensional Scaling?

Results [3] Two separate measures

Constructing the measures Example 2: The Teacher Survey (TLRP)

• ‘28 item survey to teachers 5 point Likert Scale (for frequency) Sample:110 cases from current project Rasch Rating Scale Model

B6: I encourage students to work more slowly

B24: I cover only the important ideas in a topic

Constructing the measures – Validity [Unidimensionality - Fit]

Constructing the measures:A measure of ‘pedagogical style’

I tend to follow the textbook I tend to follow the textbook closelycloselyStudents (don’t) discuss Students (don’t) discuss their ideastheir ideas

I encourage students to work I encourage students to work more quickly more quickly

I know exactly what maths the I know exactly what maths the lesson will containlesson will containStudents (don’t) invent their own Students (don’t) invent their own methods methods

I tell students which questions to I tell students which questions to tackle tackle

I teach each topic I teach each topic separately separately

Constructing the measures:A measure of ‘pedagogical style’

““It’s old fashion It’s old fashion methods, there’s a methods, there’s a

bit bit of input from meof input from me at the front and at the front and then I try to get then I try to get them working, them working,

practicing questions practicing questions as quickly as as quickly as possible…”possible…”

“… there’s a sense that I’ve achieved the

purpose…I’ve found out what they’ve come with and what they haven’t come with so…we can work with that now”

“…. from the teachers that I’ve met and talked to… it seems to me that

one of the big differences is, I mean I don’t sort of

use textbooks… [ ]…I want to get students to think about the math, I

want students to understand, I want

students to connect ideas together, to see all those things that go together and I don’t think a text

book did that…[ ].

Validation supported by qualitative data

“…“…I do tend to I do tend to teach to the teach to the syllabussyllabus now…If it’s not on I now…If it’s not on I

don’t teach it. … but I do don’t teach it. … but I do tend to say this is going to be tend to say this is going to be

on the exam…”on the exam…”

Transitional Experiences

The most challenging measure…

Analytical Framework

Instrument Development

Measures’ Construction and Validation(Rasch Model)

Model Building(Multiple Regression, GLM)

From measures to GLM Modeling - TLRP

Variables• Outcome of AS Maths (Grade, or Dropout)• Background Variables• Disposition Measures at each DP

– Disposition to go into HE (HEdisp)– Disposition to study mathematically demanding subjects in HE

(MHEdisp)– Maths Self Efficacy (MSE: overall, pure, applied)

• A score of ‘pedagogy’ based on teacher’s survey

=Outcome

Measure [A]DP(n)

Outcome Measure [A]

DP(n-1)++ +

RelatedOutcome Measures

[B,C,..]

Process variables [course,

pedagogy,..]

Background variables

Longitudinal design–DP1: 1792–DP2: 1082–DP3: 608

Resolution for some outcome variables (e.g. AS outcome)–Phone survey, School’s databases, Other databases

The TLRP Sample

Programme Gender AS UoM AS Trad Total

Male Female

351(68.8%) 159 (31.2%)

510

781 (59.9 %) 523 (40.1%)

1304

1132 (62.4%) 682 (37.6%)

1814

GCSE tier-grade Programme A* and A BHigher BIntermediate CHigher CIntermediate

AS Trad 9% (16%) 18% (50%) 31% (61%) 26% (65%) 46% (80%) UoM 12% (13%) 18% (26%) 9% (24%) 27% (54%) 18% (45%)

Results [1]: Math Dropouts

• Percentages of dropouts by course and previous attainment

•Effect Plots for a logistic regression model of dropout

VARIABLES COLLECTED

Learning Outcomes

A

DP 4

DispositionsB

Intentions / Decisions

C

BackgroundE

AS etc Grades

LearningPreparedB4LearningImportB4PercistenceFinish

MathDisp-4MSE-4

MathConfidencenceMathImportance

UniIntSTEMint

GenderUK/International

Process / ConditionsD

ChoiceReasonsChoiceInfluence

PerceptionPed@Colege

DP 5

LearningPreparedUNI

LearningImportUNI

PercistenceFinishMathDisp-5

MathConfidencenceMathImportanceMathRelevance

UniIntSTEMint

TransitionExperiencePerceptionPed@UNI

MathSupport

LanguageEthnicity

Family-in-HE

January Exams @ some places

DP 6 PercistenceFinish

MathDisp-6MathConfidencence

MathRelevance

UniIntSTEMint MathSupport SEN

End of Year 1 Exams

End year 1 outcome

Our modeling framework (TransMaths)

We hypothesized that:

Outcome of Year 1 (at University)=Entry Qualification + Dispositions + Transitional experiences +

Background Variables

An example Model (TransMaths)

Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 60.5085 2.4853 24.347 < 2e-16 *** CombTransFeel 1.0621 0.5850 1.815 0.0708 . maths[T.Yes] -3.7296 1.9013 -1.962 0.0510 . LPN -1.0616 0.5650 -1.879 0.0615 . MathCompCat 1.9893 0.3728 5.337 2.28e-07 *** MSE_DP4 -1.2712 0.6849 -1.856 0.0647 . --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 11.72 on 228 degrees of freedom (1544 observations deleted due to missingness) Multiple R-squared: 0.1233, Adjusted R-squared: 0.1041 F-statistic: 6.414 on 5 and 228 DF, p-value: 1.345e-05

The resulting Linear Regression Model

Positive effect: Positive transitional experience, previous maths qualification

Negative effect: Course, Low Participation Neighbourhood, Mathematics Self Efficacy

CombTransFeel effect plot

CombTransFeel

Yea

r1R

esul

t

60

65

70

75

-2 0 2 4 6

maths effect plot

maths

Yea

r1R

esul

t

60

62

64

66

68

No Yes

LPN effect plot

LPN

Yea

r1R

esul

t

60

62

64

66

68

1 2 3 4 5

MathCompCat effect plot

MathCompCat

Yea

r1R

esul

t

55

60

65

70

75

0 2 4 6 8

MSE_DP4 effect plot

MSE_DP4

Yea

r1R

esul

t

55

60

65

70

75

-4 -2 0 2 4 6

This is the quantitative aspect of mixed method project which also includes case studies and interviews

The multi-step methodology described helps us to create and validate our measures and then…

Use them to model (GLM- regression modeling) in order to respond to the research questions we originally set…

For instanceWhat influences students successful transitions between various

stages of education (e.g. to University)?How can we predict students’ progress at university) ?

Just a short summary

Thank you

Q & A