Computer Aided Assessment(CAA) for mathematics

Chris Sangwin & Simon Hammond

Copyright c©Last Revision Date: June 1, 2009

2

Introduction.

... NOT multiple choice questions ...

• Computer aided assessment (CAA)

• CAA with computer algebra

• Practical issuesImplementations

• Pedagogical issues

• Future directions

3

JEM - Joining Educational Mathematics

eContentPlus Thematic Network

http://jem-thematic.net/

Founder members (15):

Universitat Politecnica de Catalunya, Helsingin Yliopisto, Tech-nical University, Jacobs University, Universiteit van Amster-dam, University of Birmingham, FernUniversitt Hagen, Mathsfor More, NAG Ltd, Liguori Editore, ISN Oldenburg GmbH,RWTH Aachen University, Univ. Nacional de Educacin a Dis-tancia, Universitat Oberta de Catalunya, Universidade de Lis-boa.

4

Use of objective tests

Consider the following question:

Example question 1Determine the following integral:∫

cos(x) sin(2x)dx.

As a multiple choice question:

◦ (2/3) cos3(x) + C◦ −(2/3) cos(x) + (2/3) sin3(x) + C◦ −(2/3) cos(x) + (1/3) sin(x) sin(2x) + C◦ Don’t know.

How do we know the students don’t differentiate thecandidate solutions to check?

5

Computer algebra marking

Computer algebra systems can be used to mark work.

This checks for algebraic equivalence.

(x + 1)2 ≡ x2 + 2x + 1

Useful for marking many routine problems.

6

Fundamental idea

if simplify(sa-ta) = 0 thenmark := 1 else mark := 0

7

STACK

System overview

The STACK system:

• internet based CAA system,

• uses very simpleMaxima (computer algebra), andLATEX (type setting)

• All components open source (e.g. GPL).

8

Demonstrating the STACK system

http://www.stack.bham.ac.uk/

9

In learning and teaching

We are assessing a student provided answer.

This is an objective test.

This is

• not Multiple Choice Question;

• not string/regex match.

Other tests for the form of an answer.

10

Input of mathematics

This is a fundamental but unsolved problem.

There are a number of options

1. Strict CAS syntax. eg. 2*(x-1)*(x+1)

2. “informal” linear text syntax. eg. 2(x-1)(x+1)

x(t-1) ?

3. Graphical input tool. eg. equation editor.

4. (Pen-based input ?)

5. (Geometry applet ?)

Not all groups of students are equal.

11

Syntax innovations

Difficult to achieve!

Babbage 1830’s

“a profusion of notations [...] which threaten, if not duly cor-rected, to multiply our difficulties instead of promoting our progress”Babbage, C. (1827)

sin2(x) sin−1(x)

sin sin x = sin2 x

(composition)

12

Structure in random problem sets

In practice, the numbers often do not matter.

Tuckey, C. O., Examples in Algebra, Bell & Sons, London, (1904)

Too much randomization destroys structure.

An underlying question space.

13

Workshop task

Option A:Context: end of first calculus course. (Age 18)

Write 6 questions which test whether a student can differentiateelementary functions.

E.g. Differentiate cos(3x) with respect to x.

Option B:Context: age 11.

Write 6 questions which test whether a student can add frac-tions.

14

Randomization

1. What could you randomize?

2. What would you randomize?

3. What are some likely incorrect answers?

4. What feedback would you like to provide?

... with a view to implementing these questions live.

15

Issues

• Well-posed questions.

• Fair questions.

• Structure in question sets.Schemes of work, vs isolated questions.

• Algebraic form of answers as a goal.

16

Feedback

One third of feedback interventions decreased performance.

Kluger, A. N. and DeNisi, A., Psychological Bulletin (1996).

The nature of feedback determines its effectiveness.

17

Processing answers

Test for algebraic equivalence if simplify(sa-ta) = 0 thenmark := 1 else mark := 0

Using mainstream CAS

• Get a lot very quickly,Great for calculus and beyond.

• Elementary algebra can be a problem.

Maxima seems to be more suitable than most.

18

Every CAS is different!

Input Maple Maxima Axiom

(numbers)0.5-1/2 0.0 0.0 0.04^(1/2)

√4 2 2

4^(-1/2) 14

√4 1

212

-4^(1/2)√−4 2i 2

√−1

sqrt(-4) 2i 2i 2√−1

(indices)a^n*b*a^m anbam an+mb baman

(a^(1/2))^2 a a a(a^2)^(1/2)

√a2 |a|

√a2

(collecting terms)1+x^2-2*x x2 − 2x + 1 x2 − 2x + 1 x2 − 2x + 1

x/3+1.5*x+1/3 1.833x + .333 · · · 1.833x + 13

1.833x + 0.333 · · ·3*x/4+x/12 5

6x 5x

656x

3/(4*x)+1/(12*x) 56

1x

56x

56x

19

Input Maple Maxima Axiom

(brackets)

-1*(x+3) −x− 3 −x− 3 −x− 3

2*(x+3) 2x + 6 2(x + 3) 2x + 6

(2*x-1)/5+(x+3)/2 910

x + 1310

2x−15

+ x+32

910

x + 1310

(x-1)^3/(x-1) (x− 1)2 (x− 1)2 x2 − 2x + 1

(x^2-2*x+1)/(x-1) x2−2x+1x−1

x2−2x+1x−1

x− 1

(9*x^2+3*x)/(3*x) 13

9x2+3xx

9x2+3x3x

3x + 1

(other)

log(x^2) ln(x2

)2 log(x) log

(x2

)log(x^y) ln (xy) y log(x) log (xy)

log(exp(x)) ln (ex) x x

exp(log(x)) x x x

cos(-x) cos(x) cos(x) cos(x)

20

Issue: technical problems

• Mixed data types in polynomialsx/3 + 0.5?

• Unary minus (no simplification).

− 11− x

,−1

1− x, or

1x− 1

.

• Display,1. Implicit multiplication, (xy, x · y, x× y)2. i vs j,3.√

x vs x12 .

21

Language

Do we have a way to talk about these fine details?

Unhelpful phrases:

• simplify,e.g.221

= 4 or 221000= · · ·?

• “move over”

22

Checking for properties

To mark

Example question 2Give an odd function.

1. calculate f(x) + f(−x),2. simplify,3. check equality to zero.

23

Creating examples/instances

Some questions ask for examples of objects.

They require higher level thinking.

Such questions are rare. (11.5 questions from 486 ≈ 2.4%)Pointon and Sangwin, 2003

Perhaps because they are time consuming to mark.

STACK may mark some questions of this style.

Exemplar questions

24

Students’ answers

Students show great variety in their answer, and method.

For example, 190 students were asked for two functions thatsatisfy f ′(1) = 0.

Their answers were marked automatically.

25

The students (N = 190) gave 93 ‘different’ answers.

1st Answer Freq

uenc

y

2nd Answer Freq

uenc

y

x2 − 2x 45 x3 − 3x 29x2

2 − x 31 x2 − 2x 10x3

3 − x 11 x3

3 − x 9x2 − 2x + 1 7 (x− 1)2 8x2 − 2x + 3 7 x4

4 − x 8(x− 1)2 5 x4 − 4x 52x2 − 4x 5 ex−1 − x 1x3

3 −x2

2 5 ex−1 + e−x+1 10 4 ln(x)− x 1

26

Two strategies emerged:

JL: Ok, just take the parabola and shift it one.· · ·B: I said, x− 1 = 0, then integrated it.

27

These problems can be used to generate (short) discussions.

• sorting the data,

• methods used,

• ‘exotic’ examples.

f1(x) = 0, f2(x) = |x|(x− 2), f3(x) = e−1

(x−1)2 .

28

Automatic feedback

Sophisticated automatic feedback may be provided bycomputer algebra systems.

This

• is immediate,

• is based on properties of students’ answers,

• could be positive and encouraging,

• may be based on common mistakes,

• may be based on common misconceptions.

29

Common misconceptions

Computer algebra can also test for a type of incorrect answer.

Misconceptions may be identified by

• educational research,

• previous teaching experience,

• examining answers from previous students

30

Odd functions

On examining the odd functions given by students,

the majority of coefficients ( 6= 1) are odd,eg

3x5, 5x7, 7x5 − 3x.

Students’ concept image of an odd function requires odd coeffi-cients.

Furthermore, f(x) = 0 is odd, but was absent.

31

Functions that are odd and even.

When asked for a function that was both odd and even

35% gave the correct answer (eventually),35% failed to answer the question.

Incorrect answers revealed that 24% of the students added anodd and even function.

Examples include

x + x2, x2 + x3, x5 − x6.

The computer algebra system can test for these misconceptions.

32

Student feedback

What do you like about the system? Did you have any difficul-ties? If so please describe them.

Feedback & partial credit

i like the way that you are given credit if your an-swer is partially correct and also given guidance onachieving the full mark for that question.

I like the fact that feedback is immediate, but I donot like the fact that if I get an answer wrong I donot know where in my working I have made the error

33

Random questions

The questions are of the same style and want thesame things but they are subtly different which meansyou can talk to a friend about a certain question butthey cannot do it for you. You have to work it allout for yourself which is good.

Syntax problems

I feel the aim system is reasonably fair, however ihave lost a lot of marks in quiz 3 for simple syntaxerrors

34

Give me an example...

Recognising the turning points of the functions pro-duced in question 2 was impressive, as there are alot of functions with stationary points at x=1 andit would be difficult to simply input all possibilitiesto be recognised as answers.

35

Authoring questions

In authoring, there is tension:

1. Ability to use all features of CAS.

2. Ease of writing questions.Not making question authors into programmers.

36

Conclusion

Some important questions

• For what purposes is this tool useful?

• What properties do we want?– Not “looks correct”.– Not “select the correct answer”.

• What feedback should we give?