Aplication of CAS in automatic assessment of

math skills

Przemysław Kajetanowicz (przemyslaw.kajetanowicz@pwr.wroc.pl)

Jędrzej Wierzejewski (jedrzej.wierzejewski@pwr.wroc.pl)

Wrocław University of Technology

Agenda

• Knowledge assesment in math instruction• Automatic assessment in algebra E-

course: history, functionality, implementation, results and students’ feedback

• How automatic tests work• Recent developments:

Java+Mathematica-based tests• Future

Assessment of progress in math

• Measuring progress in math = measuring mastery in problem solving.

• Grading of math problems– Grading procedure usually reflects typical

solution structure– Common though not always applicable: partial

credit for partial solution• Assessment tools in computer-aided

learning systems– LMS-dependent – Stand-alone

E-course in algebra - overview

• Content structure– Lecture notes– Interactive exercises (Java-driven) – over 120

problem types supported– Practice tests (Java-driven)– Graded exams (Java-driven)

• Functionality of assessment tools– Random generation of data– Controlling the difficulty level at design time– Flexible grading (partial credit)– Step-by-step solution presentation– Completeness check– Initial correctness check

Lecture notes (1)

Studying…

… and exploring math

Lecture notes (2)

Studying…

…and self-testing

Exercises

Exercise… ...and solution on demand

Graphing problems and related tools

Functionality of tests (1)

Functionality of tests (2)

Dedicated tools for Gauss elimination

Functionality of tests (3)

Correct solution immediately available

Completeness check

A student can go back to test as many times as he/she wishes.

Initial correctness check

Chances to make corrections (enthusiastically greeted by students) – teachers decides on # of chances

E-course - implementation

• Spring 2005 - 55 students• Fall 2005 – 400 students• Fall 2006 – almost 1000 students• Hybrid instruction

– Classroom meetings (3 or 4 hours weekly)– (Fall 2006) grading based on

• 5 online e-exams (play the role of homework)• Final in-lab e-exam (proctored)• In-class activity (not much credit though)

Administrative grading system

• Available credit– 5 online e-exams: 5 problems, 1 hour, 25 points available;

best four exams count– In-class activity – 4 points available– In-lab final exam: 8 problems, 90 minutes, 40 points

available• Grading procedure based on student’s systematic

work– A minimum of 20 on the final necessary (but not

sufficient) to pass– 20% of online e-exams total score added to final

(available maximum = 20)– In-class credit added (maximum = 4)– Necessary total to pass = 31

Grading system in practice

Grading examples

19 2440

31

20 6

1520

0

10

20

30

40

50

60

To

tal s

core

(m

axim

um

64)

In-class credit

Online exams

Final exam

In-class credit 0 0 0 4

Online exams 20 6 15 20

Final exam 19 24 40 31

Fail Fail Very good Very goodRequired minimum

on final

Required total to pass

Fall 2005 – Final grades

3%

17%

10%

19%21%

18%

11%

0%

5%

10%

15%

20%

25%

NOTTAKEN

2 (= FAIL) 3 3+ 4 4+ 5 (VERYGOOD)

Grade

Rel

ativ

e fr

equ

ency

of

stu

den

ts

E-course results vs. traditional course (data after make-up exams)

GradePercentageof students

Fail 48.72%

Satisfactory 6.41%

Satisfactory + 17.95%

Good 3.85%

Good + 8.97%

Very good 14.10%

E-course

GradePercentageof students

Fail 84.11%

Pass 15.89%

Traditional course

(Fall 2005 results)

Question 1To what extent did you find the new form of the course (in particular, automatic tests and exams) helpful in your mastering the course material (as confronted with traditional way of learning)?

29%

30%

9%6%

26%

1

2

3

4

5

6

Question 2How willing would you be to sign up for other math courses, were they offered in a similar form?

6%

9%

11%3%

71%

1

2

3

4

5

6

Question 6How fairly, in your opinion, was your knowledge assessed by the system of automatic exams?

28%

25%

9%

9%

26%3%

1

2

3

4

5

6

Note. The exam questions were of similar type to those given on traditional exams. Presently, no statistical comparison is possible between the new and the traditional form in terms of how students judge the fairness (no survey was given to „traditionally assessed students”. The majority of students (74%) judged the system as fair (grades 4 to 6).

Students’ comments

The course is a new appealing form of study to me. It is stress-free. (…) Giving tests and exams that way implies that the grading system is fair. The course is a great thing, and studying algebra that way is truly easy. Thank you !

Student’s comments (2)

At the beginning I was terrified at the perspective of taking a course that was delivered in that way, but now I would like to wholeheartedly thank the instructors for the opportunity of being a part of this “experiment”. I can definitely say that thanks to this course I understood things that had been all Greek to me before (...) I truly believe that more courses should be taught that way. I wish to thank again for the opportunity to have been in this course.

Inside a test (Java-driven only)

• Hard-coded– Problem data generation– Solution presentation– Initial grading– Completeness check, initial correctness check– Database-related operations

• Parameter-driven– Difficulty level, problem sub-type– Scores for individual problems– Number of problems– Time assigned for solution– Specific test behaviour (completeness warning, # of tries

to make corrections etc.)– Training vs. administrative purpose of test

Java+webMathematica (1)

• Java:– Formulates problem

(generates problem data)– Accepts student’s

entry(ies)– Sends data and solution to

webMathematica• webMathematica:

– Compares solution with correct result

– Sends outcome back to Java

– Generates and sends back other elements (e.g. graphs or expressions needed for solution presentation)

Java+webMathematica (solution presentation)

Java+webMathematica (2)

Java+webMathematica (2 - solution)

Java+webMathematica (3)

Java+webMathematica (3 - solution)

• Xml:– Holds problem formulation (parametrized, so many

problems of a common type are supported)– Holds solution (parametrized)– Holds problem data sets

• Java:– Reads xml and behaves correspondingly

• Mathematica:– Encodes formulation and solution in xml, provides

necessary graphics etc.– Generates and encodes data sets in xml

Java+xml+Mathematica – version 1

Space-consuming (especially if graphics involved)

• Xml:– Holds problem formulation (parametrized, so many

problems of a common type are supported)– Holds solution (parametrized)

• Java:– Reads xml– Generates data sets

• Mathematica:– Encodes formulation and solution in xml, provides

necessary graphics etc.– NO data sets generated by Mathematica

Java+xml+Mathematica – version 2

Quick and space-saving

Java+xml+Mathematica (formulation)

<zestaw podtyp = "Y" >

<dK>8</dK>

<dX>2</dX>

<dO>OY</dO>

<dP>8</dP>

<dA>-1/8</dA>

<dB>2</dB>

<o><![CDATA[%M y %N = &nbsp;]]></o>

<o><![CDATA[ %M x + %N ]]></o>

<p>-1/8</p>

<p>2</p>

</zestaw>

Data sets - xml (including correct result)

Formulation - xml

Java+xml+Mathematica (solution)

Solution (one of steps)

Java+xml+Mathematica (v. 2)

Data sets generation - Java

Formulation & solution - xml

Java+xml+Mathematica (v. 2) - solution

Solution (parametrized) - xml

Specific values - Java

Pros and cons of automatic assessment• Automatic assessment – main highlights

– Proves far more effective (see the following slides).– Gives a student more opportunity to demonstrate skills.

• E-course: 25 problems on online e-exams + 8 on the final• Traditional: 2 x 3 problems on mid-term „paper” exams + 5

on the final– Motivates a student to systematic work. – Every student gets different problems.– Saves time.

• Automatic assessment – downsides– Certain problem types (e.g. „prove that…”) cannot be

supported („as yet” - as dr. Wierzejewski says…).– Some students focus on solution algorithms rather than

on math concepts and methods.

Forthcoming future

• E-course in Calculus • Remedial course in secondary-school math• Continuation of teaching the e-course in algebra online• Completion of e-course in linear algebra

Thank you for your attention