St. Anthony’s School

16 Jan 2013 1

Background information

2

Enhance the effectiveness of learning and teaching

3

Evaluation after Assessment

Why did students

choose A, not B, C or D?

What are the follow-up actions?

Are the questions

too difficult or too easy?

Have the students learned? Have they

studied hard?

4

5

Student MCIItem MCICronbach AlphaGraph shape

Data – Good to start

6

2 Modified Caution IndexesStudent Modified Caution Index

Item Modified Caution Index

Students’ Performance

Question items’

Performance

7

Student MCIxP Chart

2012-13 P.4 1st

Reading Exam

8

Student MCIxP Chart

2012-13 P.4 1st

Reading Exam

9

More focused strategiesType A Type B

The teaching and learning are suitable for them. Encourage them to keep on.

Know the difficult questions Miss the simple questionsCareless mistakesDon’t rely on luck

Type D Type C

Adjust the teaching contentRaise their learning motivation Need more consolidation

Answering techniquesCareless mistakesFocus on the foundationDon’t rely on luck

10

Item MCI xP Chart

2012-13 P.4 1st

Reading Exam

11

Item MCI xP Chart

Need improveme

nt

Difficult question

s

Problem question

s

12

MCI item P-value chartItem Type

1112 P.3 Final Exam

1213 P.4 First Exam

A 20 17B 1 2C 1 0D 3 7Total no. of questions

25 26According the Item Modified Caution Index, most of the items set are normal.

13

Cronbach Alpha (Reliability)

Cronbach Alpha

Reliability

0 – 0.3 low

0.7-0.9 acceptable

> 0.9 high

14

2011-2012 P.3 First Exam

Slightly Easy

15

2012-2013 P.4 First Exam

Appropriate

16

Which question

items should be improved?

17

Which question items are difficult to

our students?

18

SP Xpress analysis on ‘2011-2012 Primary 3 Third Exam Paper’SP Xpress analysis results:

Type A: 20 itemsType B: 1 itemType C: 1 itemType D: 3 items

Let’s examine one item from each category (Type B, C, D)

19

Exam item example: Type B

On which floor of the centre, can Grandpa find presents for Jack?A. 1/F B. 2/F C. 3/F D.4/F

20

Possible reasons for item being Type BType B: Item is not difficult but requires

amendments because many students couldn’t answer it.

Possible reasons:Item design flawsNot close to what teacher taught in classStudents lack specific training in reading

comic strips

21

Teacher examined the Type B itemStudents found it difficult because …

Need to process and connect many ideas all at the same timeQuestion Comic strip

Which floor? 1/F,2/F,3/F,4/Ffloor directory

… of the Centre … Star Shopping Centre

… can Grandpa find presents …

We can buy all the gifts …

presents? presentsrobotstoys(4/F)

… can Grandpa … ‘ We’ refers to ‘grandparents’

22

Recommended solutionsIncrease students’ chances in reading

English comic strips in classroom teaching or leisure reading

Teacher needs to illustrate more clearly in connecting ideas

Teacher needs to teach more explicitly which means more visual demonstrations

23

Exam item example: Type CIn which step does Jack’s mum add ‘soy sauce’ and ‘wasabi’ to the sushi?A. Step 1 B. Step 3 C. Step 6 D. Step 10

24

Possible reasons for item being Type CType C: Item is difficult and requires

amendments because many students couldn’t answer it.

Possible reasons:Confusing descriptionQuestion misleading studentsQuestion containing error(s)

25

Teacher examined the Type C itemStudents did not have the relevant personal

experience in making and/or eating sushiThe recipe did not mention dipping sushi into ‘soy

sauce’ or ‘wasabi’ after the finishWithout experience, students could not interpret the

correct answer which is the final stepThis question contains the error which requires

outside knowledge uncommon among the students

26

Exam item example: Type DAfter cutting the vegetables and crab meat, Jack’s mum ____.C.takes some rice wine vinegar and mixes it with the sushi rice

This is the answer but the option rephrased the instructions.

27

Features of Type D itemType D: Item is difficult and doesn’t require

amendments. It is suitable for screening the high ability students. Low and mid- ability students can’t answer it.

Type D items should be kept at a low proportion in an assessment.

The rephrasing is difficult for students to interpret the correct answer.

28

Analysis of TSA P.3 Reading PaperPredict P.3 students’ results in the upcoming TSA

reading assessmentCompare the overall students’ ability from different

yearsExamine the neglected elements in our teaching Assumption: TSA assessment covers the basic

competency that all students are supposed to have acquired by the end of a learning stage

29

SP Xpress analysis on ‘2011 TSA P.3 Reading Paper (3ER1)’

2011-2012 P.3

2012-2013 P.3

Mean 19.28 18.95

SD 4.78 4.76

Type A 18 20

Type B 6 3

Type C 0 1

Type D 2 230

An Interesting Fact: A common Type B item across 2 yearsPart 1B Question #1

How many chapters are there in this book?‘Chapter / Table of Contents’ can be an obstacleStudents never encounter ‘Chapter’ in storybook

teaching (replacing textbook teaching)

31

An Interesting Fact: A common Type B item across 2 years Sam buys two sandwiches with the Rainbow Fast Food

coupon. How many sandwiches can he get?C. three sandwiches (= Buy 2 + 1 Free)

Students are not familiar with ‘Buy 2 Get 1 Free’ concept even though they can understand every single word

32

An Interesting Fact: A common Type D item across 2 years “ … too. In the end, John wins the race. All the

people cheer loudly …” Read lines 11-12. What do the people do when John

wins the race?C. shout happily

Students are not familiar with synonyms

cheer loudly

shout happily33

Personal reflection1. Teachers become aware of the quality of the

assessments Accumulate professional experience in setting

better assessments

2. Teachers prepared for the unexpected and unlearned

Teachers become cautious and adjust their teaching

34

Things to do Suggest Ts reflect on their exam paper

using SP Xpress Facilitate Ts process the data with

technical supportsSet up a school based question bankNurture a culture on giving a more

constructive feedback to both Ts and Ss after assessment

35

Thank you so much for your time!