2017/7/6

1

HKASME Science Assessment Test 2017

DATE: 6 JULY 2017

(Debriefing Seminar)

HKASME Science Assessment

Test Purpose:

• To develop and to implement a science assessment test for S2&3 students in Hong Kong

• To analyse the test results and to provide feedback to schools/the administration on the strengths and weaknesses of students in learning science

2017/7/6

2

HKASME Science Assessment

Test

• Duration: 1 hour

• Paper Structure:

Section A: 24 MCQs

Section B: 2 Short-response questions

• Coverage: Science knowledge in

topics 1 to 11 of CDC Science Curriculum

S1 to S3

2017 Science Assessment Test

• Bronze Award based on a criteria-based model

• Introducing a Diamond Award (top 5%)

• Expert judgment for Gold, Silver and Bronze Award

• QP and students’ performance on individual questions available to participating schools

• More informative school reports

2017/7/6

3

Participations in 2017 SAT

No. of Participating Schools: 88

Participants S1 S2 S3 Total

Boys 14 622 654 1290

Girls 6 574 532 1112

Total 20 1196 1186 2402

Science Process Skills

• Observing, measuring and recording data (SP1)

• Comparing and classifying (SP2) • Planning and designing (SP3) • Experimenting (SP4) • Interpreting data / Inferring (SP5) • Communicating (SP6)

2017/7/6

4

Weaknesses of Participants as

Revealed from Their

Performance

Section A

Discrimination index (DI)

describes the ability of an item to distinguish between high and low scorers

ranges between -1.00 and +1.00

expected that the high-performing students select the correct answer for each item more often than the low-performing students

DI = (H-L) / ½ N

2017/7/6

5

items with p-value between 30 - 70 and DI > 0.24

“appropriate”

Understanding and Interpreting

Graphical Information

The graph below shows the variation of solubility of a

salt X with temperature:

Q.12

2017/7/6

6

In an experiment, 40 g of X is added to 100 g of water kept at

60oC and the mixture is stirred until there is no further change.

Based on the above information, which of the following

statements about the resulting mixture is correct?

*A. It is an unsaturated solution. (42%)

B. It is a saturated solution. (41%)

C. It contains undissolved X. (13%)

D. It has a mass of 100 g. (4%)

Less than half of the participants were able to choose the key

showing that they were weak in the interpretation of graphs.

Discrimination index = 0.56

Q.19

An experiment using the set-up

shown on the right can be used to

study the pressure-volume

relationship of a fixed mass of air at

room temperature.

Pressures (P) of different

magnitudes are applied onto the

plunger and the corresponding

volumes of the air inside the syringe

(V) are measured. Which of the

following graphs correctly

represents the experimental results

obtained?

2017/7/6

7

V

P

V

P

V

P

V

P

A majority of the participants knew that, under constant

temperature, the volume of a fixed mass of gas decreases with its

pressure. However, most of them did not know that the volume-

pressure relationship is non-linear.

Discrimination index = 0.22

Understanding of Abstract Science Concepts

Q.7 Toby throws a ball in air at an angle. The diagram

below shows the path of the ball and four positions, P,

Q, R and S, of the ball along the path:

In which of these positions, does the arrow represent the

direction of the force acting on the ball? (Assume that there is

no air resistance.)

2017/7/6

8

A. P (17%)

B. Q (54%)

C. R (12%)

*D. S (17%)

More than half of the participants wrongly thought that the

direction of the force is tangential to the trajectory of the ball.

They did not realize that when the ball had left Toby’s hand,

the pushing force (a contact force) would cease to act on it,

and the only force acting on the ball was the gravitational

attraction (a non-contact force).

Discrimination index = 0.58

Q.15 Peter used the set-up shown below to study the image

formed by a convex lens:

He moved the lens to and fro until a sharp inverted

image was formed on the screen. If Peter repeats the

experiment using a thicker convex lens, which of the

following combinations correctly describes the position of

the lens and the appearance of the sharp image formed?

2017/7/6

9

Position of the lens Appearance of the

image

*A. closer to the screen remains inverted (42%)

B. closer to the screen becomes upright (13%)

C. further away from the

screen remains inverted (36%)

D. further away from the

screen becomes upright (9%)

Nearly 80% of the participants knew that, under the

experimental conditions, an inverted image would be

formed. However, quite a number of them failed to realize

that a thicker convex lens bends a beam of light to a

greater extent and forms an image closer to the screen.

Discrimination index = 0.37

Classification of Animals

Q.14 Consider the animals shown below:

Which of the following features of these animals can be used

to classify them into two groups?

(1) whether or not they live in water

(2) whether or not they have antennae

(3) whether or not they have wings

2017/7/6

10

A. (1) only (12%)

B. (2) only (5%)

C. (1) and (3) only (56%)

*D. (2) and (3) only (27%)

A majority of the participants recognized that ducks

and flies have wings while elephants and frogs do not.

However, quite a number of them did not know that

frogs are amphibians, and some failed to realize that

flies do have antennae.

Discrimination index = 0.35

Weaknesses of Participants as

Revealed from Their

Performance

Section B

2017/7/6

11

Question No.

Mean S.D.

1 3.0

(30%) 2.13

(21.3%)

2 4.6

(58%) 1.79

(22.4%)

Overall 7.7

(43%) 3.28

(18.2%)

• Q.1 - to test participants’ understanding of

experimental procedures, communicative and

graph-plotting skills, and interpretation of

experimental data

• Q.2 – to test participants’ understanding of scientific

methods, the design of fair tests and the various

factors that affect a fair comparison

B1 Question on Data-handling

2017/7/6

12

(b) Word equation: chips + oxygen → carbon dioxide + water + energy

No. of

response

s

%

10 All correct 313 13.0

11 Replacing oxygen with air 2 0.1

12 Replacing energy with heat (熱/熱能) 252 10.5

13 Replacing oxygen with air and energy with

heat 5 0.2

90 Failure to give both (answers other than

oxygen/air & energy/heat) 590 24.6

91 Giving oxygen/air but failing to give

energy/heat 138 5.7

92 Failing to give oxygen/air but correctly

giving energy/heat 1053 43.8

93 Using O2 for oxygen & E for energy 2 0.1

99 Nil response 47 2.0

2017/7/6

13

90

92

(c)(i) Plot a graph to show the relationship

between ‘the mass of potato chip burnt’

and ‘the temperature rise of the water in the

boiling tube’.

Three aspects were looked for in marking the graph:

• correct labeling of axes (code: A1x, A9x);

• appropriate choice of scale (code: S1x, S9x); and

• joining the points with a straight line (code: G1x,

G9x).

2017/7/6

14

No. of responses %

A10 Plotting ‘temperature rise of water’ against ‘mass of potato chips burnt’, and both axes are correctly labeled with correct units

547 22.8

A11 Plotting ‘temperature rise of water’ against ‘mass of potato chips burnt’, and both axes are correctly labeled with some units (g/oC) missing

40 1.7

A12 Plotting ‘mass of potato chips burnt’ against ‘temperature rise of water’, and both axes are correctly labeled with correct units

240 10.0

A13 Plotting ‘mass of potato chips burnt’ against ‘temperature rise of water’, and both axes are correctly labeled with some units (g/oC) missing

22 0.9

A91

Plotting ‘temperature rise of water’ against ‘mass of potato chips burnt’, and both axes are correctly labeled with no/incorrect units (e.g. writing kg instead of g)

88 3.7

A92 Plotting ‘mass of potato chips burnt’ against ‘temperature rise of water’, and both axes are correctly labeled with no/incorrect units

40 1.7

A93 Plotting ‘final temperature of water’ against ‘mass of potato chips burnt’ or vice versa

1021 42.5

A94 Graph with one of the axes unlabeled 71 3.0

A98 Other mistakes related to axes (e.g. double axis) 33 1.4

A99 Graph without showing labelled axes/ Nil response 300 12.5

No. of

responses %

S10 Graph with appropriate scale in both x- and y-axes 831 34.6

S91 Graph with the scale in one of the axes not equally spaced

213 8.9

S92 Graph with a poor scale (e.g. the graph is highly compressed or falls out of the graph paper) in either axis

456 19.0

S93 Scale NOT at origin 128 5.3

S94 Scale using direct entry in either axis 502 20.9

S98 Other mistakes related to scale 94 3.9

S99 Graph without scale in both axes/ Nil response 178 7.4

2017/7/6

15

(Code: A94, S92, G13) (Code: A93, S94, G91)

(Code: A93, S93, G11) (Code: A94, S94, G92)

2017/7/6

16

No. of

responses %

G10 A straight line graph passing through all points and the origin

268 11.2

G11 A straight line graph passing through all points but not the origin

484 20.1

G12 A graph with line segments passing though all points and the origin

31 1.3

G13 A graph with line segments passing though all points but not the origin

153 6.4

G90 No line shown on the data points 163 6.8

G91 All or some of the points are incorrectly plotted 881 36.7

G92 A graph showing a histogram/bar chart/pie chart instead of a straight line

63 2.6

G98 Other mistakes related to graph (e.g. two lines/curves) 139 5.8

G99 Nil response 220 9.2

Mean score for plotting graph = 1.1 (out of 3)

Some participants were able to estimate the temperature rise to be

41oC, but failed to give the final temperature as 66oC.

No. of

responses %

20 No. The heat absorbed by the water in the boiling tube depends on its temperature rise. It is not related to the initial temperature of the water.

348 14.5

10 No. It is not possible to set all initial temperature at 25.0oC. 6 0.2 91 Yes. To ensure the experiment is a fair test, the initial

temperature should be kept the same. 988 41.1

92 Yes. The initial temperature is the controlled variable 221 9.2 93 Yes. Different initial temperatures will affect experimental

results/ lead to inaccurate results 333 13.9

98 Yes/No. Other irrelevant reasons (e.g. easier to compare) 439 18.3 99 Nil response 67 2.8

2017/7/6

17

91

92

93

No. of

responses %

2 Two correct reasons given 89 3.7 1 One correct reason given 744 31.0 0 Incorrect reasons/ nil response 1569 65.3

About 65% of the participants failed to give the following two reasons for

the difference in energy content:

• heat loss to surrounding, and

• incomplete burning of potato chips.

Some strange reasons such as ‘energy given out by digestion/body

consumption is different from the energy measured by the burning of

potato chips’, and ‘other substances present in the package may

increase the energy of potato chips’ were given.

2017/7/6

18

1 correct reason given:

Incorrect responses:

Q2

2017/7/6

19

Q.2(a)(i) Marking Scheme (Double-digit Coding)

Mean mark = 0.5 (out of 1)

Q.2(a) (i) Samples of Participants’ Work

93

10

92

2017/7/6

20

Q.2(a)(ii) Marking Scheme and Samples

(Similar marking scheme for dependent and controlled variables)

Mean mark for Q.2(a)(ii) = 2.4 (out of 3)

Q.2(a)(iii) Marking Scheme

Mean mark = 0.3 (out of 2)

2017/7/6

21

Q.2(a)(iii)Samples of Participants’ Work

Q.2(b) Marking Scheme

Mean mark = 1.4 (out of 2)

2017/7/6

22

Q.2(b) Samples of Participants’ Work

Analysis of 2017 SAT Results

Mean SD

Section A 14.7 (61%) 3.47 (14.4%)

B1 3.0 (30%) 2.13 (21.3%)

B2 4.6 (58%) 1.79 (22.4%)

Section B 7.7 (43%) 3.28 (18.2%)

2017/7/6

23

Comparison of Performance of

S2 and S3 Students

Section A Section B

Mean S.D. Mean S.D.

S2 14.0

(59%) 3.33

(13.9%) 7.1

(39%) 3.04

(16.9%)

S3 15.3

(64%) 3.49

(14.5%) 8.2

(46%) 3.42

(19.0%)

Comparison of the Performance

of Boys and Girls in Section A

Median (correct to the unit digit) = 15

15 correct

S2 S3 Whole group

Boys 53% 64% 59%

Girls 38% 45% 47%

2017/7/6

24

Performance of S2 Students in Section A

Performance of S3 Students in Section A

2017/7/6

25

Performance of Whole Group in Section A

2017 SAT Award Scheme

Bronze Silver Gold Diamond

Paper score / marks

24.5 – 31.5 32.0 – 36.0 36.5 – 41.5 >42.0

Section A score / MCQs

> 10 > 10 > 10 > 10

Section B

score / marks

> 4 > 4 > 4 > 4

(Paper score = 1.5 × Mark in Section A + Mark in Section B)

HKASME set up an expert panel to determine the

cut-off based on the performance of the

participants.

2017/7/6

26

2017 SAT Awards

Award type Number

Diamond 120

(5%)

Gold 351

(15%)

Silver 492

(20%)

Bronze 873

(36%)

Appreciation 566

(24%)

School-based Report –Appendix I

Analysis of Participants’ Responses to the Multiple-choice Questions (MCQs)

Q. No Skills assessed Key

% correct

Strength / weakness WG School

1 Comparing & Classifying; Understanding

B 76.6 78.8 -

2 Laboratory safety; Understanding

B 56.3 39.4 W

3 Inferring C 71.6 93.9 S

4 Planning & Designing; Fair tests

C 56.5 69.7 -

5 Inferring; Understanding D 63.0 93.9 S

6 Applying; Mathematics skills

D 85.9 97.0 -

7 Understanding; Inferring D 17.1 24.2 D

8 Measuring; Choice of apparatus

C 74.8 90.9 S

9 Understanding; Predicting

B 64.1 78.8 -

10 Inferring; Understanding A 54.2 63.6 -

2017/7/6

27

Strength in a question (indicated by “S”) means that

the school got a higher correct % by 15% or more on

that question compared to the whole group in 2016

SAT.

Weakness in a question (indicated by “W”) means

that the school got a lower correct % by 15% or

more on that question compared to the whole

group in 2016 SAT.

Questions that are poorly answered by the whole

group (1/3 correct) are represented by “D”

Question(s) showing “weaker” performance as compared to WG

WG School

Q. No. Key Most Popular

Distractor Key Most Popular

Distractor

9 B

(64.1%)

C

(26.0%)

B

(46.7%)

C

(36.7%)

12 A

(42.0%)

B

(40.5%)

A

(26.7%)

B

(40.0%)

20 D

(54.0%)

B

(27.5%)

D

(33.3%)

B

(40.0%)

2017/7/6

28

Analyzing student responses & identifying their problems

Q. No. A B C D Right

Ans.

Correct

%

Wrong

%

Diff.

Index

Disc.

Index

14 11.53

%

4.75

%

56.16

%

27.48

% D

27.48

% 72.44% 32.45% 0.35

Analysis of Participants’ Performance in

Short-response Questions – Appendix II

Mean and standard deviation

Whole Group (WG) School

Mean SD Mean SD

Question 1

(out of 10)

3.0

(30%)

2.13

(21.3%)

4.6

(46%)

1.82

(18.2%)

Question 2

(out of 8)

4.6

(58%)

1.79

(22.4%)

5.4

(68%)

1.06

(13.3%)

Section B

(out of 18)

7.7

(43%)

3.28

(18.2%)

10.0

(56%)

2.08

(11.6%)

2017/7/6

29

Double-digit coding system for Q1(c)(i) Plot a graph to show the relationship between ‘the mass of

potato chip burnt’ and ‘the temperature rise of the water in the boiling tube’

CODE ITEM Number of Response

WG School

Correct Responses : 831

(34.6%) 26

(78.8%)

S10 Graph with appropriate scale in both x- and y-axes

831 (34.6%)

26 (78.8%)

Incorrect Responses/Nil Response 1571

(65.4%) 7

(21.2%)

S91 Graph with the scale in one of the axes not equally spaced

213 (8.9%)

1 (3.0%)

S92 Graph with a poor scale (e.g. the graph is highly compressed or falls out of the graph paper) in either axis

456 (19.0%)

3 (9.1%)

S93 Scale NOT at origin 128

(5.3%) 0

(0.0%)

S94 Scale using direct entry in either axis 502

(20.9%) 3

(9.1%)

S98 Other mistakes related to scale 94

(3.9%) 0

(0.0%)

S99 Graph without scale in both axes/ Nil response 178

(7.4%) 0

(0.0%)

School-based Report – Appendix IV

(1) Diamond, Gold, Silver and Bronze Awards in 2017

(2) Overall Performance of School

Award type Number

Diamond 120 (5%)

Gold 351 (15%)

Silver 492 (20%)

Bronze 873 (36%)

Appreciation 566 (24%)

Whole Group

(WG)

School

Mean score:

(out of 54) 29.7 (55%) 34.7 (64%)

Standard

deviation: 7.61 (14.1%) 6.92 (12.8%)

2017/7/6

30

Student Performance Report (Sample)

Professional Learning Community

on Science Assessment

• Collaborative culture in the development of quality

assessment items/tasks among science teachers

• MCQs, short-response questions, double digit

coding marking

• Talks, group discussion, moderation exercise and

assignments

• TWO 3-hour Saturday meetings (19/8, 2/9)

• Meeting fee: $100

(Full refund if all 2 meetings are attended and

assignments completed.)