FACT (Formative Assessment with

Computational Technology)

Math Assessment Project

Hugh Burkhardt (Nottingham)

Malcolm Swan (Nottingham)

Daniel Pead (Nottingham)

Alan Schoenfeld (Berkeley)

David Foster (SVMI)

Arizona State University

1

Kurt VanLehn (PI)

Salman Cheema (leader)

14 grad students

ASU Center for Assured and Scalable Data

Engineering

CASCADE-IUCRC

Industry/University Collaborative Research

Center (I/UCRC) * NSF I/UCRC planning grant (NSF-IIP1464579)

CASCADE Mission

Mission: to support the innovation of data

architectures and tools that can support timely and

assured decision making to generate value.

Validate & Interpret

Act & Adapt

Sense & Integrate

Simulate & Predict

Can teachers circulate among student

groups with the computer helping?

4

Tablet

GroupGroup

For individual work, current intelligent

tutoring systems + teachers suffice

Tablet

When students work collaboratively in

small groups on a joint product, how

can tablets help the teacher?

Teacher

The FACT research problem

Given

– Successful formative assessment based on paper

– Unfortunately, the required analyses are difficult for

teachers to do, especially in real-time

Develop

– Analysis methods

– Media system: Input from students; Output to teachers

Evaluate

– Impact on student and teacher learning

– Impact on classroom processes

7

Describe the research problem

Describe FACT

Outline

8

Next

Classroom Challenges

Developed over 10 years by Mathematics

Assessment Project (map.mathshell.org)

Addresses CCSS-M

Approximately 100 available; grades 6 through 10

Widely used despite lack of publisher

Existing evaluation (Herman et al., 2015)

– Raised state test scores

– Little change in mathematical misconceptions/thinking

9

A Classroom Challenge is spread

over 3 days

10

Solve individually15 minutes of Day 1

5 minutes of Day 2 for feedback

Solve in small groups

55 minutes of Day 2

Teacher constantly analyses work-

in-progress and gives formative

feedback.

Discuss as whole class

Critique sample solutions in

groups

Solve individually 15 minutes of Day 3

A high school problem: Determine if “2000

descendants in 18 months” is realistic.

Length of pregnancy: 2 months

Age at which a female cat can first get pregnant: 4 months

Average number of litters a cat can have in one year: 3

Number of kittens per litter: 4 to 6

Age at which cat no longer has kittens: 10 years

11

What should a teacher say?

12

Length of pregnancy: 2 months

Age at which a female cat can first get pregnant: 4 months

Average number of litters a cat can have in one year: 3

Number of kittens per litter: 4 to 6

What should a teacher say?

13

Length of pregnancy: 2 months

Age at which a female cat can first get pregnant: 4 months

Average number of litters a cat can have in one year: 3

Number of kittens per litter: 4 to 6

What should a teacher say?

14

Length of pregnancy: 2 months

Age at which a female cat can first get pregnant: 4 months

Average number of litters a cat can have in one year: 3

Number of kittens per litter: 4 to 6

What should a teacher say?

15

Length of pregnancy: 2 months

Age at which a female cat can first get pregnant: 4 months

Average number of litters a cat can have in one year: 3

Number of kittens per litter: 4 to 6

What should a teacher say?

16

Length of pregnancy: 2 months

Age at which a female cat can first get pregnant: 4 months

Average number of litters a cat can have in one year: 3

Number of kittens per litter: 4 to 6

Advice from the teacher’s

manual.

17

Will these kittens grow up and have kittens?

Are they all males? Please write your assumptions

Some beautiful math, worth

pointing out to the whole class

18

Length of pregnancy: 2 months

Age at which a female cat can first get pregnant: 4 months

Average number of litters a cat can have in one year: 3

Number of kittens per litter: 4 to 6

Teacher formative assessments

decisions

What should I say to help this

student/group engage the math?

Is this something I can use with

the whole class?

19

Such decisions must be made very

rapidly and constantly

What should a teacher say?Hint: The numbers mentioned in the problem statement are: 2, 4, 3, 6 and 10.

20

What should a teacher say?

21

What should a teacher say?

22

Can the computer perceive kids’

derivations?

23

Can teachers circulate among students

with the computer helping?

24

Describe the research problem

Describe FACT

Outline

25

Next

Centaurs: Computers & humans

as nearly equal partners

Chess centaur

– Both agents propose & analyze possible moves

– Humans make the final decision

– Freestyle (fast) and correspondence (slow)

Classroom centaur

– Both agents propose & analyze possible advice to

students

– Human teacher makes the final decision

– During class (fast) and during evenings (slow)

26

FACT Media system replaces

paper with one tablet per student

27

Every student has a tablet.

Every tablet in a group edits the same poster.

28

One student

Another

student

Teachers’ Organize screen shows

students & groups. Tap to show poster.

29

Tap “TALK” button to pause student

tablets

30

TALK button

Tap “PROJECT” button to display

teacher’s screen to whole class

31

PROJECT button

Teacher’s Activity screen. Tap

row to select activity.

32

FACT Analysis system alerts

teacher to learning opportunities

33

Current FACT workflow for giving

advice to a single student/group

1. Alert

2. Peek

3. Preview

Send?

4. Sent

34

1. Alert

2. Peek

3. Preview

Send?

4. Sent

35

36

1. Alert

2. Peek

3. Preview

Send?

4. Sent

37

1. Alert

2. Peek

3. Preview

Send?

4. Sent

38

1. Alert

2. Peek

3. Preview

Send?

4. Sent

Current FACT workflow for

whole-class learning alert

39

Current status of FACT Software

– Android version stable but requires obsolete tablet

– Web app ready by end of July

Analytic power

– Product detectors for card-moving lessons – working well now

– Product detectors for handwriting lessons – next year

– Process detectors – next year

Policies for alerting

– Teachers: 2 policies explored, just starting

– Students: no policies explored yet

Evaluations

– Formative: 30 classroom trials so far; 10+ next year

– Summative: Applying for more funding

40

Take home points

Problem

– Complex, open-ended, collaborative reasoning is coming to

classrooms.

– Teachers cannot easily analyze such reasoning.

Research questions: Centaurs vs. mere tools/media

– Can computers perceive the students’ reasoning without disrupting

it or the classroom?

– Can computers can do formative analyses of complex, handwritten

problem solving?

– Can this assist rather than overwhelm the teachers?

41