David DidauresearchED - 5th September 2015

Foxy Thinking:

how to embrace

ignorance & learn to love uncertainty

I imagine this midnight moment’s forest: Something else is alive Beside the clock’s loneliness And this blank page where my fingers move. Through the window I see no star: Something more nearThough deeper within darkness Is entering the loneliness:

Cold, delicately as the dark snow, A fox’s nose touches twig, leaf; Two eyes serve a movement, that nowAnd again now, and now, and now

Sets neat prints into the snow Between trees, and warily a lame Shadow lags by stump and in hollow Of a body that is bold to come

Across clearings, an eye,A widening deepening greenness, Brilliantly, concentratedly, Coming about its own business

Till, with a sudden sharp hot stink of foxIt enters the dark hole of the head.The window is starless still; the clock ticks,The page is printed.

The Thought Fox

Ted Hughes

Even if the amount of knowledge in the world is increasing, the gap between what we know and what we think we know may be widening.

Nate Silver, The Signal & The Noise

The larger the island of knowledge grows, the longer the shoreline — where knowledge meets ignorance — extends.

Michael Smithson

Knowledge

IGNORANCE

There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns.

Donald Rumsfeld

IGNORANCE

Known knownsExplicit knowledge

Known unknowns

Educated ignorance – “the shoreline”

Unknown knowns Tacit,

implicit knowledge

Unknown unknowns

The vast ocean of ignorance

IGNORANCE

It’s very difficult to find a black cat in a dark room.

Especially when there is no cat.

• We don’t know what we don’t know• We prefer certainty to being right• It stops us thinking

The problem with certainty

Can we trust the evidence of our own eyes?

The Müller-Lyer Illusion

We can’t see when we’re wrong

We can’t see when we’re wrong

Shepard’s ‘Turning the tables’

Shepard’s ‘Turning the tables’

Michotte’s perception of causality

Michotte’s perception of causality

Michotte’s perception of causality

If it looks like a duck…

The Necker Cube

Are you a fox or a hedgehog?

The fox knows many little things, but the hedgehog knows one big thing.

Archilochus

Are you a fox or a hedgehog?For there exists a great chasm between those, on one side, who relate everything to a single central vision, one system, less or more coherent or articulate, in terms of which they understand, think and feel – a single, universal, organising principle in terms of which alone all that they are and say has significance – and, on the other side, those who pursue many ends, often unrelated and even contradictory, connected, if at all, only in some de facto way, for some psychological or physiological cause, related to no moral or aesthetic principle.

Isaiah Berlin, The Hedgehog and the Fox

Are you a fox or a hedgehog?How Foxes think How Hedgehogs thinkMultidisciplinary SpecialisedAdaptable StalwartSelf-critical StubbornTolerant of complexity Order-seekingCautious ConfidentEmpirical Ideological

✔✔

✔✔

✔✔

Foxes know more about what they don’t know

Are you a fox or a hedgehog?Foxes may have emphatic convictions about how the world ought to be. But they can usually separate that from their analysis of the way the world actually is and how it is likely to be in the near future.

Hedgehogs take a prejudicial view toward evidence, seeing what they want to see and not what is really there.

People tend to think of not knowing as something to be wiped out or overcome, as if ignorance were simply the absence of knowledge. But answers don’t merely resolve questions; they provoke new ones.

Jamie Holmes, The Case for Teaching Ignorance

After 30 years of doing such work, I have concluded that classroom teaching… is perhaps the most complex, most challenging, and most demanding, subtle, nuanced, and frightening activity that our species has ever invented…The only time a physician could possibly encounter a situation of comparable complexity would be in the emergency room of a hospital during or after a natural disaster.

Lee Shulman, The Wisdom of Practice

How can we identify good teachers?

• Observation

• Student outcomes

• Student surveys

Variation in the reliability of teacher ratings withnumber of lessons observed and number of raters (Hill et al., 2011)

Using student outcomes

Percentages of teachers placed in different quintiles of effectiveness with two equally plausible value-added models (Goldhaber et al., 2013)

9% of the teachers who are rated as the least effective with the traditional random-effects model are actually rated as the most effective with the fixed effects model.

The problem with ‘weighing’ teachers• If a scale’s accuracy

was ±50 pounds, we would not be able to say much about the weight of an individual.

• In a group of 50 men and 50 women, the men are heavier on average.

Problems with student outcomes

• Judgements depend on our assumptions

• Results vary considerably from year to year and from context to context.

Using student surveys

Clusters and indicative example statements from the Tripod student survey (Bill and Melinda Gates Foundation, 2012a)

Using student surveys• Students of teachers with the top

25% of Tripod score, make 4.6 months more progress in mathematics than those in the classrooms taught by teachers with Tripod scores in the lowest quartile.

• On average, the most effective 25% of teachers are around 130% more effective than the least effective 25%

Problems with student surveys• The Doctor Fox Effect

• Perverse incentives

We can’t reliably spot good teachers• Aggregating observation, outcomes & student

surveys :

• We can just about tell the difference between the least effective and the most effective teachers

How do you make teachers improve?

• Can you become a better teacher? Or, can the teachers in your school improve?

• Do you know exactly how to make this happen?

• Does anyone else?• How does your school support you? Or,

how do you support others?• Accountability vs. trust

A false dichotomy?

SynthesisTeachers should be

allowed to explain how their practice has resulted in pupil

progress.ThesisTeachers should experiment & develop the best ways of giving feedback in their context.

AntithesisOfsted will

evaluate progress by

checking if there is pupil response

in books.

We all want to be good, but…3 conditions which encourage people to want to be right rather than look right:• the knowledge that we will be

accountable to an audience• the audience’s views must be

unknown• the belief that the audience is well-

informed and interested in accuracy.Lerner & Tetlock,

2003

How we get accountability wrong

1. Teachers are told to use ‘two stars and wish’ to mark their books.

2. There is/isn’t evidence of ‘two stars and wish’ in this teacher’s books, therefore I am/am not happy.

Better accountability1. Teachers are told to mark students’

books in the way they believed would make the most impact.

2. There is/isn’t evidence of marking therefore I am/am not happy.

3. There is/isn’t evidence that the marking has had an impact therefore I need/do not need to ask the teacher some follow-up questions.

Intelligent accountability1. Teachers are expected to cover the

curriculum and ensure to the best of their ability that students are prepared for some kind of assessment.

2. There is/isn’t evidence of marking therefore I need/do not need to ask the teacher some follow-up questions.

Key points• Acknowledging uncertainty leads to

better decision making

• Teachers improve when they feel trusted, supported and held accountable

• Always remember, you might be wrong.

Further reading• Stuart Firestein, Ignorance: How it

Drives Science• Jonathan Haidt, The Righteous Mind• John Kay, Obliquity• Robert Proctor & Londa Schiebinger,

Agnotology: The making & unmaking of ignorance

• Nate Silver, The Signal & The Noise

There’s nothing good or bad but thinking makes

it so.

@LearningSpylearningspy.co.uk

ddidau@gmail.com