The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance...

36
Introduction The top ten list The top ten things that math probability says about the real world David Aldous 9 November 2009 David Aldous The top ten things . . .

Transcript of The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance...

Page 1: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

The top ten things that math probability saysabout the real world

David Aldous

9 November 2009

David Aldous The top ten things . . .

Page 2: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

This is part of an ongoing “probability and the real world” project. Iteach a junior/senior course, and am accumulating disorganized materialon my web site (Bing “David Aldous”). The whole program has muchscope for undergraduate projects.

There really will be a list of 10 (OK, actually 8, so the audience cansuggest more!). But first let me try to “frame” the whole topic of chanceby describing 4 corners of the topic.

David Aldous The top ten things . . .

Page 3: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

18 . IMS Bulletin Volume 38 . Issue 6

Indian Statistical Institute, BangaloreAugust 13-17, 2010

Scientific Programme Committee

web: http://www.isibang.ac.in/~statmath/icmprobsat e-mail: [email protected]

ICM Satellite Conference onProbability and Stochastic Processes

ICM Satellite Conference onProbability and Stochastic Processes

David Aldous (Univ. of California, Berkeley )Vivek S. Borkar (TIFR, Mumbai)

Mu Fa Chen (Beij ing Normal Univ., Beij ing)Alice Guionnet (ENS de Lyon, Lyon)

Takashi Kumagai (Kyoto University, Kyoto)Edwin A. Perkins (Univ. of Brit ish Columbia, Vancouver )

Rahul Roy ( Indian Statistical Institute, Delhi)Marta Sanz-Sole (Univ. de Barcelona, Barcelona)

Maria Eulalia Vares (CBPF, Rio de Janeiro)Ofer Zeitouni (Weizmann Inst. of Science, Rehovot)

Other Meetings Around the World: Announcements and Calls for Papers

NEW

Financial Support for Participation in ICM2010 (August 19–27, 2010, Hyderabad, India)w www.icm!"#".org.in

!e International Mathematical Union (IMU) and the ICM"#$# Local Organizing Committee are currently making e%orts to obtain financial support to enable as many mathematicians as possible from developing and economically disadvantaged countries to participate in ICM"#$#. !ere are three di%erent support categories (travel, registration, living support can be applied for):$) Young mathematicians from developing and economically disadvantaged countries") Senior mathematicians from developing and economically disadvantaged countries&) Mathematicians from developing countries in Asia with emphasis on countries neighboring IndiaAll mathematicians who wish to apply for support are kindly asked to complete the corresponding Application Form at the ICM"#$# website (the same form is used for all three categories). Applications may be submitted from $ July %&&' – $ January %&$&. Queries may be sent to the organization of the ICM"#$# at the address [email protected]

David Aldous The top ten things . . .

Page 4: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

Corner 1

Games of chance based on artifacts with physical symmetry

– dice, roulette, lotteries, playing cards, tossing coins, etc

Indeed the most common iconic visual image for randomness is a die (*).And the math of probability started, several centuries ago, in part withgames of chance.

BUT most of us spend little time on games of chance

Digression; Is my claim (*) true? That’s a good future undergradproject! One resource is my list of, and brief reviews of, 70 non-technicalbooks relating to Probability: just look at their cover graphics.

David Aldous The top ten things . . .

Page 5: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

Dicing With Death. Chance, risk and health.

Against the Gods: The Remarkable Story of Risk.

Struck by Lightning: the curious world of probabilities.

What are the Chances? Voodoo deaths, office gossip and otheradventures in probability.

Chances Are: Adventures in Probability.

Chance Rules: An informal guide to probability, risk and statistics.

The Jungles of Randomness.

The Drunkard’s Walk: How Randomness Rules Our Lives.

The Broken Dice, and other mathematical tales of chance.

Randomness.

These 10 books are in the “Popular Science” category; other categoriessuch as “Sports and Gaming” and “Stock Market and Finance”.

David Aldous The top ten things . . .

Page 6: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

Corner 2

In what contexts does the concept of chance arise, in ordinary lifeas an individual?

Easiest way to attempt an answer: search through blogs to examinecasual usage of specific words or phrases, e.g. “one in a million chance”.Done as undergrad project. Sample results next slide.

Fun to compare with hypothetical examples invented by a philosopher(Nicholas Rescher) in a monograph Luck: The Brilliant Randomness OfEveryday Life.

Project: articulate differences [someone, please]?

David Aldous The top ten things . . .

Page 7: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

Chance – as seen in blogs

Seeing how its like a one in a million chance to find that one personyou connect with.

I have .... syndrome. The fact that I ever became a mother was a“one in a million chance”.

i’m waiting for the day they [upcoming movie/TV filming locations]say my city which is one in a million chance

.... got this contest. It’s a one in a million chance to get somepeople .... to tell me what they think of my work.

Of course if I don’t go [to the doctor about certain symptoms],there’s that one in a million chance that I’ll be sorry I didn’t.

and they [adults] all start talking about how im too young to begoing out by myself ..... But it’s not like im going to listen to them,what happened [witnessing a mall shooting] was a once in a millionchance.

David Aldous The top ten things . . .

Page 8: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

Luck – everyday life in a Philosophy department?

winning an heiress in competition with another suitor.

contracting [a cold] on the evening of one’s opening nightperformance

your secret benefactor’s sending you that big check ...

being wounded by an assassin who mistakes one for someone else

you were inadvertently delayed and just missed crossing on theHindenberg

burglar who breaks into a house just before its owner returnswell-armed from a bear hunt

coming down with a disease for which a cure has just beendiscovered

author whose biography of a celebrity hits the bookshops just as itsprotagonist is enmeshed in a highly publicized scandal ...

David Aldous The top ten things . . .

Page 9: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

Corner 3: Global Risks

Chance, in the human world, on a larger scale –

Global Risks

Once a year you hear a news item about the World Economic Forum inDavos, Switzerland. If the attending World Leaders pay attention theywill hear a discussion of upcoming Global Risks, prepared by OECD.

David Aldous The top ten things . . .

Page 10: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

Figure 1: Global Risks Landscape 2009: Likelihood with Severity by Economic Loss

Likelihood

below 1% 1-5% 5-10% 10-20% above 20%

2-10

billi

on10

-50

billi

on50

-250

billi

on25

0bi

llion

-1tr

illio

nm

ore

than

1tr

illio

n

Sev

erity

(inU

S$)

Based on an the assessment of risks over a 10 year time horizon by the Global Risk Network

Key: Boxes indicate change since last year’s assessment

New risk for 2009

DecreasedIncreased

Stable Likelihood Severity

34

36

35

29

30

31

32

33

20

21

22

23

24

25

26

27 28 12

13

14

15

16

1718

19

11

1

3

4

5

89

10

2 67

Source: World Economic Forum 2009

ECONOMIC1 Food price volatility2 Oil and gas price spike3 Major fall in US$4 Slowing Chinese economy (6%)5 Fiscal crises6 Asset price collapse7 Retrenchment from globalization (developed)8 Retrenchment from globalization (emerging)9 Regulation cost10 Underinvestment in infrastructure

GEOPOLITICAL11 International terrorism12 Collapse of NPT13 US/Iran conflict14 US/DPRK conflict15 Afghanistan instability16 Transnational crime and corruption17 Israel-Palestine conflict18 Violence in Iraq19 Global governance gaps

ENVIRONMENTAL20 Extreme climate change related weather21 Droughts and desertification22 Loss of freshwater23 NatCat: Cyclone24 NatCat: Earthquake25 NatCat: Inland flooding26 NatCat: Coastal flooding27 Air pollution28 Biodiversity loss

SOCIETAL29 Pandemic30 Infectious disease31 Chronic disease32 Liability regimes33 Migration

TECHNOLOGICAL34 CII breakdown35 Emergence of nanotechnology risks36 Data fraud/loss

David Aldous The top ten things . . .

Page 11: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

Games of chanceChance, in individual lifeChance: global risksProbability: academic disciples outside mathematics

Corner 4: Probability in 20 academic disciplines

Some academic topics in which the math of probability plays a role.

Kinetic theory of gases (statistical physics).

Random inheritance of genes by offspring from parents; randommutation; the bottom-level mechanism on which evolution bynatural selection operates (population genetics).

Statistical estimates of probabilities for an individual, based on datafor that individual and population statistics concerning relationshipsbetween factors – e.g. credit scores; amazon.com suggestions forbooks you might like. (statistical learning theory).

Short-term fluctuations of equity prices, exchange rates etc(stochastic finance).

Systems governed by deterministic rules but so sensitive to initialconditions that long-run behavior is effectively random (chaos).

Software using randomness (algorithms).

David Aldous The top ten things . . .

Page 12: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

The top 10 list

Start with a critical analysis of freshman statistics – what do textbookssay that’s non-obvious and actually true?

1. Opinion polls actually work rather well.

Like airplanes, they only get into the news when things go wrong. I’musing opinion polls as a proxy for other textbook concepts likerandomized controlled experiments – we really do know how to determinewhether medical drugs work, but again the subject only gets into thenews when things go wrong.Data. California Field Poll – elections for Senate, Governor, President –reverse order from 2008 (Obama - McCain) back to 1982 (Bradley -Deukmejian for Governor) – percent errors were

0 1 0 1 1 3 2 3 1 2 0 1 3 3 1 3 0 3 3 1 0 2 1 0 5

and the 5 was so unusual it acquired a name: “the Bradley effect”.

David Aldous The top ten things . . .

Page 13: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

2. Separating skill from luck

My favorite quotation relevant to probability is

Chance favors the prepared mind (Louis Pasteur)

which reminds us that luck and skill are not really separable.

When we examine the specific history of “a success” (Bill Gates; militaryvictories; Superbowl winners etc) we see some mixture of skill and luck. Isit possible to say (very approximately) how much was due to skill vs luck?

Theory says:

2. You can measure the relative contributions of skill and luck inthe aggregate but not at an individual level.

David Aldous The top ten things . . .

Page 14: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

2009 baseball: the 3 teams with worst records wereWashington (.364) Pittsburgh (.385) Baltimore (.395)Without knowing anything about baseball I’m willing to bet $100 that atleast 2 of these three teams will have better records in 2010.

This is a textbook instance of the regression effect; above-averageteams one year tend to do less well the next year. It is not primarily dueto changes in a team’s ability from one year to the next.

Sport Predictions for Proportion correctHockey Top 3 72%Hockey Bottom 3 79%Football Top 3 83%Football Bottom 3 83%Basketball Top 3 77%Basketball Bottom 3 85%Baseball Top 3 66%Baseball Bottom 3 85%

David Aldous The top ten things . . .

Page 15: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

3. Coincidences

3. Coincidences are more likely than you think.

The famous birthday paradox says: with 23 people in a room, there’sabout a 50% chance that some two people have the same birthday.

There’s nothing magic about 23. For any number of people there’s aformula for the chance of some shared birthday, it just turns out that tomake this chance be 50% you need 23.

This is a theoretical prediction; and you can check how well it works withe.g. Major League Baseball teams. And it works pretty well.

David Aldous The top ten things . . .

Page 16: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

Active Roster | phillies.com: Team http://philadelphia.phillies.mlb.com/team/roster_active.jsp?c_id=phi

1 of 1 11/2/09 9:53 AM

Roster

Phillies RosterActive Roster40-Man RosterDepth ChartSearch

Coaches

Transactions

Injury Updates

Draft Results

Front Office

Broadcasters

Minor League Affiliates

Phillies tickets through

Active Roster | 40-Man Roster | Depth Chart

Active Roster Pitchers B/T Ht Wt DOB

58 Antonio Bastardo R/L 5-11 195 09/21/85

56 Joe Blanton R/R 6-3 250 12/11/80

37 Chad Durbin R/R 6-2 220 12/03/77

47 Scott Eyre L/L 6-1 225 05/30/72

35 Cole Hamels L/L 6-3 190 12/27/83

43 J.A. Happ L/L 6-6 200 10/19/82

34 Cliff Lee L/L 6-3 190 08/30/78

54 Brad Lidge R/R 6-5 215 12/23/76

46 Ryan Madson L/R 6-6 200 08/28/80

45 Pedro Martinez R/R 5-11 195 10/25/71

39 Brett Myers R/R 6-4 240 08/17/80

61 Chan Ho Park R/R 6-2 210 06/30/73

Catchers B/T Ht Wt DOB

23 Paul Bako L/R 6-3 210 06/20/72

51 Carlos Ruiz R/R 5-10 205 01/22/79

Infielders B/T Ht Wt DOB

4 Eric Bruntlett R/R 6-0 200 03/29/78

19 Greg Dobbs L/R 6-1 210 07/02/78

7 Pedro Feliz R/R 6-1 210 04/27/75

6 Ryan Howard L/L 6-4 260 11/19/79

11 Jimmy Rollins S/R 5-8 170 11/27/78

26 Chase Utley L/R 6-1 190 12/17/78

Outfielders B/T Ht Wt DOB

10 Ben Francisco R/R 6-1 190 10/23/81

29 Raul Ibanez L/R 6-2 225 06/02/72

12 Matt Stairs L/R 5-9 215 02/27/68

8 Shane Victorino S/R 5-9 185 11/30/80

28 Jayson Werth R/R 6-5 215 05/20/79

Official Info Terms of Use Privacy Policy Help/Contact Us Job Opportunities Advertising Opportunities

© 2001-2009 MLB Advanced Media, L.P. All rights reserved.

The following are trademarks or service marks of Major League Baseball entities and may be used only with permission of Major League Baseball Properties, Inc. or the relevant Major League Baseball entity: Major League, Major League Baseball, MLB, the silhouetted batter logo, World Series, National League, American League, Division Series, League Championship Series, All-Star Game, and the names, nicknames, logos, uniform designs, color combinations, and slogans designating the Major League Baseball clubs and entities, and their respective mascots, events and exhibitions.

Use of the Website signifies your agreement to the Terms of Use and Privacy Policy (updated 05.07.2009).

Search

MLB SitesMLB Sites

Register Login

News Audio & Video Scoreboard Schedule Roster Stats Community Fans Citizens Bank Park Tickets Shop Auction Mobile Fantasy

David Aldous The top ten things . . .

Page 17: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

Mathematicians conclude “coincidences are more likely than you think”based on this type of “small universe” example, but it’s a long way awayfrom the kind of real-life coincidences that people find striking.

David Aldous The top ten things . . .

Page 18: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

Some people assert: observed real-life coincidences are immensely toounlikely to be explicable as “just chance” – and so they assign spiritualsignificance.Rationalists assert: oh no they’re not. Because there are gazillions ofpossible coincidence that don’t happen.

I’m on the rationalist side, but must admit that (outside “small universe”models like the birthday paradox) there’s essentially no evidence(analogous to baseball teams evidence for the birthday paradox) thatinteresting real-life coincidences occur no more often than “pure chance”predicts.

David Aldous The top ten things . . .

Page 19: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

Here are three essential features of real-life coincidences:

coincidences are judged subjectively – different people will makedifferent judgements;

if there really are gazillions of possible coincidences, then we’re notgoing to be able to specify them all in advance – we just recognizethem as they happen;

what constitutes a coincidence between two events depends verymuch on the concrete nature of the events.

Studying real everyday life seems too difficult. Is there any other contextin which we can both get data with these features and compare totheory?

Next slide shows my best effort.

David Aldous The top ten things . . .

Page 20: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

article article specific coincidence chance

×10−8

1 Kannappa Vasishtha Hindu religious figures 12 56

2 Harrowby United F.C. Colney Heath F.C. Engl. am. Football Clubs 160 120

3 Delilah Paul of Tarsus Biblical figures 20 30

4 USS Bluegill (SS-242) SUBSAFE U.S. submarine topics 6 1 8

5 Kindersley-Lloydminster Cape Breton-Canso Canadian Fed. Elec. Dist. 110 23

6 Walter de Danyelston John de Stratford 14/15th C British bishops 1 81

7 Loppington Beckjay Shropshire villages 4 55

8 Delivery health Crystal, Nevada Prostitution 9 46

9 The Great Gildersleeve Radio Bergeijk Radio comedy programs 4 23

10 Al Del Greco Wayne Millner NFL players 3000 77

11 Tawero Point Tolaga Bay New Zealand coast 3 32

12 Evolutionary Linguistics Steven Pinker Cognitive science ??? 36

13 Brazilian battleship Sao Paulo Walter Spies Ironic ship sinkings < 1 28

14 Heap overflow Paretologic Computer security ??? 52

15 Werner Herzog Abe Osheroff Documentary filmmakers 1 92

16 Langtry, Texas Bertram, Texas Texas towns 180 53

17 Crotalus adamanteus Eryngium yuccifolium Rattlesnake/antidote < 1 80

18 French 61st Infantry Division Gebirgsjager WW2 infantry 4 45

19 Mantrap Township, Minnesota Wykoff, Minnesota Minnesota town(ship)s 810 41

20 Lucius Marcius Philippus Marcus Junius Brutus Julius Caesar associate 4 91

21 Colin Hendry David Dunn Premier league players 150 62

22 Thomas Cronin Jehuda Reinharz U.S. College presidents 32 44

23 Gosta Knuttson Hugh Lofting Authors of children’s lit. 32 31

24 Sergei Nemchinov Steve Maltais NHL players 900 16

25 Cao Rui Hua Tuo Three Kingdoms people 37 18

26 Barcelona May Days Ion Mota Spanish Civil War 5 116

27 GM 4L30-E transmission Transaxle Auto transmissions 3 37

28 Tex Ritter Reba McEntire Country music singers 8 24

David Aldous The top ten things . . .

Page 21: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

4. Long term investing

4. The Kelly criterion marks the borderline between aggressive andinsane investing.

Background: if you’re worth $500,000 then it’s irrational to be risk-aversefor small amounts – should regard “gaining $250” and “losing $250” asequal-but-opposite. But it’s rational to be risk-averse for $250,000.

In fact people are Predictably Irrational (title of recent Dan Arielybook, and item 7 on our list) in such matters, but . . . . . .

David Aldous The top ten things . . .

Page 22: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

I focus on long-term investment. Imagine you inherit a sum of money atage 25 and you resolve to invest it and not start spending it until age 65.We envisage the following setting.(i) You always have a choice between a safe investment (pays interest, norisk) and a variety of risky investments. You know the probabilities of theoutcomes of these investments. [of course in reality you don’t knowprobabilities – unlike casino games – so have to use your best guessinstead].(ii) Fixed time period – imagine a year, could be month or a day – at endyour take you gains/losses and start again with whatever you’ve got atthat time (“rebalancing”).

The Kelly criterion gives you an explicit rule for how to divide yourinvestments to maximize long-term growth rate.

David Aldous The top ten things . . .

Page 23: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

To illustrate, imagine day-trading scheme with stocks based on somestatistical non-randomness; within one day51% chance to double money; 49% chance to lose all money.Looks good – expected gain 2% per day – but don’t want to risk all yourmoney on one day. Instead use strategy: bet fixed proportion p of moneyeach day. Theory says: long-term growth rate, depends on p, but in anunexpected way.

0.02 0.04

210,000

growth rate

p

Optimal strategy: bet p = 2% of your capital each day; this providesgrowth rate 2

10,000 per day, which (250 trading days per year) becomes

5% per year.

David Aldous The top ten things . . .

Page 24: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

The numbers above depended on hypothetical assumptions. But theconceptual point is completely general. We are not assuming you canpredict the future, just that you can assess future probabilities correctly.Provided there is some risky investment whose expected payoff is greaterthan the risk-free payoff, the Kelly criterion is a formula that tells youhow to divide your portfolio between different investments.

There’s one remarkable consequence of using this strategy. To get themaximum possible long-term growth rate, using “100% Kelly strategy”,you must accept a short-term risk, of the specific form

50% chance that at some time your wealth will drop to only 50% of yourinitial wealth.

And 10%− 10% too! Of course, if not comfortable with this level of risk,you can use “partial Kelly strategy” combining with risk-free assets.

David Aldous The top ten things . . .

Page 25: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

This story is told in the popular book Fortune’s Formula by WilliamPoundstone. Maybe nothing in this story seems intellectually remarkable,but in fact something is. Consider an analogy: the light speed barrier.

[Common sense says objects can be stationary or move slowly or movefast or move very fast, and that there should be no theoretical limit tospeed – but physics says in fact you can’t go faster than the speed oflight. And that’s a very non-obvious fact. ]

Similarly, we know there are risk-free investments with low return; bytaking a little risk (risk here equals short-term fluctuations) we can gethigher low-term reward. Common sense says this risk-reward trade-offspectrum continues forever. But in fact it doesn’t. As a math fact, youcan’t get a higher long-term growth rate than you get from the “100%Kelly strategy”.

David Aldous The top ten things . . .

Page 26: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

5. Coding

5. Coding for secrecy is essentially the same as coding for efficientcommunication or storage.

The fact that most letter strings JQTOMXDW KKYSC have no meaningis what makes most simple letter substitution ciphers easy to break. In ahypothetical language in which every “monkeys on typewriters” stringhad a meaning, a letter substitution cipher would be impossible to break,because each of the 26 x 25 x 24 x .... x 2 possible decodings might bethe true message.

Now if you want to transmit or store information efficiently, you wantevery string to be possible as a coded string of some message (otherwiseyou’re wasting an opportunity) and indeed you ideally want every stringto be equally likely as the coded string of some message. This is“coding for efficiency”, but with such an ideal public code one could justapply a private letter substitution cipher and get an unbreakable “codefor secrecy”.

David Aldous The top ten things . . .

Page 27: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

6. Genetics

Genetics is one of the classic applications of probability but rather thanattempting to say anything portentious, let me just ask

6. Are you related to your 10th generation ancestors?

Genetically, that is.

You have 2 parents, 4 grandparents, 8 great-grandparents, and back 10generations you have somewhat less than 1,024 ancestors (likely not alldifferent). How many have you actually inherited DNA from? You have46 chromosomes, so if you inherited whole chromosomes you could onlybe related to 46 of the ancestors. In fact there is “crossover”, and undera simplified model, one can calculate that you inherited DNA from about370 of your 10’th generation ancestors. Less than half of them. So evenif you can prove that one of your ancestors was a relative of King GeorgeIII, this doesn’t mean you have royal blood.

David Aldous The top ten things . . .

Page 28: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

7. Predictably irrational

7. Predictable irrationality, in decisions under uncertainty.

Much psychology research since 1980 (Amos Tversky et al) involvesexperiments on “decisions under uncertainty”. Here’s a famous example:decisions can be strongly affected by how information is presented.

Imagine a rare disease is breaking out in some community. if nothing isdone, 600 people will die. There are two possible programs. To somesubjects you describe the alternatives as(A) will save 200 people(B) will save everyone with chance 1/3 and save no-one with chance 2/3

to others you describe the alternatives as(C) 400 people will die(D) no-one will die with chance 1/3; 600 people will die with chance 2/3.

Given “A or B” choice, most people choose A.Given “C or D” choice, most people choose D.

David Aldous The top ten things . . .

Page 29: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

In my undergraduate course, students do course projects, and one optionis to repeat some classic experiment. Here’s a fun example.

Subjects: college educated, non-quantitative majors.

Equipment: bingo balls (1 – 75) and 10 Monopoly $500 bills.

Draw balls one at a time; subject has to bet $500 on whether nextball will be higher or lower than last ball; prompt subject to talk(recorded) about thought process. Repeat for 5 bets.

Say: we’re doing this one last time; this time you have option to betall your money. Prompt talk.

David Aldous The top ten things . . .

Page 30: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

What is the point of this experiment?

In first part, everyone “plays the odds” – behaves and explains rationally:if this ball is 43 then more likely that next ball is less than 43, so bet thatway.

Point: what explanations do people give for their choice (last stage) ofwhether or not to bet all their money. In our experiments, about 50-50split between

risk-aversion; good or poor chances to win

feeling (or have been) lucky or unlucky.

Conclusion: even when “primed” to think rationally, people have innatetendency to revert to “luck” explanations.

David Aldous The top ten things . . .

Page 31: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

8. Efficient (?) markets

The efficient market hypothesis says (roughly) “current price reflectsrational assessement of future value based on known information” – soprice fluctuations reflect new information.

Events of the last 15 months have shaken this academic orthodoxy. Idon’t want to enter the debate over financial markets, which have manycomplicating features, so will consider something simpler in a moment.

One point to remember is that time scale matters. Our previous Kellycriterion discussion dealt with the long term, say > 10 years, wherecompounding matters. Let’s illustrate widely varying time scales.

David Aldous The top ten things . . .

Page 32: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

High frequency traders use computers that execute trades withinmilliseconds, or “with extremely low latency” in the jargon of the trade.In the U.S., high-frequency trading firms represent 2.0% of theapproximately 20,000 firms operating today, but account for 73.0% of allequity trading volume. As of the first quarter in 2009, total assets undermanagement for hedge funds with high frequency trading strategies were$141 billion, down about 21% from their high. The high frequencystrategy was first made successful by Renaissance Technologies.(Wikipedia)

Contrast with what’s perhaps the world’s largest explicit bet.Warren Buffett versus Protege Partners, LLCStakes $1,000,000 [to charity]

Over a ten-year period commencing on January 1, 2008, and ending onDecember 31, 2017, the S&P500 will outperform a portfolio of funds ofhedge funds, when performance is measured on a basis net of fees, costsand expenses.

David Aldous The top ten things . . .

Page 33: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

Instead of regular financial markets – what do you learn by being toldMicrosoft stock is at 27? – it’s conceptually simpler to considerprediction markets in which you can bet (via trading contracts) onfuture events happening or not by a specified deadline. For instance

David Aldous The top ten things . . .

Page 34: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

David Aldous The top ten things . . .

Page 35: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

2012.REP.NOM.PALIN 16:47 https://www.intrade.com/jsp/intrade/common/c_cd.jsp?conDetailID=...

1 of 2 11/4/09 8:48 AM

2012 Republican Presidential Nominee (others upon request) Sarah Palin to be Republican Presidential Nominee in 2012

New: Request A Quote

2012.REP.NOM.PALIN

Trade

Last Trade : 23.0 1.0

Suggest An Image

Last Week Last Day

Advanced graphing

Add this chart to your website/blog

Links

Related Intrade Markets

Contract Specific Rules

Search Yahoo News

Search Google

Contract Info

Currency USD

Session lo/hi 23.0 - 23.6

Life lo/hi 3.0 - 28.8

Previous Close 24.0

Open Price 23.5

Last Price 23.0

Last Trade Time

02:58 PM, GMT

Today's Volume 13

Total Volume 8749

Market Lifetime:

Sep 5 - Dec31

BID

Qty Price

40 23.0

1 20.5

1 20.4

2 20.2

8 20.1

2 17.1

1 16.5

1 16.2

3 16.0

30 15.5

5 15.4

1 15.3

1 15.2

1 15.1

25 15.0

ASK

Price Qty

23.9 4

24.0 50

24.5 25

25.8 1

26.5 2

27.4 2

27.5 50

27.8 1

27.9 30

28.0 54

28.6 6

28.7 2

28.8 20

28.9 2

29.0 120

Sarah Palin to be Republican Presidential Nominee in 2012Sarah Palin to be Republican Presidential Nominee in 2012

Examples Margin Specs Explanation

Tick Size 0.1 Contract Start Date Oct 25, 2008

Lifetime

Contract Specs

David Aldous The top ten things . . .

Page 36: The top ten things that math probability says about the ...aldous/Top_Ten/talk_old.pdf · Chance Rules: An informal guide to probability, risk and statistics. The Jungles of Randomness.

IntroductionThe top ten list

1-34-67-8

In a prediction market, a price of 25 means there’s a consensus estimatethat the chance of the event is 25%. One can’t say whether this is the“true” chance in individual cases, but theory says if you look at manydifferent contracts with today’s price around 25, then for about 25% ofthem the event will in fact happen.

What’s interesting is there are also less obvious math calculations. Iftoday’s price is 25, math says the chance that the Palin contract pricewill sometime go above 75 but then she is not selected as candidateequals 1/12. Again one can test such predictions over many contracts.

8. The actual behavior of “closed-end” speculative markets isgenerally consistent with the predictions of an “efficient markethypothesis”, that prices reflect true probabilities.

David Aldous The top ten things . . .