“Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited...
Transcript of “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited...
![Page 1: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/1.jpg)
SocietyofPetroleumEngineers2017Dis8nguishedLecturerProgramwww.spe.org/dl
JimGouveiaP.Eng.Partner Rose&AssociatesLLP
“FooledbyRandomness”-ImprovingDecisionMakingWithLimitedData
1
![Page 2: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/2.jpg)
q Asprofessionalswearecon8nuouslychallengedtomakeinformeddecisionswithlimiteddatasets.
q Ourexploita8onofUnconven8onalresourcesina8meofbudgetrestraints,lowcommodityprices,andcompe88vepressureshasdriventhedesiretogettherightanswersassoonaspossible.
q Ourdecisionson“sweetspots”,newtechnologiesandindeednewplaysareoMenbasedsimplyonthearithme8caverageoftheresultsfromafewwells.
q Wherewehaveerredasanprofessionisinhonouringlimiteddatasetswithoutconsidera8onoftherepresenta8venessofthedata.
“Fooled by Randomness”
2
![Page 3: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/3.jpg)
Outline
q BackgroundonAggrega8onPrinciples
q ReviewAggrega8onCurvesandtheirapplica8ontolimiteddatasets
q ReviewtheuseofSequen8alAccumula8onplotsforreal8mevalida8onofourdistribu8ons
q Conclusions&Recommenda8ons
3
![Page 4: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/4.jpg)
• Thereisanequalprobabilityofrollinga1toa10.
• 90%ofthe8mewewillrealizeanoutcomethatequalsorexceeds2.
• 10%ofthe8mewewillrealizeanoutcomethatequalsorexceeds10.
• Thera8ooftheP10(high)totheP90(low)is5.
• Weknowthedistribu8onofadieisdiscreteuniform,andthatwithrepeatedtrialstheaverageoutcomewillbe5.5.
ConsideraTensidedDie
Aggrega8onPrinciples101
4
![Page 5: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/5.jpg)
• Whatisourconfidencethatwewillrealizethe
meanoutcomeof5.5,aMer1dieroll,5dice
rolls,10dicerolls?
• Whatifwedevelopedanewtechnologythat
wouldimprove“Die”performanceby20%.
• Howmanydicerollswouldweneedtoconfirm
theeffec8venessofthenewtechnology?
Aggrega8onPrinciples
0=10
5
![Page 6: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/6.jpg)
• Whatwouldyouconcludeifonyourfirst
trialofthenewtechnologyyourolleda5?
• Shouldyoufeelbeberorworseaboutthenewtechnology?
• Couldweconcludethatthetechnologyfailed?
• Letsreviewapragma8csta8s8calapproachto
providequicksolu8ons.
Aggrega8onPrinciples
0=10
6
![Page 7: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/7.jpg)
Aggrega8onPrinciples
0=10
• Wearereasonablycertainwewillrolla2ormore90%ofthe8me.TheP10:P90ra8ois5.0
• Rollfivedice.Dividesumby5,repeat.Wewillaverage2ormore99.86%ofthe8me.TheP90oftheaggregatedoutcomeis3.8TheP10:P90ra8ois1.9
• Rolltendice.Dividesumby10,repeat.TheProbabilityofaveraginga2ormoreis99.999%.ThisisnotaP90!TheP10:P90ra8ois1.6
SingleDieOutcome
FiveDiceAveragedoutcome
TenDiceAveragedoutcome
2.0 10
3.8 7.2
4.3 6.7
7
![Page 8: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/8.jpg)
NumberofTimesTheDieisRolled
WithIncreasingDiceRollsTheVarianceDecreases
P90
P10
8
![Page 9: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/9.jpg)
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
5.50
6.00
6.50
7.00
7.50
8.00
8.50
9.00
9.50
10.00
0 5 10 15 20 25 30 35 40 45 50
AverageOutcomeVa
lue
NumberofDiceNumberofTimesTheDieisRolled
Aggrega8onChartsRevealHowTheVariance
DecreasesWithIncreasingDiceRolls
AggregateP10AggregateP50AggregateP90
AverageOutcomeVa
lue
9
![Page 10: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/10.jpg)
InThisAggrega8onChartTheOutcomesAre
NormalizedasaFunc8onofTheMean
AggregateP10AggregateP50AggregateP90
Percentageofth
eMeanVa
lue
0%
20%
40%
60%
80%
100%
120%
140%
160%
180%
0 5 10 15 20 25 30 35 40 45 50
Percen
tageo
fTh
eM
ean
NumberofDiceNumberofTimesTheDieisRolled1010
![Page 11: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/11.jpg)
• Thekeydriversofeconomicvalua8onsaMerproductpricearetypically:
o Reserveso Rateo Capitalcosto CycleTime
• Aseachoftheaboveisbasedonmul8plica8veprocessesthey
canbewellfibedwithlognormaldistribu8ons,with“spiked”endmembers.
• Let’sreviewanexampleofaggrega8onusingalognormaldistribu8onfores8matedul8materecovery(EUR),onaperwellbasis.
Aggrega8onAppliedtoSubsurfaceParameters
11
![Page 12: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/12.jpg)
1 5 25
P90 P10
1Well
5-WellAverage
25-WellAverage
TheImpactofAggregaKonona5&25wellprogramEUR–MMscf
Aggrega8ngEURDistribu8ons-P10:P90Ra8oof4
12
![Page 13: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/13.jpg)
Percentageofth
eMeanVa
lue
WellCount
Thesecurvesarebasedonsamplingageologicsubsetwithaknownmean.
AggregateP10AggregateP50AggregateP90
SPE175527byMcLane&Gouveia
Ageologicsub-setisaregionofrockthathasanalogousgeologicaldeposi8onalproper8es,withrela8velyconsistentbulkrockandfluidproper8es.
Aggrega8ngEURDistribu8ons-P10:P90Ra8oof4
13
![Page 14: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/14.jpg)
Percentageofth
eMeanVa
lue
WellCount
Withlargerwellcounts,theP90andP10approachtheunderlyinggeologicsubsetmeanthatthe
Aggrega8oncurveswerebasedon.
Aggrega8ngEURDistribu8ons-P10:P90Ra8oof4
AggregateP10AggregateP50AggregateP90
14
![Page 15: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/15.jpg)
• Therealityisthatbudgetsandcompe88vepressuresforceourhandsinmakingdecisionswithlimiteddata.
• Understandingtheinherentuncertaintyinourdataisnotintendedtopreventdecisionmaking.
• Thegoalshouldbeabeberunderstandingoftheinherentuncertaintyinourdatasetsandthenmakingdecisionswithknowledgeoftheirrepresenta8veness.
• Let’sreviewhowAggrega8oncurveswillguideus.
Aggrega8onPrinciples
15
![Page 16: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/16.jpg)
• NorthAmericanexperiencehasdemonstratedahighdegreeofcongruenceinP10:P90ra8osforhorizontalwellswithcommonhorizontallengthsandcomple8ons.
• P10:P90ra8osof4to5arecommonforasingleOperatorwitha
consistentcomple8ontechniqueinlateralsof5,000feet(1500+m)and20+fracturestages.
• P10:P90ra8osof2to3arebeingexperiencedwhenlateralsaredrilledusing3DSeismicdrivenGeo-steeringandconsistentcomple8ontechniqueinlateralsof3000m+.
• Thetechniquerequiresustoassumelognormalityandthevariancefromanalogousreservoirswithsimilarhorizontalwelllengthsandcomple8ontechniques.
Applica8onofAggrega8onCurves
16
![Page 17: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/17.jpg)
• Resourceplaysshowrepeatabledistribu8ons,yearoveryearforagivengeologicsubset(SocietyofPetroleumEvalua8onEngineersMonograph3).
• Caveatstothisapproach:o Horizontalwelllengthisconsistentornormalized.o Drillingandcomple8ontechniquesareanalogous.o Wearereasonablycertainthatthe“averaged”geologydoesnotvarysignificantlywithinthegeologicsubset.
• Inemergingplaystheaggrega8oncurvescanbeusedtoboundtherangeofthegeologicsubsetmeanasafunc8onofwellcount.Acri8calinsightforearlydecisionmaking.
Applica8onofAggrega8onCurves
17
![Page 18: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/18.jpg)
Aggrega8onDeriva8veApplica8onsAggrega8onCurves(TrumpetPlots)–Usedtoillustratereduc8oninuncertainty(variance)asafunc8onofincreasingwellcountusingknownanalogs• Func8onofAnaloguncertainty(P10:P90ra8o),andwellcountAggrega8onCurves–Reverseengineeredtodeterminetherangeofthemeanbasedonaknownsamplesizeandarithme8cmean• VariancebasedonAnalogs
ConfidenceCurves–Usedtocommunicateconfidenceinoutcomes• BasedonmeanandvarianceinProduc8onTypeCurves.• HelpsdefinePre-developmentstagegatethresholdsbasedonanalogdata• Confidenceisafunc8onofuncertainty(P10:P90ra8o),yourtarget(objec8ve)andthenumberofwellstobedrilled(samplesize).PilotDesign–HowmanywellsandwhataveragerateSequen8alAccumula8onPlots–Usedtoassessvalidityofforecasts• TracksactualsagainsttheforecastedP10andP90oftheaggrega8oncurves• Providesearlyfeedbackaboutthevalidityoftheforecastedparameter
![Page 19: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/19.jpg)
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
150%
160%
170%
180%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Percentageofth
eMeanVa
lue
WellCount
AggregateP10AggregateP50AggregateP90
87%
111%
Applica8onofAggrega8onCurves–KnownMeanof1,200Bopd
DetermininguncertaintyinthemeanbasedonsamplesizeP10:P90=4
TheP50~TheMean=1,200
A25wellprogramwilldelivera:P10=1,332bopdP50=1,200bopdP90=1,056bopd
![Page 20: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/20.jpg)
Aggrega8onDeriva8veApplica8onsAggrega8onCurves(TrumpetPlots)–Usedtoillustratereduc8oninuncertainty(variance)asafunc8onofincreasingwellcountusingknownanalogs• Func8onofAnaloguncertainty(P10:P90ra8o),andwellcountAggrega8onCurves–Reverseengineeredtodeterminetherangeofthemeanbasedonaknownsamplesizeandarithme8cmean• VariancebasedonAnalogs
ConfidenceCurves–Usedtocommunicateconfidenceinoutcomes• BasedonmeanandvarianceinProduc8onTypeCurves.• HelpsdefinePre-developmentstagegatethresholdsbasedonanalogdata• Confidenceisafunc8onofuncertainty(P10:P90ra8o),yourtarget(objec8ve)andthenumberofwellstobedrilled(samplesize).PilotDesign–HowmanywellsandwhataveragerateSequen8alAccumula8onPlots–Usedtoassessvalidityofforecasts• TracksactualsagainsttheforecastedP10andP90oftheaggrega8oncurves• Providesearlyfeedbackaboutthevalidityoftheforecastedparameter
![Page 21: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/21.jpg)
Falher‘H’–PeakDailyGasRateofTheFirst24Wells
P90 = 5. 6 MMscf/d P50 = 11.2 MMscf/d P10 = 22.5 MMscf/d Arithmetic Avg =12.8 MMscf/d P10:P90 ratio = 4
q Based on the 24 well sample we observe that the distribution is well fit with a lognormal distribution.
q The P90 and P10 of a randomly sampled individual well is 5.6 and 22.5 MMscf/d respectively.
q The arithmetic mean of the 24 well sample is 12.8 MMscf/d.
What is the uncertainty in the mean of this Geologic subset given the 24 well arithmetic mean of 12.8 MMscf/d?
21
![Page 22: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/22.jpg)
18
Percentageofth
eMeanVa
lue
WellCount
AggregateP10AggregateP50AggregateP90
22
DeterminingTheUncertaintyinTheMean
-BasedonSampleSize
![Page 23: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/23.jpg)
Percentageofth
eMeanVa
lue
WellCount
AggregateP10AggregateP50AggregateP90
Percentageofth
eMeanVa
lue
DeterminingTheUncertaintyinTheMean
-BasedonSampleSize
BasedontheP10:P90ra8oof4aggrega8oncurve,90%of
the8methegeologicsubsetmeanwillbe86%ormore
ofthe24wellarithme8csamplemean.
24
86%
2323
![Page 24: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/24.jpg)
Percentageofth
eMeanVa
lue
WellCount
AggregateP10AggregateP50AggregateP90
Percentageofth
eMeanVa
lue
Basedonthisaggrega8oncurve,only10%ofthe8me
willthegeologicsubsetmeanbe115%ormoreofthe24
wellarithme8csamplemean.
24
115%
DeterminingTheUncertaintyinTheMean
-BasedonSampleSize
2424
![Page 25: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/25.jpg)
Percentageofth
eMeanVa
lue
WellCount
AggregateP10AggregateP50AggregateP90
Percentageofth
eMeanVa
lue
14.7
11
12.8
86%
115%
AMer24wells,weare80%confidentthatthegeologicsub
setmeanwillbebetween11and14.7MMcfd.
DeterminingTheUncertaintyinTheMean
-BasedonSampleSize
2525
![Page 26: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/26.jpg)
P90 = 5.56 MMcf/d P50 = 11.18 MMcf/d P10 = 22.46 MMcf/d P10:P90 ratio = 4 Avg. = 12.8 MMcf/d
Falher‘H’–PeakDailyGasRate
Mean ~ P40
80% confident that the geologic sub-set mean will be between 11 and 14.7 MMcf/d
26
![Page 27: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/27.jpg)
• Employershaveanexpecta8onthattheirprofessionalscanforecasttheresultsoffutureprogramsbasedonpriorresults.
• Withincreasedsamplesizethearithme8cwellaveragewillconvergeonthetruegeologicalsubsetmean.Withlimitedwells,thebestwecandoisevaluatetheuncertaintyinthemeanofthesubset.
Forecas8ngBasedonLimitedSamples
Mean=12.8MMcfd
MMcfd
2727
![Page 28: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/28.jpg)
• E&P professionals often ignore the uncertainty in the mean value. As a consequence forecasted aggregation will converge on the mean of the sampled wells.
• This simple aggregation does not honour the irreducible uncertainty based on the original 24 well sample set.
Aggregation Curve Application –
Arithmetic Mean = 12.8 MMcfd
28 28
![Page 29: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/29.jpg)
Aggrega8onDeriva8veApplica8onsAggrega8onCurves(TrumpetPlots)–Usedtoillustratereduc8oninuncertainty(variance)asafunc8onofincreasingwellcountusingknownanalogs• Func8onofAnaloguncertainty(P10:P90ra8o),andwellcountAggrega8onCurves–Reverseengineeredtodeterminetherangeofthemeanbasedonaknownsamplesizeandarithme8cmean• VariancebasedonAnalogs
ConfidenceCurves–Usedtocommunicateconfidenceinoutcomes• BasedonmeanandvarianceinProduc8onTypeCurves.• HelpsdefinePre-developmentstagegatethresholdsbasedonanalogdata• Confidenceisafunc8onofuncertainty(P10:P90ra8o),yourtarget(objec8ve)andthenumberofwellstobedrilled(samplesize).PilotDesign–HowmanywellsandwhataveragerateSequen8alAccumula8onPlots–Usedtoassessvalidityofforecasts• TracksactualsagainsttheforecastedP10andP90oftheaggrega8oncurves• Providesearlyfeedbackaboutthevalidityoftheforecastedparameter
![Page 30: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/30.jpg)
BuildingConfidenceCurvesfromAggrega8on-Confidenceofexceeding140targetvswellcount
1well50%outcomes>140
10wellaggregate76%outcomes>140
20wellaggregate86%outcomes>140
5wells67%outcomes>140
#wells probof>140BOPD
1 50%5 67%10 76%20 86%
![Page 31: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/31.jpg)
5 10 20
30
1020
908070605040
Confi
dence-%
0
10
20
30
40
50
60
70
80
90
100
0 5 10 15 20 251NumberofWellsinProject
#wells probof>140BOPD
1 50%5 67%10 76%20 86%
BuildingConfidenceCurvesfromAggrega8on-Confidenceofexceeding140targetvswellcount
![Page 32: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/32.jpg)
VariableRateConfidenceCurves-Confidenceofexceedingatargetvswellcount
5 10 20
30
1020
908070605040
1NumberofWellsinProject
175
150
140125
100
BuiltusingCrystalBallExcelfunc8on“CB.GetCertaintyFN”
Confi
dence-%
Toincreasetheconfidenceofdelivery:• Increasenumberofwells• Reducetarget
![Page 33: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/33.jpg)
Aggrega8onDeriva8veApplica8onsAggrega8onCurves(TrumpetPlots)–Usedtoillustratereduc8oninuncertainty(variance)asafunc8onofincreasingwellcountusingknownanalogs• Func8onofAnaloguncertainty(P10:P90ra8o),andwellcountAggrega8onCurves–Reverseengineeredtodeterminetherangeofthemeanbasedonaknownsamplesizeandarithme8cmean• VariancebasedonAnalogs
ConfidenceCurves–Usedtocommunicateconfidenceinoutcomes• BasedonmeanandvarianceinProduc8onTypeCurves.• HelpsdefinePre-developmentstagegatethresholdsbasedonanalogdata• Confidenceisafunc8onofuncertainty(P10:P90ra8o),yourtarget(objec8ve)andthenumberofwellstobedrilled(samplesize).PilotDesign–HowmanywellsandwhataveragerateSequen8alAccumula8onPlots–Usedtoassessvalidityofforecasts• TracksactualsagainsttheforecastedP10andP90oftheaggrega8oncurves• Providesearlyfeedbackaboutthevalidityoftheforecastedparameter
![Page 34: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/34.jpg)
IndesigningPilotsthekeyques8onsshouldbe:• Howmanywellsshouldbeinourpilot?• Whatconfidencelevelinthedatadowerequire?• Whatratedoweneedtovalidatebeforeproceedingtothenextstageofdevelopment?
LetsreviewanexampleofPilotdesign
DecisionMakinginUnconven8onalReservoirs
![Page 35: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/35.jpg)
PilotDesign:Step1–BuildAnalogDistribu8ons
TheperwellPeakratedatacanbepresentedviathelogcumula8veprobabilityplotorafrequencyplot.
140bopdistheP50ofthedistribu8on.Weknowthat50%ofthe8meanewwellwillmeetorexceedthattarget.50%ofthe8meanewwellwillsamplelessthanorequalto140bopd.
![Page 36: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/36.jpg)
• Thesevaluesaresetbythefirm’sleadership.ForexampleallnewplaysmustprovideafullcycleRORof15%ormore.
• TheAssetteamwilldevelopacurveofdailyaverageproduc8onratedividedbypeakrateversus8mefortheirselectedanalog.
• Thepeakrateisthenreducedinthedevelopmentmodelun8lthefullcycleeconomicreturnequalstherequired15%ROR.
• ThisisthereverseengineeredPeakratethatwewilltestagainstinourPilotdesign.
• Wewilluseabreak-evenrateof100bopdtobuildanexample.
PilotDesign:Step2–ReverseEngineeraMinimumEconomicTargetofPeakRateinbopd.
![Page 37: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/37.jpg)
• Howmanywells,andatwhatperwellaveragepeakratewillourPilotneedtoproducetovalidatea50%and90%confidencethatthenewzonewillmeetorexceedManagement’sexpecta8ons
• Ifdecisionsaretobemadeonan50or90%confidenceinterval,wewillneedtoreverseengineertherequired“Pilottarget”basedonanaveragedperwelloutcomefromourpilot.
• Wewillneedtoassumelognormalityandthevariance(P10:P90ra8o)basedonourbestavailableanalog(s).
• The“Up8ck”ofthe“break-evenrateof100bopdwillbeafunc8onofthepilotwellcurveandthederivedvaluefromtheP90andP50aggrega8oncurves.
PilotDesign:Step3
![Page 38: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/38.jpg)
• EntertheP90andP50Aggrega8oncurvesatthenumberofpilotwellsbeingevaluated.Readofftherespec8ve%ofthemeanvalues.
• TheP50=97%ofthemean
• TheP90=70%ofthemean
PilotDesign:Step4
PercentageoftheM
eanVa
lue
WellCount
AggregateP10AggregateP50AggregateP90
70%
97%
• Tobe50%confidentthatourdevelopmentwillmeetorexceedtheminimumthresholdvalueof100bopd,the5wellpilotaveragemustmeetorexceed(100/0.97)103bopd.
• Tobe90%confidentthatourdevelopmentwillmeetorexceedtheminimumthresholdof100bopd,the5wellpilotaveragemustmeetorexceed(100/0.7)142.9bopd.
![Page 39: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/39.jpg)
PilotDesign-The“NoRegrets”Rates
• Wedeterminedthatweneedtorealizeanaveragerateofatleast103bopd/welltobe50%confidentthatthenewareawouldrealizeabreakevenROR.Ques8onis,whatifyoutested100bopd,wouldyouwalkknowingthereisclosetoa50%chanceyouwillbeembarrassed?
• Thisbringsaboutaconsidera8onofwhatwewillrefertoasthe“NoRegretsRate”.Thatproduc8onrateatwhichyouwouldhavenoregretswalkingaway,knowingthatitmightactuallywork.Therearetwokeyfactorstoconsider:
o Determininganoregretsvolumeonabreak-evenbasis
o Determiningthelikelihoodoftheplaymee8ngyoudesiredeconomichurdlesgiventhatitfailedtomeetyour50%breakevenrate.
![Page 40: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/40.jpg)
Determininganoregretsvolumeonabreak-evenbasis
• Weneedafrankdiscussionwithourdecisionmakersbeforethewellresultsandouremo8onscomeintoplay.
• Youwillneedtohaveaddi8onalaggrega8oncurvestoshowa25%confidencelevelplusothervaluestheymayrequest.R&Arecommendstheaddi8onofaP25andP10value.
• Rerunthenumbersusingyourcorporatehurdleratesinaddi8ontothe“break-even”RORvalues.
PilotDesign-The“NoRegrets”Rates
![Page 41: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/41.jpg)
PilotDesign-“ChallengesinPilo8ngNewTechnology”
• Baseforecastuncertaintyo InSAGDwellwehaveobserveda+/-20%varianceo AtthePadlevelthiscandecreaseto+/-5%o Measurementprora8onissuesamplifiedatthewelllevel
• Timedelaytoseeimprovementso Impactofforecastuncertaintyiscompounded
o Impa8ence
• Incrementalchangesaresmallrela8vetootheruncertain8es
![Page 42: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/42.jpg)
Aggrega8onDeriva8veApplica8onsAggrega8onCurves(TrumpetPlots)–Usedtoillustratereduc8oninuncertainty(variance)asafunc8onofincreasingwellcountusingknownanalogs• Func8onofAnaloguncertainty(P10:P90ra8o),andwellcountAggrega8onCurves–Reverseengineeredtodeterminetherangeofthemeanbasedonaknownsamplesizeandarithme8cmean• VariancebasedonAnalogs
ConfidenceCurves–Usedtocommunicateconfidenceinoutcomes• BasedonmeanandvarianceinProduc8onTypeCurves.• HelpsdefinePre-developmentstagegatethresholdsbasedonanalogdata• Confidenceisafunc8onofuncertainty(P10:P90ra8o),yourtarget(objec8ve)andthenumberofwellstobedrilled(samplesize).PilotDesign–HowmanywellsandwhataveragerateSequen8alAccumula8onPlots–Usedtoassessvalidityofforecasts• TracksactualsagainsttheforecastedP10andP90oftheaggrega8oncurves• Providesearlyfeedbackaboutthevalidityoftheforecastedparameter
![Page 43: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/43.jpg)
• Withsuchalargedegreeofinnateuncertaintyhowdoweassureourmanagementteamthatourprogramsareontrack?
• WecanusetheAggrega8oncurvestodetermineour80%confidenceintervalsasafunc8onofwellcount.
• Byployngouractualresultsagainstthe80%confidencebandswearegenera8ngwhatarereferredtoas“Sequen8alAccumula8onPlots”.
• Thisgraphicalapproachprovidesanearlyindica8onofpossibleissuesandfacilitate“real-8me”earlydecisionmaking.
MakingBeberDecisionsBasedonLimitedData
![Page 44: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/44.jpg)
Sequen8alAccumula8onPlots(SAP)
• SAPPlotsarebasedontheAggrega8onCurves• Ratherthanpresen8ngdataastheaveragewellinamul8plewellprogram,thedataispresentedasthetotalP10-P50-P10valueforthefullwellcount
• Usefulforcomparingactualtoexpectedoutcomes– Sothatexpecta8onscanbeadjustedand,ifnecessary,theprogramcanbehaltedifunderperforming
![Page 45: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/45.jpg)
Accumula8ng
1
2
3
4
6
P10:
P90RATIO
WeconverttheAggrega8oncurve,whichpresentstheaveragewellasafunc8onofwellcount,totheaccumula8ngtotals,inthisplot,bymul8plyingthe%ofthemeanfromtheaggrega8oncurvebythemeanvalueandbythetotalnumberofwells.
5
WellCount
Accum
ula8
ngM
Mcfd
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
150%
160%
170%
180%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
DeterminingTheUncertaintyinTheMeanBasedonSampleSizeP10:P90 =4(P1,P99 spike)
Percen
tageofthe
MeanVa
lue
WellCount
AggregateP10AggregateP50AggregateP90
![Page 46: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/46.jpg)
Accumula8ng
P10:P90Ra8o
1
2
3
4
5
6 P10:
P90RATIO
46
Sequen8alAccumula8onPlot-P10:P90Ra8o=4
![Page 47: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/47.jpg)
Thebluebarsareeachindividualwell’speakdailygasrateusingtherighthandYaxis.
Accum
ulaK
ngM
Mcfd Increm
entalMMcfd
4747
PeakDailyGasRate-Falher‘H’
![Page 48: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/48.jpg)
CaseStudy–NorthernMontney,WCSBTheIni8alAnalogProduc8onTypeCurveWasBasedonRegionalHeritageMontney.IsThisTypeCurveRepresenta8ve?
GreendotsareMontneyHorizontalsattheendof2012.
NorthernMontneyArea
RegionalMontneyArea
48
![Page 49: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/49.jpg)
AnalogueBest3Mo.Avg.GasRate
BasedonRegionalHeritageMontneyselectedanaloguewells
P10/P90=4.2
49
![Page 50: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/50.jpg)
Sequen8alAccumula8onPlot
ForecastbasedonRegionalHeritageAnaloguewells.ActualsbasedoncasestudyOperator’s Northern Montney wells coming on-stream from the period from08-2009through12-2009(nextwellon-streamin07-2010).
50
![Page 51: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/51.jpg)
Forecast based on Regional Heritage Analogue wells. Actuals based on casestudy Operator’s NorthernMontneywells coming on-stream from the periodfrom08-2009through03/2011(nextwellson-streamin08/2011).
51
Sequen8alAccumula8onPlot
![Page 52: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/52.jpg)
ComparisonofBest3Mo.Avg.GasRate
Basedonactualresults-to-datefromcasestudyOperator’sNorthernMontneyrevisedistribu8on to reflect mean from case study Operator’s Northern Montney wellswithsamevariance(P10/P90=4.2)fromanaloguewells.
Mean=3,206
Mean=4,137
52
![Page 53: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/53.jpg)
Sequen8alAccumula8onPlot(RevisedMeantoReflectResults-To-Date)
ForecastbasedonrevisedmeanusingcasestudyOperator’sNorthernMontneyactualresults-to-date.ActualsbasedoncasestudyOperator’sNorthernMontneywellscomingon-streamfromtheperiodfrom08-2009through03/2011.
53
![Page 54: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/54.jpg)
54
57
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
CumulativeRa
te-MMcfd
WellCount
Actual ForecastP90 ForecastP50 ForecastP10 CumActual
Actuals-MMcfd
4
6
2
8
1
5
7
3
Sequen8alAccumula8onPlot(RevisedMeantoReflectResults-To-Date)
ForecastbasedonrevisedmeanusingOperator’sNorthernMontneyfirst31wells.Actualsarefromthenext16wellsdrilledwithrevisedtechnology.
54
![Page 55: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/55.jpg)
HowtoErode$1BillioninShareholderValue
Forecastsarebasedonthebreakeventypewellcurve
AllCaseStudyOperator’sWells
![Page 56: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/56.jpg)
• AsProfessionalswetendtorelyontheobserveddatawithoutacknowledgingandunderstandingtherepresenta8venessofthesampleddata.
• Allowingsta8s8cstospeakforthemselvesrequireslargewellcountsthatareoMennotprac8calinhighcostcompe88veplays.
• Aggrega8oncurvesarepragma8capproachesthatprovideinsigh{ulillustra8onsoftheinnateuncertaintyinourlimiteddatasets.
• Theobservedvarianceindrillingprogramsdoesnotalwaysimplythatthingsarechanging.
Conclusions
56
![Page 57: “Fooled by Randomness”...“Fooled by Randomness” - Improving Decision Making With Limited Data 1 q As professionals we are con8nuously challenged to make informed decisions](https://reader034.fdocuments.net/reader034/viewer/2022042100/5e7d24cf9122c672583e89fb/html5/thumbnails/57.jpg)
SocietyofPetroleumEngineers2017Dis8nguishedLecturerProgramwww.spe.org/dl
JimGouveiaP.Eng.Partner Rose&AssociatesLLP
“FooledbyRandomness”-ImprovingDecisionMakingWithLimitedData
57