How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04...

33
How risky is the software you use? https://shmoo18.cyber-itl.org Cyber Independent Testing Lab { Sarah Zatko , Tim Carstens , Patrick Stach , Parker Thompson , mudge } @ CITL

Transcript of How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04...

Page 1: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Howriskyisthesoftwareyouuse?

https://shmoo18.cyber-itl.org

CyberIndependentTestingLab

{SarahZatko,TimCarstens,PatrickStach,ParkerThompson,mudge}@CITL

Page 2: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

WeareCITL

• Anon-profitorganizationbasedinUSA• FoundedbySarahZatko &mudge

• Mission:toimprovethestateofsoftwaresecuritybyprovidingthepublicwithaccuratereportingonthesecurityofpopularsoftware

• FundingfromtheFordFoundation• PartnerswithConsumerReportshttps://www.consumerreports.org&TheDigitalStandardhttps://thedigitalstandard.org

Page 3: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the
Page 4: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Somethinglikethis,butforsoftwaresecurity.

Page 5: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Howdoyoudothisforsoftwaresecurity?

Page 6: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the
Page 7: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Scores&Histograms

HardenedGentoo

Ubuntu16LTS

SamsungUN55KS9000

LG55UH8500

Page 8: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

SecurityToday:

Youcanleadthepackbymasteringthefundamentals.

Visio LG Samsung UbuntuP55-E1 49UJ7700 UN55KS9000 16.04

#binaries 504 1740 4243 4991aslr 98% 67% 80% 100%

stackDEP 99% 99%* 99%* 99%64bit 0% 0% 0% 98%RELRO 100% 4% 9% 96%

stackguards 68% 1% 57% 79%fullyfortified 7% 0% 6% 11%

partialfort 43% 1% 37% 42%

hasgood 3% 3% 25% 4%hasrisky 68% 66% 67% 67%hasbad 28% 34% 23% 28%hasick 3% 5% 5% 3%

Page 9: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Ourgoals

1. Remainindependentofvendorinfluence2. Automated,comparable,quantitativeanalysis3. Actasauserwatchdog

• Non-goal:findanddisclosevulnerabilities• Non-goal:tellsoftwarevendorswhattodo• Non-goal:performfreesecuritytestingforvendors

Page 10: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Threebigquestions

1. Whatworks?

2. Howdoyourecognizewhenit’sbeingdone?

3. Who’sdoingit?

Page 11: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Thebasicidea

Page 12: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

InformationTheoryPerspective

• Givenapieceofsoftware,wecanask1. Overall,howsecureisit?2. Whatareallofitsvulnerabilities?

• (1)appearstoaskforless-infothan(2)

• OurQuestion:Developanheuristicwhichcanefficientlyanswer(1)butnotnecessarily(2)

Page 13: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

StepOne:StaticMeasurements

• Complexity• Functionscalled• Safetyfeatures

Yearsinthefieldgiveusagoodstartingpoint– lookforthesamethingswe’dlookatwhentryingtopickasofttargettoexploit.

But,thisfielddoesn’tknowenoughaboutimpact/effectivenessofbestpractices.

Page 14: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

EarlyPromise

Browser “Underground”ExploitPrice

MicrosoftEdge $80,000

GoogleChrome $80,000

AppleSafari $50,000

MozillaFirefox $30,000

Page 15: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Step2:Fuzzing!Lotsofit.

• Fuzzingprovidesatestable,recognizedwaytoroughlymeasuresoftware’s“security”• Themorerobustsoftwareiswhenfuzzed,thelesslikelyitistobeexploitable• Ifwecouldfuzzeverything,wewouldn't’tevennecessarilyneedtheheuristics• Butwecan’t,so

Page 16: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Step3:Profit!Bayes!(1/3)

• Forsomesoftwares,weknowthatwecan’tcomputeP(s issecure)

• Asasurrogate,wecancomputeprobabilitiesofdifferentfuzzingoutcomes,like:

Ph,k =P(h unitsoffuzzingagainsts yields<k uniquecrashes)

Page 17: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Step3:Profit!Bayes!(2/3)

• Fuzzingisexpensive,sowe“goBayesian”• LetM beanobservablepropertyofsoftware

• Examples:iscompatiblewithRELRO,has“lowcomplexity,”etc• Forrandoms inS,considertheconditionalprobabilities

Ph,k(M) =P(h fuzzingons yields<k uniquecrashes|M istrueofs )

• Whatwewant:WhichM havePh,k(M)>0.5 forlargelog(h)/k?

Whichindicators(M)canbeusedtopredictfuzzingperformance?

Page 18: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Step3:Profit!Bayes!(3/3)

Indicatorsmightnotbecausal,andthat’sOK:

• ItcouldbethatM’spresenceliterallypreventscrashes

• ButitcouldalsobethatM ismostlyonlyfoundinsoftwarewrittenbyteamswhoshipreliablesoftware

• Ifyou’relookingforsecurity,whatdifferencedoesitmake?

Page 19: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

IndicatorMinerals

Wanttofind:• Diamond (USGeologicalSurvey)

Lookfor:• Garnet

(Moha112100@Wikipedia)

• Diopside(RobLavinsky)

• Chromite(WeinrichMinerals,Inc.)

Page 20: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Step4:Reports

Whileweworkongatheringdataanddevelopingourmodel,we’realso•Developingreports•BuildingrelationshipswithpartnerorganizationslikeConsumerReports• Lookingforsecurityorgstosharedatawith

Page 21: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

TheProgressionofCITLTech

Static(Prototype)

Static(Extensible)

AFL CITL-fuzz NEWFUZZERToday

FirstData Firstreports FinalModel&Reports

Page 22: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

AppliedStaticAnalysis

• Lotsofarchitectures:x86-*,ARM-*,MIPS-*• Lotsofoperatingsystems:Windows,Linux,OSX• Lotsofbinaryformats:PE,ELF,MachO• Eachwiththeirownapp-armoringfeatures

• Lotsofversionsofeachoftheabove!

Page 23: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

OSComparisons• Windowslagsinstackguards,buthasgoodusageofCFI• LinuxdoesmoresourcefortificationthanOSX• Windowshasthebestfunctionhygiene• Linux’sfunctionhygieneisslightlyworsethanOSX’s

Ubuntu Windows OSX16.04 10 10.13.1

64bit 97% 66% 77%aslr 100% 99% 100%dep 99% 98% 100%

stack_guards 79% 40% 73%fullyfortified 11% 2%

partialfort 42% 33%cfi 92%

good 4% 19% 29%risky 67% 30% 60%bad 28% 3% 24%ick 3% 0% 2%

Page 24: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

LinuxBrowsers– Ubuntu16.04• Scoresareallveryclose,Firefoxwinsbyanoseinstaticanalysis• Chrome’ssandboxisn’tfactoredintoscoreyet

• Allhaveinconsistentfunctionhygiene• OperatakesahitforlackofRELRO• Chromelagsbehindinfortificationuse

Chrome Firefox Operaversion 63.0.3239.13257.0.4 50.0.2762.4564bit 100% 100% 100%aslr 100% 100% 100%dep 100% 100% 100%relro 86% 100% 11%

stack_guards 86% 87% 100%partialfortification 29% 70% 56%

functionsgood 12% 4% 22%risky 86% 91% 100%bad 62% 61% 89%

scores5th% 35 64 4350th% 58 78 4895th% 71 86 65

Page 25: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

OSXBrowsers• FirefoxandOperahadallbinaries64bitwithASLR,StackDEP• Firefoxalsomademostuseofstackguardsandfortification• ChromeistheonlyonetoenableHeapprotectionflag• Safariisn’tusingsourcefortificationmuch• Scoresareveryclose,allnear95thpercentileforHighSierra(71)• SamegeneraloutcomeasinLinux

Chrome Firefox Opera Safari63.0.3239.13257.0.4 50.0.2762.4511.0.1

count 9 19 8 2564bit 89% 100% 100% 88%aslr 89% 100% 100% 100%dep 100% 100% 100% 100%heap 11% 0% 0% 0%

stack_guards 78% 95% 88% 68%partialfortification 33% 47% 38% 4%

good 33% 37% 25% 8%risky 89% 95% 100% 44%bad 44% 68% 38% 8%

scores5th% 33 43 38 2450th% 51 56 51 5195th% 63 71 63 64

Page 26: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

Windows10Browsers• Scoresareveryclose,butEdgewinsbyahair• 95th percentileis64forWin10

• Chromehasmore32bitbinariesthantheothers• Edgeistheonlyonewith100%CFI• ChromeandOperadobetteronstackguards• Firefoxtakesahitbecauseitexcelsinneither,hasmoreriskyfunctions

Chrome Edge Firefox Operaversion 63.0.3239 41.16299 57.0.4 50.0.2762count 31 7 31 1664bit 62% 100% 94% 100%dep 100% 100% 100% 100%aslr 100% 100% 100% 100%cfi 13% 100% 13% 38%

stackguards 94% 57% 61% 94%functions

good 0% 0% 3% 0%risky 9% 0% 16% 0%bad 9% 0% 0% 0%

scores5th% 23 44 7.5 4450th% 44 64 44 4495th% 64 64 44 64

Page 27: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

OSXTimeProgression• Lookedatfourversionsfrom10.10.5through10.13.1• 7.7%increaseinpercentofbinariesthatare64bit• 2%increaseinuseofstackguards,goodfunctions• HeapprotectiondecreasecorrelateswithASLRincrease?• HighSierrashowssignificantdecreasein#ofbinaries(~400fewer)

OSX OSX OSX OSX total10.10.5 10.11.6 10.12.6 10.13.1 change

#binaries 6449 6456 7017 662264bit 69% 71% 73% 77% +8aslr 99% 99% 100%* 100%* +1

heap 5% 5% 4% 4% -1stack_guards 71% 71% 72% 73% +2

goodfunctions 27% 27% 27% 29% +2riskyfunctions 62% 62% 60% 60% -2badfunctions 25% 25% 24% 24% -1

Page 28: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

SafariTimeProgression• NewbinariesintroducedinHighSierragenerallydecreasedperformance• Overallincreasesin64bitandstackguards,butnotconsistently• Functionhygienegotabitworse,especiallyinHighSierra• PartialsourcefortificationintroducedinHS

Safari totalinOSX 10.10.5 10.11.6 10.12.6 10.13.1 change

#binaries 9 13 22 2564bit 83% 92% 86% 88% +5*

stack_guards 67% 69% 73% 68% +1partialfortification 0% 0% 0% 4% +4

goodfunctions 17% 15% 9% 8% -9risky 50% 36% 38% 44% -6*bad 0% 8% 5% 8% +8

Page 29: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

MiningUsefulSpectre Gadgets• FocusonBTBpoisoningakaVariant2widgets• UseDFAtolocatethispattern:• Opreg1,[base(+index)]• BaseorIndexeitherattackercontrolledorusefuldata

• …(anythingthatdoesn’tdestroydatainreg1)• Op[base(+index)],reg2orOpreg2,[base(+index)]• Wherebaseorindexarereg1

• Tl;dr:load,loadorstore

Page 30: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

MiningUsefulSpectre Gadgets

Page 31: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

CITL:Impact

• We’vebeenreportingbugs• FirefoxonOSXwasmissingASLR(theyfixeditquick!)• Severalpatches&bugssubmittedtoLLVM&Qemu

• We’veinspiredothers• Bigshout-outtotheFedoraRedTeam

• We’vepartneredtocoverbroaderdomains• ConsumerReportshttps://www.consumerreports.org

• TheDigitalStandardhttps://thedigitalstandard.org

Page 32: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

CITL:TodayandTomorrow

• Wearebuildingthetoolingnecessarytocomputethesurrogatesecurityscoresat-scale

• Inthemeantime,ourstaticanalyzersarealreadymakingsurprisingdiscoveries:seeourrecenttalksatDEFCON/Blackhat

• Advicetosoftwarevendors:MakesureyoursoftwareemployseveryexploitmitigationourTABhaseverheardof!

Page 33: How risky is the · risky 67% 30% 60% bad 28% 3% 24% ick 3% 0% 2%. Linux Browsers –Ubuntu 16.04 •Scores are all very close, Firefox wins ... and Tomorrow •We are building the

https://shmoo18.cyber-itl.org

CyberIndependentTestingLab

{SarahZatko

,TimCarstens

,ParkerThompson

,PatrickStach

,mudge

}@CITL


Risky actions & Risky places - sakraplatser.abe.kth.se

Risky actions & Risky places - sakraplatser.abe.kth.se

Акция Фармаси В Десятку 16.04 30.04

Акция Фармаси В Десятку 16.04 30.04

Tox-Ick Monster

Tox-Ick Monster

Ick’ words

Ick’ words

Circuito quarto de milha 16.04

Circuito quarto de milha 16.04

The ick Family Set - Carl's Corner CD Files/Toons...ick Family Word Slide ick ick Family Word Slide (Sound Blending) Print on vellum, cut, and laminate for durability. Cut the top

The ick Family Set - Carl's Corner CD Files/Toons...ick Family Word Slide ick ick Family Word Slide (Sound Blending) Print on vellum, cut, and laminate for durability. Cut the top

R ICHARD D ICK PEASLEE

R ICHARD D ICK PEASLEE

New Doc 04-09-2020 16.04

New Doc 04-09-2020 16.04

instalacion ubuntu 16.04

instalacion ubuntu 16.04

16.04 part 2

16.04 part 2

Risky Deal Risky Business

Risky Deal Risky Business

CamScanner 06-20-2020 16.04

CamScanner 06-20-2020 16.04

Count down to Mahara 16.04

Count down to Mahara 16.04

Revista ICK

Revista ICK

Fig. 16.04

Fig. 16.04

Sucking Ppt 16.04

Sucking Ppt 16.04

Catalog ORIFLAME C06/2012 > 16.04 – 06.05.2012

Catalog ORIFLAME C06/2012 > 16.04 – 06.05.2012

Kasper Ohm un ick - foruq.com

Kasper Ohm un ick - foruq.com

Neuerscheinungen zum 16.04 - naxos.de

Neuerscheinungen zum 16.04 - naxos.de

SIGUIENDO OBY ICK

SIGUIENDO OBY ICK