Sans Analyst Program Vmware09

8/9/2019 Sans Analyst Program Vmware09

1/13

Sponsored by VMware

IT Audit for the Virtual

EnvironmentA SANS Whitepaper September 2009Written by: J. Michael Butler and Rob Vandenbrink

Introduction:

It All Boils Down

to PII

Similarities and

Differences

Practical

Applications


2/13

SANS Analyst Program 1 IT Audit for the Virtual Environment

Introduction: It All Boils Down to PII

Industryrequirements, governmentagency directives, andederalandstatedisclosure laws

(startingwithCaliorniasSB1386)haveonegoal incommon:Protect personal and privateinformation.Itreallydoesntmatterwhetherwearetalkingaboutcreditcardinormation,

bankaccountnumbers,socialsecuritynumbers,healthdataorinsuranceinormation.Inact,

insteadopersonalinormation,someorganizationsareocusedonprotectingutilityinrastruc -

tures,suchaspowerplants,telecommunications,orgaslines.Althoughtheinormationrequir-

ingprotectioninsuchacaseisnotpersonal,thesamesecurityandauditprinciplesstillapply.

So,toachievecompliance,ITgroupscheckpoliciesandproceduresagainstrules,regulations,

and directives. They ollow best practices and builddeense-in-depth. IT auditors, SAS70

auditors,andPCIQSAs(QualiedSecurityAssessor)meetwiththeoperationsteams,whose

responsesshowthattheyare,indeed,compliantthatis,untilwestarttalkingaboutvirtualiza-tion.Inthisrealm,auditorsareusuallyataloss.

Virtualizationisgainingpopularitybecauseoitspromiseoincreasedreturnoninvestment

(ROI)byreducingthedatacenterootprintandpowerrequirements.Gartnerestimatesthat

morethanourmillionvirtualserverswillbedeployedby2009,andthatnumberwillgrow

to660millionby2011.1Accordingtoa recentSANSLogManagementSurveyomorethan

700ITproessionals,49percentorespondentsarecurrentlycollectinglogdataromvirtual

machines,and68percentothosepredictthat,in2010,nearly70percentotheirlogswill

comeromvirtualmachines.2

Asorganizationsmoveaheadwiththeirvirtualizationprograms,theyneedtounderstandthe

securityandauditimplicationsinthelayersandeaturespresentedbyvirtualmachinearms,

andtheirVMMs(virtualmachinemanagers).

For starters, virtualization introduces a new layer known tomost as Hypervisor, which is

VMwaresvirtualmachinemanager. Virtualizationalsocreatesanewenvironment inwhich

virtualmachinesystemsconnectedviavirtualnetworkinteraces,virtualroutersandvirtual

switchesandtraversingvirtualnetworkpathsaredynamicallymovingaround.Inaddition,

virtualization introducesnewstorageconsiderationsaroundvirtualdrives,networkstorage

systemsandberchannels.

1GartnerResearch.GartnerSaysVirtualizationWillBetheHighest-ImpactTrendinInrastructureandOperationsMarketThrough2012,April,2008,www.gartner.com/it/page.jsp?id=638207

2JerryShenk.SANSAnnual2009LogManagementSurvey,April,2009,www.sans.org/reading_room/analysts_program/logMgtSurvey_Apr09.pd


3/13


Allthesenewlayers,devicesandtracrequiremanagementandprotectionjustastheywould

itheywerephysicalmachinesandnetworks.Butwhatdoauditorsneedtoknowinorderto

successullylocateandensuresecureprocessesaroundsensitivedatatraversingthisnewvir-

tualenvironment?

Unortunately, at thisearly stage oadoption, there is littleguidancewithin theregulatoryrameworksonhowtoaddressnewauditissuespresentedwithvirtualization.Thepurposeo

thispaperistohelpITmanagersandauditorscometogetherandunderstandthevirtualiza -

tionprocessandthenewriskandauditareasthistechnologypresents.Italsooersguidance

ondevelopingauditreviewprocessesthatcanbeappliedtovirtualization,includinghowto

usevirtualizationtoenhanceauditprocesses.

Forpurposesobrevity,thispaperwillocusonPCIDSSauditinaVMwareenvironment,which

iscurrentlythemostwidelyusedvirtualizationplatorm. VMwaresownreportsclaimthat

itsESXdevelopmentplatormisinuseby95percentotheglobalortune500companies.3

AccordingtoCNNinJanuaryothisyear,VMwarehadcaptured85percentothemarketorallvirtualizationimplementations.4AlthoughtheseprincipalsareVMwareandPCIDSSspe-

cifc,theycanbeappliedtomostregulations/mandates,aswellastomostenterprisevirtual

environments.

3www.vmware.com/technology/whyvmware/virtualization-customers.html

4money.cnn.com/2009/01/19/technology/shambora_vmware.ortune/index.htm


4/13


Similarities and Dierences

Virtualmachinesneedtobesecuredandauditedexactlyastheyareinphysicalnetwork-server

environments.Suchmeasuresincludetheamiliarproceduresandcheckliststhatweveused

allalong:Systemhardeningandsecurity,changecontrol,blockingounauthorizedequipment

andapplications, network segmentation,monitoring, logging, alerting, anddocumentation

thatsupportsauditsotheseprocesses.

Thebenetoauditinginthevirtualworldisthatvirtualserverarmsaremorecentralizedand

thereoremoreeasilymanaged.Asanexample,PCIDSSaddressescongurationandchange

controlappliedtoinitialandnalcongurations. BecauseVMwareiseasilyauditableusing

scripts,auditsorchangecanbedoneperiodically,perhapsdaily,withalarmstotriggeroncon-

gurationchanges.Thesechangescanthenbecomparedtodocumentationindicatingwhat

changeshavebeenapproved,andcanalsoveriythatapprovedchangesoccurredinagreed

uponmaintenancewindows.Therearemanytoolsavailabletoauditchangecontrolinavirtualinrastructurethatwillgen-

erallyauditagainstthemainregulatoryrameworkrequirementsaswell. Theyalso include

theirviewocompliancetoseveralrameworks,including:VMwareHardeningGuide,PCI-DSS,

SOX,HIPAA,GLBAandISO17799(inshort,mostrameworksexceptorJSOX).

Inadditiontocommercialtools,VMwareoersAPIsandcommandlinetoolstopermitaudit

operationsromPerlscriptsromtheESXServiceConsoleorvSpherecommandlineinterace.

AuditsooverallenvironmentsaregenerallycarriedoutusingthePowershellAPIsagainstthe

vCenterviewotheworld. AuditsusingthesetoolscancapturenotonlymanyotheESX

speciccontrols,butalsothecontrolsthatareonlyseenromvCenter,suchastheimpacto

VMotion(migration),HA(highavailability)orFT(aulttolerance)oncomplianceorseparationodutiesenorcedthroughpermissionsonuseraccounts. However,somecontrolsarebest

assessedromtheESXhoststhemselves.ESXFirewallsettings,orinstance,arenotalways

accuratelyrefectedinthevCenterconsole,andshouldbeassessedbothromvCenterand

romthehostitsel(usingtheesxcg-rewallcommand).

Finally,mostotheinormationthatisrequiredorauditpurposesisavailablebymanually

navigatingthevCenterconsole.However,therearetwochallengesinusingthisapproach:

repeatabilityandormatting.Manualapproachestoauditmustbebackedupwithstringent,

documentedmanual processes to ensure thatsuccessive audits areactually assessingthe

samecontrolsinthesameway.Moreimportant,collectingauditinormationmanu-

allyorcesauditorstocreateandormattheirauditromscratch,otenmanu-allytranscribinginormationromaGUIscreenorinsomecasesrelying

ongraphicalscreenshots.Suchprocedurescanresultinerrorsand/or

changesinauditmetricsastheGUIchangesacrossversions.More-

over,themanualproceduresaddsignicantlytothetimerequired

toassembleanaudit,whencomparedtobasinganauditontext-

based inormation collected and preormatted by commercial

audittoolsorscript-basedtoolsets.


5/13


Practical Applications

Practicalapplicationsoauditinvirtualenvironmentsmayvarydependingonconguration

andinteroperabilityissues.Despitetheseenvironmentaldierences,auditprogramsshould

containtheollowingcontrolareas:

Audit and Inrastructure Planning

The single largest issuewithrespecttoPCIcompliance is separationovirtual serversand

devicesandurtherseparationromtheguestoperatingsystem(i.e.,PCIsection2.2.1).Sepa-

rationovirtualserver,switches,portgroupsonvirtualswitches,andseparationotheguest

operatingsystemsromtheserviceconsoleareclearlydenedintheVMwareVirtualInra -

structure.

However,thesethingsarenotspelledoutinanycurrentregulatoryramework.Thisambiguity

meansthattheeasiestapproachorauditorsistotakethewordseparatetomeanseparate

hardware,andsimplyinsistonseparateserversoraseparatephysicalserverstohousedata

thatallsunderPCIrequirements.Withpropersegmentation,congurationandcontrols,how-

ever,separatehardwareshouldnotbenecessary.

PCIauditorsotenrecommendPCIVLANsratherthanseparatehardware,becauseVLANsarewell

understoodbytheQSAcommunity.APCIVLANisconsideredbymosttobeaseparateenough

networksegmentationtechniquebetweenvirtualserverarms,solongastheVLANscanpass

teststoindicatethatappropriatecontrolsareinplace.Hopeully,thenextversionothePCI-DSS

rameworkwilloerasimilarapproachtowardvirtualizedinrastructuresegmentation.

Itisimportant,however,toensurethatPCIcomplianceismaintainedwhenthemoreadvanced

eaturesovirtualinrastructuresareemployed.Forinstance,VMotion,highavailability(HA),

ault tolerance (FT),distributed resourcescheduling (DRS), distributedpowermanagement

(DPM),andevenbasicsystemadministrationprogramsallhavetheabilitytomoveahosttoa

dierentnetworksegment.Anyothesemovescanchangethesecuritypostureoahostand

itsassociatedPCI-governeddata.

ThePCIcommitteeonvirtualizationisworkingonsecurityguidanceorvirtualenvironments

thatmaybeinsertedinthenextversionothePCIstandard.Currently,however,such

guidancedoesnotexist.Iyourcompanyisplanningasubstantialnancial

orprojectoutlayoravirtualinrastructure,itsagoodideatoinvolvea

QSA(preerablyyourregularauditororauditrm)provideawritten

opinionontheinrastructureaspartothedesignprocess.


6/13


Confguration

Itisacommonpracticetocreatenewvirtualmachinesromgoldimages.TheseareVMimages

thatarecompletelyinstalledandcustomizedtoaparticularenvironmentandsecuritystan-

dard,whichpromotesaconsistent,auditableserverenvironment.However,thereisahidden

riskinthisapproach,whichisthepotentialormiscongurationoserversandsystemsasthey

replicate,spinup,spindownandmovearoundinthedynamicvirtualenvironment.Standard

updatemechanisms(auto-updateromtheInternetorrominternalcorporateupdateservers)

willapplypatches.However,mandatedcongurationupdatesareotenappliedinacatch-as-

catch-can,out-o-processmanner,resultinginnon-uniormserverbuilds.

Change controlprocedures should beupdated so that changes aecting servers are also

appliedtotheirbaseimages.Thiskeepsallimagesinsyncwithcurrentsecurityandopera-

tionalrequirementsandisassimpleasaddingaeldtothechangecontrolormtoensurethat

thisstepisntorgotten.Forauditors,theprocessshouldberequestedandveried.Itsalso

agoodideatogobackandidentiyaewrecentupdatesinchangecontroltoveriythatthe

changeshavebeenappliedtorelevantgoldimages.

AuditorsmusttakesimilarstepswithHypervisor,ensuringtheexistenceoahardened,gold

buildoHypervisor,andthendocumentinghowithasbeenmanagedandmaintainedwith

updates,patches,auditlogs,andalerting/reportingochanges.

Networkcongurationmustalsobecontrolledandmaintained. Here,thevCenterinterace

allowsor logicalnamingschemasornetworksegments,serversandstorageinrastructure

components.Thisallowsorganizationstoconstructasel-documentedinrastructure,where

componentsthatallunderPCIregulationsareclearlyidentiedbynameineveryadministra-

tiveviewwithinvCenter. Iimplemented,thisapproachonamingcomponentsalsomakes

auditingorcomplianceeasier,asthemapviewswithinvCentershowtherelationshipbetween

thevariousPCIcomponentsintheinrastructure.


7/13


Visibility

Thismappedview intothePCI componentso the inrastructureisoneothemajorsecu-

ritybenetsovirtualization. TheHypervisoradministrationconsole(vCenterin thecaseo

VMware)givestheadministratorullvisibilityintonetwork,storage,resourcemanagementand

administrativeconguration.TheMapViewwithinvCentergivestheauditoracompletepic-tureovirtualmachineconnectivitytothevirtualnetworkandvirtualstorageinrastructures,

aswellasanyseparationrequiredorPCICompliance. vCenteralsograntsacommoninter-

aceorseveraloperationaltasks,includinglogging,perormanceandresourceutilization,and

overallutilizationostorage.Thisbirdseyeviewothedatacenterissimplynotpossibleina

traditionaldatacenterwithconnectionsbetweenphysicaldevices.

Separation o Duties

Bydeault,theVMwareadministratorhasullrightstoallactivitiesintheinrastructure.In

manycases(particularlyinsmallerenvironments),thisdeaultisnotchangedduringsetup

anduse.Worse,thisleveloullrightsaccessiscopiedintomirrorimagesandallotheraspects

othevirtualmachineswithinthedatacenter.Wherethedeaultpermissionsarenotchanged,

then,asinglebreachotheadministratorsaccesscouldleadtoanattackergainingullowner-

shipotheentireserverarm.

Inaddition,ailingtochangethepermissionscongurationsmakesthephrasewhowatches

thewatchmen?veryrelevantinthiscase.So,itisincumbentontheITgrouptoproperlyimple-

mentchangecontrol,separationoduties,congurationmanagement,andproperloggingto

mitigatethisexposure.UsingthevCenterinterace,someotheollowingseparationoduties(SOD)optionscanbeachieved:

Serveradministratorscanbegivenpoweron/powerorightstotheirownserversandnoothers.

Networkadministratorscanbegrantedtherightstopatchserversintovirtualswitchesandcreatevirtualswitches.

VMwareadministratorscanbegrantedtherightstodeploynewVMsbutnottomod-iyexistingVMs.

Auditorscanbegivenview-onlyrightstoallcongurationinormationintheinrastructure.

I implementedcorrectly, SOD inthevirtualnetwork serverenvironment

canbeenorcedatatechnicallevelthatisnotpossibleinthephysical

environment.Forinstance,inthephysicalworld,anetworkadministra-

torcouldpressthepowerbuttononaserver,oraserveradministrator

couldpatchhisorherserverintoanetworkswitch.Inavirtualworld,

bothotheseactivitiescanbedeniedwithtechnicalcontrols.


8/13


Storage Virtualization

Storagevirtualizationhasbeencommonindatacentersordecades.Localstoragevirtualization

(commonlyRAID)doesnotgenerallyhaveasignifcantimpactonPCIandotherregulatoryrame-

works.However,everyothervirtualizationmethodostorageinrastructurecertainlydoes.

FiberChannelisgenerallyviewedasthepremierstoragemechanismandispresentinalmost

alldatacenters. However,perormancein FibreChannel isalmostalwaysattheexpenseo

security.EventhoughassistedencryptionisanoptioninmanyHBAs(hostbusadapters),its

rarelyimplementedorperormancereasons.Asaresult,FibreChanneldataisalmostalways

transportedincleartext.

Becauseothis,FibreChannelarchitecturesaresusceptibletoattacksoseveraltypesthatare

analogoustoattacksinthephysicalEthernetworld,includingsessionhijackingandman-in-

the-middleattacks.WWN(worldwidename)spoonginFibreChannelcorrespondstoMAC

addressspoongintheEthernetworld,whilezonehoppingisverysimilartoVLANhoppingonEthernetswitches.LUN(logicalunitnumber)maskingattacksaresimplyavariationonWWN

spoongviewedromthestorageprocessorratherthantheHBAperspective.

Othertypesotransportalsocommunicateinplaintext.iSCSI(Internetsmallcomputerstorage

interace)andNFS(networklesystem)arealmostalwaystransportingdataincleartexton

thevirtualnetwork.Asuccessulman-in-the-middleattackwillotentargetthedataitsel,but

iSCSIcredentialsoeraninterestingalternative.BecauseiSCSIusessimpleCHAP(Challenge

HandshakeAuthenticationProtocol),oncethecredentialhashiscaptured,theactualcreden-

tialsarenotrequiredtoimpersonatethesupplicanthostandhijackthesession.Thisisoten

calledaPasstheHashattack.

Forthesereasons,bothiSCSIandNFSaregenerallyimplementedondedicatedVLANsordedi-

catedstoragenetworks.Documentation,changecontrol,andcongurationmanagementare

allgoodapproachestomitigationorisksinstoragevirtualization.

Network Virtualization

Networkvirtualizationisanotherinrastructurethatshouldbeconsideredinthecontexto

audit,complianceandsecurity.Therearetwomainvirtualnetworkstoconsider:Virtu-

alizationotheLANusingVLANs,virtualizingtheWANusingMPLS(multiproto-

collabelswitching)orramerelay(inolderWANinrastructures).


9/13


VLANsoerexcellentlocalnetworksegregationasrequiredinthePCIspecication,andare

otenrecommendedbyauditorsbecausetheyareeasilyimplementedandoeracost-eec-

tivealternativetoadedicatedswitchorPCIservices.InactPCIVLANisacommonindustry

term.However,careshouldbetakenwhenimplementingVLANsorsegregation.Traditional

VLANsareotensusceptibletonestedVLANattacks,whichinvolveusingdouble-encapsulated

802.1qramestojumpromoneVLANtoanother(orinstance,romageneralpurposeVLAN

toaPCIVLAN).Inaddition,asimplemiscongurationanerrorinanACL(accesscontrollist)

orinstancecanexposedata.

Ciscoandotherswitchvendorshaveexcellentdocumentationonremediationortheissueo

VLANjumping,butthereissimplynosubstituteorcareincongurationollowedbyperi -

odicpenetrationtestingusingavarietyocommercialtoolsavailableorvirtualmachineenvi-

ronments.

ItsalsoimportanttonotethatVMwaresvirtualswitchimplementationisnotsusceptibleto

manyothecommonVLANandotherlayer2attacks.Thispointisotenoverlookedinconver-sationswithauditors,soitisimportanttospecicallybringthisinormationorward.

Themorecommonriskin thesevirtualnetworkinrastructuresismisconguration. Itisnot

uncommontohavealinktoaremoteoceunavailableonaMondaybecausemaintenance

wasdoneonSunday and a routerwas rebootedwithoutsaving its runningconguration.

TheseMondayopsOOPSsituationshavebeencommonoraslongastherehavebeenWANs.

Whatmostpeopledonotconsiderinsuchupdateandrepairsituationsisthattherunlinkis

stillconnectedtosomething.Itmaybeconnectedtoanunswitchedsegmentortosomeother

customersnetworkasituationnotcommonlydetected. This is, ineect, reclassiyingthe

WANnetworkromatrustednetworktoanuntrustednetwork,whichinvokesthePCIrulesto

encryptingdataintransitovertheuntrustedsegment.

The technical control that can mitigatebothmisconguration and malicious attacks is to

encryptvirtualWANdatausingastrongalgorithmovertheMPLSorotherWAN.Also,secure

yourWANinteracesusingACLs,permittingonlyencryptedtrac.

5VMwarevSphereOnlineLibrary,VirtualSwitchProtectionandVLANs.http://pubs.vmware.com/vsp40_e/server_cong/wwhelp/wwhimpl/common/html/wwhelp.htm#hre=c_virtual_switch_protection_and_vlans.html(accessedAugust2009).


10/13


Disaster Recovery

Disasterrecoveryisanareathatcaneasilybenetromtheuseovirtualizationtocreatespon-

taneousrollovercapabilitytoositestorage.Businesscontinuityplanningisallaboutpre-

paringordisasterssobusinessoperationsaremaintainedduringadisaster,so,duringthis

planning,organizationscanlosesightosecurityandcompliancerequirements.Forinstance,itisverycommontoseeallcriticalhostsreplicatedorrestoredtoasinglevirtualinrastructure

withouttheseparationthatisrequiredorPCIcompliance.

Disasterrecovery(DR)operations,bytheirnature,involvethemostcondentialandsensitive

dataandmostessentialprocessesinthecorporation.Somevirtualauditprogramrequirements

toconsiderindisasterrecoveryplanninginclude:

Theproductionrewall,intrusionpreventionandIPSpostureshouldbemaintainedat

theDRsite.Itherewallrulesaredierent(i.e.,therewallrulesarenotenableduntil

adisasterisdeclared),thentheDRrewallshouldbeauditedregularly.ChangecontrolshouldbeimplementedsuchthattheDRsiteandtheprimarysiteare

keptinlock-step.ThelastthingyouwantisabreachbecausetheDRrewallhasnt

beenpatchedorupdatedsinceitwasinstalled.

LogmonitoringortheDRsiteshouldbetreatedwiththesamerigorastheprimary

datacenter.DonottrytosaveonSIMlicensesbynotcoveringyourDRsite!Thelast

thingyouwantistohaveabreachandtotallymisstheincident.

TheDRsiteshouldbeauditedandpen-testedasanentityseparateromtheprimary

site,withthesamerequencyandrigor.

Finally,replicationtotheDRsiteshouldbeencrypted.


11/13


Summary

Asstudiesandstatisticsshow,virtualizationisalreadyuponus.Butalongwiththecostsavingsandsmallerootprintsoeredbyvirtualization,therearenew

security,managementandauditresponsibilitiesthatmustbeaddressed. The

sameauditobligationsorhardwareenvironmentsmustnowbeappliedtovir-

tualnetworks.However,therearealsomanynewauditprogramareastoincor-

porateasaresultovirtualizationvisibility,congurationmanagement,net-

workmanagement,disasterrecovery,andmore.

Thereisnoclearcontractual,regulatoryorlegalguidanceastohowtosecure

andauditinavirtualizedenvironment.Soorganizationsneedtoaligntheirvir-

tualizationprojectswithauditproceduresbeorethesevirtualizationrequire-mentsaredened.Whenitcomestoachievinganddocumentingcompliance,

toolsnativetothevirtualmachineproductsoeragoodstartingpoint. 6Com-

monlyusedthirdpartytoolsthathavedoneagoodjobwithauditcontrolsin

thephysicalworldarealsoaddingvaluetothevirtualizationauditprocess.

Ultimately, security and IT stas should beworking together to continually

assesstheaudit/riskareasintroducedbyvirtualization.Withproperprogram

guidelinesandcontrols,virtualmachinenetworksshouldbeeasiertomonitor

anddocumentorauditorsbecauseothemorecentralizednatureovirtual

machinearmsandthemanagementcapabilitiesprovidednativelyandthrough

thirdpartytools.

6AnexampleorVMware:www.vmware.com/les/pd/vi35_security_hardening_wp.pd


12/13


About the Author

J. Michael Butler, CISA, GSEC, EnCE, GCFAisaninormationsecurityconsul-tantwithLPS,aleadingproviderocomputerservicestothemortgageindustry.

Butlersresponsibilitieshaveincludedinternalauditoinormationsystemsand

inrastructure,inormationsecuritypolicies,(alignedtoISOandaddressinged-

eralandstatedisclosurelaws),enterprisesecurityincidentmanagementplan-

ning,computerorensics,servicedelivery,anddistributedsystemssupport.He

hasalsobeeninvolvedinauthoringSANSsecuritytrainingcourseware,position

papers,articles,andblogs. Butlerhasmorethan27yearsoexperienceinthe

computerindustry.

Rob Vandenbrink, MSISEandGIACadvisoryboardmember,iscoauthorand

instructoro the SANS Institutes comprehensive course titledVirtualization

SecurityandOperations.Since1981,hehasworkedinallacetsonetwork-

ing and security, and has been a consultant at Metaore (www.metaore.ca)

since1994.Vandenbrinkspracticecoversinternationalclientsinthenancial,

manuacturingandhealthcaresectors.Hiscurrentprojectsandinterestsinclude

PowershellautomationoVMware,VMwaresecurity,scriptingonCiscoIOS,and

securityinFibreChannelarchitectures,amongotherareas.HeholdsaBachelors

degreeinmechanicalengineeringromUniversityoWaterlooandisworkingtowardaMastersdegreeininormationsecurityattheSANSTechnologyInsti -

tute(www.sans.edu).


13/13


SANS would like to thank this papers sponsor:

Sans Analyst Program Vmware09

Documents

Transcript of Sans Analyst Program Vmware09