Principal Product Manager, Splunk Senior Software Engineer ... · How search works in 6.5. 7 search...
49
Copyright © 2016 Splunk Inc. Alex James Principal Product Manager, Splunk Search Optimization Karthik Sabhanatarajan Senior Software Engineer, Splunk &
Transcript of Principal Product Manager, Splunk Senior Software Engineer ... · How search works in 6.5. 7 search...
Copyright©2016Splunk Inc.
AlexJamesPrincipalProductManager,Splunk
SearchOptimizationKarthikSabhanatarajanSeniorSoftwareEngineer,Splunk
&
Demo
A B C D L EA B C D L EA B C D LA B C D LA B C D
A B C D L EA B C D L EA B C D LA B C D
Motivation:Taleoftwosearches
3
searchSourceType lookupL evalE searchA&L&E
Diskrawdata&index
searchSourceType&A lookupL searchL evalE searchE
• 10,000,000indexhits• 1,000,000eventscreated(i.e.extractions)• 1,000,000lookups• 1,000,000evals• 1,000,000filters• Produces100,000matchingevents
• 7,000,000fewerindexhits• 700,000 fewereventscreated• 700,000fewerlookups• 800,000fewerevals• Net500,000lessfilters• ProducesIDENTICAL100,000matchingevents
TOTALWORK
SAVINGS
OptimizationPrinciplesDoaslittleworkaspossible– Retrieveonlytherequireddata– Moveaslittledataaspossible– Parallelizeasmuchworkaspossible– Setappropriatetimewindows
ImplicationsbasedonSplunkArchitecture– Filterasmuchaspossibleintheinitialsearch– Join/Lookuponlyonrequireddata– Evalontheminimumnumberofeventspossible– Delaycommandsthatbringdatatothesearchheadasmuchaspossible.
4
NewinSplunk6.5
Howsearchworksin6.4.
6
searchsourcetype=access-*(status=401orstatus=403)|lookup usertogroupuserOUTPUTgroup|where src_category=“email_server”
search lookup
1)Spliton‘|’andcreateprocessorpipeline
2)Distributebetweenindexandsearchheads,passargumentsandexecute
search lookup whereIndexer1
search lookup whereIndexer2
combine
Searchhead
where
Howsearchworksin6.5.
7
searchsourcetype=access-*(status=401orstatus=403)|lookup usertogroupuserOUTPUTgroup|where src_category=“email_server”
search lookup
1)ParseintoAST
search lookupIndexer1
search lookupIndexer2combine
Searchhead
searchsourcetype=access-*(status=401orstatus=403)src_category=“email_server”|lookup usertogroupuserOUTPUTgroup
2)OptimizeAST
3)ConstructPipelinefromAST
JSONAST
OptimizedJSONAST
4)Distributebetweenindexandsearchheads,passargumentsandexecute
Demo
Whatoptimizationsaredone?
9
Pushingpredicatestotheleft(ordown)– For*any*streamingcommandsthatdon’tmodifyafield:
– |rangemap field=scoreF=0-64D=65-69C=70-79B=80-89A=90-100|wherehost=mail30– |wherehost=mail30|rangemap field=scoreF=0-64D=65-69C=70-79B=80-89A=90-100
– SpecialHandlingforsomecommands:ê Rename– |renamesrc asip |whereip=“192.1.2.13”– |wheresrc=“192.1.2.13”|renamesrc asip
ê Eval– |evalsrc=if(isnull(src)ORsrc=“”,“unknown”,src |wheresrc =“192.1.2.13”– |wheresrc =“192.1.2.13”|evalsrc=if(isnull(src)ORsrc=“”,“unknown”,src
ê Byclausefilters– |statscountbyclientip|searchclientip=“192.0.0.0/8”– |searchclientip=“192.0.0.0/8”|statscountbyclientip
Search/Wheremerging– searchERROR|search404|wheresourcetype=“windows”– searchERROR404sourcetype=“windows”
Whatoptimizationsarecominglater?
10
PredicateSplittingPredicateNormalizationCollapsingconsecutivecommandsConvertingEvalFunctionsintoSearchfiltersifpossibleProjectionEliminationRe-usingprevioussearchresults
Whatdoesthismeanforyou?
11
FasterSearchesUpgradeto6.5Scanfor‘inefficientsearches’– Especiallyinscheduledworkloads...
UsetheJobInspectortoseeoptimizationinactionOptimizefurthermanuallyifneeded
Q&A
12
Disclaimer
14
Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose
containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor
functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.
MigratingSlidesforMac1. Forbestresults,simplypasteyourslidesintothis
template.
2. ApplyslidelayoutsusingtheLayout buttonundertheFormattab.
3. IfLayoutstilldoesnotreflectthedesiredMasterLayout,chooseResetLayouttoDefaultsettings.
4. Deleteunwantedtemplateslides(anyslidesafterLastSlide).
5. ChooseSaveAstosavethefilewithoutoverwritingthetemplate.
15
MigratingSlidesforPC1. Forbestresults,simplypasteyourslidesintothistemplate.
– Pastingafterabulletslideisrecommended
2. Reviewallslidesandmakeformattingadjustmentsasneeded– OntheHome ribbon,clickLayout andselectthecorrectslidelayout– ClickReset toresetallslideelementstothedefaultsizeandposition– Checkforhiddentext,suchaswhitetextonawhitebackground
3. Deleteunnecessarytemplateslides4. SaveAstosavethefilewithoutoverwritingthetemplate
16
SlideMasters• Whenimportingslidesfromanotherpresentation,the
SlideMastersassociatedwiththoseslidesmayalsoimporttothistemplate.Thisisa‘feature’ofPPTandcannotbeturnedoff.
• TodeleteunwantedSlideMasters:– makesureallslidesinthepresentationhavethenew
templateSlideMasterLayoutsassigned(first16SlideMastersshownunderLayout)
– GotoView/MastertodeleteanyunwantedSlideMasters
• ThelastSlideMasterinthistemplateiscalledLastSlide.AnySlideMastersafterthisslidewerelikelyimportedfromanotherpresentationandcanbedeleted(ifnolongerusedbyanyslides.)
17
ImportantTips• Thistemplateusesareducedslidesize.Youmayhavetomanuallydecreasethesizeofsomeitemssuchasstrokes andfonts.
• Iffontsappearbiggerthandesired,remembertoassignaLayout toyourslideandResettoDefaultSettings.
• Ifpagenumbersdonotappearorarethewrongformatting,remembertoassignaLayout toyourslideandResettoDefaultSettings.
• Thecolorsinyourgraphicswillautomaticallybeshiftedtothenewpalette.Pleaseadjustasneeded.
18
Agenda
AgendaItemAgendaItemAgendaItem
19
2012GoalsandObjectivesExample
GoalItemGoalItemGoalItem
20
SampleTitle,66pt.Calibri
21
SampleTitle,66pt.Calibri
22
Subhead
TitleOnlySlide,60pt.Calibri
23
TitleOnlySlide,54pt.Calibri
24
Samplewithscreenshot
25
Screenshothere
SampleTwo-columnFormatSubhead
26
Sampletwo-columnformat
• Sampletwo-columnformat,sentence– Secondbullet
Sampletwo-columnformat
SplunkObjectStyleandColor
27
Hardware ProductBusiness/Corporate
HighlightOnlyGenericVirtualization
Generic
Thesearesuggestedusesforcolorsonly.
AssignDefaultObjectStyle
28
ApplyingSplunkObjectStyle
29
ToapplytheSplunk objectstyletoanyshape:1. Selecttheshapewiththedesiredstyle2. ClickonFormatPainter(paintbrush)toolintoolbar3. Applystyletoanynewshape
Logos
30
CorporateLogo ProductLogo
Logos
31
Logos
32
Splunk Icons
33
search barchart lock cloud opencloud checkmark envelope
storage- 3storageiPhoneiPadandroid
server indexer forwarder searchhead desktop laptop
datacenter
Splunk server
firewall
Splunk IconsCont’d
34
application virtualmachine virtualserver network wwworglobal tools
logfile RFID router loadbalancer script shoppingcart
user users gears/settings gear messaging tag/ticket
document
alert
gps tower
Splunk Icons
35
Checkmark InfoAlert StopiPhoneiPadAndroid
Twitter Facebook LinkedIn RSS YouTube ShoppingcartGPSTower
Healthcare Hospital Officebuilding VoIPPhone Support POSCardReader RFID
Splunk Icons
36
SecurityIcons
37
FirewallAttacker,Generic
Attacker,Insider
Attacker,Nation/State
Botnet Key
Malware MalwareDocument
MalwarePackaged
SecurityBadge
SecurityServer
Shield VirusFootsteps
TheInternetofThingsIcons
38
POSCardReader
RFIDElectricCar
EMVReaderInternetofThings Meter Factory
SignatureCapture
Arrows
39
TableExample
40
ColumnTitle ColumnTitle ColumnTitle ColumnTitle
Text Text Text Text
Text Text Text Text
Text Text Text Text
Text Text Text Text
Text Text Text Text
TableExample
41
ColumnTitle ColumnTitle ColumnTitle ColumnTitle
Text Text Text Text
Text Text Text Text
Text Text Text Text
Text Text Text Text
Text Text Text Text
SampleCustomerSuccess
42
Customerlogohere
CustomernameCustomercompany
“SplunkmakesitcheaperandeasierforHughestoanalyzenetworktrafficforenterprisecustomersaswellasmanagebandwidthforconsumerandsmallbusinesscustomers.”
BulletplaceholderBulletplaceholderBulletplaceholder
Screenshotorgraphichere
TimelineChart
43
Q1 Q2 Q3 Q4
Milestone Event
ChartExample
44
PlannedActual
Number
Number
Number
Number
Number
FY09 FY10FY08PreviousYear
N%growthoverFYxx
QuoteBox
45
“Apessimistseesthedifficultyineveryopportunity;anoptimistseestheopportunityineverydifficulty.”
-WinstonChurchill
QuoteBox
46
QuoteBox
47
WhatNow?
48
Relatedbreakoutsessionsandactivities…
THANKYOU
