WSO2Con EU 2016: Integrate APIM to Third-party Tools: Creating an Agent for ELK
-
Upload
wso2-inc -
Category
Technology
-
view
351 -
download
0
Transcript of WSO2Con EU 2016: Integrate APIM to Third-party Tools: Creating an Agent for ELK
WSO2APIManagersta1s1cs:integra1onwiththeELKstack
DominiqueDebailleuxSeniorArchitectEMOXA
Agenda
• APIManagementtools:whatisitabout?• APIU1liza1onmetrics…whatfor?• WSO2APIManagerinanutshell• Sta1s1cs:WSO2DASapproach• YetAnotherSolu1on?• Drawmean• PuJngitalltogether• Onestepbeyond• FAQ• Almostforgot!• WSO2APIM:opensource…andopenminded
APIManagementtools:whatisitabout?
APIManagementtooling:whatisitabout?
• Ali%lereminder,justincaseyoulivedinacaveintheNunavutduringthelast3years…• APImanagement(APIM)
– Publishing,promo1ngandoverseeingAPIs– Inasecure,scalableenvironment.– Alsoincludesthecrea1onofendusersupportresourcesthatdefineand
documenttheAPI.• APIMtoolsmainfocus:
– Design&deployAPIs– Exposethemtointernalandexternaldevelopers– Ensurethattheyaresecuredandfunc1onal– Getra1ng&u1liza1onfeedbacks
APIManagementtooling:whatisitabout?
• APIMtoolsbasics:– Developerportal:engagementw/partnersordevelopers– SLA&trafficmanagement– Highavailability&scalability– Consistencyacrossimplementa1onsandversions– Cachingmechanism– Security&authoriza1on– Easytodeployandtest– APIDocumenta1on– Sta1s1cs&Analy1cs
APIU1liza1onmetrics…whatfor?
APIU1liza1onmetrics…whatfor?
• Helptodiagnos1chowpeopleuseAPIsandgetinsightsforyourbusiness• Alloworganiza1ons
– tounderstandAPIu1liza1on– tomakesuretheneedsofdevelopersandapplica1onsusingtheAPIare
beingmet– toplanaccordinglyforfutureenhancementsorcapaci1es
• Willalsobeusedtohelpbothmonitorandmone1zetheAPIsexposed
APIU1liza1onmetrics…whatfor?
• APIMtoolscollectandstorerun1meinforma1onaboutAPIsandtheirconsumerssuchas:– howo\enareAPIsbeingused(average,peakrequests,response1me,…)– whichAPIsarebeingused,– whoisregistreredandisusingtheAPIs,– SLAcomplianceorexpecta1on,– …– well,anydatathatismeaningfulltotheorganiza1ons
WSO2APIManagerinanutshell
WSO2APIManagerinanutshell
• h^p://wso2.com/products/api-manager/
• FreeandopensourceAPIMsolu1on– Designs&publishesAPIsinminutes– Includesadeveloperportalw/selfregistra1on(Applica1onStore)– GovernsAPIs(lifecycle,visibility,versions)– APIcrea1onandUsersregistra1onvalida1onthroughworkflows(WSO2
BPS)– ProvidessecuritythroughAPIsaccesscontrol(OAuth2)– …
WSO2APIManagerinanutshell
• FreeandopensourceAPIMsolu1on(cont’d)– …– On-premise&Saasversion– HA&Scalable– Limitedoverhead– Highlyextensible&customizable– ProvidesAnaly5cs(WSO2DAS)– LeveragesonWSO2ESBandWSO2IS
WSO2APIManagerinanutshell
• Fourcomponentswhichcanbesplitondifferentservers
• Publisher(managementwebapp)– APIdesignandconfigura1on– Publica1ontowardtheStoreandtheGateway
• Store(portalwebapp)– APIscatalogue– Subscrip1ons– Accesskeygenera1on(Oauthtoken+ClientID&secret)– APIstest(sandbox&produc1on)
WSO2APIManagerinanutshell
• Gateway(run1mecomponent)– APIProxy– Appliespolicies(security,thro^ling)andcollectssta1s1cs– Passesthewebservicecalltothebackend– LeveragesonWSO2ESB
• KeyManager(securitycomponent)– Tokenmanagement– Authoriza1onmanagement
14
Backend
APIGatewayExtract
accesstokenValidate
accesstokenApplyQoSpolicies
Publishsta1s1cs
Customextensions
APIStore
KeyManager
APIPublisher
Client
Publisher
1
1’
2
2
3
4 5
6 7 8
9
Sta1s1cs:WSO2DASapproach
16
Backend
APIGatewayExtract
accesstokenValidate
accesstokenApplyQoSpolicies
Customextensions
Publishsta1s1cs
Client
Publisher
5
1
2 3
4
APIPublisherAPIStore
APIMsta1s1cstables
request responsefault
othereventstreams
2
32’
3’
2’’
3’’
2’’
3’’
5
Streamdatatable
Summarydatatable
Eventflow
17
Backen
d APIMsta1s1cstables
BAMmediator
Workflowexecutor
In
APIMgtUsageHandler
Out
APIMgtResponseHandler
Fault
APIMgtFaultHandler
APIMgtUsageDataBridgeDataPublisher
mediate(mc)
mediate(mc)
mediate(mc)
publishEvent(DTO)
APIGateway(ESBinside)
Publishsta1s1cs
Usagepublisher
RequestPublisherDTO
Streamdatatable
Summarydatatable
SPAR
K
Analy1cs
Eventreceivers
Eventstore
Processedeventstore
APIMgtUsageDataPublisher
+publishEvent(RequestPublisherDTO)+publishEvent(ResponsePublisherDTO)+publishEvent(FaultPublisherDTO)
APIMgtUsageDataBridgeDataPublisher
RequestPublisherDTO
Requeststreamdefini=on
ResponsePublisherDTO
Responsestreamdefini=on
FaultPublisherDTO
Faultstreamdefini=on
APIMgtFaulHandler
+mediate(MessageContext)
APIMgtUsageHandler
+mediate(MessageContext)
APIMgtResponseHandler
+mediate(MessageContext)
DASPublisherimplementa1on
Handlers&Mediators
StreamDTOs
YetAnotherSolu1on?
YetAnotherSolu1on?
• DASisagreatso\warebutwhatifonealreadyhasaBAMandSAMsolu1oninplace?
• Opera1vesarenoteagertoaddasecond(third,…)tool• Someonlyneedplainsta1s1cswithfirstlevelhistorycapabili1es
(summupsandsimpleaggrega1ons)thatmaynotrequirehighcompu1nganalysismodel
• …notnecessarily->userstory
APIMgtUsageDataPublisher
+publishEvent(RequestPublisherDTO)+publishEvent(ResponsePublisherDTO)+publishEvent(FaultPublisherDTO)
YourAPIMgtUsagePublisher
RequestPublisherDTO
Requeststreamdefini=on
ResponsePublisherDTO
Responsestreamdefini=on
FaultPublisherDTO
Faultstreamdefini=on
APIMgtFaulHandler
+mediate(MessageContext)
APIMgtUsageHandler
+handleRequest(MessageContext)
APIMgtResponseHandler
+mediate(MessageContext)
YOURPublisherimplementa1on
YoukeepthesameStreamDTOs
22
Backen
d
BAMmediator
Workflowexecutor
In
APIMgtUsageHandler
Out
APIMgtResponseHandler
Fault
APIMgtFaultHandler
YourAPIMgtUsageDataPublisher
mediate(mc)
mediate(mc)
mediate(mc)
publishEvent(DTO)
APIGateway(ESBinside)
Publishsta1s1cs
Usagepublisher
RequestPublisherDTO
Yourpreferredanaly1cssolu1on
goesthere
Withsomeadjustments
here
Drawmean
• ELK– PipelineofthreeopensourceprojectsElas1csearch,Logstashand
Kibana– Takedatafromanysource,anyformat– Search,analyze,– Visualizeitinreal1me
Drawmean
• Logstash:3stagespipeline– Inputs-Responsibleforcollectthedatafromvarioussourcessuch
aslogs,syslogfiles,shippers,beats,redis,…– Filters–makescondi1onaltransforma1ons(grok,drop,clone,
1me,geoip,…)like• parsing–usingregularexpressions–,• addingfieldsortags,• formaJngstructureslikeJSON,…
– Outputs-sendsthedatatovariousdes1na1ons,likeaElas1cSearchcluster,filesortoolslikegraphiteornagios
Drawmean
• Elas1csearch:RESTfuldataindexerandsearchengine– Solu1ontomakesearchesandanalysisonasetofdataviaaquery
language– Addressesstructured,unstrucureddataand1meseries– Distributed,reliableandscalable– Easymanagement(elas1csearch-headtools)– LeveragesonLucene
Drawmean
• Kibana:web-basedlightandeasy-to-usedashboardtool– QueriesElas1csearchindexesanddiscoversdata– Letyouinteractwithdataviagraphics(selec1ons,filters,1me
zooming– Histograms,piecharts,1meseriesandevengeomapscreatedand
populatedonthefly– Visualsthatcanbecombinedintodashboards– SecuredaccessviaShieldplugin
Drawmean
Drawmean
Beats,tcp,h^p,udp,jmx,puppet…
Jdbc,Couchdb,Sqlite,Redis,ES…
Kaua,Rabbitmq,Pipe,…
File,Log4j,…
55outputplugins…
50inputplugins…
PuJngitalltogether
APIMgtUsageDataPublisher
+publishEvent(RequestPublisherDTO)+publishEvent(ResponsePublisherDTO)+publishEvent(FaultPublisherDTO)
APIMgtUsageLogPublisher
+publishEvent(RequestPublisherDTO){log.info(datafromRequestPublisherDTO);}…
RequestPublisherDTO
Requeststreamdefini=on
ResponsePublisherDTO
Responsestreamdefini=on
FaultPublisherDTO
Faultstreamdefini=on
APIMgtFaulHandler
+mediate(MessageContext)
APIMgtUsageHandler
+handleRequest(MessageContext)
APIMgtResponseHandler
+mediate(MessageContext)
LOGPublisherimplementa1on
31
Backen
d
BAMmediator
Workflowexecutor
In
APIMgtUsageHandler
Out
APIMgtResponseHandler
Fault
APIMgtFaultHandler
APIMgtUsageLogPublisher
mediate(mc)
mediate(mc)
mediate(mc)
publishEvent(DTO)
APIGateway(ESBinside)
Publishsta1s1cs
Usagepublisher
RequestPublisherDTO
statslogfiles
PuJngitalltogether
Logstashoutputw/sta1s1cs
PuJngitalltogether
PuJngitalltogether
PuJngitalltogether
PuJngitalltogether
PuJngitalltogether
• Easyuh?– Oneclasstocreate– Oneconfigura1onlinetomodify– Somefunwith– That’sit
• Let’strysomethingmorechallenging
Onestepbeyond
Onestepbeyond
• WhatifIwanttoenrichtheprovidedsta1s1cswithmyownones?– Nocorrela1onbetweenrequestandresponse– Nospaceforyoubeloveddata
• Well,let’srestartfromtheclassdiagram
APIMgtUsageExtendedLogPublisher
+publishEvent(ExtendedRequestPublisherDTO)+publishEvent(ExtendedResponsePublisherDTO)+publishEvent(ExtendedFaultPublisherDTO)
ExtendedRequestPublisherDTO
Requeststreamdefini=on
ExtendedResponsePublisherDTO
Responsestreamdefini=on
ExtendedFaultPublisherDTO
Faultstreamdefini=on
APIMgtFaulLogHandler
+mediate(MessageContext)
APIMgtUsageLogHandler
+handleRequest(MessageContext)
APIMgtResponseLogHandler
+mediate(MessageContext)
ExtendedLogPublisher
RequestPublisherDTO
Requeststreamdefini=on
ResponsePublisherDTO
Responsestreamdefini=on
FaultPublisherDTO
Faultstreamdefini=on
Extrafieldstolog• Correla1onId• BusinessData
+getextravaluesfromthe
mc
Exis1ngsta1s1csdata
41
Backen
d
BAMmediator
Workflowexecutor
In
APIMgtUsageLogHandler
Out
APIMgtResponseLogHandler
Fault
APIMgtFaultLogHandler
APIMgtUsageExtendedLogPublisher
mediate(mc)
mediate(mc)
mediate(mc)
publishEvent(DTO)
APIGateway(ESBinside)
Publishsta1s1cs
Usagepublisher
ExtendedRequestPublisherDTO
WSO2APIMlogfiles
APImedia1onisimpacted
Onestepbeyond Request
Response
Logstashoutputw/businessdataand
correla1onid
Logstashoutputw/sta1s1cs
Onestepbeyond
FAQ
FAQ
• WhatifIwanttopopulatetheWSO2APIMsta1s1cstablesanyway?– DASapproach,– DIYapproach
• WhatifIdon't(wantto)usetheELKstack?• Whataboutthe1.xversions?• WhatifIrepublishtheAPI(“onestepbeyond”case)?• Whatabouttheoverhead?
Almostforgot!
Almostforgot
• WSO2PreferredPartner• WSO2TrainingFrance• UsingWSO2solu1onsSince2013• BarcelonaWSO2Con2014speaker• SOAArchitects
co-authorsofthebook“SOA–aguideforarchitects”
WSO2APIM:opensource…andopenminded
WSO2APIM:opensource…andopenminded
• WSO2APIMistherighttooltomanageyourAPIs.• MostAPIToolshavetoconnecttoanexis1ngecosysteminplace• WSO2APIMhasprovenitscapabili1estoembracereallifecases
andtoadapt.• Extensioncapabili1esofWSO2APIMisunchallengedand"part
ofitsstrength”• An“elas1c”solu1on…inaway
ThankYou!
#WSO2ConEU
Shareyourfeedbackforthissessionwso2con.com/app