API and Big Data Solution Patterns
Transcript of API and Big Data Solution Patterns
API and Big data Solution Patterns
Srinath Perera Senior Software Architect
About WSO2
• Providing the only complete open source componen:zed cloud pla=orm
– Dedicated to removing all the stumbling blocks to enterprise agility
– Enabling you to focus on business logic and business value
• Recognized by leading analyst firms as visionaries and leaders – Gartner cites WSO2 as visionaries in all 3 categories of
applica:on infrastructure – Forrester places WSO2 in top 2 for API Management
• Global corpora:on with offices in USA, UK & Sri Lanka – 200+ employees and growing
• Business model of selling comprehensive support & maintenance for our products
150+ globally posi:oned support customers
Outline • What is API Management
• What is Big Data?
• API manager + Bigdata solu:on
• API + Big data PaVerns
• Conclusion
Photo by John Trainoron Flickr hVp://www.flickr.com/photos/trainor/2902023575/, Licensed under CC
APIs
• An API is a business capability delivered over the Internet to internal or external consumers – Network accessible function – Available using standard web protocols – With well-defined interfaces – Designed for access by third-parties
• A Managed API is: – Actively advertised and subscribe-able – Available with SLAs – Secured, authenticated, authorized and protected – Monitored and monetized with analytics
Typical Use Cases
• Expose APIs for public consumption – Extend your business through APIs. • Expose APIs for internal consumption – Manage the APIs used in internal applications – Detect Usage Patterns – Internal Monetization
• Control Access to Cloud Services – Manage and Secure access from internal
applications to cloud services (SalesForce, Google Apps, etc.)
WSO2 API Management Platform
WSO2’s API Management Vision • Create APIs – WSO2 Application Server, Data Services Server and ESB
• Find and subscribe/buy APIs – API Store and Governance
• Manage, secure and protect APIs – API Management and Gateway
• Monitor and Monetize APIs – API Monitoring and Analytics
• Develop, host and run API-based applications in a Platform-as-a-Service – WSO2 Stratos and Cloud Development Platform – API Cloud
API Manager Components
API Gateway Processing Flow
API Manager Product and Platform
What is Big data?
• There is lot of data available – E.g. Internet of things
• We have computing power
• We have technology • Goal To know – To Explain – To predict
• Challenge is the full lifecycle
Drivers of Big Data
Big Data Architecture
Collecting Data • Data collected at sensors and sent to big data
system via events or flat files • Event Streams: we name the events by its
content/ originator
• Get data through – Point to Point – Event Bus (WSO2 MB)
• E.g. Data bridge (come with WSO2 CEP/ BAM) – a thrib based transport that do about 400k events/ sec
Storing Data • Historically we used databases
– Scale is a challenge: replication, sharding
• Scalable options – NoSQL (Cassandra, Hbase) [If
data is structured] • Column families Gaining Ground
– Distributed file systems (e.g. HDFS) [If data is unstructured]
• New SQL – In Memory computing, VoltDB
• Specialized data structures – Graph Databases, Data structure
servers hVp://www.flickr.com/photos/keso/363133967/
Making Sense of Data • To know (what happened?) – Basic analytics +
visualizations – Interactive drill down – E.g. MapReduce, Hive, CEP
• To explain (why) – Data mining, classifications,
building models, clustering (e.g. Mahout, Map Reduce)
• To forecast – Neural networks, decision
models (e.g. Mahout)
WSO2 Big Data Story
API Gateway + Big Data
API + Big Data Patterns • Big Data as API enabler
– Monitor and control: API analytics for an organization
– Analytics: Billing and Metering – API Store (Market Place)
• Targeted Marketing (Recommendations, Deals, Search)
• Context sensitive prizing
– Deployment optimization – Governance: Planning and Proactive
action • APIs as a Big Data Enabler
– Easy way to find and get access to data – Data Market Place – API Façade for exposing Analytics/ Data – Controlled delegation of data
hVp://www.flickr.com/photos/ooohoooh/1350774047/
Pattern 1: Monitor and Control
Monitor and Control (Contd.) • API Manager as single entry point to come in and go
out • Capture data – client, what API, when? $$ revenue .. • Analyze and Monitor KPIs
– CIO Dashboard – Alerts
• React to trends faster • Greater control
– Audit – Subscriptions policies – Usage tiers
• Policy enforcement – Make sure everything is managed
Example: Business value driven Org • Customer facing site is a
composite (e.g. like Gmail page) • Each widget back by API or APIs • Each widget added removed in
runtime • Widget shown based on business
value • Each API owner get resource
compared to business value • We can use API management as
the tool to realize this
Pattern 2: Billing and Metering • API store detect all access • Contains information about
subscriptions, amount of data transferred etc.
• We can use this to – Meter, control and optimize – Enforce usage tiers (e.g. Quota
via CEP)
• Generate Billing and reports • Supported by WSO2 API
manager now
Pattern 3: API Store (Marketplace)
Pattern 4: Deployment Optimization
App Factory - Applications Management
App Factory (Contd.) • Applications Management – Creation – Continuous Build / Deploy – Promotion across environments
• Team Management – Add developers, QA engineers and DevOps for your
application • Bugs/Features Tracking • Integrated with API Store • Activity Dashboards • Built on top of Jaggery framework • Skinnable
Pattern 5:API lifecycle Governance • Take control of your API life cycle
(develop, test, production) with WSO2 App Factory
• You can apply analytics to your API lifecycle – How long does it take to develop and
API? – How often are they updated? – What are the dependencies of my
API? – What are most effective versions? – What steps takes most time? – What APIs are actively developed? – What one have more issues, how fast
are they get fixed?
Pattern 6: Find datasets
• Most big data apps need Good data
• Hard to find good data – No good way to find – Hard to access – No clear way to buy
• APIs is a great way to find useful data
• Also to provide controlled access
e.g. Andy Stanford-Clark used public data about cruisers to track where availability.
Pattern 7: Data Marketplace • Find, (buy), use data • Great way to monetize the
output of big data projects – Sell the analyzed traffic data – Sell stock quote analytics
• API subscribe model can help • Usage tiers can be based on
frequency or resolution of data
Pattern 8: Controlled Data Delegation • Only expose the part of your data set of a processing
without exposing credentials of the database to the user
Big data in API Store
APIs in Big data System
Usecase • Trigger an alert when new version is 10% more
slower than the older versions – Expose the API through API store – Configure API manager with BAM location – BAM integration will pump stats (api, version, latency,
time) – Write a BAM query to calculate the latency through
time and store it in a DB – Write a CEP query to detect if new version is slower
Define eventTable APIStatHistory .. Select from APIStat as s join APIStatHistory as h on (h.latency -
s.latency)<=h.latency*.1
Conclusion
• APIs and Big Data has received much attentions
• We looked at WSO2 offerings in both cases – WSO2 API Manager, WSO2 App Factory – WSO2 BAM, WSO2 CEP
• We looked at some patterns on how both can work together
• We looked at how we can implement them
Engage with WSO2
• Helping you get the most out of your deployments • From project evalua:on and incep:on to development and going into produc:on, WSO2 is your partner in ensuring 100% project success
Questions?