Innovations in Data Grid Technology with Oracle Coherence
-
Upload
bob-rhubart -
Category
Technology
-
view
2.289 -
download
1
description
Transcript of Innovations in Data Grid Technology with Oracle Coherence
1 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Innovations in Data Grid Technology with Oracle
Coherence
Randy Stafford | Architect At-Large
Oracle Product Development | Cloud Application Foundation
2 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
The following is intended to outline general product use and direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Presentation Objectives
To Understand…
1) What is driving the need for Distributed Caching and In-Memory Data Grid products?
2) How Coherence improves performance, enables linear scalability, and provides availability for shared services
3) Coherence’s Strategic Direction and Latest Features
4 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Internet Scale
Problem: How to deal with the flood of customer demand for services that occurs when you expose your infrastructure to a large, potentially unbounded network.
- Or, stated more positively -
Opportunity: How to build infrastructure that allows you to scale business and improve margins by exposing your services to a large, potentially unbounded network.
5 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
The Internet Scale Problem is Multi-Faceted
• Virtually unlimited number of customers
• Services exposed to partners’ overuse
• Customers’ experience relies on partners’ services
• Global and Real-time
6 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Scaling Infrastructure to Meet the Problem
• Applications must act on data in real-time
– Access, query, and aggregate the data
– Trigger business actions based on changes to data
• Must scale predictably and cost-effectively
– Failure to do so results in loss of revenue and cost overruns
– No time to re-architect in Internet time
• Need a new paradigm in data management
7 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
More. Faster. Cheaper.
•Scale to more users and more data
•Scale predictably and cost-effectively
•Act on data in real-time
•Reduce spending on backend systems
Get on the Grid
8 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Coherence
• Coherence: in-memory data grid
– Distributed caching to applications
– Scalable extreme transaction
processing
– Real-time eventing, query, and
map/reduce aggregations
– Abstraction from back-end data
sources
– High-availability to applications
End Users
Application Servers
Mainframes, Databases, etc.
Application Objects
9 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Fusion Middleware Complete, Open, Integrated, Best-in-Class
Development Tools
User Engagement
Social Web Mobile
Enterprise Management
Identity Management
Content Management
Business Intelligence
Business Process Management
Service Integration
Data Integration
Cloud Application Foundation
10 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Presentation Objectives
To Understand…
1) What is driving the need for Distributed Caching and In-Memory Data Grid products?
2) How Coherence improves performance, enables linear scalability, and provides availability for shared services
11 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Enterprise Application Scalability Challenges
Service Clients Browsers, supplier and partner clients, application clients, mobile apps, and other service consumers.
Service Implementation Custom applications, BPM processes, service bus endpoints, UI services, and other service providers.
Ease of Scaling
Shared Services RDBMS, cloud services, supplier and partner services, mainframe applications, and so on.
• Grows and scales naturally
• Increasing load • Computationally
intensive work • Highly dependent on
shared services
• Complex and expensive to scale
• High latency, SPOB, and SPOF
12 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Traditional Clustering
• Dependent on shared service
latency
• Increase in size of cluster leads to
increase in load on shared service
Service Node 1
Shared Service
Service Node 2
Service Node N
Cached Objects
Cached Objects
Cached Objects
Service Clients
• Inconsistent view of data across
cluster nodes
• No possibility for in-memory
analytics
• Limited in-memory cache due to
• Capacity implications
• Java garbage collection
13 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Enter the Coherence Data Grid
• Clustered Caching
– Consistent view of data
across cluster
• Partitioned Caching
– Increase in size of cluster
adds capacity but does not
decrease performance
• Distributed Data Processing
– Enables a scalable platform
for high performance in-
memory analytics
Service Node 1
Ease of Scaling
Service Clients
Shared Services
In-memory Data Grid
Service Node 2
Service Node N
14 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
How to Integrate with Coherence
Two Options
Custom integration through Coherence API
- or –
Through Existing ‘Switch-On’ OOTB Integrations
Oracle Coherence*Web: HTTP session state
replication with WebLogic, GlassFish, Others
Oracle TopLink Grid: JPA object-relational data
Other Oracle Products that integrate Coherence
15 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
ActiveCache: Premier WebLogic integration
• Distributed HTTP Session State Replication via WebLogic SPI
• Configuration, Lifecycle and Monitoring of Coherence Clusters
and Servers via WebLogic Admin Console & MBeans
• Dependency Injection in JEE modules
– Configure in web.xml, use annotations to inject named
cache
– Alternative to CacheFactory.getCache() lookup
16 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Gartner, Magic Quadrant for Enterprise
Application Servers
This Magic Quadrant graphic was published by Gartner, Inc. as part of a larger research note and should be evaluated in the context of the entire report. The Gartner report is available here: http://www.gartner.com/technology/reprints.do?id=1-17GUO5Z&ct=110928&st=sb
Source: Gartner, Magic Quadrant for Enterprise Application Servers – Massimo Pezzini, Yefim V. Natis, Kimihiko Iijima, Daniel Sholler, Raffaella Faveta – September 26, 2011
17 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Coherence Basic Inteface
Custom Integration
• Application uses
Coherence API to
store/access its data
• Application cache
properties are configured
externally through XML
file
• Basic cache access
through NamedCache
Map interface
NamedCache neCache = CacheFactory
.getCache("NetworkElement");
…
NetworkElement ne = new
NetworkElement(“ID-321”, “OC-
196”, “Nortel Networks);
neCache.put(“ID-321”, ne);
…
NetworkElement ne =
neCache.get(“ID-321”);
18 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Management
Partitioned Caching
• Extreme Scalability: Automatically, dynamically and transparently partitions the data set across the members of the grid.
• Pros: – Linear scalability of data capacity
– Processing power scales with data capacity.
– Fixed cost per data access
• Cons: – Cost Per Access: High percentage
chance that each data access will go across the wire.
• Primary Use: • Large in-memory storage environments
• Parallel processing environments
19 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Management
Partitioned Fault Tolerance
• Automatically, dynamically and transparently manages the fault tolerance of your data.
• Backups are guaranteed to be on a separate physical machine as the primary.
• Backup responsibilities for one node’s data is shared amongst the other nodes in the grid.
20 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Management
Near Caching
• Extreme Scalability and Extreme Performance: The best of both worlds between the Replicated and Partitioned topologies. Most recently/frequently used data is stored locally.
• Pros: – All of the same Pros as the Partitioned
topology plus…
– High percentage chance data is local to request.
• Cons: – Cost Per Update: There is a cost
associated with each update to a piece of data that is stored locally on other nodes.
• Primary Use: – Large in-memory storage environments
with likelihood of repetitive data access.
21 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Source Integration
Read Through / Write Through
• Read Through
– All data reads occur through cache
– If there is a cache miss, the cache will load the data from the data
source automatically
• Write Through
– All data writes occur through cache
– Updates to the cache are written synchronously to the data
source
22 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Source Integration
Write Behind
• All data writes occur through cache
• Updates to the cache are written asynchronously to the
data source
23 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Processing
Parallel Query
• Programmatic query mechanism
• Queries performed in parallel across the grid
• Standard indexes provided out-of-the-box and supports implementing
your own custom indexes
24 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Processing
Continuous Query Cache
• Automatically, transparently and dynamically maintains a view locally
based on a specific criteria (i.e. Filter)
• Same API as all other Coherence caches
25 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Processing
Invocable Map
• The inverse of caching
• Sends the processing to where the data is in the grid
– Once and only once guarantees
– Processing is automatically fault-tolerant
• Processing can be:
• Targeted to a specific key or collection of keys
• Targeted to any object that matches a specific criteria (i.e. Filter)
26 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Data Processing
Events
• Build Complex Event-Driven Apps
• Java Bean Model
• Key-Based
• Filter-Based
27 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Presentation Objectives
To Understand…
1) What is driving the need for Distributed Caching and In-Memory Data Grid products?
2) How Coherence improves performance, enables linear scalability, and provides availability for shared services
3) Coherence’s Strategic Direction and Latest Features
28 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Release Overview
• Oracle Coherence 3.7.1
– Released September 23
– Strategic investment in:
• Exalogic Innovation
• Ease of Use
• Integration Points
– Significant hardening
© 2011 Oracle Corporation
29 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Elastic Data Virtual Memory for Your Data
Cache Server 1 Cache
Server 1
Heap Heap
Flash Flash
Cache Server 2 Cache
Server 2
Heap Heap
Flash Flash
Cache Server 3 Cache
Server 3
Heap Heap
Flash Flash
• Revolutionizes the scale of data grids
• Use block storage for cached data
– Optimized for flash
– Works well with NAS, Disk, etc.
• Simplifies capacity planning and deployment
– Configure amount of on-heap data to store
– Overflow data written to block storage
– Data stored in buffers until flushed to journal
– Reduces chances of Out-Of-Memory errors
30 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Scalable Distributed Cache to Scalable, Event-Driven Real-Time Applications
Coherence Cluster
Coherence Cluster
Replication
Per Member Cache
Store/Loader API
Data Source
Map API Map API Real-Time Client API Real-Time Client API
Query Filter API
Query Filter API
Aggregation (Map-Reduce)
API
Aggregation (Map-Reduce)
API
Grid Processing API
Grid Processing API
Java / C++ / .NET Java / C++ / .NET
REST (Python, PHP,…)
REST Support Simplified Integration
31 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Coherence API Types
Map API Map API Real-Time Client API Real-Time Client API
Query Filter API
Query Filter API
Aggregation (Map-Reduce)
API
Aggregation (Map-Reduce)
API
Grid Processing API
Grid Processing API
Scalable Distributed Cache to Scalable, Event-Driven Real-Time Applications
Coherence Cluster
Coherence Cluster
WAN Replication
WAN Replication
Per Member Cache
Store/Loader API
Data Source
Java / C++ / .NET Java / C++ / .NET
REST (Python, PHP,…)
32 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Query Explain Plan
• Evaluate query cost and
index effectiveness
• Quicker time to market of
optimized Coherence
solutions
• Explain Plan
• Provides the estimated cost of
evaluating a filter as part of a
query operation
• Trace • Performs the associated query
• Provides the actual cost of
evaluating a filter as part of a
query operation.
© 2011 Oracle Corporation
33 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Live Objects
New Coherence Incubator Project
• Building blocks for an event-
driven finite-state machine – …that is distributed
– …and scalable
– …and resistant to machine failure
• Architectural possibilities:
– Stated Event Driven Processing
– Ripple Effect
34 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Coherence on Exalogic Engineered System Optimized Scalability and Performance in a Box
• Coherence optimized for Exabus
• Pre-configured network/compute nodes
• Elastic Data: Expand Capacity with Flash
• Easy deployment as demand spikes
• Grow from ¼ to multi-rack
4x Throughput,
1/6th Latency!!!
35 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Coherence Integrations
Meta-data Services (MDS)
Real-time Decision
SOA Infrastructure
Oracle ADF Oracle GlassFish
Oracle Access Manager
Oracle Data Integration
TopLink
WebLogic Portal
Oracle Service Bus
Service Delivery Platform
WebLogic Server
BI Publisher
Oracle PeopleSoft
WebCenter
Oracle BEAM
ATG
Oracle CEP
Oracle Entitlement Services
Oracle Secure Token Service
Oracle IGBU
Oracle CGBU
ATG Hosted Services Data Cache Simple Clustering
Data Grid Advanced Capabilities
Oracle Identity Federation
Oracle Web Services Manager
SOA Technology Adapters
SOA Business Rules
SOA Human Workflow
SOA BPEL
SOA Coherence Adapter
Oracle Golden Gate
36 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
GoldenGate/Coherence Integration
• Simplifies Coherence use in shared
database environment
– Propagates DB updates to Coherence
– GoldenGate captures changes to
database tables
– TopLink maps database changes to
cached objects
– Availability—2012
– GoldenGate 11gR2
– Available as patch to WebLogic 12.1.1
and 11.1.1.6/10.3.6
– Oracle database only
Database
Coherence Enabled
Application
Coherence
Read
/Write Th
rou
gh
Custom Packaged
Application
Direct D
atabase U
pd
ates
Update Coherence
Update Cohere
nce
GoldenGate Java Client
TopLink Grid Coherence
Cache Updater
Datab
ase Log Stream
Cap
ture
<entity
class="Address">
<table
name="ADDR”/>
<attributes>
<id name="id">
<generated-
value />
</id>
<version
name="ver“/>
</attributes>
</entity>
JPA Metadata
37 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Enterprise Pack for Eclipse (OEPE)
• Coherence Project Configuration
– Facets
– Library Management
– Descriptor Generation
• Runtime Configuration
– Launch Config Editor
– Run/Deploy/Debug from IDE
• Configuration Support
– tangosol-coherence-override.xml editor
– Validation
– Context Sensitive Help Integration
– Cache Configuration Editor (OEPE 11.1.1.8)
© 2011 Oracle Corporation
38 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Enterprise Manager 12C
Applications and
Business Services
Platform as a Service
Infrastructure
as a Service
Plan
Setup
Build
Test
Deploy
Monitor
Manage
Meter & Charge
Optimize
DBaaS MWaaS
D P V &
DEEP PERFORMANCE VISIBILITY & ALERTS-BASED MONITORING
R - JVM D REAL-TIME JVM DIAGNOSTICS FOR COHERENCE NODES*
C C D CENTRALIZED CACHE DATA MANAGEMENT
A P AUTOMATED PROVISIONING
39 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
•Completely customizable performance views – save different views for different monitoring requirements of the enterprise
•Proactive Alerts and notifications based on granular thresholds
•Topology view to show associations and dependencies
•Log monitoring – generate Alerts based on log patters
•Advanced monitoring – push replication, reap sessions, transactional caches, etc
Deep Performance Visibility & Alerts Based Proactive Monitoring
•Real-time threads analysis – find call stack, locks, method local variables, etc
•Real-time heap analysis – heap distribution in different spaces (eden, perm-gen, etc), garbage collection
•Differential heap analysis – pin point heap leaks by comparing two snapshots
Real-time JVM Diagnostics for Coherence Nodes
Oracle Enterprise Manager 12C Monitoring & Diagnostics
40 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
• Query based data operations
• Central UI for key cache operations – view, update, import, export, purge, add/remove indexes
• Drastically saves time in cache operations and makes it easy for administrators
• Save queries for future references
Centralized Cache Data Management
• Maintain ‘gold image’ in the software library
• Provisioning new cluster or add nodes to an existing cluster
• Support Unicast Addrress as well as WKA based clusters
Automated Provisioning
Oracle Enterprise Manager 12C Administration & Provisioning
41 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
For More Information
• General Information: http://coherence.oracle.com
• Coherence Training: http://education.oracle.com
• Coherence Discussion Forum: http://forums.oracle.com
• Coherence User Group on Linkedin
• “Oracle Coherence 3.5” by Aleks Seovic
• Email: [email protected]
42 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Q & (hopefully) A