Gemfire

Copyright © 2005, GemStone Systems Inc. All Rights Reserved.

GemFire Enterprise Data Fabric

GemStone Systems Inc.

Michael I Lazar, Federal Technology Director

[email protected] 301 325 8405

Steve Rixse, Director Public Sector

[email protected] 301 664 8494

2 Copyright © 2006, GemStone Systems Inc. All Rights Reserved.

GemStone at a Glance

Founded 1982, with over 200 installed customers, in the Global 2000. Includes Bear Sterns, JP Morgan, Merrill Lynch, Northrop Grumman, NCI

Unique and patented expertise in object management, high performance caching and data distribution technologies.

Profitable, well funded and experienced management team.

Office Locations - Beaverton OR, Santa Clara CA, New York City, Washington DC; 60% in R&D, 20 % in Sales and Marketing

Dedicated Federal and Financial teams

24x7x365 global support.


What do we do ?

Distributed Operational Data Infrastructure Not just a distributed cache Key semantics of a database – ACID Key semantics of a Message bus

Enable data sharing and event notifications At memory speeds

Enable apps to continuously analyze and react to very fast moving data

A Data Fabric for the Enterprise


What do we do - GemFire Value proposition

Enable notification of data changes to users and other applications

Create framework for high performance data access

Scale applications to meet unpredictable demands for information

Boost performance across applications without increasing other hardware/software requirements

Reduces network load and can work over low bandwidth networks

Enhance the performance, scalability, and network characteristics of other software.

Single product with multiple uses, easy to implement and little to no management requirements

Standards compliant interfaces


GemFire Enterprise

GemFire Enterprise (GFE Java and GFE C++)

A scalable and high-performance software data infrastructure that

Caches and distributes data in multiple formats (Java, C++, XML) across distributed applications

Connects to backend systems like databases, messaging systems, etc.

Provides high data availability mechanisms

Supports distributed transactions and standards-based querying

Highly instrumented

GemFire Real-Time Events

In memory complex event processing (CEP) solution that facilitates agile, event-driven architectures by analyzing streams of data in conjunction with other static/historical data sources JDBC and SQL based, with Continuous Queries Client side materialized view Messaging paradigm with connected and disconnect clients Highly instrumented


EDF - Where Does it Fit?

High AvailabilityData Management Data Distribution

MainframesHistorical DataCustomer Data

Enterprise Data FabricDistributed Caching

High Availability

Custom Application

Data Sources

Event Processing

Data distribution

Data Modeling Data Access

Application Access

Analytics Portals Execution Reporting

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East

West

North

SQL XML Java C++/ C#

Grid Computing SOA EDA


Analytics Portals Execution Reporting

What is GemFire EDF - Key Features?

Virtualize data and events from many sources

Manage and partition data in distributed main-memory for

blazing performance

Provide data access forapplications in multiple

formats to supportheterogeneity

High-speed data distribution

(TCP/IP, Multicast) across

multiple nodes- Move data on demand

Replicate data to mirrornodes synchronously for

high availability

Overflow or persist to disk to scale

to large data volumesand for data recovery

Standards-based Querying

and Continuous Querying on structured data and streaming data

Distributed transactionsfor reliable operations

Query

Event

Streams

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East

West

North

SQL XML Java C++/ C#

Disk

Grid Computing SOA Event-Processing

ENTERPRISE DATA FABRIC


GEMFIRE ENTERPRISE DATA FABRICJava, C/C++/C#, XML:DB, SOAP Interfaces

High AvailabilityData Management Data Distribution

Transactions, Logging, Authentication, Discovery Session Management, EntitlementsESB

Business Process

Composite Applications: e.g,,

Web Portals

File systemsData WarehousesDatabases

Custom Application

Data Sources

Process

State

SOA – Composite Applications • Authentication

• Session/Process State Management

• Reference data access

• XML Data Transformation


SOA - Composite Processes

Web Service 1

Composite

Application Flow

User/Process

Web Service 2 Web Service 3 Web Service N

• Authentication

• Session State Management

• Reference data access

• XML Data Transformation

GemFire EnterpriseGemFire Enterprise

Web Service 1 Web Service 2 Web Service 3 Web Service N


SOA - Authentication

Web Service 1

Composite

Application Flow

User/Process




Identify and Authenticate - Get Authorizations (IA&A)

Travel over network

Calls Policy Server/LDAP repository

Calls RDBMS

Repeat for each step in application flow

Often more time than actual service step

GemFire eliminates redundant lookups

This is also used for

Authentication Management for Enterprise Portals

GFE GFE


SOA - Session/Process State Management

Web Service 1

Composite

Application Flow

User/Process




In SOA process state is either

Returned as XML between steps

Often stored by workflow manager in RDBMS

Stored in RDBMS by state aware services

GemFire removes latency introduced by network and RDBMS

GemFire GemFire

This is also used for

Session State Management for Enterprise Portals


SOA – Reference Data

Web Service 1

Composite

Application Flow

User/Process




Service calls back-end data source to

look up same information over and over

GemFire removes need to make redundant calls to back end


SOA – XML data transformation

Web Service 1

Composite

Application Flow

User/Process




Service returns same information over and over

(for a period of time) If it is called with the same request

GemFire removes need to retrieve data from back end, and

transform it into XML.


SOA – XML data transformation

Web Service 1

Composite

Application Flow

User/Process




Service returns same information over and over (for a period of time)

If it is called with the same request

GemFire removes need to retrieve data from back end, and

transform it into XML.

GemFire also provides

• Optimized XML data representation

• Less Memory

• Less Java GC/overhead

• Optimized XML data access


GemFire Design Patterns

Application Survivability and Network Usage Reduction

Web Browsers

App ServerWeb ServicesApplications

App Server or SOA applications

Wide Area Network

PDAs

And Wireless DevicesWireless

AppServer GemFire

Cache

GemFire Cache


Real-TimeData Streams

GemFire Real-Time Events

Continuously analyze (execute continuous

queries onevent streams)

Identify patterns of interest through

correlation with other data

Selectively notify systems and personnel of relevant changes and updates in real-timeFinancial Markets Data

Battlefield Information

RFID Data

Wireless Network Data

Register

Continuous Queries

GF Real Time Events Functional Overview


Summary Highlights

Solve numerous challenges in SOA and distributed systems environments

Enhance Scalability, Performance, Reliability of systems

Potentially reduce hardware and software costs

Easy to insert into new and existing systems Highly Available, Highly instrumented, Highly

configurable NSA Acquisition Security approved

Copyright © 2005, GemStone Systems Inc. All Rights Reserved.

EDF - Key Technical Concepts

GemStone Systems Inc.

This presentation contains information that is CONFIDENTIAL to GemStone Systems Inc.

Do not distribute or copy this presentation without express written permission.


Enterprise Data Fabric - What is it ?

Middle tier Distributed Data Infrastructure Operational Data Combines Distributed Caching (main-memory),

DB semantics, Reliable Messaging/ Intelligent Routing and Continuous Analytics

Distributed Data Space

Data warehouses

Applications

Rational databases


Where is the data in the Fabric?

All across the Network

NETWORK IS THE DATABASE

CacheCacheCache

In Local process memory Copies in multiple nodes


Enterprise Data Fabric - Physical View

All across the Network

NETWORK IS THE DATABASE

CacheCacheCache

Rationaldatabases

Spread across many nodes

On Disk

Or lazily fetched from external source

Clustered ApplicationClustered Application


5 Key Features

1. Data Storage Optimized concurrent main-memory structures Topology

• Embedded in application process• Client caches and Cache servers• Partitioned across many servers• n-way in-memory replication for HA• On Disk

– Overflow to disk– Recoverable disk regions – Asynchronous or

Synchronous


5 Key Features

2. Access Model

Object Caching API (get()/put()). Querying through Xpath, OQL and SQL XML /Web Services (HTTP/SOAP) APIs in Java, C++, and C# JTA Transactions Management APIs – JMX


5 Key Features

3. Data Distribution Model Consistency Model

• Distribution without ACKs• With ACKs• With global locking

Reliable Pub-Sub semantics• Novel declarative “role” based model to ensure

message delivery • Dealing with Slow subscribers, net/node failures, etc

Multiple transports• TCP, Reliable UDP MultiCast


5 Key Features

4. Integration with External Data Sources

A Framework for read-through, write-through, and write-behind

Custom Connectors link to DBMS, Middleware Messaging, EII solutions

Components - Loaders, Writers and Listeners


5 Key Features

Dealing with high event rate RFID streaming data, etc

Many clients can subscribe to portions of the Data Fabric Express interest using

complex query

CQ engine continuously analyses data events Continuously calculates how

the query view has been impacted

Ships “delta” and merges into client result set. Notifications delivered with

guaranteed latency threshold

5. Continuous Analytics

Gemfire

Technology

Transcript of Gemfire