DDS:The data-centric futurebeyond message-based integration

© 2010 Real-Time Innovations, Inc.

February 2011

Gerardo Pardo-Castellote, Ph.D.Co-chair OMG DDS SIGChief Technology Officer, Real-Time Innovations, Inc.

2

Systems that interact with the Real World

Must adapt to changing environment

Cannot stop processing the information

Live within world-imposed timing

Beyond traditional interpretation of real-time

© 2010 Real-Time Innovations, Inc.

3

TRENDS: Growing Information Volume Lowering Decision Latency Increasing System Availability Accelerating technology insertion and

deployment

Next-generation systems needs: Performance Scalability Robustness Integration & Evolution

Challenge: More Data, More Speed, More Sources

© 2010 Real-Time Innovations, Inc.

“Real World” Systems are integrated using a Data Model

Grounded on the “physics” of the problem domain– Tied to the nature of the sensors and real objects in the system

(vehicles, device types, …)

Provides governance across disparate teams & organizations– Central authority can define data model necessary for interoperability

Increased decoupling from use-cases and components– Avoids over constraining applications

Open, Evolvable, Platform-Independent– The use-cases, algorithms might change between missions or

versions of the system

4

Realizing this data-model requires a middleware infrastructure

App AppApp

© 2010 Real-Time Innovations, Inc.

Data Centricity 101

Everyday Example: Calendaring

Alternative Process #1:

1. Email: “Meeting Monday at 10:00.”

2. Email: “Meeting moved to Tuesday.”

3. Email: “Here’s dial-in info for meeting…”

4. Rick: “Where do I have to be? When?”

5. Rick: (sifting through email…)

6

Example: Calendaring

Alternative Process #2:

1. Calendar: (add meeting Monday at 10:00)

2. Calendar: (move meeting to Tuesday)

3. Calendar: (add dial-in info)

4. Rick: “Where do I have to be? When?”

5. Rick: (check calendar)

7

What’s the Difference? State.

Things have attributes and characteristics– The meeting will run 1:00–2:00

in the conference room.– My friend’s phone number is

555-1234 and he’s currently grooming his cat.

– The car is blue and is traveling north from Sunnyvale at 65 mph.

…whether they exist in the real world, in the computer, or both

…whether or not we observe or acknowledge them

“State” (“data”) is a snapshot of those attributes and characteristics.

Best Practice: operate on state directly, not dialogs about state.

Data-Centricity =

Describing the world

as it is

Implication: State of the world can be maintained by infrastructure, not each app

the part of you care about

at a certain point in time

Not Data-Centricity =

Saying anything else… “Hey you: go do this.”

“The thing changed in this way.”

Implication: State must be inferred, reconstructed, managed by each app

(Sometimes called “message-centricity”: focus on what’s said vs. what is)

Why is it better to just describe the world?

Reconstructing the state of the world is hard– Must infer based on all previous messages– Maintaining all these messages is expensive– Each app makes these inferences

=> duplicate effort

People make mistakes– Many copies of state => may be different =>

bugsvs.

– Uniform operations on state => fewer bugs

So it’s “better.” Who cares?

Faster to implement=> Save time and money

Easier to integrate and update=> Protect your investment

More reliable systems=> Protect your business

12

Before We Forget: the Definition

A data-centric architecture:

1. …is based on a data model that is:– Appropriately documented—

i.e. understandable by humans– Formally defined—

i.e. understandable by machines– Discoverable—i.e. can be found during execution

2. The data model is independent of any domain-specific functionality / application.– i.e. made of nouns, not verbs

3. The (instantiated) data model is the only authoritative source of state in the system.

For example, data structures in IDL file.

Calendar Event =• Start Time• Duration• Location• Organizer

DDS Lets You Observe a Changing World

Other data-centric technologies:– Databases: SQL– Web: HTTP (mostly)

…assume the world changes slowly

…use network resources inefficiently

…are highly centralized

State

ServerApp

App

AppApp

App

App

SlowA few updates/sec

Not scalable100 apps => 100x load

UnreliableFailure here kills many apps

DDS Lets You Observe a Changing World

DDS:

…allows you to observe frequent changes

…uses network resources efficiently

…is decentralized

App App App App App App

Fast100,000’s updates/sec

ScalableLoad indep. # apps

ReliableNo single pt. failure

Managedwith QoS

State: Global Data Space

DDS Lets You Observe a Changing World

JBC-P replaced home-brew messaging w/ DDS:

Tracks 20x more objects with fewer failures

…with 97% less code (1.5M lines 50K)

…with 99% less CPU resources (88 cores 0.8)

App App App App App App

Fast100,000’s updates/sec

ScalableLoad indep. # apps

ReliableNo single pt. failure

Managedwith QoS

State: Global Data Space

DDS Lets You Observe a Changing World

Domain: world you’re talking about

Topic: group of similar things– Similar structure (“type”) what– Similar way they change when

over time (“QoS”) how

Instance: individual thing

DataWriter: source of observations about part of the world (topic)

DataReader: observer of part of the world (topic)

Domain(e.g. Yellowstone Park)

Topic(e.g. bears in the park)

Instance(e.g. Yogi the bear)

The OMG DDS Standard

© 2009 Real-Time Innovations, Inc. 19

DDS: Standard Data-Centric middleware for Application Integration

Data Distribution Service

StreamingData

Sensors Events

Real-TimeApplications

EnterpriseApplications

Actuators

Web-EnabledDDS

2011

Family of Specifications

© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL 20

DDSImplementation

Network / TCP / UDP / IP

App

DDSImplementation

App

DDSImplementation

DDS Spec

2004

DDSInteroperablity

2006

UML Profilefor DDS

2008

DDS forLw CCM

2009

DDS X-Types

2010

DDSSecurity

20112010

DDS-STD-C++DDS-JAVA5

App

DDS adopted by key programs

DISR– Mandates DDS for Pub-Sub API– Mandates DDS-RTPS for Pub-Sub Interop

US Navy Open Architecture– Mandates DDS for Pub-Sub

SPAWAR NESI– Mandates DDS for Pub-Sub SOA

European Air Traffic Control– DDS used to interoperate ATC centers

UK Generic Vehicle Architecture– Mandates DDS for vehicle comm.– Mandates DDS-RTPS for interop.

© 2010 Real-Time Innovations, Inc.

22

Key Civilian Programs Have Adopted DDS

Air-Traffic Management

INDRA / THALES Eurocontrol

Spain, UK, Germany, France, Italy, Switzerland

Finance

Citi, PIMCO

High Speed Trading, Compliance checking

Automotive

VW, BMW

Driver Assistance, Car Ethernet Bus,

Automation/Scada

Schneider PLCs

Factory communications

Large Telecopes

ESO, NRAO

Coordinated Mirror Control

SCADA

Grand Coulee Dam

Non-stop Turbine operation

© 2010 Real-Time Innovations, Inc.

Key A&D Programs Have Adopted DDS

Land Vehicles

UK GVA

Base10, Nexter

Vehicle control

UAS Ground Stations & Vehicles

Predator, SkyWarrior

General Atomics Ground Stations

Korea FFX Frigate

Samsung-Thales

Combat Management system

Ship Self Defense System

Reagan Class Aircraft Carrier

Combat Management System

Aegis Weapon System

Lockheed Martin

Radar, weapons, displays, C2

AWACS

Radar System

© 2010 Real-Time Innovations, Inc.

Data-Centric Qos-Aware Pub-Sub Model

Source(Key) Latitude Longitude Altitude

UAV1 37.4 -122.0 500.0

UAV2 40.7 -74.0 250.0

UAV3 50.2 -0.7 2000.0

PersistenceService

RecordingService

Virtual, decentralized global data space

24

CRUD operations© 2010 Real-Time Innovations, Inc.

25

Demo: Publish-SubscribeShapesDemo

DataReader“Alarm”

DomainParticipant

DataWriter

“Alarm”

DomainParticipant

DDS communications model

Participants scope the global data space (domain)

Topics define the data-objects (collections of subjects)

Writers publish data on Topics

Readers subscribe to data on Topics

QoS Policies are used configure the system

Listeners are used to notify the application of events

ListenerOffered

QoS Listener

Got newdata

Requested

QoS

New subscriber

!

example

© 2009 Real-Time Innovations, Inc. 27

Demo: Real-Time Quality of Service

Content filter

Time-based filter

History

Deadline

ShapesDemo

Analyzer

© 2009 Real-Time Innovations, Inc. 28

Real-Time Quality of Service (QoS)

QoS Policy

DURABILITY

HISTORY

READER DATA LIFECYCLE

WRITER DATA LIFECYCLE

LIFESPAN

ENTITY FACTORY

RESOURCE LIMITS

RELIABILITY

TIME BASED FILTER

DEADLINE

CONTENT FILTERS

Vo

lati

lity

User Q

oS

Del

ive

ry

Presen

tation

Red

un

dan

cy

Infr

astr

uct

ure

Transp

ort

QoS Policy

USER DATA

TOPIC DATA

GROUP DATA

PARTITION

PRESENTATION

DESTINATION ORDER

OWNERSHIP

OWNERSHIP STRENGTH

LIVELINESS

LATENCY BUDGET

TRANSPORT PRIORITY

DDS builds Higher quality, Lower TCO Systems

Presence

Discovery

Content-Based Delivery

Scalable pub-sub

Real-Time QoS

Qos Monitoring

Historical Cache

Durable Data

Availability

Redundancy & Failover

Security Guard Hooks

DDS Global

Data Space

Messaging & Caching

Event Processing

Database Bridge

Persistence& Durability

RecordingRedundancy & Failover

SQL

Pre-built components address many challenging use-cases

30

CompComp

Comp

© 2010 Real-Time Innovations, Inc.

31

Integrating components to generic middleware technology

Comp Comp Comp Comp

Middleware Artifacts

DataModel

Custom Mapping

Custom Integration

Akin to implementing an OO design on a Procedural Language:Requires mapping inheritance, encapsulation, exceptions, …

© 2010 Real-Time Innovations, Inc.

32

Integrating components to data-centric middleware technology

Comp Comp Comp Comp

DDS Global Data Space

DataModel

Standard Mapping(*)

Standard API

No custom mappings / code necessaryDirect support for data-centric actions: create, dispose, read/take

© 2010 Real-Time Innovations, Inc.

Example: Message-Centric LegacyDefine message-sets / handshakes

33

Pu

blish

Su

bscribe

x=float(45.6)

y=float(78.9)

id=“AA123”0x0000000641410102030042366666429DC

“My app knows this means dispose.”

Component or Use-case based

Schema,Limited QoS)

Nothing to base filters, xforms on

Error checking dev integration

Self-describing data is verbose

© 2010 Real-Time Innovations, Inc.

Example: Modern Data-Centric DesignStart with Data Model / Schemas / Meaning

34

Pu

blish

Su

bscribe

Data Schema

x : float

y : float

id : string (key)

New

45.6

78.9

“AA123”

Update

56.7

89.0

“AA123”

New

65.4

32.1

“DL987”

Dispose

“AA123”

X

Map this into XML; rows + cols

Express content-based filters

Propagate data efficiently

© 2010 Real-Time Innovations, Inc.

© 2009 Real-Time Innovations, Inc. 35

Demo: What it took to make a demo like this

Detecting presence

History cache

Deleting objects

Detecting Applications

ShapesDemo

OMG DDS Security:How to Secure DDS?

DDS Entities are authenticated

DDS Entities access only domains/Topics/… they are allowed to

DDS data integrity and confidentiality is provided

Non-repudiation is enforced

DDS provides availability through reliable access to data

36

….while maintaining DDS’s high performance

OMG DDS Security: A Pluggable Architecture

OMG RFP accepted in Dec 2010OMG RFP Response due in June 2011

OMG standard Dec 2011-Mar 201237

Protocol optimized for disadvantaged networks

Full peer-to-peer protocol– No required brokers or servers

Adaptable via Qos– Reliability, timeouts, message priority

Native multicast support– Fully uses transport multicast, if available– Handles reliability, avoids duplicates

Supports disconnected media– Based on UDP robust to disconnects

Efficient data encapsulation– Binary CDR is 20 X better than XML/SOAP

Built-in availability and durability– Durable & Persistent data– Historical cache– Failover support

DDS Interoperability Wire Protocol adopted in 2007

© 2010 Real-Time Innovations, Inc.

39

Summary

Real-World Systems & Systems of Systems facing information volume, velocity, and mgmt. challenges

Common solution is integration around a Data Model

DDS is a family of OMG specifications that directly supports data-centric publish-subscribe communications

DDS includes portable API’s for C, C++, Java, etc. and an Interoperable Wire Protocol

Use of DDS results in reduced programming, decreased cost, and lowered risk

Cost and Interoperability are the key drivers© 2010 Real-Time Innovations, Inc.

Thank You

40© 2010 Real-Time Innovations, Inc.