Rocket U2 Databases - distributed matters · Rocket U2 is installed at 30,000 sites around the...

and the MultiValue model

Rocket U2 Databases

Dan McGrath, Product Manager Previously – Developer & Systems Architect

3

Today we will cover…

4

Today we will cover…

5

Rocket Software (www.rocketsoftware.com) is a global

software development firm that builds enterprise products

and delivers enterprise solutions in the following segments:

Rocket is engaged in business and technology

partnerships with IBM, EMC, Fujitsu, HP Enterprise Services,

Avaya, Motorola, Epicor, and many others.

The company is headquartered in Boston, Massachusetts,

USA.

Rocket Software

©2012 Rocket Software, Inc. All Rights Reserved.

Business Intelligence and Analytics

Storage, Networks, and Compliance

Application Development, Integration, and Modernization

Database Servers and Tools.

6 ©2012 Rocket Software, Inc. All Rights Reserved.

Rocket Worldwide Locations

7

10,000+ customers

1,000+ ISV partners

10+ OEM partners

15+ R&D labs

100+ products

600+ engineers

1,000+ employees

22 years

Rocket by the Numbers


8

Rocket Brands


9

Rocket U2 is installed at 30,000 sites around the world.

Users include:

Who Uses Rocket U2?


• US Higher Education 40%

• Top 400 North America energy product companies 50%

• Fortune 100 companies 55%

• Automobile dealerships in North America 80%

• Mexico’s emergency response systems 80%

• Airlines checking luggage in Australia/New Zealand 100%

Now, the real stuff…

11

UniData & UniVerse

12

Windows AIX HP-UX

Solaris RHEL SuSE

U2 Database - Platforms

13

• Efficient disk I/O

• Table joining not required – nested data

• Flexible storage structure

MultiValue (MV) DB Engine

• Map MV to Relational Tables

• ODBC/JDBC access

• Optional Data-type Enforcement

• Federated Database System

Relational DB Engine

UniData & UniVerse

14

Built-in JSON & XML support

Customizable security framework

Customizable DB commands

Automatic Data Encryption

Federated Database System

Replication

UniData & UniVerse

15

Files

Accounts

Instance Localhost

Production Staging

VOC [Files…]

Dev

Database Structure

16

VO

C

Keywords

Commands

Files

Scripts

Database Structure

17

File types

Hash Tables

Static

Dynamic

Locally Distributed

OS Directories

18

File Structures

Hash table

Settings

• # Groups

• Block size

• Hash Algorithm

Header

• # Groups

• Block Size

• Hash Algo

Group 0

• IDs/Pointers

• Data

Group 1

• IDs/Pointers

• Data

Group 2

• IDs/Pointers

• Data

Overflow

• IDs/Pointers (2)

• Data (1)

Block size Block size Block size Limited by OS

Example file: 3 groups

19

File Structures

Dynamic Hash table Settings

• Hash Table’s settings

• Split/Merge %

Header

• # Groups

• Block Size

• Hash Algo

• Split/Merge

Group 0

• IDs/Pointers

• Data

Group 1

• IDs/Pointers

• Data

Group 2

• IDs/Pointers

• Data

Example file: Initial 3 groups & first split

Group 3

• IDs/Pointers

• Data

Linear extension; buddy system

Separate Overflow file


• Data (1)

20

File Structures

Locally Distributed Hash

table

Settings

• Hash Table’s settings

• Distribution Method

Part 0

Header

Groups

Overflow

Part …

Header

Groups

Overflow

Part n

Header

Groups

Overflow

21

File Structures

Directory

ID: OS filename

Data: OS file contents

Inherits OS & File system

limits

22

Data Structures - MultiValue

Records

•Key/Value pairs

•1-n columns

• Can be 0-bytes

Column types:

•Single value

•List (ordered)

•List of lists (ordered)

•Dynamic columns & lists

• Indexable ‘Virtual’ Columns

•‘Virtual’ Lookups

Features

23

Data Structures – ‘people’ <0> <1> <2> <3>

a45fe2 Tony the Pony

Jon Skeet goo.gl/C2z9f

goo.gl/n44vS

Unicorns goo.gl/0GPMb

goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u

5bd4e6 Jeff Atwood Unicorns goo.gl/MimWd

Estimates goo.gl/PCrfW

24

Data Structures <0> <1> <2> <3>



goo.gl/n44vS


goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u



25




goo.gl/n44vS


goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u



26




goo.gl/n44vS


goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u



27

• @ID = Key

• Dictionary ≈ Schema

• Dictionary Item = Column Definition

• Field/Attribute = Column

• File = Table/Collection

Terminology

Metadata - MultiValue

28

• Logical file = VOC entry

• Physical file = OS entry

• Multiple logical per physical Files

• Logical file / primary dictionary

• View with other dictionaries Sharing

• Dictionaries optional

• Not enforced; ‘views’ Schema-less


29

Dictionary Item Types

• Direct (indexable)

• Virtual - Local (indexable)

• Virtual - Lookup (indexable-ish)

• Associations


30

Metadata – Direct

@ID <1> <2> <3>



goo.gl/n44vS


goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u

@ID

Type: Direct (Data Descriptor)

Location: 0

Column: Single

31

Metadata – Direct

@ID Name Topics URLs



goo.gl/n44vS


goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u

Name

Type: Direct

Location: 1

Column: Single

Topics

Type: Direct

Location: 2

Column: List

URLs

Type: Direct

Location: 3

Column: List of Lists

32

Metadata – Associations

@ID Name Topics URLs



goo.gl/n44vS


goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u

Topics

Location: 2

Column: List

Association: Interests

URLs

Location: 3

Column: List of Lists

Association: Interests

Interests

Type: Association

Items: Topics, URLs

33

Metadata – Virtual (Local)

@ID Name Topics URLs NumTopics Soundex



4

J523 goo.gl/n44vS


U526 goo.gl/MimWd

Boat Programming B316

jQuery goo.gl/iWs1u j260

NumTopics

Type: Virtual

Location: DCOUNT(Topics, @VM)

Column: Single

Soundex

Type: Virtual

Location: SUBR(‘soundex’, Topics)

Column: List

34

Metadata – Virtual (Lookup)

@ID Name Topics URLs CreditCard



goo.gl/n44vS


goo.gl/MimWd

Boat Programming

jQuery goo.gl/iWs1u


3787-344936-71000 Estimates goo.gl/PCrfW

CreditCard

Type: Virtual

Location: TRANS(‘creditCards’,@ID, 1, ‘X’)

Column: Single

35

Metadata – Sharing

@ID

Type: Direct

Location: 0

Column: Single

byteLen

Type: Virtual

Location: LEN(@RECORD)

Column: Single

JSON

Type: Virtual

Location: SUBR(‘toJSON’, @RECORD, @FILENAME)

Column: Single

36

Locking

Header

• # Groups

• Block Size

• Hash Algo

Group 0

• IDs/Pointers

• Data

Group 1

• IDs/Pointers

• Data

Group 2

• IDs/Pointers

• Data

Overflow


• Data (1)

Write: Exclusive Group Lock

Read: Shared Group Lock

Example file: 3 groups

37

• SQL at the command-line

• Schema management tools

SQL Engine embedded

• Explicitly by schema generation

• On-the-fly using ‘UNNEST’ keyword

‘Virtual’ tables for SQL

• Able to run native commands too JDBC & ODBC

access

• Ignore completely

• Log violations to disk

• Raise errors and prevent action

Data-type enforcement

SQL Engine & Nested Tables

38

Store data in an external DB

Access as if it was local

Drivers for:

• SQL Server

• DB2

• Oracle

Driver API exposed

• We use this API to write our drivers

Federated Database System

39

File level encryption

Column level encryption

Record key encryption

Index encryption

Multiple password protection

of ‘Master Key’

‘Wallets’

Grouping / Temporary access

Automatic Data Encryption

40

Publisher 1-n Subscribers

Account (full environment) replication

File replication

Organize into ‘groups’

Can Publish & Subscribe from same machine

Replication


41

3 modes:

Deferred – Sync every ‘x’ time

Immediate – Commits once staged

Real-time – Commits once sync’d

Replication


42

Replication Publishing System

uvrepmanagerxadmin

uvpubuvpublistener

uvsh

Publisher FilesLog Extension

Files (LEF)

Replication Buffer

Transaction Control

Area (TCA)

Subscribing System

uvrepmanager xadmin

Transaction Control

Area (TCA)

Replication Buffer

Log Extension

Files (LEF)Subscriber Files

uvrw

uvsub

sublistener submain

Subscriber Packet

File (SPF)

43


uvrepmanagerxadmin

uvpubuvpublistener

uvsh


Files (LEF)

Replication Buffer

Transaction Control

Area (TCA)

Subscribing System

uvrepmanager xadmin

Transaction Control

Area (TCA)

Replication Buffer

Log Extension


uvrw

uvsub

sublistener submain

Subscriber Packet

File (SPF)

44


uvrepmanagerxadmin

uvpubuvpublistener

uvsh


Files (LEF)

Replication Buffer

Transaction Control

Area (TCA)

Subscribing System

uvrepmanager xadmin

Transaction Control

Area (TCA)

Replication Buffer

Log Extension


uvrw

uvsub

sublistener submain

Subscriber Packet

File (SPF)

45


uvrepmanagerxadmin

uvpubuvpublistener

uvsh


Files (LEF)

Replication Buffer

Transaction Control

Area (TCA)

Subscribing System

uvrepmanager xadmin

Transaction Control

Area (TCA)

Replication Buffer

Log Extension


uvrw

uvsub

sublistener submain

Subscriber Packet

File (SPF)

46


uvrepmanagerxadmin

uvpubuvpublistener

uvsh


Files (LEF)

Replication Buffer

Transaction Control

Area (TCA)

Subscribing System

uvrepmanager xadmin

Transaction Control

Area (TCA)

Replication Buffer

Log Extension


uvrw

uvsub

sublistener submain

Subscriber Packet

File (SPF)

47


uvrepmanagerxadmin

uvpubuvpublistener

uvsh


Files (LEF)

Replication Buffer

Transaction Control

Area (TCA)

Subscribing System

uvrepmanager xadmin

Transaction Control

Area (TCA)

Replication Buffer

Log Extension


uvrw

uvsub

sublistener submain

Subscriber Packet

File (SPF)

48


uvrepmanagerxadmin

uvpubuvpublistener

uvsh


Files (LEF)

Replication Buffer

Transaction Control

Area (TCA)

Subscribing System

uvrepmanager xadmin

Transaction Control

Area (TCA)

Replication Buffer

Log Extension


uvrw

uvsub

sublistener submain

Subscriber Packet

File (SPF)

Application Server

50

“We need SMART Models, THIN Controllers, and DUMB Views”

51

U2 Database

•Model

Controller View

Application Server

52

U2 Database

•Model

Controller View

Application Server

53

Application Server


U2 Database

24/7 Phantoms

Callable Subroutines for Business Logic

Session Memory

Customizable DB Commands & Keywords

Customizable Security Framework

Select result ‘snapshots’

54

VOC

SELECT

PersonModel

CreditCardModel

stdSELECT

Commands & Keywords


55

VOC

SELECT

• V SELECT

PersonModel

• C /byteCode/PersonModel

CreditCardModel

• C /byteCode/CreditCardModel

stdSELECT

• V SELECT

Customizable DB Commands


56

VOC

SELECT

• C /byteCode/CustSelect

PersonModel


CreditCardModel

• C /byteCode/CreditCardModel

stdSELECT

• V SELECT

Customizable DB Keywords


57

VOC

SELECT

• V SELECT

• Security SecControl

PersonModel



SecControl

• C /byteCode/SecControl



58


VOC

SELECT

• V SELECT


EVAL

• K EVAL


SELECT people WITH name EVAL “TRANS(‘users’, @ID, 1, ‘X’)”


59

Sharing within a session

Sharing between sessions

Save to disk

• Resume same processing after outage

Select result ‘snapshots’


Additional tooling


XAdmin

62

SOAP Web Services

• Automatic WSDL generation

RESTful Web Services

• JSON-based

Web Services

Expose U2

as a

Web Service

63

Business Intelligence & Analytics

64

Business Intelligence & Analytics

65

Network/Systems Monitoring


67

AeroText: Unstructured Data


68



69



70



71



72



73



74



75

Rocket Software; enterprise solutions

Rocket U2 DBs; Airlines to 911

Flexible data structure & metadata

Plays nice with Relational DBs

Summary


76

www.rocketsoftware.com/u2

Free personal editions

Email: [email protected]

Twitter: @itcmcgrath

Tech Conferences: U2 University

UK: 26-28 June, Australia: 14-16 August

u2u.rocketsoftware.com

What’s next?


http://www.rocketsoftware.com/u2

mailto:[email protected]

http://u2u.rocketsoftware.com/

77

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.

WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

IN ADDITION, THIS INFORMATION IS BASED ON ROCKET SOFTWARE’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY ROCKET SOFTWARE WITHOUT NOTICE.

ROCKET SOFTWARE SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

• CREATING ANY WARRANTY OR REPRESENTATION FROM ROCKET SOFTWARE(OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR

• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF ROCKET SOFTWARE.

Disclaimer


78

The following are trademarks or registered trademarks of

Rocket Software, Inc.: Dynamic Connect, SystemBuilder, U2,

U2 Web Development Environment, UniData, UniVerse, and

wIntegrate.

IBM, the IBM logo, AIX, and DB2 are trademarks of IBM in the

United States and other countries.

Oracle and Java are registered trademarks of Oracle

and/or its affiliates.

Microsoft, SQL Server, Windows, and Excel are trademarks of

the Microsoft group of companies.

UNIX is a registered trademark of The Open Group.

Other company, product, and service names mentioned

herein may be trademarks or service marks of others.

Trademarks and

Acknowledgements


Rocket U2 Databases - distributed matters · Rocket U2 is installed at 30,000 sites around the...

Documents

Transcript of Rocket U2 Databases - distributed matters · Rocket U2 is installed at 30,000 sites around the...