Lucas Jellema (CTO, AMIS)

10 Mei 2016

The evolution of the Database – role of the database in application architecture

2

Four years ago – here at Fontys

3

Agenda

• Wie of wat is AMIS?• Hoe zit het met die Database?• Vragen & Opmerkingen

AMIS

• Leuke organisatie!• Oracle & Java specialisten (pakweg 90 man & vrouw)• Gevestigd in Nieuwegein• Opgericht in 1991 door studenten van de Universiteit Twente

– A M I S• Kennisdeling: SIGs, conferenties,

blog (sinds 2004)– http://technology.amis.nl

• Opdrachten in – Heerlen, Joure, Almelo, Nijmegen, Deventer– Utrecht, Den Haag, Amsterdam, Rotterdam– New Caledonia, Mongolië, Canada,

Denemarken, België, Koeweit, Faroër eilanden, Duitsland, Verenigde Staten, …

• Stage-opdrachten– Mail: eva.van.der.kleij@amis.nl

5

The Presenter:Lucas Jellema

• Lives in The Netherlands (close to Amsterdam)

• Started doing Oracle in 1994 with Oracle Consulting (Oracle Designer, Forms, Database)

• Joined AMIS in 2002 – now working as CTO, Consultant (Solution Architect, Technical Lead, Programmer, Product Owner) and Trainer

• Oracle ACE (2005) & ACE Director (2006)• Author of ‘Oracle SOA Suite 11g Handbook’

(Oracle Press, 2010), ‘Oracle SOA Suite 12c Handbook’ (2015)• Presenter at Oracle OpenWorld, JavaOne and

many Oracle and Java User Group Conferences• Frequent blogger at http://technology.amis.nl• Active with SQL & PL/SQL, Java EE & ADF,

SOA, BPM & more Fusion Middleware

7

AMIS

• Projecten rond (enterprise) web applicaties en portalen, B2B integratie, “mobilisering”, 24/7 beschikbaarheid, User Experience, business process management, digitale communicatie, self service, STP

• Klanten waaronder:– Politie Nederland, Mediq, Gezondheidsdienst voor Dieren, Raad voor de Rechtspraak,

NS, ProRail, Eneco, Pensioenfondsen (APG, MN), financiële instellingen, logistieke bedrijven, software producenten en een heel groot Social Network

• Technische kreten– Database, Middleware

SOA, APIs, BI, IoT,Big Data, UX, HTML 5,(No)SQL, Real Time,Java, grid, IdM, XML,Cloud, BPM, VM, Docker,Node.js, Angular, ScrumProvisioning, ScalabilityEngineered Systems

8

De Nieuwegein-Eindhoven connectie

9

THE EVOLUTION OF THE DATABASE – ROLE OF THE DATABASE IN APPLICATION ARCHITECTURE

Overview

• Role of the Database in Modern Architecture • More than a container for data• Evolution of the Oracle Database• Recent Trends around the Database• Q & A

11

Decentralized Silos to central, consolidated Layers

12

Decentralized Silos to central, consolidated Layers (2)

13

14

Consolidated, Layered Architecture

Data

Business

Interface

15

Consolidated, Layered Architecture

Data

Business

Interface

16

Cloud Power

PaaS

More than a container for data

• “We could also do that in the database”• in the database? Huh?

RDBMS ≈>>

Brief history of Oracle Database

• 1970: Paper “A Relational Model of Data for Large Shared Data Banks” by Ted Codd (IBM)

• 1977: Software Development Laboratories was founded• Late ‘70s – Project “Oracle” – executed by SDL for the CIA• 1979: SDL became Relational Software, Inc. and introduced Oracle V2

(built in PDP-11)– Basic SQL functionality (query, join) but no transactions

• 1982: RSI became Oracle Corporation• 1983: Oracle 3 - rewritten in C, ported to UNIX platforms and extended

with COMMIT and ROLLBACK• 1984: Oracle 4 – read consistency• 1985-1990: Oracle 5, 5.1 and 6: client/server, distributed queries, row

level locking & hot back ups• 1993: Oracle7… more than just a database

PL/SQL VM

(Oracle7, 1993)PL/SQL VM

PL/SQL Virtual Machine

• Ever since Oracle7, the Oracle RDBMS contains a PL/SQL Virtual Machine

• That means: write [PL/SQL] once, run anywhere (as long as the RDBMS is running)

• This portability of PL/SQL across platforms is pretty extraordinary!

PL/SQL

1993 200910g7.0 8.0 8i 9i

internet

xml

JVM inside

ANSI/standardsinterMedia

Spatial

OLAP/BI/Analytics

Data Warehouse/ETL

The Grid

11g

PL/SQL:Triggers, Packages

SOA

SOX

Constraints

20011997 2004

Data Mining

Semantic Query

OO

RegExp

‘13XE

NoSQL

Evolution of the Oracle Database

12c

CBO

RAC

Flashback & Total Recall

Data Guard

R

24/7 EBR

Cloud

Multi-platform porting

Materialized Views & Query Rewrite

SQL2Hadoop

Pattern Matching

Temporal DB

Multitenant

In Memory DB

JSON

Data Masking

‘15

Backup and recovery

• Back up is not relevant….• .. unless you have an established and test recovery strategy!• Backup should be done ‘hot and on line’• Recovery should be

– Unnecessary– Online– Focused– Quick

• Oracle Backup & Recovery facilities– Data Guard – (remote)

synchronized Stand By Database

– Flashback Recovery – RMAN

24/7 – High Availability

• Unplanned Downtime– Real Application Clusters (RAC)– Data Guard Standby Database for fail over– Hot Patching– Back-up Air Conditioning & Power Generator

• Planned Downtime– Online Redefinition– Edition Based Redefinition for zero down time database application upgrades– NoSQL “read ahead & write behind” cache layer

Compression

• When the system’s performance is I/O bound– Reading/writing data from/to storage cannot keep up at the same rate as the CPUs

• and there is spare capacity in the CPUs to alleviate the I/O burden– Through a reduction in the volume of data to be read and written – by zipping before write and unzipping after read

• The same information gets across in fewer bytes and therefore with less I/O

• It even reduces storage requirements

Flashback and Total Recall

• Oracle Database has long had two capabilities for ‘remembering’ the old situation regarding records– Uncommitted transactions whose changes are local– Read consistency, even for long running queries

• In 9i, 10g and 11g this featurehas been extended to remember‘the old situation’ for far longer– Undo Data => Flashback Archive

• This allows for – Historical queries– Trend analysis over time– Point-in-time recovery

• At database, schema, tableor even record level

• 12c adds Valid Time Modeling (aka TemporalValidity): database keeps track of begin and enddate of the valid period of records

Valid time aware flashback queries

• Select all employees

• Select all employees who were employed at a certain moment in time

• Perform all queries as if we traveled back in time

• Example: Run an end-of-year report at the end of January

SELECT * FROM EMP AS OF TIMESTAMP TO_TIMESTAMP('01-JUN-2014 12.00.01 PM')

EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_time ( 'ASOF' , TO_TIMESTAMP('29-MAR-2014 13:14:15') );

SELECT * FROM EMP

28

Flash Forward: Looking into the future…

OUR_PRODUCTSNAME PRICE

select name, pricefrom our_products

29

Looking further into the future…

OUR_PRODUCTSNAME PRICE

select name, pricefrom our_products

begin DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME ( level => 'ASOF' , query_time => TO_TIMESTAMP('01-10-2018', 'DD-MM-YYYY') );end;

30

Current situation …

OUR_PRODUCTSNAME PRICE

select name, pricefrom our_products

begin DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME ( level => 'CURRENT');end;

31

All data in the table(the default setting)

OUR_PRODUCTSNAME PRICE

select name, pricefrom our_products

begin DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME ( level => 'ALL');end;

Role of the Database in Modern Architecture

• Single Point of …• What goes where …

Reaching out from the database [platform]

Database

Java Class processing an HTTP request: The servlet

• The JVM can handle HTTP requests via the Servlet Container• HTTP requests are routed to a custom Java Class that writes the (usually

HTML response)• The Servlet infrastructure handles the actual HTTP response to the

invokerJVM

Servlet

http request

http responseWebLogic Server

PL/SQL Package processing an HTTP request: The embedded PL/SQL gateway

• The Database (the PL/SQL VM) can handle HTTP requests via the Embedded PL/SQL Gateway

• HTTP requests are routed to a custom PL/SQL Package that writes the (usually HTML response)

• The EPG infrastructure handles the actual HTTP response to the invoker

http request

http response Custom PL/SQL package

EPG

htp

Reaching out from the database

Database

dbms_epg(mod_plsql)

http

XMLDB

httpftp

webdav

utl_http

http

AdvancedQueuing

AQjms

Queue

RESTWebService

File, O/S

JDBC, SQL Net, DB Link

APEXGUIhttp

Database: single point of …Scalability | Availability | Consistency

• Single Point of …• What goes where …

38

Relational Databases

• Based on relational model of data (E.F. Codd), a mathematical foundation

• Uses SQL for query, DML and DDL• Transactions are ACID (Atomicity, Consistency, Isolation, Durability)

– All or nothing– Constraint Compliant– Individual experience [in a multi-session environment] (aka concurrency)– Down does not hurt

39

ACID comes at a cost

• Transaction results have to be persisted in order to guarantee D• Concurrency requires some degree of locking (and multi-versioning) in

order to have I• Constraint compliance (unique key, foreign key) means all data hangs

together (as do all transactions) in order to have C

• ACID means:– All data closely together (and sometimes far from

applications and users)– All transactions centrally organized– All data persisted as part of the transaction

• ACID means:– Hard to scale out (horizontally)

• Oracle offers scale up with Real Application Clusters

40

Databases re-evaluated

• Not all use cases require ACID (or can afford it)– Read only (product catalog for web shops)– Inserts only and no (inter-record) constraints– Big Data collected and “dumped” in Data Lake (Hadoop) for subsequent processing– High performance demands

• Not all data needs structured formats or structured querying and JOINs– Entire documents are stored and retrieved based on a single key

• Sometimes – scalabile availability is more important than Consistency – and ACID is sacrificed– CAP-theorem states: Consistency [across nodes], Availability and Partition tolerance

can not all three be satisfied

41

NoSQL and BASE

• NoSQL is a label for a wide variety of databases that lack some aspect of a true relational database– ACID-ness, SQL, relational model, constraints

• Cassandra, MongoDB, Redis, CouchDB, …

• BASE as alternative to ACID:– basically available, soft state, eventually consistent

42

Oracle Database 12.2. Sharding for huge scale-up

• Multiple (potentially distributed) partitions– Individually consistent – Mutually joinable

Recent Trends around the Database

• Self managing => role of DBA• Cloud• Appliances & Engineered Systems (the Oracle Exadata machine)

– From I/O bound back to CPU bound – Software in Silicon

• Oracle Database Release 12c– Multitenant, In Memory Database, Temporal Validity, Data Masking, JSON, Node.js

Cloud

45

The Cloud Pyramid

46

HaaS?

47

Why is the cloud hot?

• Web Scale for the rest of us– Availability, Scalability, Security, …

• No specific expertise or even effort for consuming services– No need to acquire skills, build up experience and hire special resources

• No physical facilities – room, cooling, cleaning, physical security, power

• No/Low initial investment (low entry level)• Pay per use (no gain, no pain) (CAPEX => OPEX)• Flexible up and down scaling (capacity on demand)• Quick Ramp-Up, Rapid Start• Enterprise functionality for small businesses

– Out of reach in the on premises world• Accessibility from anywhere• Global market place • Lifting the burden from your shoulders

SaaSPaaSIaaS

48

Cloud adoption in bottom-up steps

IaaS/PaaS

Self study

PoC

Training

Load Test

Func Test

Peak, Failover

Peripheral Applications

Backup

BI

Edge Systems

Core Systems & Secure Data

[Distributed] Development

BPO

Database Cloud

• Automated (self-service) provisioning of database resources• Multi-tenant• Metering and per-usage billing• 24/7 uptime – hot patching, fail-over• Fine grained recovery, upgrade, authorizations• Scalable

• ‘The Oracle Public Cloud’ IaaS

PaaS

SaaS

50

CEO Cloud

51

You need a grid

Nah, you don’t need

a cloud

What about just a cute

little private cloud?

Are you sure you wouldn’t

prefer an engineered

system?

Today, we announce

the Oracle PubliC CloudA preview

of the beta of

the prototype is almost

…

Nimbus, Cirrus, Stratus, Incus,

Pannus, … any cloud under the sun. And

yes that takes a while!

The boss is off sailing. I’m here to

launch some new cloud services

52

53

Bring Oracle’s leading Infrastructure, Technology, Business Applications, and

Information to customers and partners anywhere in the World through the Oracle

Cloud

Oracle Cloud: Mission

55

Four Cloud Tier approach

56

=≈

57

Oracle Public Cloud Stack

58

Oracle Public PaaS Platform

59

Hardware and Software Engineered to work together

Appliances & Engineered Systems

• Pre-configured, installed, plug’n’play• One stop solution in case of issues• Software/hardware mutually optimized (*

– Infini-band, Storage Cells, Flash-memory (between RAM & Disk)

61

Software in Silicon

62

Hardware and Software Engineered to work together

• Leveraging tailored hardware optimizations for best performance and security

Data Masking

• Gartner reports that: data masking should be mandatory for enterprises using copies of sensitive production data for application development, analytics or training.

• They also believe the market is expanding into production and unstructured data protection.

Data Redaction

• At runtime, you can optionally have the query results modified to reset/scramble/randomize sensitive data– Through ‘data redaction’ policies associated with tables and view and applied at

query time

• Because the data is masked in real-time, Data Redaction is well suited to environments in which data is constantly changing.

• You can create the Data Redaction policies in one central location and easily manage them from there.

SQL engine SQL

POLICY

POLICY

RESULTS

Oracle 12c (released 2013, first update Summer 2014, R2 in 2016)C is for…

Complete

Complementary

Cloud

Consolidation

ContainerCrowd

Control

Core

CentralCool Carefree

Classified

DBA is resource intensive

Managing dozens of databases means…

– Installing– Configuring– Securing– Monitoring– Patching– Upgrading– Backing up

• many database instances on potentially a large number of machines

• Using dedicated resources for each individual database instance– 20 processes– Memory for SGA– Disk space for generic objects such as most of the SYS schema

Multitenant Architecture

Oracle Database Architecture

• Requires memory, processes and database files

ERP

Database Files

BackgroundProcesses

MemoryUtilized

RAM

Oracle Database Architecture

• Separate memory and processes required for each database

ERP

Database Files

MemoryUtilized

BackgroundProcesses

RAM

CRM

Database Files

BackgroundProcesses

MemoryUtilized

RAM

DW

Database Files

BackgroundProcesses

MemoryUtilized

RAM

Pluggable Database Architecture

ERP

Database Files

MemoryUtilized

BackgroundProcesses

CRM

Database Files

MemoryUtilized

BackgroundProcesses

RAM

DW

Database Files

MemoryUtilized

BackgroundProcesses

RAMRAM

• Memory and processes required at container level only

New Pluggable Database Architecture

• Memory and processes required at container level only

RAM ERP CRM DW

BackgroundProcesses

MemoryUtilized Container Database

Multitenant Architecture

78

Big News…

79

Oracle Database In Memory Option

80

Hybrid, dual-format data storage transparent to application

81Free Software!!!Oracle Technology Networkhttp://otn.oracle.com

• Gratis download van vrijwel alle Oracle producten• Gebruiken voor studie, onderzoek, prototype development, …• Geen beperking in tijd of functionaliteit• Betalen als je een

systeem in productieneemt

• Handig: Pre-BuiltVirtual Machines– PHP– BigData– SOA & BPM– Java EE– BI– Solaris– Oracle Database EE

82

Oracle XE Database – free!

http://www.oracle.com/technetwork/database/database-technologies/express-edition/overview/index.html

83

Summary

• Database is core of enterprise IT– Persistent when the plug is pulled

• Consolidation is important theme– More efficient usage of hardware resources and

of human resources– Agile scaling – quick, simple, cheap– Cloud as ultimate target– Also: “logical consolidation” – centralize data

• Database is bottleneck – a single point of failure• Logical consistency demands require transactions to use locks on data –

scalability at database level is reduced as a result– CAP theorem – consistency, availability and partition tolerance can not all be

achieved• Evolution of databases continues rapidly – the relevance of the Database

Platform is quite strong today (and what about tomorrow?)• IT is a smart career choice – and AMIS is a fun company

• Blog: technology.amis.nl– On Oracle, SQL, Java, SOA, BPM & more

• Email: lucas.jellema@amis.nl

• : lucasjellema

• : www.amis.nl, info@amis.nl +31 306016000 Edisonbaan 15,

Nieuwegein