Database Group, Georgia Tech 1 SQL SQL - intergalactic dataspeak [Stonebraker]
One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker
-
Upload
michael-stokes -
Category
Documents
-
view
25 -
download
0
description
Transcript of One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker
![Page 1: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/1.jpg)
© Copyright StreamBase®. Proprietary & Confidential.www.streambase.com 1
One Size Fits All: An Idea Whose Time has Come and Gone
Michael Stonebraker
![Page 2: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/2.jpg)
www.streambase.com 2
Alternate TitleAlternate Title
The elephants are selling 30 year old “bloatware”
That is not good at anything
And you should send them to the “home for old software”
![Page 3: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/3.jpg)
www.streambase.com 3
Three Financial Services MarketsThree Financial Services Markets
Stream processing (electronic trading)
Tick stores (data warehouses)
OLTP (transaction processing)
![Page 4: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/4.jpg)
www.streambase.com 4
Stream ProcessingStream Processing(Electronic Trading)(Electronic Trading)
A feed comes out of the wall
Compute a “secret sauce” looking for events of interest
Trade based on the result
But only if you are more nimble than the next guy….
![Page 5: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/5.jpg)
www.streambase.com 5
Traditional RDBMS ModelOutbound Processing
Store the data before processing!
LatencyWhat if the data is not important?
Too many processes! Optimized for business data
processing Where you don’t trust the app.
Queries
Memory
Disk
Updates
Processing
Too slow to be interesting!
![Page 6: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/6.jpg)
www.streambase.com 6
Stream Processing Engine with StreamSQL
Database paradigm (SQL) a good one
But need a different architecture
Straight through processing
No task switches
Lightweight scheduling
Inbound Processing
Memory
Disk
StreamBase Application
Event Data
Queries
Alerts Actions
Alerts Actions
Streambase Application
![Page 7: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/7.jpg)
www.streambase.com 7
• Example: Every minute for every stock I am trading: Calculate VWAP (vol. weighted avg. price) for my trades & all trades Alert whenever my personal trading execution is inferior to market
5 Streambase operators, 30 min to build Streams of “tuples” (time-series data) flow through query
Queries run continuously
StreamSQL Application Example
”
Market_Feeds
My_Buys
Alerts
![Page 8: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/8.jpg)
www.streambase.com 8
StreamSQL Will Dominate Rule Engines
Essentially all applications entail a mix of stored and real-time dataStreamSQL covers both kinds of data in a single paradigmA rule engine must switch paradigms
StreamSQL amenable to compilationKnow what is the next event to processIn contrast, hard to figure this out in a rule engine
![Page 9: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/9.jpg)
www.streambase.com 9
Performance Benchmark
Financial Services Application:
Construct a virtual feed of “first arrivers” on a low end Linux machine
Relational DB: 11,000 messages/secStreambase: 300,000 messages/secAnother StreamSQL vendor: 20,000 messages/sec
Result: Streambase was a factor of 27 faster
![Page 10: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/10.jpg)
www.streambase.com 10
Tick Stores Tick Stores (and Other Warehouse Applications)(and Other Warehouse Applications)
Store all market data for the last 10 years
To back test “secret sauce” models
To answer ad-hoc queries – “how many times has X happened”
Typical size – 100 Tbytes
Append only
![Page 11: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/11.jpg)
www.streambase.com 11
Terminology -- “Row Store”
Record 2
Record 4
Record 1
Record 3
E.g. DB2, Oracle, Sybase, SQLServer, …
![Page 12: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/12.jpg)
www.streambase.com 12
Rotate Your Thinking 90 Degrees Rotate Your Thinking 90 Degrees
Column stores read only the columns required
Not all of them
Compression works better
By a factor of 2-3 against the elephants
No record headers
Which are big ticket items
No padding to byte or word boundaries
![Page 13: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/13.jpg)
www.streambase.com 13
Benchmark SummaryBenchmark Summary
Vertica has been baked off about 30 times
Typically against the incumbent
Has yet to win by less than a factor of 30 against a
row store
Beats most other column stores by around 10X
KX is the only system to come within an order of
magnitude
![Page 14: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/14.jpg)
www.streambase.com 14
Maybe Elephants are Good Maybe Elephants are Good at OLTP……at OLTP……
OLTP is a main memory market
Not a disk-based one
Transactions are short and have no I/O or user stalls
Run to completion (single threaded)
Disaster Recovery (and HA) a requirement
Build it into the bottom of the system
![Page 15: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/15.jpg)
www.streambase.com 15
TPC-C Performance TPC-C Performance on a Low-end Machineon a Low-end Machine
Elephant
850 TPS (1/2 the land speed record per processor)
H-Store (so far – a university prototype)
70,416 TPS (41X the land speed record per processor)
Factor of 82!!!!!
![Page 16: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/16.jpg)
www.streambase.com 16
Implications for the ElephantsImplications for the Elephants
They are selling “one size fits all”
Which is 30 year old legacy technology that is good at nothing
![Page 17: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/17.jpg)
www.streambase.com 17
Pictorially:
OLTPData Warehouse
Streaming data
DBMS apps
![Page 18: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/18.jpg)
www.streambase.com 18
The DBMS Landscape – Performance Needs
OLTPData Warehouse
Streaming data
low
high
high
high
![Page 19: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/19.jpg)
www.streambase.com 19
One Size Does Not Fit All -- Pictorially
Open source
Vertica H-Store
successors
StreambaseElephants get only “the crevices”
![Page 20: One Size Fits All: An Idea Whose Time has Come and Gone Michael Stonebraker](https://reader035.fdocuments.net/reader035/viewer/2022062721/568136f8550346895d9e899c/html5/thumbnails/20.jpg)
© Copyright StreamBase®. Proprietary & Confidential.www.streambase.com 20
Thank You
Member
Corporate Headquarters181 Spring StreetLexington, Massachusetts 02421+1 866 STRMBAS+1 866 787 6227+1 781 761 0800
New York City Office220 West 42nd Street, 20th FloorNew York, New York 10036+1 866 STRMBAS+1 866 787 6227
Reston, Virginia Office11921 Freedom Drive, Suite 550 Reston, VA 20190+1 703 608 6958
London Office107-111 Fleet StreetLondon EC4A 2ABUnited Kingdom+44 (0)20 7936 9050