James Corcoran, Head of Engineering EMEA, First Derivatives, "Simplifying Big Data Architecture...
33
Simplifying Big Data Architecture Paris 1 st December 2016
-
Upload
dataconomy-media -
Category
Data & Analytics
-
view
132 -
download
0
Transcript of James Corcoran, Head of Engineering EMEA, First Derivatives, "Simplifying Big Data Architecture...
Simplifying Big Data Architecture
Paris 1st December 2016
2 | KX SYSTEMS
Agenda
• About
• Kx Technology
• Solutions
• Technical Overview
• Resources
• Q & A
3 | KX SYSTEMS
About
First Derivatives plc
People1,600 FTEs
LocationsIreland, London, New York,
Singapore,Sydney, Tokyo, Toronto, Zurich
HistoryFounded in 1996
Public listed CompanyAlways Profitable
Divisions
KxBig Fast Data
Capital Markets Consulting
Domain expertsTechnology experts
MRP/Prelytix Digital marketing
resources
Our Software
KxStreaming Analytics, In Memory Computing, - the market standard
Tools for KxDevelop, Deploy,
Visualize
Industry SolutionsPowered by Kx
Implemented and supported by us
4 | KX SYSTEMS
Kx Technology Overview
• Integrated columnar database & programming system
• Streaming, real time and historical data
• Built for massive data volumes
• In-database analytics
• Parallelism
• Compression
• More
The kdb+ database Features
• Interpreted
• Event-driven
• Functional
• Array / Vector
• Query
• Time-series
q Programming Language
5 | KX SYSTEMS
Tools for Kx Solutions
Solutions Overview
• Dashboards for Kx
• Analyst for Kx
• Control for Kx
• Stream for Kx
• Kx for Algos
• Kx for Flow
• Kx for Monitoring
• Kx for Pharma
• Kx for Surveillance
• Kx for Sensors
• Kx for Telcos
Visualise&
Discover
Architect&
Manage
Enterprise Concepts
6 | KX SYSTEMS
kdb+
Technical Overview
7 | KX SYSTEMS
kdb+
• Integrated columnar database & programming system
• Streaming, real time and historical data
• 32 bit free version
• 64 bit licensed version
8 | KX SYSTEMS
kdb+ Features
• Standard OS & hardware
• In-database analytics
• Compression
• Parallelism
• Interfaces
9 | KX SYSTEMS
q programming language
• Functional
• Array/Vector
• Query
• Interpreted
• Time-series
10
q LanguageSeamless database and data manipulation language • We are fast not only due to data architecture, our native
programming language, q, runs inside the database not in separate processes with costly data passing.
• We are one of the few fully 64 bit databases and unique in having time as a native type, with nanosecond resolution and a full set of operations over time.
Quick to Deliver• Unlike many compile-link and run approaches q is
dynamic allowing much shorter development and deployment cycles
Allows you to get more from less code• q is terse - this means a smaller code base to maintain
and debug (one line in q versus many of Java).
Other Key Features• Array/Vector Language• Parallelism • Native Compression
Java
Project Euler Problem 1* in Java
Project Euler Problem 1* in q
*Find the Sum of all multiples of 3 or 5 below 1000. For more information on the Euler Programming Problems see - https://projecteuler.net
11
Sample q query
select open: first price, high: max price, low: min price, close: last price from trade where date = 2013.05.01, sym = `AAPL
12
q vs. standard SQL
• q supports order for faster execution• q eliminates many joins• Standard SQL does not enforce row order• Built-in support for CSV, JSON & XML• higher productivity & less maintenance
13
Parallel queries in q
Serial:
calcEvent each eventTypesParallel:
calcEvent peach eventTypes
14
Time-Series Analysis
• Native temporal data types• high-precision (nanosecond)• temporal arithmetic• asof join/bi-temporal joins• join time-series on-the-fly
15
Time-Series Analysis cont’d
select count i by userAgent,5 xbar time.minute from events where date within (.z.D - 7;.z.D), eventType=`login
aj[`time; select time,price from trade where date=last date,sym=`MSFT; select time,bid,ask from quote where date=last date,sym=`MSFT]
16
Streaming, Real-Time and Historical Data
17 | KX SYSTEMS
Sample Architecture
18 | KX SYSTEMS
Kx Solutions
Technical Overview
19 | KX SYSTEMS
Dashboards for Kx
20 | KX SYSTEMS
Analyst for Kx
21 | KX SYSTEMS
Streams for KxFast Data Management
• Clustered services
• Load-balancing
• Failover
• Automated recovery
• Query Manager
• Messaging Server
• Alert Framework
• File Watchers & Loaders
• Scheduling
22 | KX SYSTEMS
Kx
Resources
23 | KX SYSTEMS
Resources – Wiki Page
24 | KX SYSTEMS
Resources – Kx Community
25 | KX SYSTEMS
Resources – Forum & Google Group
26 | KX SYSTEMS
Resources – Github
27 | KX SYSTEMS
Resources – Q For Mortals
28 | KX SYSTEMS
Resources – Q Tips
29 | KX SYSTEMS
Resources – Free 32-bit download
30 | KX SYSTEMS
Resources – Kx Community Meetups
31 | KX SYSTEMS
Resources – Lecture Series
32 | KX SYSTEMS
Thank You.
kx.com@kxsystems
33 | KX SYSTEMS
Appendix
• Kx Wiki • Kx Community • Google Group • Kx Github • Q For Mortals• Q Tips • STAC benchmarks• Free version• Kx Meetups• FD Lecture Series
