Wayne State University & DataStax: World's best data modeling tool for Apache Cassandra

World’s Best Data Modeling Tool

for Apache Cassandra

1 © 2015. All Rights Reserved.

Artem Chebotko

Andrey Kashlev

1 Cassandra Data Modeling Methodology

2 The KDM Tool

3 Live Demo: IoT

4 Live Demo: Media Cataloguing

5 Future Work


Data Modeling Process

• Data requirements

• Application requirements

• Schema Design

• Optimization


Cassandra Data Modeling Methodology

© 2015. All Rights Reserved. 4

Conceptual

Data Model

Application

Workflow

Logical

Data Model

Physical

Data Model Mapping Optimization

Methodology Models


Model Representation

Conceptual Data Model ERD

Application Workflow Model Graph

Logical Data Model Chebotko Diagram

Physical Data Model Chebotko Diagram, CQL

Methodology Protocols


• Conceptual-to-logical mapping

– Mapping rules

– Mapping patterns

• Physical optimizations

– Partition size analysis

– Duplication factor analysis

– Keys, aggregation, transactions, …

Example


SELECT timestamp, value FROM …

WHERE location = ? AND parameter = ? AND timestamp > ?

ORDER BY timestamp DESC

n

parameter value

1

timestampid location

Sensor Measurementrecords

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

1

Example



WHERE location = ? AND parameter = ?

AND timestamp > ?


n

parameter value

1



Mapping Entity and Relationship Types

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

1 2

Example




AND timestamp > ?


n

parameter value

1



Mapping Equality Search Atributes

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

sensor_data

location K

parameter K

timestamp C↑

id C↑

value

1 2 3

Example




AND timestamp > ?


n

parameter value

1



Mapping Inequality Search Attributes

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

sensor_data

location K

parameter K

timestamp C↑

id C↑

value

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

1 2 3 4

Example




AND timestamp > ?


n

parameter value

1



Mapping Ordering Attributes

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

sensor_data

location K

parameter K

timestamp C↑

id C↑

value

sensor_data

location K

parameter K

timestamp C↓

id C↑

value

1 2 3 4 5

Example




AND timestamp > ?


n

parameter value

1



Mapping Key Attributes

Methodology Pros and Cons

Correctness

Completeness


Complexity

Time investment

Human Errors Happen …


Automation


Complexity

Time investment

Human Error


2 The KDM Tool

3 Live Demo: IoT


5 Future Work


The KDM Tool

• Streamlines the methodology

• Guides the user

• Automates data modeling tasks:

– Conceptual-to-logical mapping

– Physical optimization

– CQL generation


KDM Automation Workflow




Design

Conceptual

Data Model

Step1

Solution

architect



Design

Conceptual

Data Model

Specify

Access

Patterns

Solution

architect

Step1 Step2

Solution

architect



Design

Conceptual

Data Model

Specify

Access

Patterns

Generate

Logical

Data

Models

KDM Solution

architect

Step1 Step2 Automated

Solution

architect



Design

Conceptual

Data Model

Specify

Access

Patterns

Generate

Logical

Data

Models

Select

Logical

Data

Model

KDM Solution

architect

Step1 Step2 Step3 Automated

Solution

architect

Solution

architect



Design

Conceptual

Data Model

Specify

Access

Patterns

Generate

Logical

Data

Models

Select

Logical

Data

Model

Generate

Physical

Data

Model

KDM Solution

architect

Step1 Step2 Step3 Automated Automated

Solution

architect

Solution

architect KDM



Design

Conceptual

Data Model

Specify

Access

Patterns

Generate

Logical

Data

Models

Select

Logical

Data

Model

Generate

Physical

Data

Model

Configure

Physical

Data

Model

KDM Solution

architect

Step1 Step2 Step3 Step4 Automated Automated

Solution

architect

Solution

architect

Solution

architect KDM



Design

Conceptual

Data Model

Specify

Access

Patterns

Generate

Logical

Data

Models

Select

Logical

Data

Model

Generate

Physical

Data

Model

Configure

Physical

Data

Model

Generate

Physical

Schema

KDM Solution

architect

Step1 Step2 Step3 Step4 Automated Automated Automated

Solution

architect

Solution

architect

Solution

architect KDM KDM



Design

Conceptual

Data Model

Specify

Access

Patterns

Generate

Logical

Data

Models

Select

Logical

Data

Model

Generate

Physical

Data

Model

Configure

Physical

Data

Model

Generate

Physical

Schema

Download

CQL

Script

KDM Solution

architect

Step1 Step2 Step3 Step4 Step5 Automated Automated Automated

Solution

architect

Solution

architect

Solution

architect

Solution

architect KDM KDM


2 The KDM Tool

3 Live Demo: IoT


5 Future Work



• KDM:

– automates most complex tasks

– eliminates human error

– simplifies data modeling

– guides

– is a general purpose tool

Summary


• build new data models

• verify existing data models

• teach/learn data modeling

How Can KDM Help You?


2 The KDM Tool

3 Live Demo: IoT


5 Future Work


Future Work

• Materialized views


Future Work


• User Defined Types


Future Work



• Analysis and physical optimization


Future Work




• Support for application workflow design


Future Work




• Support for application workflow design

• Support for Chebotko Diagrams


Sign up for KDM – it’s FREE!

• KDM: kdm.dataview.org

• Methodology: academy.datastax.com

• Planet Cassandra blog posts:

– KDM: An Automated Data Modeling Tool for Apache

Cassandra, Pt. 1, Pt. 2

• Artem Chebotko, Andrey Kashlev, Shiyong Lu,

“A Big Data Modeling Methodology for Apache Cassandra”,

IEEE International Congress on Big Data, 2015.


http://kdm.dataview.org/

http://academy.datastax.com/

Acknowledgements

• Andrey Kashlev would like to thank:

– Dr. Shiyong Lu

– Anthony Piazza

• Artem Chebotko would like to thank:

– Anthony Piazza

– Patrick McFadin

– Jonathan Ellis

– Tim Berglund


Thank you

Wayne State University & DataStax: World's best data modeling tool for Apache Cassandra

Technology

Transcript of Wayne State University & DataStax: World's best data modeling tool for Apache Cassandra