A Scalable Data Transformation Framework

using the Hadoop Ecosystem

Raj NairDirector–Data Platform

Kiru Pakkirisamy CTO

AGENDA

• About Penton and Serendio Inc

• Data Processing at Penton

• PoC Use Case

• Functional Aspects of the Use Case

• Big Data Architecture, Design and Implementation

• Lessons Learned• Lessons Learned

• Conclusion

• Questions

About Penton

• Professional information services company

• Provide actionable information to five core markets

Agriculture Transportation Natural

Products

Infrastructure Industrial Design

& Manufacturing

Success StoriesSuccess Stories

EquipmentWatch.com Govalytics.com

Prices, Specs, Costs, Rental Analytics around Gov’t capital spending

down to county level

SourceESB NextTrend.com

Vertical Directory, electronic parts Identify new product trends in the natural

products industry

About Serendio

Serendio provides Big Data Science

Solutions & Services for

Data-Driven Enterprises.

www.serendio.com

Data Processing at PentonData Processing at Penton

What got us thinking?

• Business units process data in silos

• Heavy ETL

– Hours to process, in some cases days

• Not even using all the data we want • Not even using all the data we want

• Not logging what we needed to

• Can’t scale for future requirements

Data Processing Pipeline New

features

New

Insights

New

Products

Biz ValueAssembly Line

processing

The Data Processing Pipeline

InsightsProducts

Data Processing Pipeline

Penton examples

• Daily Inventory data, ingested throughout the day

(tens of thousands of parts)

• Auction and survey data gathered daily

• Aviation Fleet data, varying frequency

AnalyzeIngest, store

Clean, validateApply Business Rules

Map

Analyze

Report

Distribute

Slow Extract, Transform and Load = Frustration + missed business SLAs

Won’t scale for future

Various data formats, mostly unstructured

Current Design

• Survey data loaded as CSV files

• Data needs to be scrubbed/mapped

• All CSV rows loaded into one table

• Once scrubbed/mapped data is loaded into main tables

• Not all rows are loaded, some may be used in the future

What were our options?

Adopt Hadoop Ecosystem

- M/R: Ideal for Batch Processing

- Flexible for storage

- NoSQL: scale, usability and flexibility

Expand RDBMS options

- Expensive

- Complex

HBASE OracleSQL

Server

Drools

POC Use CasePOC Use Case

Primary Use Case

• Daily model data – upload and map

– Ingest data, build buckets

– Map data (batch and interactive)

– Build Aggregates (dynamic)

Issue: Mapping timeIssue: Mapping time

Functional AspectsFunctional Aspects

Data Scrubbing

• Standardized names for fields/columns

• Example - Country

– Unites States of America -> USA

– United States -> USA

Data Mapping

• Converting Fields - > Ids

– Manufacturer - Caterpillar -> 25

– Model - Caterpillar/Front Loader -> 300

• Requires the use of lookup tables and partial/fuzzy

matching strings

Data Exporting

• Move scrubbed/mapped data to main RDBMS

Key Pain Points

• CSV data table continues to grow

• Large size of the table impacts operations on rows in a single

file

• CSV data could grow rapidly in the future

Criteria for New Design

• Ability to store an individual file and manipulate it easily

– No join/relationships across CSV files

• Solution should have good integration with RDBMS

• Could possibly host the complete application in future

• Technology stack should possibly have advanced analytics

capabilities

NoSQL model would allow to quickly retrieve/address

individual file and manipulate it

Big Data ArchitectureBig Data Architecture

Solution Architecture

REST API

CSV and Rule Management Endpoints

CSV

Files

Master database of

Products/ Parts

Current

Push

Updates

Insert

RDB-> Data Upload UI

API Calls Launch

Data manipulation APIs exposed through REST

layer

Existing Business

Applications

HBASE

HADOOP HDFS

Current

Oracle

Schema

Insert

Accepted

DataMR Jobs

Survey

RESTDrools

Use HBaseas a store for CSV

files

Drools –for rule

based data scrubbing

Operations on individual files in UI through

Hbase Get/Put

Operations on all/groups of

files using MR jobs

Hbase Schema Design

• One row per HBase row

• One file per HBase row

– One cell per column qualifier (simple and started the development

with this approach)

– One row per column qualifier (more performant approach)

Hbase Rowkey Design

• Row Key

– Composite

• Created Date (YYYYMMDD)

• User

• FileType

• GUID• GUID

• Salting for better region splitting

– One byte

Hbase Column Family Design

• Column Family

– Data separated from Metadata into two or more

column families

– One cf for mapping data (more later)

– One cf for analytics data (used by analytics

coprocessors)

M/R Jobs

• Jobs

– Scrubbing

– Mapping

– Export

• Schedule

– Manually from UI– Manually from UI

– On schedule using Oozie

Sqoop Jobs

• One time

– FileDetailExport (current CSV)

– RuleImport (all current rules)

• Periodic

– Lookup Table Data import– Lookup Table Data import

• Manufacture

• Model

• State

• Country

• Currency

• Condition

• Participant

Application Integration - REST

• Hide HBase AP/Java APIs from rest of

application

• Language independence for PHP front-end

• REST APIs for

– CSV Management– CSV Management

– Drools Rule Management

Lessons LearnedLessons Learned

Performance Benefits

• Mapping

– 20000 csv files, 20 million records

– Time taken – 1/3rd of RDBMS processing

• Metrics

– < 10 secs vs (Oracle Materialized View)

• Upload a file• Upload a file

– < 10 secs

• Delete a file

– < 10 secs

Hbase Tuning

• Heap Size for

– RegionServer

– MapReduce Tasks

• Table Compression

– SNAPPY for Column Family holding csv data– SNAPPY for Column Family holding csv data

• Table data caching

– IN_MEMORY for lookup tables

Application Design Challenges

• Pagination – implemented using intermediate REST layer and

scan.setStartRow.

• Translating SQL queries

– Used Scan/Filter and Java (especially on coprocessor)

– No secondary indexes - used FuzzyRowFilter

– Maybe something like Phoenix would have helped

• Some issues in mixed mode. Want to move to 0.96.0 for • Some issues in mixed mode. Want to move to 0.96.0 for

better/individual column family flushing but needed to 'port'

coprocessors (to protobuf)

Hbase Value Proposition

• Better response in UI for CSV file operations - Operations

within a file (map, import, reject etc) not dependent on the

db size

• Relieve load on RDBMS - no more CSV data tables

• Scale out batch processing performance on the cheap (vs

vertical RDBMS upgrade)

• Redundant store for CSV files

• Versioning to track data cleansing

Roadmap

• Benchmark with 0.96

• Retire Coprocessors in favor of Phoenix (?)

• Lookup Data tables are small. Need to find a better alternative

than HBase than HBase

• Design UI for a more Big Data appropriate model

– Search oriented paradigm, than exploratory/ paginative

– Add REST endpoints to support such UI

Wrap-UpWrap-Up

Conclusion

• PoC demonstrated

– value of the Hadoop ecosystem

– Co-existence of Big data technologies with current solutions

– Adoption can significantly improve scale

– New skill requirements

Thank You

Rajesh.Nair@Penton.com Rajesh.Nair@Penton.com

Kiru@Serendio.com