Post on 15-Apr-2017
http://purdygoodengineering.com http://anant.us
Introduction● Section 1: Understanding the Technology
○ Big Picture○ Accumulo ○ Spark○ Example Code
● Section 2: Use Cases○ Multi-Tenant Data Processing○ Machine Learning / Graph Processing in Spark ○ Example ML + Graph on Business Data
● Questions and Answers● Contact Information
http://purdygoodengineering.com http://anant.us
● Section 1: Understanding the Technology○ Big Picture
■ Why Accumulo■ Why Spark
○ Accumulo■ Key/Value Structure■ Table Structure■ Cell Level Security■ Splits■ Reads (scans)■ Writes (upserts)■ Deletes
○ Spark■ Batch/Streaming■ Machine Learning■ Graph Processing
○ Example Code■ Writing to Accumulo■ Reading from Accumulo■ Shell
Section 1: Understanding the Technology
http://purdygoodengineering.com http://anant.us
Section 1: Big Picture● Accumulo
○ Scalable, sorted, distributed key/value store with cell level security● Spark
○ General compute engine for large-scale data processing■ Batch Processing■ Streaming■ Machine Learning Library■ Graph Processing
● Use Spark for Compute and Accumulo for storage for a security distributed scalable solution
http://purdygoodengineering.com http://anant.us
● Section 1: Understanding the Technology○ Big Picture
■ Why Accumulo■ Why Spark
○ Accumulo■ Key/Value Structure■ Table Structure■ Cell Level Security■ Splits■ Reads (scans)■ Writes (upserts)■ Deletes
○ Spark■ Batch/Streaming■ Machine Learning■ Graph Processing
○ Example Code■ Writing to Accumulo■ Reading from Accumulo■ Shell
Section 1: Understanding the Technology
http://purdygoodengineering.com http://anant.us
Section 1: Accumulo: Key Structure
(image from accumulo.apache.org)
http://purdygoodengineering.com http://anant.us
Section 1: Accumulo: Key StructureAccumuloTableDesign
RDBMTableDesign
http://purdygoodengineering.com http://anant.us
Section 1: Accumulo: Table Structure
● Each table has many tablets (distributed across nodes)● Tablet servers are replicated (default is 3)● Each row resides on the same tablets
○ A Row Id design strategy needs to ensure binning is evenly distributed
○ Each table has “splits” which determine binning○ If Row Ids are still too large; a sharding strategy is
required
http://purdygoodengineering.com http://anant.us
Section 1: Accumulo: Cell Level Security
● Each cell (or field) has its own access control determined by visibility
● Each user has authorizations which correspond to visibilities
● Only fields with visibilities which a user has authorization to access can be retrieved by that user
● Visibilities have limited logic such as AND and OR ○ e.g. private | system public & dna_partner
http://purdygoodengineering.com http://anant.us
Section 1: Splits
● Each table has a default split● Splits can be added to tables● Accumulo auto splits when tablets get to large● Table splits and tablet max size can is configurable● Row ids are generally hashed to support distribution● Example splits based on hashing
○ 0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f
http://purdygoodengineering.com http://anant.us
Section 1: Accumulo Reads
● Reads (are scans)○ Scanner○ BatchScanner (parallelizes over ranges)
● MapReduce/Spark○ AccumuloInputFormat (one field at a time)○ AccumuloRowInputFormat (one row at a time)
http://purdygoodengineering.com http://anant.us
Section 1: Accumulo: Writes
● Writes○ Writer○ BatchWriter (parallelizes over tablets)
● MapReduce/Spark○ AccumuloOutputFormat○ AccumuloFileOutputFormat (bulk ingest)
● Both use Mutations to write to accumulo
http://purdygoodengineering.com http://anant.us
Section 1: Accumulo: Mutations (write and delete)
● Mutations are used to write and delete● Mutation.put (to write)● Mutation.putDelete (to delete)● Writes are Upserts (insert or updates)
http://purdygoodengineering.com http://anant.us
Section 1: Accumulo
● accumulo.apache.org● Download accumulo● Examples● Documentation
Concerned about scalling; how about 4T Nodes, 70T edges in a graph => see link http://www.pdl.cmu.edu/SDI/2013/slides/big_graph_nsa_rd_2013_56002v1.pdf
http://purdygoodengineering.com http://anant.us
● Section 1: Understanding the Technology○ Big Picture
■ Why Accumulo■ Why Spark
○ Accumulo■ Key/Value Structure■ Table Structure■ Cell Level Security■ Splits■ Reads (scans)■ Writes (upserts)■ Deletes
○ Spark■ Batch/Streaming■ Machine Learning■ Graph Processing
○ Example Code■ Writing to Accumulo■ Reading from Accumulo■ Shell
Section 1: Understanding the Technology
http://purdygoodengineering.com http://anant.us
Section 1: Spark: MapReduce first
● Hadoop MapReduce (batch processing)○ Mapping○ Reducing○ Chain jobs○ 95% IO (each job must read/write to disk)○ scalable
http://purdygoodengineering.com http://anant.us
Section 1: Spark
● Batch Processing - MapReduce (many more functions)● Streaming - mini batch processing● Machine Learning - MLLib● Graph Processing - GraphX● Many Languages - (Java, Scala, Python, R)
http://purdygoodengineering.com http://anant.us
Section 1: Spark
● spark.apache.org● Download spark ● Example code● Documentation
http://purdygoodengineering.com http://anant.us
○ Spark■ Batch/Streaming■ Machine Learning■ Graph Processing
○ Example Code■ Writing to Accumulo■ Reading from Accumulo■ Shell
Section 1: Understanding the Technology● Section 1: Understanding the Technology
○ Big Picture■ Why Accumulo■ Why Spark
○ Accumulo■ Key/Value Structure■ Table Structure■ Cell Level Security■ Splits■ Reads (scans)■ Writes (upserts)■ Deletes
http://purdygoodengineering.com http://anant.us
Section 1: Example Code
Simple Examples for bookkeeping with spark and accumulo
https://github.com/matthewpurdy/purdy-good/tree/master/purdy-good-spark/purdy-good-spark-accumulo
http://purdygoodengineering.com http://anant.us
Section 2: Use Case(s) Machine Learning and Graph Processing● Multi-Tenant Data Processing● Machine Learning / Graph Processing in Spark ● Example Usecase of ML + Graph on Business Data
http://purdygoodengineering.com http://anant.us
Section 2: Multi-Tenant Data Processing NeedsCustomer (C) (P) & (C) Provider (P)
Team Customer Private Customer Data shared w/ Provider
Private Provider Data for Economy of Scale
SalesMarketing
IBM IndicatorsRelationshipsClassification
Classification ModelRelationship Graph
MarketingFinance
Apple IndicatorsCorrelationPrediction
Correlation ModelPrediction Model
SalesMarketingFinance
Microsoft IndicatorsRelationshipsCorrelationPrediction
Correlation ModelPrediction ModelRelationship Graph
Finance Google IndicatorsCorrelationPrediction
Correlation ModelPrediction Model
http://purdygoodengineering.com http://anant.us
Section 2: Multi-Tenant Data Processing NeedsCustomer (C) (P) & (C) Provider (P)
C User C Team C Management C ManagementP Analytics
P AnalyticsP Support
CU ManagerCU Employee
CT Sales CM Executive CM ExecutiveCU ManagerPA * / PS *
PA * / PS *
CU ManagerCU Employee
CT Marketing CM Executive CM ExecutiveCU ManagerPA * / PS *
PA * / PS *
CU Employee CT Research CM Executive CM ExecutiveCU ManagerPA * / PS *
PA * / PS *
CU Employee CT Finance CM Executive CM ExecutiveCU ManagerPA * / PS *
PA * / PS *
http://purdygoodengineering.com http://anant.us
Section 2: Multi-Tenant Data Processing Needs● Analyze Sales Team successes (Closed Accounts) to recommend companies
to target for Marketing campaigns. ● Analyze Sales Team User social account against social network users against
recommended companies to create Call Lists ● Correlate historic Marketing (Traffic & Conversions) with historic Sales (Leads
& Closed Accounts) data with historic Finance ( Revenue & Profit) to Predict Sales from current Marketing & Sales activities
http://purdygoodengineering.com http://anant.us
Section 2: Out of the Box : MLLib in Spark● Classification● Regression● Decision Trees● Recommendation● Clustering● Topic Modeling● Feature Transformations● ML Pipelining / Persistence
● “Based on past performance in the companies in the CRM, the most successful sales have come from these categories, so go after these companies.”
http://purdygoodengineering.com http://anant.us
Section 2: Out of the Box : MLLib in Spark● Load Data● Extract Features● Train Model● Find Best Model● Use Model to Predict
http://purdygoodengineering.com http://anant.us
Section 2: KeystoneML - End to End ML
http://keystone-ml.org/
http://purdygoodengineering.com http://anant.us
Section 2: Out of the Box : GraphX in Spark● PageRank● Connected components● Label propagation● SVD++● Strongly connected components● Triangle count
● “Based on the social graph of sales team members and the companies in your CRM, talk to the companies you are most “closest” to.
http://purdygoodengineering.com http://anant.us
Section 2: Out of the Box : GraphX in Spark● Load Nodes RDD● Load Vertices RDD● Create Graph from
Nodes & Vertices RDD● Run Graph Process /
Query● Get Data
http://ampcamp.berkeley.edu/big-data-mini-course/graph-analytics-with-graphx.html
http://purdygoodengineering.com http://anant.us
Section 2: Out of the Box : GraphX in Spark● Load Edges into Graph● Run Page Rank● Load Nodes into RDD● Join Users RDD with
Rank
http://purdygoodengineering.com http://anant.us
Contact InformationMatthew Purdy
● matthew.purdy@purdygoodengineering.com● http://www.purdygoodengineering.com ● https://www.linkedin.com/in/matthewpurdy● https://github.com/matthewpurdy
Rahul Singh
● rahul.singh@anant.us● http://www.anant.us● http://www.linkedin.com/in/xingh● https://github.com/xingh