Real Time Analytics via Spark & Scala | Spark & Scala Fundamentals | Spark & Scala Architecture

of 22 /22
Slide 1 © 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Real Time Analytics via Spark & Scala

Embed Size (px)

Transcript of Real Time Analytics via Spark & Scala | Spark & Scala Fundamentals | Spark & Scala Architecture

  1. 1. Slide 1 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Real Time Analytics via Spark & Scala
  2. 2. Slide 2 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Session Objectives Introduction to Big Data and Hadoop Understanding Spark Why Spark? Note on Scala Getting answers to interview questions on Spark Getting your doubts cleared
  3. 3. Slide 3 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Big Data and its Challenges
  4. 4. Slide 4 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Big Data and its Challenges Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications Systems / Enterprises generate huge amount of data from Terabytes to and even Petabytes of information Its very difficult to manage such huge data Get Started with Spark and Scala
  5. 5. Slide 5 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Who Generates Big Data? Have you ever wondered how Google, Facebook or LinkedIn manages to store and utilize the huge data? Today, it is becoming a problem for all of us to manage such BIG DATA. Get Started with Spark and Scala
  6. 6. Slide 6 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Spark - Lightens Fast Cluster Computing Apache Spark is an open source big data processing framework Spark is used at a wide range of organizations to process large datasets 1. Easy to develop 2. Interactive shell 3. Out of box functionality Get Started with Spark and Scala
  7. 7. Slide 7 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Why Spark? 100x Faster than Speed: Run programs up to 100x faster than Hadoop Map Reduce in memory, or 10x faster on disk. Get Started with Spark and Scala
  8. 8. Slide 8 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Why Spark? Ease of Use: Allows to write applications quickly in Java, Scala or Python. Get Started with Spark and Scala
  9. 9. Slide 9 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Why Spark? Generality: Combines SQL, streaming, and complex analytics. Get Started with Spark and Scala
  10. 10. Slide 10 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Why Spark? Runs Everywhere: Runs on Hadoop, Mesos, standalone, or in the cloud. Get Started with Spark and Scala
  11. 11. Slide 11 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Spark Architecture Spark Architecture includes following three main components: Data Storage API Management Framework API (Scala, Java, Python) Storage (HDFS other formats) Distributed Computing (Stand-alone, Mesos, YARN) Compute Interface Data Management Get Started with Spark and Scala
  12. 12. Slide 12 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Spark Ecosystem Spark Core Engine Shark (SQL) Spark Streaming (streaming) MLLib (Machine Learning) GraphX (Graph Computation) SparkR (R on Spark) BlinkDB (Approximate SQL) Alpha/Pre-alpha Get Started with Spark and Scala
  13. 13. Slide 13 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com The Spark Users Get Started with Spark and Scala
  14. 14. Slide 14 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Scala Scala is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way It integrates features of object-oriented and functional languages Publicly released in January 2004 on the JVM Platform and a few months later on the .NET platform Get Started with Spark and Scala
  15. 15. Slide 15 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Why Scala? Scala is a type-safe JVM language that incorporates both object oriented and functional programming into an extremely concise, logical, and extraordinarily powerful language Scala makes use of a lot of functional syntactic sugar that has become popular with developers and makes many developers characterize Scala as a more functional language Scala is Statically typed Language Being used heavily for future Big data and developments frameworks like Spark, Akka, Scalding etc. Get Started with Spark and Scala
  16. 16. Slide 16 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Job Trends Spark and Scala Get Started with Spark and Scala
  17. 17. Slide 17 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Why SkillSpeed? Course Curriculum from Industry Experts Instructor Led Live Virtual Sessions Lifetime access to Course Content via LMS 100% Placement Assistance 24x7 Support Get Started with Spark and Scala
  18. 18. Slide 18 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Course Topics Module 1 Getting Started / Introduction to Scala Module 2 Scala Essentials and Deep Dive Module 3 Introducing Traits and OOPS in Scala Module 4 Functional Programming in Scala Module 5 Spark and Big Data Module 6 Advanced Spark Concepts Module 7 Understanding RDDs Module 8 Shark, SparkSQL and Project Discussion Get Started with Spark and Scala
  19. 19. Slide 19 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Corporate Partners Get Started with Spark and Scala
  20. 20. Slide 20 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Lines open 24/7 To know more about the course, Please contact: IND +91-90660-20904 USA 1866-607-6547 (Toll Free) Or reach us at [email protected] Contact us.. Get Started with Spark and Scala
  21. 21. Slide 21 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Image References Google images credit for google, Facebook and LinkedIn LOGO and Snapshots http://pixshark.com/big-data-comic.htm http://findicons.com/icon/66444/user_group http://www.virtualizor.com/tour https://accounts.it.et.byu.edu/ http://www.clipartsfree.net/tag/server.html http://www.gopixpic.com/16/time-clock-icon-png-download http://blog.smartbear.com/requirements/how-to-interview-users-to-find-out-what-they-really-want/ http://www.lincs.fr/research/areas/big-data/ http://www.counsellingpages.co.uk/ http://langfordsconsultancy.com/langfords-training-support-package/ http://cbsepathshala.blogspot.in/2012/05/physics-class-x-chapter-electricity.html http://mmatycoon.com/tycoontimes/tycoontimesstory.php?SID=1010

Spark as a distributed Scala
Spark as a distributed Scala
Spark with Scala Developer - Sevenmentor Pvt. Ltd · Apache Spark 2 x Installation 1. Download release and set 2. Working with eclipse 3. Installing Scala IDE with spark 4. Testing
Spark with Scala Developer - Sevenmentor Pvt. Ltd · Apache Spark 2 x Installation 1. Download release and set 2. Working with eclipse 3. Installing Scala IDE with spark 4. Testing
Experiences Using Scala in Apache Spark
Experiences Using Scala in Apache Spark
An Introduction to Scala for Spark programmingenglishonlineclub.com/pdf/An Introduction to Scala for... · 2019. 9. 22. · An Introduction to Scala for Spark programming Nastaran
An Introduction to Scala for Spark programmingenglishonlineclub.com/pdf/An Introduction to Scala for... · 2019. 9. 22. · An Introduction to Scala for Spark programming Nastaran
Scala & Spark Msc. Johannes Härtel PTT18/19 Msc. Marcel ... · What is Scala? - Scala is a general purpose programming language. - Scala provides support for functional programming
Scala & Spark Msc. Johannes Härtel PTT18/19 Msc. Marcel ... · What is Scala? - Scala is a general purpose programming language. - Scala provides support for functional programming
Building a High-Performance Database with Scala, Akka, and Spark
Building a High-Performance Database with Scala, Akka, and Spark
Mahout Scala Bindings and Mahout Spark Bindings for Linear ...Mahout Scala Bindings and Mahout Spark Bindings for Linear Algebra Subroutines WorkingNotesandManual Dmitriy Lyubimov∗
Mahout Scala Bindings and Mahout Spark Bindings for Linear ...Mahout Scala Bindings and Mahout Spark Bindings for Linear Algebra Subroutines WorkingNotesandManual Dmitriy Lyubimov∗
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala
Scala and Spark Bindings manual - Apache Mahout · PDF fileMahout Scala Bindings and Mahout Spark Bindings for Linear Algebra Subroutines WorkingNotesandManual Dmitriy Lyubimov∗
Scala and Spark Bindings manual - Apache Mahout · PDF fileMahout Scala Bindings and Mahout Spark Bindings for Linear Algebra Subroutines WorkingNotesandManual Dmitriy Lyubimov∗
Database Systems CSE 414 - courses.cs.washington.eduSpark Interface •Spark supports a Scalainterface •Scala = extof Java with functions/closures –will show Scala/Spark examples
Database Systems CSE 414 - courses.cs.washington.eduSpark Interface •Spark supports a Scalainterface •Scala = extof Java with functions/closures –will show Scala/Spark examples
Spark编程基础（Scala 版）》
Spark编程基础（Scala 版）》
Spark Architecture · Spark Architecture Spark Shuffle ... Spark Shuffle Spark DataFrames . ... – Entry point of the Spark Shell (Scala, Python, R) – The place where SparkContext
Spark Architecture · Spark Architecture Spark Shuffle ... Spark Shuffle Spark DataFrames . ... – Entry point of the Spark Shell (Scala, Python, R) – The place where SparkContext
Fast, Distributed Geoprocessing with Scala, Spark and GeoTrellis
Fast, Distributed Geoprocessing with Scala, Spark and GeoTrellis
CRT020: Spark Certification Guide (Scala) · 2020-01-24 · How you should prepare for CRT020 Spark Scala/Python (Databricks) Certification Exam? ... Why Cloudera CCA175 Hadoop and
CRT020: Spark Certification Guide (Scala) · 2020-01-24 · How you should prepare for CRT020 Spark Scala/Python (Databricks) Certification Exam? ... Why Cloudera CCA175 Hadoop and
An Overview of Spark DataFrames with Scala
An Overview of Spark DataFrames with Scala
Reactive Dashboard’s Using Apache Spark · Prerequisite for cluster setup Spark runs on Java 6+, Python 2.6+ and R 3.1+. For the Scala API, Spark 1.4.1 uses Scala 2.10. Java 8 sudo
Reactive Dashboard’s Using Apache Spark · Prerequisite for cluster setup Spark runs on Java 6+, Python 2.6+ and R 3.1+. For the Scala API, Spark 1.4.1 uses Scala 2.10. Java 8 sudo
Zaharia spark-scala-days-2012
Zaharia spark-scala-days-2012
Big data apache spark + scala
Big data apache spark + scala
Scala+spark 2nd
Scala+spark 2nd
Spark rdd operations in scala part 2
Spark rdd operations in scala part 2