Big data: Loading your data with flume and sqoop

22
Loading data in Hadoop 2 with SQOOP and Flume Christophe Marchal | Software Architect

description

Studying Hortonworks stack, I created this 10 minutes presentation. http://hortonworks.com

Transcript of Big data: Loading your data with flume and sqoop

Page 1: Big data: Loading your data with flume and sqoop

Loading data in Hadoop 2

with SQOOP and Flume

Christophe Marchal | Software Architect

Page 2: Big data: Loading your data with flume and sqoop

Problem to solve

Page 3: Big data: Loading your data with flume and sqoop

Hortonworks stack

Page 4: Big data: Loading your data with flume and sqoop

Batch Loading vs Stream Loading

Page 5: Big data: Loading your data with flume and sqoop

SQOOP

HCatalog

Page 6: Big data: Loading your data with flume and sqoop

SQOOP 1: Import

Page 7: Big data: Loading your data with flume and sqoop

SQOOP 1: Export

Page 8: Big data: Loading your data with flume and sqoop

SCOOP 2

Page 9: Big data: Loading your data with flume and sqoop

Flume

AgentWeb Server

Source

Channel

Sink

HDFSAgent

Source

Channel

Sink

Agent

Source

Channel

Sink

Agent

Source

Channel

Sink

Web ServerWeb

Server

Page 10: Big data: Loading your data with flume and sqoop

Multi agent flow

Page 11: Big data: Loading your data with flume and sqoop

Consolidation flow

Page 12: Big data: Loading your data with flume and sqoop

Flume vs SQOOP

● distributed

● reliable (transaction)

● available (backup

routes)

● collecting data

● aggregating data

● Data imports

● Parallelizes data

transfer

● Copies data quickly

Page 13: Big data: Loading your data with flume and sqoop

Flume example

Page 14: Big data: Loading your data with flume and sqoop

Flume example

Page 15: Big data: Loading your data with flume and sqoop

Flume example

Page 16: Big data: Loading your data with flume and sqoop

SQOOP: import HDFS

Page 17: Big data: Loading your data with flume and sqoop

SQOOP: import HDFS

Page 18: Big data: Loading your data with flume and sqoop

SQOOP: import HDFS

Page 19: Big data: Loading your data with flume and sqoop

SQOOP: import Hive

Page 20: Big data: Loading your data with flume and sqoop

SQOOP: import Hive

Page 21: Big data: Loading your data with flume and sqoop

SQOOP: import Hive

Page 22: Big data: Loading your data with flume and sqoop

Thanks

Christophe Marchal | Software Architect @toff63

https://twitter.com/toff63
https://twitter.com/toff63

Big data components - Introduction to Flume, Pig and Sqoop

Big data components - Introduction to Flume, Pig and Sqoop

Big Spatial Data - Esri · HBase Impala Spark Pig Cascading Sqoop Flume. What Next? Load data! hadoop fs -put N:\07_2014_gps.txt \user\vehicle\data.txt •Command line interface for

Big Spatial Data - Esri · HBase Impala Spark Pig Cascading Sqoop Flume. What Next? Load data! hadoop fs -put N:\07_2014_gps.txt \user\vehicle\data.txt •Command line interface for

Apache Flume & Apache Sqoop Data Ingestion to Apache ... · HBase and Hive and then export the data back after transforming it using Hadoop MapReduce. ... , enabling fast and simple

Apache Flume & Apache Sqoop Data Ingestion to Apache ... · HBase and Hive and then export the data back after transforming it using Hadoop MapReduce. ... , enabling fast and simple

Big Data Hadoop & Spark · 2021. 1. 8. · FLUME, SQOOP, AND HBASE 7.1 Apache Sqoop introduction 7.2 Importing and exporting data 7.3 Performance improvement with Sqoop 7.4 Sqoop

Big Data Hadoop & Spark · 2021. 1. 8. · FLUME, SQOOP, AND HBASE 7.1 Apache Sqoop introduction 7.2 Importing and exporting data 7.3 Performance improvement with Sqoop 7.4 Sqoop

Distributed Systems Fall 2014 Zubair Amjad. Outline Motivation What is Sqoop? How Sqoop works? Sqoop Architecture Import Export Sqoop Connectors Sqoop.

Distributed Systems Fall 2014 Zubair Amjad. Outline Motivation What is Sqoop? How Sqoop works? Sqoop Architecture Import Export Sqoop Connectors Sqoop.

Advanced Sqoop

Advanced Sqoop

Big Data Workflows with Oozie and Sqoop - GitHub · Big Data Workflows with Oozie and Sqoop. What is Oozie? •A workflow engine for actions in a Hadoop cluster ... •Support parallel

Big Data Workflows with Oozie and Sqoop - GitHub · Big Data Workflows with Oozie and Sqoop. What is Oozie? •A workflow engine for actions in a Hadoop cluster ... •Support parallel

Apache Flume™ and Apache Sqoop™ Data Ingestion to Apache ...

Apache Flume™ and Apache Sqoop™ Data Ingestion to Apache ...

Teradata Connector for Hadoop User Guide · Web viewFor example, it can be integrated with Sqoop so Sqoop users can use the Sqoop command line interface to move data between Teradata

Teradata Connector for Hadoop User Guide · Web viewFor example, it can be integrated with Sqoop so Sqoop users can use the Sqoop command line interface to move data between Teradata

Hortonworks: We Do Hadoop.hortonworks.com/wp-content/uploads/2013/11/Webinar.HDP2_.201311… · Hortonworks: We Do Hadoop. ... Hadoop Pig HCatalog Hive HBase Sqoop Flume Oozie Zookeeper

Hortonworks: We Do Hadoop.hortonworks.com/wp-content/uploads/2013/11/Webinar.HDP2_.201311… · Hortonworks: We Do Hadoop. ... Hadoop Pig HCatalog Hive HBase Sqoop Flume Oozie Zookeeper

Kelly TechnologiesSQOOP Introduction to SQOOP Use of SQOOP Connect to different databases SQOOP commands Import Detailed scenarios on importing data / perfor- mance tuning Export Eval

Kelly TechnologiesSQOOP Introduction to SQOOP Use of SQOOP Connect to different databases SQOOP commands Import Detailed scenarios on importing data / perfor- mance tuning Export Eval

Mesoscon2017 - Streaming Data Pipelines on Mesos - …...Spark job #1 Spark job #2 Flume Sqoop DBs Slave Node DiskDiskDisk Node Mgr Data Node Master Resource Manager Name Node. 8 YARN

Mesoscon2017 - Streaming Data Pipelines on Mesos - …...Spark job #1 Spark job #2 Flume Sqoop DBs Slave Node DiskDiskDisk Node Mgr Data Node Master Resource Manager Name Node. 8 YARN

Sqoop on Spark for Data Ingestion-(Veena Basavaraj and Vinoth Chandar, Uber)

Sqoop on Spark for Data Ingestion-(Veena Basavaraj and Vinoth Chandar, Uber)

Flume 1.8.0 User Guide — Apache Flume · of event data including but not limited to network traffic data, social-media-generated data, email messages and pretty much any data source

Flume 1.8.0 User Guide — Apache Flume · of event data including but not limited to network traffic data, social-media-generated data, email messages and pretty much any data source

Sqoop User Guide

Sqoop User Guide

Total Duration: 80 Hours (10 Days)€¦ · Pig, Hive, Impala, HBase, Sqoop, Flume, Oozie, Zookeeper, Spark and Storm. The course will also include Spark, along with hands-on integration

Total Duration: 80 Hours (10 Days)€¦ · Pig, Hive, Impala, HBase, Sqoop, Flume, Oozie, Zookeeper, Spark and Storm. The course will also include Spark, along with hands-on integration

Lightening-Speed Data Processing with Big Data...Flume Data Ingestion Apache Flume is a tool designed for streaming the unstructured data into the Hadoop Eco System. It can read massive

Lightening-Speed Data Processing with Big Data...Flume Data Ingestion Apache Flume is a tool designed for streaming the unstructured data into the Hadoop Eco System. It can read massive

Sqoop In Actionstatic.roncoo.com/lecturer/da2ea00e057547a18320914bef7dc9e4.pdf · • Sqoop is a tool designed to transfer data between Hadoop and relational databases or mainframes.

Sqoop In Actionstatic.roncoo.com/lecturer/da2ea00e057547a18320914bef7dc9e4.pdf · • Sqoop is a tool designed to transfer data between Hadoop and relational databases or mainframes.

Sqoop 2 Introduction

Sqoop 2 Introduction

SQOOP HCatalog Integration Venkat Ranganathan Sqoop Meetup 10/28/13.

SQOOP HCatalog Integration Venkat Ranganathan Sqoop Meetup 10/28/13.