Integrating Event Streams and File Data with Apache Flume and ...
Transcript of Integrating Event Streams and File Data with Apache Flume and ...
![Page 1: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/1.jpg)
Joey Echeverria | April 13, 2015
Integrating Event Streams and File Data with Apache Flume and Apache NiFi
![Page 2: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/2.jpg)
Data integration
![Page 3: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/3.jpg)
Data integration
• Multiple data source
![Page 4: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/4.jpg)
Data integration
• Multiple data source • Questions
![Page 5: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/5.jpg)
Challenges
![Page 6: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/6.jpg)
Challenges
• Unique sources
![Page 7: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/7.jpg)
Challenges
• Unique sources – Format
![Page 8: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/8.jpg)
Challenges
• Unique sources – Format – Schema
![Page 9: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/9.jpg)
Challenges
• Unique sources – Format – Schema – Protocol
![Page 10: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/10.jpg)
Challenges
• Unique sources – Format – Schema – Protocol – Batchiness
![Page 11: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/11.jpg)
Challenges
• Unique sources – Format – Schema – Protocol – Batchiness
• Big data
![Page 12: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/12.jpg)
Traditional (Hadoop) approach
![Page 13: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/13.jpg)
Traditional (Hadoop) approach
• In so far as anything with Apache Hadoop can be called “traditional”
![Page 14: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/14.jpg)
Traditional (Hadoop) approach
• Identify source class
![Page 15: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/15.jpg)
Traditional (Hadoop) approach
• Identify source class – Event streams
![Page 16: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/16.jpg)
Traditional (Hadoop) approach
• Identify source class – Event streams – Database tables
![Page 17: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/17.jpg)
Traditional (Hadoop) approach
• Identify source class – Event streams – Database tables – Files
![Page 18: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/18.jpg)
Traditional (Hadoop) approach
• Map class to system
![Page 19: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/19.jpg)
Traditional (Hadoop) approach
• Map class to system – Event streams è Apache Flume
![Page 20: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/20.jpg)
Traditional (Hadoop) approach
• Map class to system – Event streams è Apache Flume – Database tables è Apache Sqoop
![Page 21: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/21.jpg)
Traditional (Hadoop) approach
• Map class to system – Event streams è Apache Flume – Database tables è Apache Sqoop – Files è hdfs dfs -put?
![Page 22: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/22.jpg)
Integrate in the repository
![Page 23: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/23.jpg)
Integrate in the repository
• Ingest raw data
![Page 24: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/24.jpg)
Integrate in the repository
• Ingest raw data – Raw database tables?
![Page 25: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/25.jpg)
Integrate in the repository
• Ingest raw data – Raw database tables? – Raw events?
![Page 26: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/26.jpg)
Integrate in the repository
• Ingest raw data – Raw database tables? – Raw events?
• MapReduce jobs for ETL
![Page 27: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/27.jpg)
Use case
![Page 28: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/28.jpg)
Use case
• Completely contrived for this presentation, but maybe you really want to do this
![Page 29: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/29.jpg)
Use case
• Data sources
![Page 30: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/30.jpg)
Use case
• Data sources – Twitter fire hose
![Page 31: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/31.jpg)
Use case
• Data sources – Twitter fire hose*
*1%
![Page 32: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/32.jpg)
Use case
• Data sources – Twitter fire hose* – My tweet archive
*1%
![Page 33: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/33.jpg)
Use case
• Data sources – Twitter fire hose* – My tweet archive
• Goal
*1%
![Page 34: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/34.jpg)
Use case
• Data sources – Twitter fire hose* – My tweet archive
• Goal – Identify the user most similar to me
*1%
![Page 35: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/35.jpg)
(Mostly) traditional solution
![Page 36: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/36.jpg)
(Mostly) traditional solution
Twi$er
Tweet Archive HDFS
![Page 37: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/37.jpg)
(Mostly) traditional solution
Twi$er
Tweet Archive
Flume
HDFS
![Page 38: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/38.jpg)
(Mostly) traditional solution
Twi$er
Tweet Archive
Flume
HDFS
Twi$er Source
![Page 39: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/39.jpg)
(Mostly) traditional solution
Twi$er
Tweet Archive
Flume
HDFS
Twi$er Source Channel
![Page 40: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/40.jpg)
(Mostly) traditional solution
Twi$er Twi$er Source Channel HDFS
Sink
HDFS Tweet Archive
Flume
![Page 41: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/41.jpg)
(Mostly) traditional solution
Twi$er Twi$er Source Channel HDFS
Sink
HDFS Tweet Archive
Kite CLI
Flume
![Page 42: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/42.jpg)
Demo
![Page 43: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/43.jpg)
Drawbacks
![Page 44: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/44.jpg)
Drawbacks
• Two ingest systems
![Page 45: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/45.jpg)
Drawbacks
• Two ingest systems – Separate monitoring
![Page 46: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/46.jpg)
Drawbacks
• Two ingest systems – Separate monitoring – Separate failure modes
![Page 47: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/47.jpg)
Drawbacks
• Two ingest systems – Distinct monitoring – Distinct failure modes – Distinct debugging
![Page 48: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/48.jpg)
Drawbacks
• Two ingest systems – Distinct monitoring – Distinct failure modes – Distinct debugging
• Manual integration
![Page 49: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/49.jpg)
Drawbacks
• Two ingest systems – Distinct monitoring – Distinct failure modes – Distinct debugging
• Manual integration – Kite CLI with cron
![Page 50: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/50.jpg)
Enter Apache NiFi
![Page 51: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/51.jpg)
Enter Apache NiFi
![Page 52: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/52.jpg)
Bounded context
![Page 53: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/53.jpg)
Bounded context
• You control all the parts
![Page 54: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/54.jpg)
Bounded context
• You control all the parts – Protocols
![Page 55: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/55.jpg)
Bounded context
• You control all the parts – Protocols – Schemas
![Page 56: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/56.jpg)
Bounded context
• You control all the parts – Protocols – Schemas – Formats
![Page 57: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/57.jpg)
Bounded context
• You control all the parts – Protocols – Schemas – Formats – Changes
![Page 58: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/58.jpg)
NiFi strengths
![Page 59: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/59.jpg)
NiFi strengths
• Generic data flow
![Page 60: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/60.jpg)
NiFi strengths
• Generic data flow • Built-in editor/monitor
![Page 61: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/61.jpg)
NiFi strengths
• Generic data flow • Built-in editor/monitor • Varying object size
![Page 62: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/62.jpg)
NiFi strengths
• Generic data flow • Built-in editor/monitor • Varying object size • Traditional sources
![Page 63: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/63.jpg)
NiFi strengths
• Generic data flow • Built-in editor/monitor • Varying object size • Traditional sources
– Files, FTP, SFTP, HTTP, etc.
![Page 64: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/64.jpg)
NiFi limitations
![Page 65: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/65.jpg)
NiFi limitations
• Streaming sources
![Page 66: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/66.jpg)
NiFi limitations
• Streaming sources – ListenHttp
![Page 67: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/67.jpg)
NiFi limitations
• Streaming sources – ListenHttp – ListenUdp
![Page 68: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/68.jpg)
NiFi limitations
• Streaming sources – ListenHttp – ListenUdp – GetKafka
![Page 69: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/69.jpg)
Enter Apache Flume
![Page 70: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/70.jpg)
Enter Apache Flume
• Streaming from the start
![Page 71: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/71.jpg)
Enter Apache Flume
• Streaming from the start • Rich set of sources/sinks
![Page 72: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/72.jpg)
Enter Apache Flume
• Streaming from the start • Rich set of sources/sinks
– Apache Avro, Apache Thrift, Twitter, NetCat, Syslog
![Page 73: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/73.jpg)
Enter Apache Flume
• Streaming from the start • Rich set of sources/sinks
– Apache Avro, Apache Thrift, Twitter, NetCat, Syslog – HDFS, IRC, Hbase, Kite
![Page 74: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/74.jpg)
Cake
![Page 75: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/75.jpg)
Cake
• NiFi combines ingest contexts
![Page 76: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/76.jpg)
Cake
• NiFi combines ingest contexts • Flume requires static stream configuration
![Page 77: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/77.jpg)
Cake
• NiFi combines ingest contexts • Flume requires static stream configuration • I want both
![Page 78: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/78.jpg)
Flume architecture
Source
Channel
Sink
![Page 79: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/79.jpg)
Flume è NiFi
![Page 80: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/80.jpg)
Flume è NiFi
• Source/Sink
![Page 81: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/81.jpg)
Flume è NiFi
• Source/Sink • Event
![Page 82: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/82.jpg)
Flume è NiFi
• Source/Sink • Event • Channel
![Page 83: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/83.jpg)
Flume è NiFi
• Source/Sink è Processor
![Page 84: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/84.jpg)
Flume è NiFi
• Source/Sink è Processor • Event è FlowFile
![Page 85: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/85.jpg)
Flume è NiFi
• Source/Sink è Processor • Event è FlowFile • Channel è FlowFile Queue/Connection
![Page 86: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/86.jpg)
Solution
![Page 87: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/87.jpg)
Solution
• NiFi processors to run Flume sources/sinks
![Page 88: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/88.jpg)
Solution
• NiFi processors to run Flume sources/sinks • Prototype
![Page 89: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/89.jpg)
Solution
• NiFi processors to run Flume sources/sinks • Prototype • http://bit.ly/flume-processors
![Page 90: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/90.jpg)
Demo
![Page 91: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/91.jpg)
Summary
![Page 92: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/92.jpg)
Summary
• Integrating data is challenging
![Page 93: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/93.jpg)
Summary
• Integrating data is challenging • Managing multiple systems adds complexity
![Page 94: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/94.jpg)
Summary
• Integrating data is challenging • Managing multiple systems adds complexity • NiFi supports generic data flow
![Page 95: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/95.jpg)
Summary
• Integrating data is challenging • Managing multiple systems adds complexity • NiFi supports generic data flow • NiFi can be extended to solve new use cases
![Page 96: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/96.jpg)
Joey Echeverria [email protected] @fwiffo
![Page 97: Integrating Event Streams and File Data with Apache Flume and ...](https://reader033.fdocuments.net/reader033/viewer/2022052406/58a1ab021a28abac578ba069/html5/thumbnails/97.jpg)
Big Data Meets IT Ops