End to End Processing of 3.7 Million Telemetry Events per Second using Lambda Architecture

End to End Processing of 3.7 Million Telemetry Events per Second Using Lambda Architecture

Saurabh Mishra Raghavendra Nandagopal

Who are We?

Saurabh MishraSolution Architect, Hortonworks Professional Services @draftsperson [email protected]

Raghavendra NandagopalCloud Data Services Architect, Symantec@[email protected]

Cloud Platform EngineeringSymantec - Global Leader in Cyber Security

- Symantec is the world leader in providing security software for both enterprises and end users

- There are 1000’s of Enterprise Customers and more than 400 million devices (Pcs, Tablets and Phones) that rely on Symantec to help them secure their assets from attacks, including their data centers, emails and other sensitive data

Cloud Platform Engineering (CPE)- Build consolidated cloud infrastructure and platform services for next generation data powered

Symantec applications

- A big data platform for batch and stream analytics integrated with both private and public cloud.

- Open source components as building blocks

- Hadoop and Openstack

- Bridge feature gaps and contribute back

Agenda

• Security Data Lake @ Global Scale• Infrastructure At Scale• Telemetry Data Processing Architecture• Tunable Targets• Performance Benchmarks• Service Monitoring

Security Data Lake @ Global Scale

Security Data Lake @ Global Scale

Products

HDFS

Analytic Applications, Workload Management (YARN)

Stream Processing (Storm)

Real-time Results(HBase, ElasticSearch)

Query(Hive, Spark SQL)

Device Agents

Telemetry Data

Data Transfer

Threat ProtectionInbound Messaging (Data Import, Kafka)

Physical Machine , Public Cloud, Private Cloud

Lambda Architecture

7

Speed Layer

Batch Layer

Serving Layer

Complexity Isolation Once Data Makes to Serving Layer via Batch , Speed Layer can be Neglected

Compensate for high latency of updates to serving layer Fast and incremental algorithms on real time data Batch layer eventually overrides speed layer

Random access to batch viewsUpdated by batch layer

Stores master datasetBatch layer stores the master copy of the Serving layer Computes arbitrary Views

Infrastructure At Scale

Yarn Applications in Production- 669388 submitted- 646632 completed- 4640 killed- 401 failed

Hive in Production- 25887 Tables- 306 Databases- 98493 Partitions

Storm in Production- 210 Nodes 50+ Topologies

Kafka in Production- 80 Nodes

Hbase in Production- 135 Nodes

ElasticSearch in Production- 62 Nodes

Ambari

Infrastructure At Scale

Centralized Logging

and Metering

IronicAnsible

Cloudbreak

Hybrid Data Lake

OpenStack(Dev)

350 Nodes

Metal(Production)600 Nodes

Public Cloud

(Production)

200 Nodes

Telemetry Data Processing Architecture

Telemetry Data Processing Architecture

Telemetry Data Collector

Telemetry GatewayRaw Events

Data Centers

Avro SerializedTelemetry Avro Serialized

Telemetry

Opaque TridentKafka Spout

Deserialized Objects

Transformations Functions

Transformation Topology

Trident Streams (Micro batch implementation, Exactly Once semantics)

Persist Avro Objects

Avro Serialized Transformed Objects

ElasticSearch Ingestion TopologyOpaque

TridentKafka Spout

Trident ElasticSearch Writer Bolt

HBase Ingestion Topology

TridentHBase Bolt

Trident Hive

Streaming

Opaque TridentKafka Spout

YARN

Hive Ingestion Topology

Identity Topology

Tunable Targets

Operating SystemTuning Targets

● Operating System● Disk● Network

Tunables● Disable Transparent Huge Pages

echo never > defrag and > enabled● Disable Swap● Configure VM Cache flushing● Configure IO Scheduler as deadline● Disk Jbod Ext4

Mount Options- inode_readahead_blks=128,data=writeback,noatime,nodiratime

● Network Dual Bonded 10gbps

rx-checksumming: on, tx-checksumming: on, scatter-gather: on, tcp-segmentation-offload: on

KafkaTuning Targets

●Broker●Producer●Consumer

Tunables● replica.fetch.max.bytes ● socket.send.buffer.bytes ● socket.receive.buffer.bytes● replica.socket.receive.buffer.bytes● num.network.threads ● num.io.threads ● zookeeper.*.timeout.ms

Type

Metal 2.6 GHzE5-2660 v3

12 * 4TBJBOD

128 GBDDR4 ECC

Cloud AWS: D2*8xlarge

v 0.8.2.1

KafkaTuning Targets

● Broker● Producer● Consumer

Tunables● buffer.memory● batch.size● linger.ms● compression.type● socket.send.buffer.bytes

Type


12 * 4TBJBOD

128 GBDDR4 ECC


v 0.8.2.1

KafkaTuning Targets

●Broker●Producer●Consumer

Tunables

● num.consumer.fetchers● socket.receive.buffer.bytes● fetch.message.max.bytes● fetch.min.bytes

Type


12 * 4TBJBOD

128 GBDDR4 ECC


v 0.8.2.1

StormTuning Targets

● Nimbus● Supervisors● Workers and Executors● Topology

Tunables● Nimbus High Availability - 4 Nimbus Servers Avoid downtime and performance degradation.● Reduce workload on Zookeeper and Decreased

topology Submission time. storm.codedistributor.class = HDFSCodeDistributor

● topology.min.replication.count = 3 floor(number_of_nimbus_hosts/2 + 1)

● max.replication.wait.time.sec = -1● code.sync.freq.secs = 2 mins● storm.messaging.netty.buffer_size = 10 mb● nimbus.thrift.threads = 256

Type


2 * 500GBSSD

256 GBDDR4 ECC

Cloud AWS: r3*8xlarge

v 0.10.0.2.4

StormTuning Targets


Tunables● Use supervisord to control Supervisors● Supervisor.slots.ports = Min (No of HT Cores ,

TotalMem of Server/Worker heap size)● Supervisor.childopts = -Xms4096m -Xmx4096m -

verbose:gc-Xloggc:/var/log/storm/supervisor_%ID%_gc.log

Type


2 * 500GBSSD

256 GBDDR4 ECC


v 0.10.0.2.4

StormTuning Targets


TunablesRule of Thumb! - Use Case of Storm – Telemetry Processing

● CPU bound tasks 1 Executor Per worker ● IO Bound tasks 8 Executors Per worker, ● Fixed the JVM Memory for each Worker Based on

Fetch Size of Kafka Trident Spout and Split Size of Bolt -Xms8g -Xmx8g -XX:MaxDirectMemorySize=2048m -XX:NewSize=2g -XX:MaxNewSize=2g -XX:+UseParNewGC -XX:MaxTenuringThreshold=2 -XX:SurvivorRatio=8 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32768 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly -XX:-CMSConcurrentMTEnabled -XX:+AlwaysPreTouch

Type


2 * 500GBSSD

256 GBDDR4 ECC


v 0.10.0.2.4

StormTuning Targets


Tunables● Topology.optimize = true● Topology.message.timeout.secs = 110 ● Topology.max.spout.pending = 3● Remove Topology.metrics.consumer.register- AMBARI-13237

Incoming queue and Outgoing queue● Topology.transfer.buffer.size = 64 – Batch Size● Topology.receiver.buffer.size = 16 – Queue Size● Topology.executor.receive.buffer.size = 32768 ● Topology.executor.send.buffer.size = 32768

Type


2 * 500GBSSD

256 GBDDR4 ECC


v 0.10.0.2.4

StormTuning Targets


Tunables● topology.trident.parallelism.hint = (number of

worker nodes in cluster * number cores per worker node) - (number of acker tasks)

● Kafka.consumer.fetch.size.byte = 209715200 ( 200MB - Yes! We process large batches)

● Kafka.consumer.buffer.size.byte = 209715200● Kafka.consumer.min.fetch.byte = 100428800

Type


2 * 500GBSSD

256 GBDDR4 ECC


v 0.10.0.2.4

ZooKeeper

Tunables● Keep data and log directories separately and on different

mounts

● Separate Zookeeper quorum of 5 Servers each for Kafka, Storm, Hbase, HA quorum.

● Zookeeper GC Configurations -Xms4192m -Xmx4192m -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -verbose:gc -Xloggc:/var/log/zookeeper/zookeeper_gc.log

Type


2*400 GB SSD

128 GBDDR4 ECC

Cloud AWS: r3.2xlarge

v 3.4.6

Tuning Targets● Data and Log directory● Garbage Collection

ElasticsearchType


14 * 400 GBSSD

256 GBDDR4 ECC

Cloud AWS: i2.4xlarge

Tunables● bootstrap.mlockall: true● indices.fielddata.cache.size: 25%● threadpool.bulk.queue_size: 5000● index.refresh_interval: 30s● Index.memory.index_buffer_size: 10%● index.store.type: mmapfs● GC settings: -verbose:gc -Xloggc:/var/log/elasticsearch/elasticsearch_gc.log -

Xss256k -Djava.awt.headless=true -XX:+UseCompressedOops -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+DisableExplicitGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:ErrorFile=/var/log/elasticsearch_err.log -XX:ParallelGCThreads=8

● Bulk api● Client node

v 1.7.5

Tuning Targets● Index Parameters● Garbage Collection

HbaseTuning Targets

● Region server GC● Hbase Configurations

Tunablesexport HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:+UseConcMarkSweepGC -Xmn2500m -XX:SurvivorRatio=4 -XX:CMSInitiatingOccupancyFraction=50 -XX:+UseCMSInitiatingOccupancyOnly -Xmx{{regionserver_heapsize}} -Xms{{regionserver_heapsize}} -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintPromotionFailure -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:${HBASE_LOG_DIR}/hbase-gc-regionserver.log.`date +'%Y%m%d%H%M'`

Type


14 * 400 GBSSD

256 GBDDR4 ECC

Cloud AWS: i2.8xlarge

v 1.1.0

HiveTuning Targets

● Table Structure● Partition and Bucketing

Scheme● Orc Tuning

Tunables

● Use strings instead of binaries.

● Use Integer fields.

Type


14 * 6TBHDD

256 GBDDR4 ECC

Cloud AWS: d2*8xlarge

v 1.2.1

HiveTuning Targets



Tunables● Partitioning by Date Timestamp.

● Additional partitioning - Resulted in explosion of number of partitions, small file size and inefficient ORC compression.

● Bucketing: If two tables bucket on the same column, they should use the same number of buckets to support joining

● Sorting : Each table should optimize its sorting. The bucket column typically should be the first sorted column.

Type


14 * 6TBHDD

256 GBDDR4 ECC


v 1.2.1

HiveTuning Targets



Tunables

● Table Structure , Bucketing and Partition and Sorting

Impact ORC Performance.

● ORC Stripe Size default 128MB Balanced Insert and

Query Optimized.

● ORC use ZLIB Compression. Smaller data size

improves any query.

● Predicate Push Down.

Type


14 * 6TBHDD

256 GBDDR4 ECC


v 1.2.1

No of Yarn Containers Per Query

orc.compress ZLIB high level compression (one of NONE, ZLIB, SNAPPY)

orc.compress.size 262144 number of bytes in each compression chunk

orc.stripe.size 130023424 number of bytes in each stripe

orc.row.index.stride 64,000 number of rows between index entries (must be >= 1000)

orc.create.index true whether to create row indexes

orc.bloom.filter.columns "file_sha2" comma separated list of column names for which bloom filter should be created

orc.bloom.filter.fpp 0.05 false positive probability for bloom filter (must >0.0 and <1.0)

Hive StreamingTuning Targets

● Hive Metastore Stability● Evaluate BatchSize & TxnsPerBatch

Tunables● No Hive Shell Access only Hiveserver2.

● Multiple Hive Metastore Process

○ Compaction Metastore - 5 - 10 Compaction thread

○ Streaming Metastore - 5 - Connection pool

● 16 GB Heap Size.

● Metastore Mysql Database Scalability.

● Maximum EPS was achieved by Increasing Batch Size and keeping TxnPerBatch as Smaller.

Type


14 * 6TBHDD

256 GBDDR4 ECC


v 1.2.1

Performance Benchmarks

Benchmarking Suite

Kafka Producer Consumer Throughput Test

Storm Core and Trident Topologies

Standard Platform Test Suite

Hive TPC

Kafka Producer and Consumer Tests

The benchmark set contains Producer and Consumer test executing at various message size.

Producer and Consumer Together● 100 bytes● 1000 bytes - Average Telemetry Event Size ● 10000 bytes● 100000 bytes● 500000 bytes● 1000000 bytes

Type of Tests● Single thread no replication● Single-thread, async 3x replication● Single-thread, sync 3x replication● Throughput Versus Stored Data

Ingesting 10 Telemetry Sources in Parallel

Storm Topology

The benchmark set custom topologies for processing telemetry data source transformation and ingestion

which simulates end to end use cases for real time streaming of Telemetry.

● Storm Trident HDFS Telemetry Transformation and Ingestion

● Storm Trident Hive Telemetry Ingestion

● Storm Trident Hbase Telemetry Ingestion

● Storm Trident Elasticsearch Telemetry Ingestion

Ingesting 10 Telemetry Sources in Parallel

Standard Platform TestsTeraSort benchmark suite ( 2TB , 5TB, 10TB)RandomWriter(Write and Sort) - 10GB of random data per nodeDFS-IO Write and Read - TestDFSIONNBench (Write, Read, Rename and Delete)MRBench Data Load (Upload and Download)

TPC-DS 20TBTPC-DS: Decision Support Performance Benchmarks

● Classic EDW Dimensional model

● Large fact tables

● Complex queries

Scale: 20TB

TPC-DS Benchmarking

Service Monitoring

Service Monitoring Architecture

Kafka Monitoring

ElasticSearch

Storm Kafka Lag

Storm Kafka Logging Collection

Thank YouQ&A

End to End Processing of 3.7 Million Telemetry Events per Second using Lambda Architecture

Technology

Transcript of End to End Processing of 3.7 Million Telemetry Events per Second using Lambda Architecture