EXEM Big Data - s3-ap-northeast-1. FTP HDFS Kafka MQTT FTP HTTP Kafka Realtime Distributed Message...

download EXEM Big Data - s3-ap-northeast-1.   FTP HDFS Kafka MQTT FTP HTTP Kafka Realtime Distributed Message Broker Spark Streaming ‹¤‹œê°„

of 12

  • date post

    01-May-2018
  • Category

    Documents

  • view

    221
  • download

    6

Embed Size (px)

Transcript of EXEM Big Data - s3-ap-northeast-1. FTP HDFS Kafka MQTT FTP HTTP Kafka Realtime Distributed Message...

  • Smart Factory

    EXEM Big Data

    XDP

    FlamingoAnalytics

    FlamingoManagement

    /

    IoTSensors

    Web/SNS

    Log

    RDBMS

    File

    Crawler

    ETL

    ESB

    Sqoop

    Flume

    SFTP/FTP

    OpenAPI

    Dashboard

    Open API

    Service

    Service

    Service

    AnalyticsModule

    Big Data PlatformManagement Data Catalog Open API

    Big Data Infra Management Data Management

    Unified Big Data Query(SQL)

    HadoopConnector

    RStudio

    TableauSpotfire

    SPSSSAS

    Data Integrator

    Hive Pig Mahout

    Flamingo MLlib

    Spark In-Memory MapReduce

    YARN( )

    R

    HDFS()

    NoSQL

    /

  • Data Lifecycle&

    Governance

    Falcon

    Atlas

    Data Workflow

    Sqoop

    Flume

    Kafka

    NFS

    WebHDFS

    TOOLS

    Zeppelin FlamingoAnalytics

    MapReduce

    Solr/

    ElasticSearch

    HDFSHadoop Distributed File System

    Administration

    Authentication

    AuthorizationAuditing

    Data Protection

    Provisioning,Managing,

    &Monitoring

    FlamingoManagement

    Ranger

    Knox

    Atlas

    HDFS Encryption

    Ambari

    Cloudbreak

    ZooKeeper

    Oozie

    Pig

    SQLHiveTez

    NoSQL

    HBaseAccumuloPhoenix

    Silder

    StormSilder

    YARN:

    ISV Engines

    Spark

    (Hadoop)

    (R)

    (Hive)

    (Flume)

    (Pig)

    (Sqoop)

    (Mahout)

    (Spark)

  • h3

    h2

    h1

    v

    W 3

    W 2

    W 1

    RBM

    SigmoidBeliefNetwork

    h0 h0

    X0

    h0

    X0

    h3

    X2

    h2

    X2X0

    =

    time

    A A A A A

    Convolutional layer

    layer 1FC 6layer 2

    layer 3 layer 4 layer 5

  • OPEN API MES

    Gateway

    MQTT

    HTTP

    TCP/IP

    JMS

    MES

    Spark Sqoop TimeSeriesDB

    (TSDB)MapReduce

    Hive R PigData Hub SQL on Hadoop

    Log Collector HDFS Cloud Storage Service

    /

    OPENAPI

    OPENAPI

    ,

    S/W()

    OPEN API MES

  • RFIDPLC

    M2M

    I/O

    RFID/USN

    M2M

    I/O

    Access Point

    KIOSK

    K-ICT

    (Flamingo)

    DB

    IoT (Flamingo)

    Apache Tajo

    Apache Spark

    Flum

    e

    Apache

    DBMS

    Storm

    Cascading

    Sentry

    Sq

    oop

    MapReduce

    Hive R Pig

    HDFS

    SocialData

    IoT

  • R

    Management

    time1

    windowat time1

    windowat time2

    windowat time3

    time2

    window basedoperation

    time3 time4 time5

    ISO/IEC30128

    MQTT

    HTTP

    IOT

    Realtime Analytics

    Flamingo

    one M2M

    Kaf

    ka

    /

    MQTT

    FTP

    HDFS

    Kafka

    MQTT

    FTP

    HTTP

    Kafka

    Realtime Distributed Message Broker

    Spark Streaming

    Batch Analytics

    R/RStudio

    Hive, Tajo,HAWQ

    HDFS

    Hadoop EcoSystem

    SQL onHadoop Spark ZooKeeper Sqoop

    Hive

    MapReduce

    R Pig

    Flume HDFS

    HD

    FS

    MR, Spark

    Oozie

    HDFS

  • Hive DB, Table, Partition UI

    ,