INDUSTRY 4.0 OPEN ARCHITECTURE FOR DATA COLLECTION...
Transcript of INDUSTRY 4.0 OPEN ARCHITECTURE FOR DATA COLLECTION...
INDUSTRY 4.0 – OPEN ARCHITECTURE FOR DATA COLLECTION AND TRANSPORT
Bologna - 17/05/2019
Agenda
› Industry 4.0 – definition and principles
› Architecture for data collection and transport for Industry 4.0
› Enabling technologies
▪ Apache Kafka
▪ Docker
▪ Kubernetes
2
Industry 4.0 – definition and principles
3
Industry 4.0 - Definition
“Industry 4.0 is a term applied to a group of rapid transformations in the design, manufacture, operation and service of manufacturing systems and products […] everything in and around a manufacturing operation (suppliers, the plant, distributors, even the product itself) is digitally connected, providing a highly integrated value chain”
http://www.europarl.europa.eu/RegData/etudes/BRIE/2015/568337/EPRS_BRI(2015)568337_EN.pdf
4
Industry 4.0 - Definition
1st Industrial revolution
1784 – mid 19° century
Water and steam-poweredmechanical manufacturing
2nd Industrial revolution
Late 19° century -1970s
Electric-powered mass production based on the division of labour
3rd Industrial revolution
1970s - Today
Electronics and information technology drives new levels of automation of complex tasks
4th Industrial revolution – Industry 4.0
Today -
Electronics and information technology drives new levels of automation of complex tasks
5
http://www.europarl.europa.eu/RegData/etudes/BRIE/2015/568337/EPRS_BRI(2015)568337_EN.pdf
I4.0 - Cornerstones
The application of information and communication technology (ICT) to digitise information and integrate systems
Cyber-physical systems that use ICTs to monitor and control physical processes and systems. These may involve embedded sensors, intelligent robots that can configure themselves to suit the immediate product to be created, or additive manufacturing (3D printing) devices
Network communications including wireless and internet technologies that serve to link machines, work products, systems and people, both within the manufacturing plant, and with suppliers and distributors
Simulation, modelling and virtualisation in the design of products and the establishment of manufacturing processes
Collection of vast quantities of data, and their analysis and exploitation, either immediately on the factory floor, or through big data analysis and cloud computing
Greater ICT-based support for human workers, including robots, augmented reality and intelligent tools
6
I4.0 - Cornerstones
The application of information and communication technology (ICT) to digitise information and integrate systems
Cyber-physical systems that use ICTs to monitor and control physical processes and systems. These may involve embedded sensors, intelligent robots that can configure themselves to suit the immediate product to be created, or additive manufacturing (3D printing) devices
Network communications including wireless and internet technologies that serve to link machines, work products, systems and people, both within the manufacturing plant, and with suppliers and distributors
Simulation, modelling and virtualisation in the design of products and the establishment of manufacturing processes
Collection of vast quantities of data, and their analysis and exploitation, either immediately on the factory floor, or through big data analysis and cloud computing
Greater ICT-based support for human workers, including robots, augmented reality and intelligent tools
7
Industry 4.0 – Key factors and challenges
▪ Business practices
▪ Corporate standards
▪ Security/safety policies and procedures
▪ Business and Application requirements
▪ Regulatory compliance
▪ Risk management
8
Cornerstones of a joint collaboration
▪ Industry (SACMI)▪ Business domain expertise (regulations, value stream, etc…)▪ Cyber-physical systems expertise
▪ Academia (UniBO)▪ Private/public infrastructure interoperability▪ Protocol interoperability, efficiency, and performance tuning/evaluation▪ Digital Twins
▪ Enterprise Architecture (Imola)▪ Large-scale system design and integration on private/public infrastructures▪ Software development/delivery process governance▪ Data Governance and Cybersecurity
9
Architecture for data collection and transport for Industry 4.0
10
Project goal
Design and implementation of a data collection and transportplatform/architecture for Industry 4.0 scenarios
11
Project requirements
▪ OT principles and regulations should be guaranteed; (Physical) Safety AND Cybersecurity
▪ Heterogeneity: need to accomodate a large number of (mainly legacy) equipments and data formats
▪ Scaling: need to accomodate ever-increasing volumes of connectedequipments (new plants) and message rates
▪ Near real-time processing requirements: business/safety-critical actions happen at the equipment level
12
Data collection and transport architecture – logical layers
13
• Pluggable output towards multiple data stores
• Multiple data analysis and reporting platformsStorage/analysis layer
• Data format modeling and definition
• Data format validation and transformationInteroperability layer
• Asynchronous, publish-subscribe model
• Scalable message transport
Transport/messaging layer
• HW/SW integration via possibly legacy and/or proprietaryprotocols
• Different equipment models and HW/firmware versionsData collection layer
Data collection and transport architecture – architecture
14
Enabling technologies
Apache Kafka
15
Transport layer – Apache Kafka
Apache Kafka is a streaming platform with three key capabilities
▪ Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system
▪ Store streams of records in a fault-tolerant durable way▪ Process streams of records as they occur
Apache Kafka typical use cases▪ Real-time streaming data pipelines used for data aggregation, processing, and transport;▪ Event-reactive streaming applications used for fraud detection, data validation, email sending
confirmation;▪ Applications for real-time data analytics, stream processing, log aggregation, messaging, audit
trail, sync for cooperative nodes.
16
Publish-subscribe model
17
Publisher Subscriber
Topic SubscriberPublisher
Publisher
Subscriber
Topic SubscriberPublisher
Publisher Subscriber
Subscriber
Apache Kafka Topics and Partitions
18
Publisher
Consumer group 1
Topic – Football Teams
Publisher
Publisher
Partition 1 - A-M Team names
Partition 2 - P-Z Team names
offset:0Name: Boca
Juniors
offset:1Name: Arsenal
offset:0Name:
Real Madrid
offset:1Name:
Borussia Dortmund
Consumer group 2
Consumer process
Consumer process
Consumer process
• Partitions and consumer groups allow for scalability
• Partitions are spread across differentmachines
• Partitions split commit log by given criteria• Partitions preserve message order
Apache Kafka – Brokers and Cluster
19
Publisher
Consumer group 1
Publisher
Publisher
Partition 2 - P-Z Team names
offset:0Name:
Real Madrid
Partition 1 - A-M Team names
offset:0Name: Boca
Juniors
offset:1Name: Arsenal
offset:1Name:
Borussia Dortmund
Consumer group 2
Consumer process
Consumer process
Consumer process
BROKER 2
BROKER 1
KAFKA CLUSTER
Apache Kafka – Apache Zookeeper coordination
20
Publisher
Consumer group 1
Publisher
Publisher
Partition 2 - P-Z Team names
offset:0Name:
Real Madrid
Partition 1 - A-M Team names
offset:0Name: Boca
Juniors
offset:1Name: Arsenal
offset:1Name:
Borussia Dortmund
Consumer group 2
Consumer process
Consumer process
Consumer process
BROKER 2
BROKER 1
KAFKA CLUSTERAPACHE ZOOKEEPER
Apache Kafka commit log
21
Interoperability layer – Kafka Connect
APACHE Kafka Connect – framework to define connectors to stream data into and out of Kafka
▪ A common framework for Kafka connectors
▪ REST interface
▪ Automatic offset management
▪ Distributed and scalable
22
Interoperability layer – Kafka Connect
23
Elasticsearch SinkConnector
KafkaTopic
PostgreSQL DB Source Connector
Amazon S3 SinkConnector
Amazon S3
Enabling technologies
Containers & Docker
24
Infrastructure - Containers
25
PHYSICAL INFRASTRUCTURE
HOST OS
HYPERVISOR
GUEST OS
LIBS/BIN
APP
GUEST OS
LIBS/BIN
APP
GUEST OS
LIBS/BIN
APP
PHYSICAL INFRASTRUCTURE
HOST OS
DOCKER ENGINE
LIBS/BIN
APP
LIBS/BIN
APP
LIBS/BIN
APP
VIRTUAL MACHINE
CONTAINER
Infrastructure - Containers
▪ containers include an application/service together with its
dependencies
▪ containers share kernel with other containers
▪ containers run as isolated processes
▪ higher efficiency w/r to virtualization
▪ images are the cornerstone in crafting declarative/automated, easily
repeatable, and scalable services and applications
26
Infrastructure - Docker
An open platform for distributed applications for developers and sysadmins
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
https://docs.docker.com/engine/
27
What is Docker?
▪ Docker consists of:▪ The Docker Engine - our lightweight and powerful open
source containerization technology combined with a work flow for building and containerizing your applications.
▪ Docker Hub - our SaaS service for sharing and managing your application stacks.
28
Docker inception
29
▪ 2013: Docker comes to life as an open-source project at dotCloud Inc.
▪ 2014: company changed name to “Docker Inc.” and joined the Linux Foundation
▪ 2015: tremendous increase in popularity
▪ Today:
https://blog.docker.com/2018/03/5-years-later-docker-journey/
Docker - Under the hood
▪ Standard Bodies: Open Container Initiative (OCI), Cloud Native Computing Foundation (CNCF)
▪ OCI Image specification
▪ OCI Runtime Specification
▪ runc runtime (formerly libcontainer)
▪ an abstraction/unification layer to decouple Docker from kernel-specific container features (e.g. LXC, libvirt, ...)
▪ The Docker Images:
▪ copy-on-write filesystems (e.g. AUFS)
▪ The Go programming language
▪ a statically typed programming language developed by Google with syntax loosely based on C
30
Docker Architecture
▪ Docker daemon – The Docker daemon listens for Docker API requests and manages Docker objects such as images,
containers, networks, and volumes.
▪ Docker client – The Docker client (docker) is the primary way that many Docker users interact with Docker. When you
use commands such as docker run, the client sends these commands to the docker daemon, which carries them out.
▪ Docker registries – A Docker registry stores Docker images. Docker Hub and Docker Cloud are public registries that
anyone can use, and Docker is configured to look for images on Docker Hub by default. You can even run your own
private registry. Docker registries are the distribution component of Docker.
Docker objects
Docker imagesA Docker image is a read-only template. For example, an image could contain an
Ubuntu operating system with Apache and your web application installed. Images are used to create Docker containers. Docker provides a simple way to build new images or update existing images, or you can download Docker images that other people have already created. Docker images are the build component of Docker.
Docker containersDocker containers are similar to a directory. A Docker container holds everything
that is needed for an application to run. Each container is created from a Docker image. Docker containers can be run, started, stopped, moved, and deleted. Each container is an isolated and secure application platform. Docker containers are the run component of Docker.
Enabling technologies
Kubernetes Container OrchestrationPlatform
33
Infrastructure orchestration ( Kubernetes, Swarm, Mesos)
▪ Manage the lifecycle of execution environments (containers) in a cluster
▪ Check the state of the containers in the worker nodes
▪ Simplify the implementation of :
▪ High availability (HA) with load balancers
▪ advanced deployment strategies:
▪blue/green deployment, canary release, ...
▪rollback in case of problems
▪ health checks
34
Kubernetes container orchestration platform
Kubernetes is an open source system for managing containerized applications across multiple hosts; providing basic mechanisms for deployment, maintenance, and scaling of applications.
Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale using a system called Borg, combined with best-of-breed ideas and practices from the community.
Kubernetes is hosted by the Cloud Native Computing Foundation (CNCF)
https://github.com/kubernetes/kubernetes
35
Infrastructure - Kubernetes
36
Physical Infrastructure
VM VM VM VM VM VM VM VM
Cluster 1 Cluster 2
Container Orchestration Engine
APP1 APP1
Docker runtime Docker runtime Docker runtime Docker runtime Docker runtime
APP1APP2 APP3APP3
Live demo!
37
Data collection and transport architecture screenshots
38
▪ Più di 20 anni di esperienza nell’Enterprise IT
▪ Consulenza e Skill Transfer su Architetture, Integrazione e Processo
▪ OMG Influence Member, JSR 312 Expert Group, CSI, WWISA, OpenESB Key Partner, NetBeansStrategic Partner
▪ La comunita’ italiana dedicata a Java
▪ 10 anni di articoli, pubblicazioni, libri, eventi, training
▪ Dai programmatori agli architetti
▪ Piu’ di 1.000.000 pagine lette al mese
▪ Business partner in progetti con alto grado di innovazione
▪ Padroni in tecnologie e architetture mobile
▪ Competenti in architetture dell‘informazione, UX e Design
39