D2.8 First prototype (V1) - s293afedef7b6d099.jimcontent.com...of Apache Flink for easily analysing...

PROTEUS Scalable online machine learning for predictive analytics and real-time

interactive visualization

687691

D2.8 First prototype (V1) Lead Author: AMIII

With contributions from: TREE, DFKI, BU, LMDP, TRI Reviewer: LMDP

Deliverable nature: Demonstrator (D)

Dissemination level: (Confidentiality)

Public (PU)

Contractual delivery date: 31/05/2017

Actual delivery date: 31/05/2017

Version: 1.0

Total number of pages: 88

Keywords:

PROTEUS Deliverable D2.8

687691 of 88

Abstract

The appearance of defects in the products produced by ArcelorMittal represent an important cost which is

aimed to be reduced by the company. To this regard, the present document details three industrial use cases

proposed by ArcelorMittal which are aimed to be solved, and have guided the implementation of the first

prototype in PROTEUS.

This deliverable focuses on the software requirements and technical design of the first prototype integrated

as a result of PROTEUS, jointly with the setup of the integration and validation environment, and the

definition of the KPIs involved and their evaluation after the prototype execution phase.

Deliverable D2.8 PROTEUS

687691 of 88

Executive summary

In previous deliverables it was presented the general requirements and objectives of the industrial use case

related to the early-stage detection of defects in the Hot Strip Mill. These requirements tackled software and

hardware issues whereas the objectives focused their goals in the early detection of defects in general.

The current deliverable focuses on several goals, among which we can find the detailed description of the

industrial use cases. These cases derive from the data provided by ArcelorMittal which cover data-at-rest and

data-in-motion. Thus, a detailed description of three industrial use cases is provided jointly with the

objectives that are expected to be achieved in each one. This will drive the implementation of the prototype.

From this specific use cases, the technical goals derive, like the one proposing a declarative language on top

of Apache Flink for easily analysing data streams and batch datasets using the integrated processing engine

(first prototype). Moreover, in this document it is also presented the prototype of a stream library on top of

Apache Flink capable of querying the data streams of the industrial scenario (moments, heavy hitters, event

detection, subsampling, statistics) devoted to the analytic part of the project (i.e. SOLMA). Finally, specific

requirements and features demanded at the visualisation dashboard for considering both types of data are

specified.

The current deliverable details the software implementation and setup of the first prototype in PROTEUS

following the aforementioned objectives according to the business use cases covering some alarming issues

and pattern discovery aiming to detect and understand the appearance of flatness defects. Besides integrating

the technology components produced by PROTEUS as a result of the research WPs (WP3, WP4 and WP5),

the work performed in this task and documented in this deliverable also included the implementation of

specific components that are required for the recreation of the operational environment, following the actual

data production workflow and timeline, as a simulation of the actual production environment in the industrial

context (i.e. in the form of configurable data producers, tailored to the specific performance of the real

scenario).

Finally, some Key Performance Indicators (KPIs) have been defined as the ways to measure to what extent

the objectives, both business and technical objectives, have been guaranteed and fulfilled with the current

technology.


687691 of 88

Document Information

IST Project

Number

687691 Acronym PROTEUS

Full Title First prototype (V1)

Project URL www.proteus-bigdata.com

EU Project Officer Martina EYDNER

Deliverable Number D2.8 Title First prototype (V1)

Work Package Number WP2 Title Industrial case: requirements, challenges,

validation and demonstration

Date of Delivery Contractual M18 Actual M18

Status version v.1 final ■

Nature report □ demonstrator ■ other □

Dissemination level public ■ restricted □

Authors (Partner) AMIII

Responsible Author Name E-mail

Partner AMIII Phone

Abstract

(for dissemination)

The appearance of defects in the products produced by ArcelorMittal represent a

representative cost which is aimed to be reduced by the company. To this

regard, the present document details three industrial use cases proposed by

ArcelorMittal which are aimed to be solved.

Moreover, other software requirements are detailed jointly with the definition of

the KPIs involved and their evaluation after the prototype execution phase.

Keywords Use cases, software requirements, KPIs, prototype

Version Log

Issue Date Rev. No. Author Change

24/04/2017 0.1 AMIII ToC and section 1

08/05/2017 0.2 TREE, AMIII Refined section1 and section 2

10/05/2017 0.3 AMIII, TREE, DFKI Additional inputs on the technical

sections

11/05/2017 0.4 AMIII, TRI KPI first version

13/05/2017 0.5 AMIII Use cases revised

15/05/2017 0.6 AMIII, TREE Minor refinements on the technical

design and setup

16/05/2017 0.7 AMIII Document integrations

18/05/2017 0.8 AMIII, TREE Final refinements

22/05/2017 0.9 AMIII Common sections (abstract,

summary, conclusions)

31/05/2017 1.0 AMIII Final release

http://www.proteus-bigdata.com/


687691 of 88

Table of Contents

Executive summary ........................................................................................................................................... 3 Document Information ...................................................................................................................................... 4 Table of Contents .............................................................................................................................................. 5 Abbreviations .................................................................................................................................................... 6 1 Introduction: scenario definition, functional design and objectives ........................................................... 7

1.1 Specific business use cases for the prototype (V1) ............................................................................. 7 1.1.1 Control of the key process parameters ......................................................................................... 7 1.1.2 Better quality of the flatness of the products ............................................................................... 8 1.1.3 Better quality of the flatness and process .................................................................................. 10

1.2 Data production flow in the industrial environment ......................................................................... 11 1.2.1 Data description ......................................................................................................................... 11

2 Technical design ....................................................................................................................................... 13 2.1 Overview ........................................................................................................................................... 13 2.2 Cluster infrastructure......................................................................................................................... 14

2.2.1 Hardware Architecture ............................................................................................................... 14 2.2.2 Base Software & Services Architecture: services under Hortonworks data platform and

CouchBase ................................................................................................................................................ 16 2.2.3 The PROTEUS core contributions ............................................................................................ 19

2.3 Visual Analytics Platform ................................................................................................................. 21 2.3.1 Backend ..................................................................................................................................... 21 2.3.2 Front-end.................................................................................................................................... 22

2.4 Setup ................................................................................................................................................. 22 2.4.1 Required packages before installation ....................................................................................... 22 2.4.2 Operating System ....................................................................................................................... 23 2.4.3 Server and Agents ...................................................................................................................... 23 2.4.4 Hortonworks Data Platform deployment ................................................................................... 25 2.4.5 Proteus Environment ................................................................................................................. 25 2.4.6 Additional Storage System: Couchbase ..................................................................................... 28 2.4.7 Visual Analytics Infrastructure .................................................................................................. 29 2.4.8 Setup Summary .......................................................................................................................... 29

3 Validations & KPIs .................................................................................................................................. 31 3.1 KPI review ........................................................................................................................................ 31 3.2 Use case evaluation ........................................................................................................................... 32

3.2.1 Defective coil detection KPIs .................................................................................................... 32 3.2.2 Process improvement KPIs ........................................................................................................ 33

3.3 KPIs for AMIII data analysis, visualisation and visual analytics platform ....................................... 33 3.3.1 Data extraction-related KPIs ...................................................................................................... 33 3.3.2 Data analysis-related KPIs ......................................................................................................... 34 3.3.3 Data visualization-related KPIs ................................................................................................. 34 3.3.4 KPIs relevant to the PROTEUS specific components ............................................................... 34

3.4 Big data KPIs .................................................................................................................................... 35 3.5 Summary ........................................................................................................................................... 36

4 Conclusions .............................................................................................................................................. 37 5 References ................................................................................................................................................ 38 Annex A Coil data formats for streaming data .......................................................................................... 39

A.1 Variables measured on 1 Dimension................................................................................................. 39 A.2 Variables measured on 2 Dimensions ............................................................................................... 39 A.3 HSM data .......................................................................................................................................... 39


687691 of 88

Abbreviations

KPI: Key Performance Indicators

SOLMA: Scalable Online Machine Learning Algorithm

PEACH: Proteus Elastic Cache


687691 of 88

1 Introduction: scenario definition, functional design and

objectives

Deliverable D2.1 presented the general requirements and objectives in order to address the early-stage

detection of defects in the Hot Strip Mill. For this industry, the early detection of defects in the process of

steel production is a key point due to the remarkable economic impact that they have later on. For example,

the cost of transformation of defective material to make it useful (by degradation of the type of material for

instance) will represent an economical loss or, in the worst case, the rejection of the material by the

customers, which represents a bad impact for the reputation of the enterprise. Thus, to overcome or mitigate

all these situations, the sooner the defects are detected, the sooner the process can be modified/stopped in

order to save these expenses.

Once the general context of the problem is recalled, the functional design will be summarized. The role of

ArcelorMittal, as the end-user industrial partner, is to specify the industrial use case to take advantage of the

technologies that will be developed in order to fulfil the industrial requirements. Thus, to this extent, Proteus

platform will be designed and deployed in order to adjust to ArcelorMittal’s necessities.

In deliverable D2.7 was already specified the software prototypes that will be deployed and integrated for the

ArcelorMittal use case, describing the infrastructure to get actual data in real-time from the factory sensors

and historical registers from the systems without affecting to the daily functioning steelmaking process.

Hardware and software requirements and restrictions to the integration with existing systems were also

specified.

The current deliverable focuses on several goals, among which we can find the description of the industrial

use cases which derive from the available data-at-rest and data-in-motion with the aim of early detecting the

appearance of defects and minimizing the effect of having missed any of them.

In this section, a detailed description of three use cases is provided jointly with the objectives that are

expected to solve.

1.1 Specific business use cases for the prototype (V1)

In the Hot Strip Mill several parameters that affect the final dimensional properties of the obtained coils are

measured. Some example of these parameters would be temperatures, tension in the rollers, vibrations,

speeds of lamination, etc. All these data are collected mainly from a sensor network installed across the

facility. Those parameters will define the properties of the final coil and therefore, it is essential to obtain and

analyse them.

Defects introduced in early processes of steel transformation, such as the Hot Strip Mill, have a great

economic impact due to the costs of posterior transformations prior to detecting the defect. The coils with

defects continue through other facilities until they are identified as defective. Identifying a greater number of

defective coils would reduce the amount of money that is wasted by the company. Therefore, the goal is to

analyse the data provided by the inspection system in the Hot Strip Mill in order to

Detect as many defective coils as possible and thus, stop producing defective subsequent coils

Reassign the detected defective coils to other uses that require lower quality standards, redefining

new industrial routes.

To this extent, this first prototype describes the specific business use cases which arise as an attempt to detect

and minimize the impact of the defects in this facility.

1.1.1 Control of the key process parameters

As mentioned before, there is a wide network of sensors spread along the HSM measuring online process

data parameters. As might be expected, a correct and deep control of them would allow ArcelorMittal to be

more efficient in its process. In a more precise way, a detailed control of the parameters would help to be


687691 of 88

aware about which are the unbreakable margins that these parameters should not cross in order to prevent the

appearance of defects, breakages, etc.

Given the large amount of parameters provided, it is not feasible to make a manual inspection to detect

deviations in some of the variables. Therefore, a crucial goal would be to develop a method able to monitor

all these stream data and trigger alarms when the values of the variables are out of a “safety zone”. These

alarms would represent a useful and agile tool for the plant operators that would allow them to take decisions

and develop actions. According to these decision and actions, it would be possible to prevent the appearance

of a defect or, in the worst case, if the defect is already present, would allow them to evaluate which is the

best way to proceed. For instance, if it is possible to see and warn that the tension of a coil is increasing too

much, then the operator could decide to open a bit the stands with the rollers in order to prevent a breakage

of the coil strip. This would be a predictive action. By contrast, in the case that there is already a flatness

issue (the speed tension has exceed a threshold which implies flatness), a possible action would be to stop

the rolling process or to automatically downgrade the quality of the coil to a less exigent client.

To these control purposes, the tasks to be defined would be

a system computing online averages and variances representing a “safety zone” for each of the given

variables. This is a simple and interesting approach that would allow plant operators to track the

evolution of the time series and detect when certain variable is exceeding the “normal-functioning”

range. This range should be adaptable by the operators so that moving the variability up and down, a

more conservative or more permissive region is obtained

a system providing a visual alarm easy to interpret by the operators (I.e., apart from the “safety zone”

which can help them to visualize the evolution of the series, the alarm will help the operators by

focussing on the risky parts.)

The visualizations are a key point in this use case as it would allow the experts to have the general view in a

quick glance. Thus, the visualization tool should be able to visualize the stream data and provide :

the online representation of the “safety zone”, given by average and variability of the chosen

variable. This tool should allow the users to tune the parameters to make the region more or less

wide (depending on their goals end expert knowledge)

an alarm representation highlighting the risky/problematic zones

Figure 1: Representation of the zone, where in red are highlighted the alarms that should be

given

1.1.2 Better quality of the flatness of the products

There are three main parameters to take into account regarding the quality of the coils: thickness, width and

flatness. However, the latest is the hardest to predict due to its variability. There are different factors

affecting the flatness of the coils although they are not known in advance.

Then, as flatness of the product entail expensive implications, the main goal of this use case is to predict the

flatness of the coils. To this extent, we should be able to

to extract insights from the flatness of previous coils to make flatness predictions of future coils


687691 of 88

to develop predictive models which are updated iteratively with the new incoming flatness

data, i.e., after each coil has finished and therefore, new flatness measures are available for that coil.

To sum up the goal of the use case, the following diagrams clarifies the available data, the models required

and the goal pursued by them:

Figure 2: Representation of the approach proposed

In the diagram the data sources are represented in orange. These two types of data are:

the flatness from past and current coils, which is only obtained once the coil has finished the

rolling process

In grey is represented the desired model whose final goal is to predict the flatness of a coil. Due to the time

delay between the prediction of the flatness and the real measurement of this variable, the model has to

follow an iterative process where in each step it is:

predicting the flatness from the particular coil that is being produced in this step based on previous

coils

update the model parameters once the coil has been produced and the real flatness is known (New

updated model) . This second step is really important as, once we know the predicted flatness for a

coil and its real flatness, it is possible to make adjustments in the models that make it more accurate

for future coils. In other words, you are helping the model to learn the mistakes he made in the

flatness prediction in step 1 to adjust its parameters to make a better prediction in step 2. This point

would be represented in Figure 4. In the diagram the shadowed zone represents the time when the

coil is still in the lamination process, an thus, there is no flatness information about this coil. For the

first coil, in this zone there are only other stream variables whereas for subsequent coils the real

flatness of previous coils is also available. The model has to start to have the first real flatness to

start the iterative process of flatness prediction.

Figure 3: Representation of the iterative model update

A last remark would be that, without loss of generality, we cannot assume that the corrected model

would be updated with the real flatness information of the previous coil. I.e., in some cases it could


687691 of 88

be necessary to use the information of coils further back, as the previous one is not ready on time to

make the correction. In other words, it could be a time gap between the end of the processed coil and

the creation of its real flatness, which imply that the model might be updated some coils after the

processed one.

Then, once the model/method is done it should be able to provide :

the prediction of the degree of flatness in each moment as the coil is passing along the process

an error of the prediction estimate as the models are being updated. In other words, after each update

of the models (in all the intermediate steps), measure the error that they have made (flatness

prediction versus real flatness)

1.1.3 Better quality of the flatness and process

As an extension of the previous section, the goal could go one step beyond and expand the goals of flatness

prediction to try to find the causes why this flatness is taking place. Thus, we aim to find patterns in the

process variables that might explain/predict the appearance of flatness. These patterns would represent

different aspects from the process variables, such as peaks, pronounced slopes, valleys which will be

associated with certain degrees of flatness.

Thus, an approach would be to exploit the power of the state-of-the-art techniques of time series pattern

extraction to relate the flatness variables with the shape of the remaining variables. The main idea would be

to find patterns in the process variables and be able to associate certain trends with specific degrees

of flatness. For instance, a dummy example will be that when the time series representing the

temperature starts with a peak and end with a sharp fall the flatness is going to be low.

to define the time series classes based on the flatness map. The correspondence would be something

like: when the flatness is zero the coil is perfectly flat whereas when the values are far from zero (in

both positive and negative directions) the degree of flatness decreases accordingly.

In a visual way, it would be desirable to be able to

highlight the important patterns that are being considered

design appropriate alarms according to the shape of the series

Figure 4: Making use of the knowledge that high peaks at the start of the coil for certain

variable induce a higher risk of low flatness (red), trigger an alarm


687691 of 88

1.2 Data production flow in the industrial environment

1.2.1 Data description

Datasets used on the present prototype were provided by Arcelor Mittal as the final user of PROTEUS. Coil

production data consists on different measures that are made available during the production process at

different moments in time:

Time-series: these data are produced and available in real time during the coil production. It consist

of 53 different variables that are measured and made available in real time by different sensors

deployed in the actual production environment. These variables present a variety of formats,

combining both 1D and 2D information, and are produced at different rates (depending on the

specific variable) ranging from 50 milliseconds to around 1 second period.

HSM: historical and final production coil data (known as HMS data). It provides more than 7,000

variables as aggregated information per coil at the end of the production process, thus this is

available only once the production has finalised for one particular coil.

Flatness: measures of the flatness of the resulting coil. It consists of 3 variables (2 of them as 1D

flatness, and an additional 2D flatness measure) following a format that is compliant with the one

used in time-series, and synchronised with it by the spatial information (x, y). This information is

measured and available only after the coil has been fully produced, and after a certain delay due

to the production environment setup and infrastructure. As previously introduced in previous

sections and also in previous deliverable D2.1, this information represents one of the main targets

for the real time learning and predictive processes to be enabled by PROTEUS, since the

capability to predict flatness in real time (without the need to wait until the process has finalised to

get the actual measures) would be key to improve business operations.

The figure below provides an overview of the process:

Figure 5: Summary of data production flow


687691 of 88

The diagram represents the following: on top left the real time data process is represented. These are the 53

variables, defined as stream data previously, which are generated meanwhile the coils are being processed.

Some examples are provided, as the positions or the coiled. Then, on top right are the flatness data, which are

the three flatness variables generated once the coil has finished the lamination process. On bottom right are

represented the third and last type of data, the historical HSM data.

In the centre part of the diagram is represented the temporal evolution of the coil (from the start to the end of

the production process) and the available data in each moment. I.e., the first block represents when the coil is

being produced and at this point there is only real time data process available. Once the process has finished,

there are HSM historical data and flatness measure process. The flatness, as was mentioned in a previous

comment, is placed at the very end of the second block as in some cases there is some delay which makes

that the real flatness is not provided until minutes later. In any case, once it is known, it becomes another

input variable for the next iteration of the model (as the white arrow shows).

For further information about the data formats and schemas, please see Annex A.


687691 of 88

2 Technical design

2.1 Overview

The Proteus environment will be deployed on a computer cluster located at Treelogic headquarters. The

cluster consists of four virtual machines with the Hortonworks Data Platform 2.5 distribution, which is

capable of being integrated with external services such as Apache Flink and CouchBase. These four virtual

machines are distributed on a master-slave architecture and are in charge of providing the hardware resources

to achieve the full integration of the components of the demo. An overview of the architecture and the data

flow is shown in the following image:

Figure 6: The overall distributed architecture of PROTEUS

The raw data starts its life within an ETL process (1) to send such data to a distributed HDFS cluster along

the four nodes. At this point Kafka takes the data (2) to start the ingestion on processing side with Apache

Flink Cluster (3) over Proteus Hybrid Engine, PEACH and SOLMA library developed on the project, or

directly to CouchBase Server as another parallel ingestion (3) using Kafka Connect in order to persist data

without any process.

The cluster deployment Cluster deploy is based on a lambda architecture with two processing layers: a speed

layer and a batch layer.

Batch layer is in charge of process historical data persisted on the CouchBase server. Using a

specific Flink engine developed by Proteus team (SOLMA library, PEACH library and DSL), this


687691 of 88

layer will retrieve all data needed in order to improve some models that would be applied in the

speed layer.

Speed layer is in charge of process all incoming data in a real-time fashion. This layer could use

some specific models created or improved by the batch layer in order to process and analyse data,

obtaining results in near to real time requests by operator user.

At the end of both Flink processing stages, data will be stored in the CouchBase cluster (4) and sent as a real-

time stream to the Visual Analytics Platform where it is gathered by an Apache Kafka consumer at the back

end (6). The processed data will be improved by the continuous improvement process of the models

developed by the machine learning libraries (5).

The Visual Analytics Platform comprises a back-end application for authentication and data access; and a

front-end client (8) where the user can create and customise visualizations of the historical data (7) and the

results of the Proteus Hybrid Engine (6).

This is only an overview of the architecture of PROTEUS for a quick understanding. In the following

sections the services, integration and deployment will be explained in depth.

2.2 Cluster infrastructure

All the services in charge of storing, loading, transporting and processing data are deployed in a cluster

composed of four virtual machines in master-slave architecture

2.2.1 Hardware Architecture

The cluster is composed by one physical machine which is virtualized by VMWare Workstation

10.0.5.52125 in four virtual nodes.

Table 1: Cluster Physical Machine

Cluster Physical Machine

Processor 2x Xeon E5 2650 v2

RAM 64 GB

HDD 4 TB

The original machine is divided in four virtual nodes with the follow hardware virtualized


687691 of 88

Figure 7: Cluster details

Virtual Node 1: Master

Operating System Centos 7

RAM 12 GB

HDD 1 TB

Virtual Node 1: Slave 01

Operating System Centos 7

RAM 16 GB

HDD 1 TB

Cluster Physical Machine: Slave 02

Processor Centos 7

RAM 16 GB

HDD 1 TB

Cluster Physical Machine: Slave 03

Processor Centos 7

RAM 16 GB

HDD 1 TB


687691 of 88

2.2.2 Base Software & Services Architecture: services under Hortonworks data

platform and CouchBase

The platform selected to manage the cluster and the services is Hortonworks, which offers an installation

service that includes most of the necessary services. The platform can be later completed with the installation

of new services which can coexist with the services of the original package of the platform without

problems. The services and packages deployed and their functionality are described below.

The cluster uses a Hortonworks distribution, concretely the Hortonworks Data Platform on his 2.5 version.

This platform packages the most typical services of the Hadoop Ecosystem. Of this range of services

included by Hortonworks, are selected to be installed and active on the cluster only the necessary to achieve

the necessities of processes like data ingestion, data storage, data exploitation and management. This services

are described in the next sections.

2.2.2.1 Data Ingestion: Apache Kafka

Apache Kafka is a distributed streaming platform which enables publication and subscription to streams of

records. It gives the option of storing the records generated by a stream in a fault-tolerant way and it can

process the stream of records as they occur. A typical Kafka cluster is composed by the following elements:

Brokers: A Kafka cluster is formed by one to multiple brokers to maintain load balance. These

broker instances can handle thousands of reads and writes per second and TB of messages. In the

Proteus architecture there are three Kafka brokers located in the Slave 01, Slave 02 and Slave 03.

Kafka Producers: These are the elements responsible to push data to the brokers. In the real

environment only one coil is produced at a time. In Proteus deployment there will be a single

Producer.,

Kafa Consumer: Responsible to pull the data from a topic with the help of the cluster coordinator.

Although it belongs to the Kafka deployment, the Kafka Consumer is located in the Visual Analytics

Platform, so it will be described in its corresponding section. In Proteus Deployment there will be 3

All the components of the Kafka cluster are coordinated by one instance of Apache ZooKeeper, another

typical service of the Hadoop Ecosystem also included in the deployed Hortonworks Data Platform.

ZooKeeper is responsible of coordinating the brokers, which are stateless, and maintain the cluster state. The

architecture of the cluster is shown below

Figure 8: Kafka producer

Built using the API of Apache Kafka combined with the HDFS API, the Proteus Producer is a small project

capable of taking formatted data from the HDFS and generate a simulated streaming of a real production

environment. To achieve this, the Producer has one attribute called COIL_SPEED which is responsible of


687691 of 88

setting the time used on the production of one coil. By default it is 120 seconds, similar to the real industry

scenario of AMIII.

The way to get one coil produced in that 120 seconds is as follows: The producer reads all the data from the

HDFS to one coil on particularly. This data is stored in one temporary buffer, with the total count of lines for

these coil. In the buffer it is calculated all the distances between two positions x, which result is multiplied

by the result of the division between COIL_SPEED and the total count of lines of the coil, getting the final

delay between registers. This operation makes the producer capable of producing all the values for the same

x position in the same time, while the positions with bigger differences between the x position, will arrive at

different moments.

To get a big amount of data in real time and achieve the big data challenge, the producer is multithreaded, so

it is possible to set a bigger number of Coil on streaming production.

All the data is sent to one of the Kafka topics which is created with only one partition. Topics are divided

according to the production flow settings, as earlier described in section 1:

Real-time: who manages all the time-series data with 1 dimension (position x) and 2 Dimension

(position x and position y), produced and available in real time (streaming) during the coil

production.

HSM: who manages all HSM data, produced as aggregated information at the end of the process and

available only once the coil production has finalised.

Flatness: who manages the flatness data variables, produced as measures of the flatness of the

resulting coil, and available only after a certain delay after the coil production has finalised.

The data stream is composed by messages which create objects of type Coil. This Coil objects have the

format as explained at Annex A.

2.2.2.2 Data Storage: HDFS

The Hadoop Distributed File System (HDFS) is a distributed file system that shares many similarities with

traditional file systems. The HDFS have features like a highly fault-tolerance and a high throughput access to

data, making it the most common choice for applications that have large data sets like the industry scenario

subject of Proteus. The HDFS has a master/slave architecture composed by a single NameNode and a

variable number of DataNodes.

The NameNode is the master server. Its function is to manage the file system namespace and regulate the

access to files while the DataNodes are responsible for serving read and write requests from the file system's

clients and perform block creation, deletion and replication upon instruction from the NameNode.

In the Proteus architecture exists one NameNode located in the master node which coexists with one of the

DataNodes. Each of the remaining DataNodes are deployed for each slave node of the cluster. So, the HDFS

cluster is composed by one NameNode and four DataNodes deployed with the following architecture


687691 of 88

Figure 9: HDFS: Block replication

2.2.2.3 Data Storage: CouchBase

In addition to HDFS, the prototype architecture is also considering the use of CouchBase for data storage.

Couchbase is responsible to store the historical data in a reliable way. The applications developed with JSON

are fully compatible because Couchbase can speak JSON natively and supports binary for all the data storage

needs. Also, Couchbase comes with a developer-friendly query language, an optionally MapReduce for

simple, efficient and comprehensive data retrieval.

A Couchbase cluster composed by four nodes, hosts the database, which store documents. Each of these

documents is uniquely named in the database and it can be readed, updated or writed thanks a RESTful

HTTP API provided by Couchbase. Each of these documents are the primary unit data in Couchbase and

consists of any number of fields of varying types (text, number, boolean, lists, etc) and attachments with an

extra metadata maintained by the database system.

2.2.2.4 Data Exploitation: Apache Hive

The Apache Hive data warehouse software facilitates reading, writing and managing large datasets residing

in distributed storage using SQL language, the easiest way for a big majority of users. Apache Hive auto

translates the SQL query to MapReduce, and its structure can be projected onto data already in storage. The

Hive client provides a command line tool and a JDBC driver to make the queries.

Apache Hive is the tool selected in the Proteus architecture to perform queries about the raw and log data.

Exploring the Hadoop cluster capabilities with optimizations layers like Apache Tez and the creation of ORC

tables, Apache Hive is capable of making queries against the data stored in the HDFS getting back the results

in a low latency time, making possible to compare them with the results of the processing


687691 of 88

Figure 10: HIVE architecture

2.2.2.5 Management Services: Apache Ambari and CouchBase Dashboard

The Apache Ambari project is aimed at making Hadoop management simpler by developing software for

provisioning, managing and monitoring Apache Hadoop clusters. Ambari also provides an intuitive, easy-to-

use Hadoop management web UI backed by its RESTful APIs.

To make the deployment over the Proteus cluster and manage all the services, it is used the Horton Works

Dataplatform, which includes Ambari as the default management tool. All the operations realized over the

services, like changing between different configurations files in the services, checking the status and logs of

the services and resources consumption can be realized from a web interface friendly, very easy to use and

understand.

In addition, the Web Console is the main tool for managing the CouchBase environment. It is accessed by

navigating to the IP and PORT designated of a node running. From these dashboard it is possible to access to

the Cluster section, where information about the RAM and disks usage among others are provided. It is also

possible to make queries with against the data included in the cluster.

2.2.3 The PROTEUS core contributions

PROTEUS is taking advantage of the Apache Flink as the core platform (together with several state-of-the-

art technologies and services as previously described in the previous section), then further researching and

developing specific technology to enhance this platform. Apache Flink is an open-source framework for

distributed stream processing for high-performing, always-available, and accurate data streaming

applications. It is capable of providing results that are accurate even in the case of out-of-order or late-

arriving data. It is stateful and fault-tolerant and capable of working at large scale, and running on thousands

of nodes with good throughput and low latency.

Because Flink Engine is capable to work with batch and streaming data, it is the core tool to make all the

predictive processing, but in PROTEUS, the original Flink capabilities are being enhanced thanks the

including of the consortium developed tools, i.e., a fork of Apache Flink able to perform hybrid processing, a


687691 of 88

Scala-embedded engine-independent DSL called Lara, the SOLMA library, responsible to make all the

operations of the predictive analysis of the data. Each of these tools, will be described in the next sections.

Figure 11: The PROTEUS, Flink-based architecture

2.2.3.1 Data Processing: the PROTEUS Hybrid Engine

PROTEUS Hybrid Engine is a fork of Apache Flink. It features an extended support to operations that mix

streaming and batch processing. This is achieved through side inputs for stream operators, i.e., an operator

processes records coming from the stream as well as preloaded historical data. As a result, side inputs enable

hybrid processing within a stream operator.

2.2.3.2 Declarative Language: Lara

Lara is an engine-independent declarative language that allows programmers to author complete end-to-end

data analysis programs that are automatically parallelized. Lara is based on Emma, a deeply embedded in

Scala which enables authoring scalable programs using an abstract data types (called DataBag) and control

flow constructs. The main observation behind Lara is that data analytics first perform some preprocessing,

which naturally fits the dataflow paradigm, and then they perform some machine learning tasks, which are

expressed as linear algebra operations with iterations. Therefore, Lara aims to provide a common

environment that allows to define pipelines that mixes relational and linear algebra. To this end, Lara extends

Emma by introducing a Matrix data type meant for machine learning workloads. Secondly, Lara enables

joint optimizations over both relational and linear algebra as having a single pipeline made of both

operations enables holistic optimizations.


687691 of 88

2.2.3.3 Proteus Elastic Cache: PEACH

Peach (Proteus Elastic Cache) is a distributed key-value cache that can be used both inside and outside of the

Proteus scope. The cache aims to provide low latency responses on a distributed elastic deployment with

fault-tolerance capabilities. As a generic design, the cache could be integrated within Apache Flink to

speedup computing processes.

2.2.3.4 Scalable Online Machine Learning Algorithms: SOLMA

Solma is a scalable online machine learning library (including classification, clustering, regression, ensemble

algorithms, graph oriented algorithms, linear algebra operations, and anomaly and novelty detection)

implemented on top of Apache Flink using the hybrid processing capabilities.

2.3 Visual Analytics Platform

The visual analytics platform of Proteus architecture is deployed in a different machine which does not

belong to the cluster. All the machines are under the same subnet and have full visibility each others, but are

independent.

The analytics platform is designed as standalone software and can be run on commodity hardware and

plugged to the processing engine and historical database, running in a cluster. Its user interface features a

dashboard that provides querying, monitoring and visualization of both data-at-rest from the data storage

layer and data-in-motion streaming from the data processing engine.

This analytics platform is implemented as a client-server architecture comprising a back-end server that

provides user accounts and authentication; and a multi-platform client that can be used both as a web or a

desktop application.

2.3.1 Backend

The backend for the visual analytics infrastructure is implemented as a Java EE application, built on top of

the Spring Framework and deployed in an Apache Tomcat Server. It provides user authentication, querying

for the historical database using Spring Data and a Kafka consumer to collect the messages from the

processing engine.

2.3.1.1 User authentication and security

The visual analytics platform incorporates the Spring Security framework to preserve confidentiality of the

Proteus dataset. This framework provides customizable authentication, authorization and protection against

attacks like session fixation, clickjacking, cross site request forgery, etc.

2.3.1.2 Kafka Consumer

Data processed in the cluster infrastructure is written into Kafka topics (real-time, hsm & flatness) to be later

read by a Kafka consumer from the visual analytics platform.

2.3.1.3 Database connection

The visual analytics platform is able to query historical data stored in the CouchBase server, but it could be

connected to other kinds of databases for other use cases, both relational and non-relational. To enable these

queries, the backend employs Spring Data, which provides a common API for the data access layer to

abstract the programming model from the underlying data store.


687691 of 88

2.3.1.4 Datasources API

The datasources API allows bidirectional communication of the visualization at the client side of the

platform with the server side. This API can be used from the client-side to list all the available data sources,

either batch or streaming, and receive data through various protocols, namely WebSocket, HTTP or HTTP/2.

It can be extended to enable other protocols in the future.

2.3.2 Front-end

2.3.2.1 Proteus Dashboard

Proteus Dashboard is the user interface for the visual analytics platform. It is a cross-platform application

that can run either as a web application or a desktop application by means of the Electron framework. The

application allows the user to access a dashboard through a login screen where he can create and customize

Proteic visualizations that can then be connected to a set of data sources in the back-end of the visual

analytics platform. The dashboard is based in the ng2-admin framework, built with cutting edge, open source

technologies like Angular, Bootstrap 4, Webpack and Electron.

2.3.2.2 Proteic

Proteic is the data visualization library for the web created for the interactive visualization of the Proteus data

and the results of the incremental analytics engine. It is integrated in the Proteus Dashboard to display data

retrieved from the back-end of the visual analytics platform. The library features a convenient declarative

API for the creation of multiple kinds of visualizations suited for high volumes of both streaming and static

data. Its API allows the retrieval of data over the various communication protocols available in the

Datasources API of the back-end component.

This component is built and distributed as a standalone open source library that can be used in any data

visualization scenario for the web. It is implemented in the TypeScript language, and follows the latest

standards and good practices in web development to achieve high performance, compatibility and ease of

development. The design process of the library accounts for usability and accessibility of the various

visualizations, following design guidelines like responsive web design and offering alternative colour

palettes for colour blind-users.

2.4 Setup

2.4.1 Required packages before installation

Just before deploying the Hortonworks Data Platform in the virtual nodes, it is necessary to check that the

following software packages are installed in each of the virtual nodes:

Yum

Rpm

Scp

Curl

Tar

Wget

Open SSL ( v1.01, build 16 or later)

Python 2.7

Also, all nodes must be in same network in order to communicate among themselves.


687691 of 88

2.4.2 Operating System

The operating system in each of the nodes is Centos 7. By default, Centos 7 uses Chrony as synchronization

service, which will be uninstalled by NTP synchronization.

2.4.3 Server and Agents

2.4.3.1 Ambari Sever

The Ambari Server will be installed through yum. To achieve the deployment, it will be necessary download

and install the repository from the Ambari Server will be downloaded:

wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo -O

/etc/yum.repos.d/ambari.repo

After the properly check, the installation of Ambari-Server can be made:

yum install ambari-server

Before launch Ambari-Server, it is necessary make the setup:

ambari-server setup

In this setup, it will be checked the next points:

SELinux it will be necessary to be deactivated.

Ambari-Server will ask if the user want to “Customize user account for ambari-server daemon”. The

answer is no, by the way the installation is made as root user.

The iptables it will be necessary to be deactivated.

The version of JDK to install, 1.8 will be the selected choice and it will be necessary to accept the

user contract.

When the Ambari-Server will ask by a database to store the server states, it will be selected the

default option, where Ambari-Server uses the PostgreSQL by defect.

When the setup is ended, the Ambari-Server can be launched and checked.

ambari-server start

ambari-server status

2.4.3.2 Ambari Agents

With the Ambari-Server running propertly, is moment to install the Ambari-Agents, which will be need to be

installed manually. The first step consist in download the official repository from Hortonworks, like it was

made in the Ambari-Server in the three nodes.

wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo -O

/etc/yum.repos.d/ambari.repo

After check than the repository is correctly installed, it can be deployed the Ambari-Agent


687691 of 88

yum install ambari-agent

When the agent finishes the installation, it will be needed to edit the next file

File: /etc/ambari-agent/conf/ambari-agent.ini

[server]

hostname=<FQDN_Selected_for_Master>

url_port=8440

secured_url_port=8441

[agent]

logdir=/var/log/ambari-agent

piddir=/var/run/ambari-agent

prefix=/var/lib/ambari-agent/data

;loglevel=(DEBUG/INFO)

loglevel=INFO

data_cleanup_interval=86400

data_cleanup_max_age=2592000

data_cleanup_max_size_MB = 100

ping_port=8670

cache_dir=/var/lib/ambari-agent/cache

tolerate_download_failures=true

run_as_user=root

parallel_execution=0

alert_grace_period=5

alert_kinit_timeout=14400000

system_resource_overrides=/etc/resource_overrides

; memory_threshold_soft_mb=400

; memory_threshold_hard_mb=1000

[security]

keysdir=/var/lib/ambari-agent/keys

server_crt=ca.crt

passphrase_env_var_name=AMBARI_PASSPHRASE

ssl_verify_cert=0

[services]

pidLookupPath=/var/run/

Finally, ambari agents need to be started in all the machines

ambari-agent start


687691 of 88

2.4.4 Hortonworks Data Platform deployment

If the previous setup ends properly, a web interface will be showed in the FQDN selected for the Master in

the port 8080 by default, for example:

cluster.master.local:8080

The web interface corresponds to Apache Ambari management tool, where will be possible to select the

option “Create a Cluster” and follow all the steps defined by the “Install Wizard”. The principal decisions to

take are the next options:

1. Select the Hortonworks Data Platform version. In this case, the choice is Hortonworks Data

Platform 2.5.3.0 with the public repository.

2. The next step consists in define all the FQDN of each node who compose the cluster. One name

by line and with the option of manual register for the Ambari agents. Then the agents will be

registered against the server and the necessary checks will be made, to check than the process

ends correctly.

3. Finally it will be needed to select all the services to deploy in the cluster. They are:

HDFS

Yarn + MapReduce 2

Apache Hive

Kafka

2.4.5 Proteus Environment

2.4.5.1 Flink Hybrid Engine

First step is to download the official Flink Hybrid Engine DFKI repository on master node:

git clone https://github.com/proteus-h2020/proteus-engine.git

After that, it requires to be compiled and make and packaged:

cd flink

mvn clean package -DskipTests

Finally, on Build-Target folder, Hybrid Engine is installed

2.4.5.2 PEACH

Peach works over Redis database, so first of all Redis would be deployed in PEACH node:

wget http://download.redis.io/releases/redis-<your_version>.tar.gz

tar xzf redis-<your_version>.tar.gz

cd redis-<your_version>

Make

./src/redis-server


687691 of 88

1. Compile the project to obtain the executable scripts:

mvn install -DskipTests

2. Move to the target directory to launch the PeachServer:

cd peach-redis-server-dist/target/peach-redis-server-dist-*/

3. Launch Peach with the Redis backend. Notice that Redis must be up and running

./bin/peach-redis-server-app

2.4.5.3 SOLMA

In order to use SOLMA library, in every maven proyect it is needed to add attached code inside pom.xml

file:

<dependencies>

<dependency>

<groupId>eu.proteus</groupId>

<artifactId>proteus-solma_2.10</artifactId>

<version>0.1-SNAPSHOT</version>

<scope>compile</scope>

</dependency>

<dependency>

<groupId>org.apache.flink</groupId>

<artifactId>flink-clients_2.10</artifactId>

<version>1.4-SNAPSHOT</version>

<scope>compile</scope>

<exclusions>

<exclusion>

<groupId>log4j</groupId>

<artifactId>*</artifactId>

</exclusion>

<exclusion>

<groupId>org.slf4j</groupId>

<artifactId>slf4j-log4j12</artifactId>

</exclusion>

</exclusions>

</dependency>

</dependencies>

<build>

<plugins>


687691 of 88

<plugin>

<groupId>net.alchim31.maven</groupId>

<artifactId>scala-maven-plugin</artifactId>

<configuration>

<recompileMode>incremental</recompileMode>

</configuration>

<executions>

<execution>

<id>scala-compile-first</id>

<phase>process-resources</phase>

<goals>

<goal>add-source</goal>

<goal>compile</goal>

</goals>

</execution>

<execution>

<id>scala-test-compile</id>

<phase>process-test-resources</phase>

<goals>

<goal>add-source</goal>

<goal>testCompile</goal>

</goals>

</execution>

</executions>

</plugin>

<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-assembly-plugin</artifactId>

<version>2.4</version>

<configuration>

<descriptorRefs>

<descriptorRef>jar-with-dependencies</descriptorRef>

</descriptorRefs>

<archive>

<manifest>

<mainClass>solma.TestSolma</mainClass>

</manifest>

</archive>


687691 of 88

</configuration>

<executions>

<execution>

<phase>package</phase>

<goals>

<goal>single</goal>

</goals>

</execution>

</executions>

</plugin>

<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-dependency-plugin</artifactId>

<executions>

<execution>

<id>copy-dependencies</id>

<phase>package</phase>

<goals>

<goal>copy-dependencies</goal>

</goals>

<configuration>

<useBaseVersion>false</useBaseVersion>

</configuration>

</execution>

</executions>

</plugin>

</plugins>

</build>

2.4.6 Additional Storage System: Couchbase

Couchbase installation requires next steps:

1. Disable THP (Transparent Huge Pages). THP feature of the Linux kernel must be disabled on

systems running Couchbase Server. Transparent Huge Pages (THP) is a Linux OS feature that

conceals much of the complexity of using actual Huge Pages as well as automates the creation of

contiguous memory space. It is enabled by default in some Linux Operating systems, but not all.

2. Download official rmp package in all machines:

curl -O http://packages.couchbase.com/releases/couchbase-release/couchbase-release-1.0-2-x86_64.rpm

3. Install rpm package:


687691 of 88

sudo rpm -i couchbase-release-1.0-2-x86_64.rpm

sudo yum update

sudo yum install couchbase-server

4. After that, access via dashboard manage interface on port 8091 in master node.

5. Check in “Start new cluster”

6. In port 8091 for all slave nodes, select “join a cluster now” option. Select FQDN master node and

user name with password configured in that server.

2.4.7 Visual Analytics Infrastructure

The visual analytics platform is distributed as a standalone WAR file that contains both the front and back

ends of the web application alongside all its dependencies. It requires Java 1.7 and Apache Tomcat 7 and it

should be deployed by placing the WAR file in the webapps directory of the Tomcat 7 installation directory.

The application requires setting some configuration values before running in a production environment and it

is shipped with a default settings file that contains both the configuration values and its documentation. Its

location can be passed to the application by setting the SPRING_CONFIG_LOCATION environment

variable.

2.4.8 Setup Summary

The table provides a summary of the different technologies and versions used for the setup of the prototype:

Cluster

Hadoop Ecosystem under Hortonworks Data Platform

Hortonworks Data Platform 2.5.3.0

HDFS 2.7.3

NameNode Master

Secondary NameNode Slave 01

DataNode Master, Slave 01, Slave 02, Slave 03

NFSGateway Slave 01, Slave 02, Slave 03

YARN + MapReduce2 2.7.3

ResourceManager Master

App Timeline Server Master

NodeManagers Slave 01, Slave 02, Slave 03

YARN Clients Master, Slave 01, Slave 02, Slave 03

Tez 0.7.0

Client-only service

Hive 1.2.1


687691 of 88

Hive Metastore Slave 01

HiveServer 2 Slave 03

MySQL Server Slave 03

WebHCat Server Slave 03

Hive Clients Master, Slave 01, Slave 02, Slave 03

HCat Clients Master, Slave 01, Slave 02, Slave 03

Oozie 4.2.0

Oozie Server Slave 01

Oozie Clientes Slave 01, Slave 02, Slave 03

Zookeeper 3.4.6

Nodes Master, Slave 01, Slave 02, Slave 03

Ambari Infra + Ambari Metrics 0.1.0

Infra Solr Instance Master

Infra Solr Clients Master, Slave 01, Slave 02, Slave 03

Kafka 0.10.0

Kafka Brokers Master, Slave 01, Slave 02, Slave 03

External Services

Apache Flink - Hybrid Engine 1.0

PEACH 1.0

SOLMA 1.0

DLS Language 1.0

Couchbase 4.5.1-2844 Community Edition (build-2844)

Couchbase servers Slave01, slave02, slave03

Visual Analytics Infrastructure

Proteus Dashboard

PROTEIC.JS


687691 of 88

3 Validations & KPIs

3.1 KPI review

The identification of Key Performance Indicators (KPIs) within PROTEUS is an ongoing, iterative process

that commenced with the work in Deliverable 2.6. In D2.6 an initial set of KPIs were provided for measuring

the performance of any future PROTEUS solution, within the parameters in which the project had progressed

up to that point. At that stage three manufacturing/analysis processes representing defined PROTEUS

scenarios were presented, these being: data process; visualisation; and online learning. From these the

following KPIs were presented:

Data process:

the amount of data being processed

the number of data queries happening at any given time

the speed at which the data is normally processed determined by the workload

Visualization:

identification of defects/relevant variables

response time from interactive query like zoom

amount/percentages data is critical to detect the defects

Online learning:

improvement of defects detection

reduce defects

Following that initial step, and as the use-cases based on the needs of industry crystallised, these initial KPIs

have been revised and expanded as part of this deliverable to reflect the further development of the project

towards the first prototype.

In addition to the developments in relation to the scenario and technical specifications of the first prototype,

the project has also used interfaces with other projects to further develop the KPIs with respect to the two

missions of the PROTEUS impact assessment:

1. To identify the extent to which the PROTEUS solution provides added value for Arcelor Mittal

2. To identify improvements in large-scale, predictive analytics for machine learning more generally

Despite some notable efforts industry and academics broadly agree that big data analytics has yet to create

agreed, standard benchmarks [1]. Instead, benchmarks are focused on particular analytical tools (e.g.,

MapReduce [1] [2]) or in specific contexts (e.g., Internet services [3]). This makes it difficult to create

standardised benchmarks across different tools, scenarios and analytic contexts. Nevertheless, projects like

the EC-funded Holistic Benchmarking of Big Data, HOBBIT, project are using different case studies to

create benchmarks and KPIs for big data across different platforms. The benchmarks to be created by the

HOBBIT project fall within eight specific categories:

Data Storage Benchmark

Data Extraction Benchmark for unstructured data

Linking benchmark

Data analytics benchmark

Data Extraction Benchmark for Sensor Data

Faceted Browsing Benchmark

Versioning Benchmark

Question Answering Benchmark

The HOBBIT benchmarks are divided between different stages in the data value chain (e.g., data storage,

extraction, linking, etc.) and different analytic processes (browsing, question answering, etc.). Not all of


687691 of 88

these will be relevant to PROTEUS, however the delivery of the first version of these benchmarks in M18 of

the HOBBIT project will provide an opportunity for PROTEUS partners to review them in detail and work

with HOBBIT partners to identify the extent to which their benchmarks are useful for PROTEUS analytic

processes.

Furthermore, the current benchmarking activity has a dependency with the delivery of the first PROTEUS

prototype, and as such, a more comprehensive KPI and benchmark identification is premature, given the

need to evaluate the first prototype against the scenarios to understand more precisely what should be

measured, what the measurement tool might be and what the targets should be. Nevertheless, this chapter

represents the development, to date, of the project’s consideration of potential useful KPIs and benchmarks

for the use case scenario and for the broader improvements in data analytics to be evaluated within

PROTEUS. In this deliverable, given the industrial scenario and technical specifications presented, the KPIs

have be divided into three main categories, i.e., KPIs for end-user industrial partner, KPIs for data analysis

and visualisation, and KPIs for data platform. This deliverable only includes KPIs that are relevant for the

first prototype.

3.2 Use case evaluation

Use case information presented above indicates some KPIs that would be useful for evaluating the added

value in efficiency and defective coil detection for AMIII provided by PROTEUS. The prototype addresses

initial uses cases that would allow end users to react to defective coils.

Use-Case One Use-Case Two

Problem: Tension of a coil increasing leads to

breakage of the coil strip

Problem: When speed tension exceeds a certain

threshold, this implies a flatness issue

First prototype solution: Provide predictive

actions through triggering an alarm

First prototype solution: Provide predictive

actions through triggering an alarm

Operator Response: Open the stands with the

rollers

Operator Response: Stop the rolling process or

automatically downgrade the quality of the coil

With respect to these two particular use cases, the project is currently considering the following KPIs. The

scenario-related KPIs are divided between KPIs directly related to defective coil detection and KPIs related

to process improvements for AMIII:

3.2.1 Defective coil detection KPIs

Temperature: number of alerts triggered whereby the temperature exceeds a pre-identified “normal

functioning” range – specifically where temperature exceeds Mean + Deviation or Mean - Deviation.

Roller tension: number of alerts triggered whereby the roller tension exceeds a pre-identified “normal

functioning” range – specifically where tension exceeds Mean + Deviation or Mean - Deviation.

Rejection ratios: ratio of coils at high original requisite [high] quality standards vs. coils downgraded yet

acceptable for lower quality uses [defective yet usable]; ratio of coils at high original requisite [high] quality

standards vs. coils that cannot be re-tasked for lower quality uses [defective and unusable]; ratio of coils

downgraded yet acceptable for lower quality uses [defective yet usable] vs. coils that cannot be re-tasked for

lower quality uses [defective and unusable]

These KPIs will enable the project to evaluate how well the prototype is functioning at identifying defective

coils given the production sensor parameters (e.g., deviations from the agreed mean range in relation to

temperature or roller tension) during the production process. It will also enable a baseline measurement of

the functioning of the first prototype in relation to the rejection ratios. This will allow both an identification

of improvements in later versions of the PROTEUS solution and provide data to enable the project to

measure some of the process improvement KPIs presented below.


687691 of 88

3.2.2 Process improvement KPIs

Number of false alarms emitted per measurement variable: incorporating both false-positives and false-

negatives from alarms within the Hot Strip Mill monitoring processes during the coil production (i.e., sensor

results for temperature, roller tension, lamination, flatness, etc.).

Refresh time: time required for the system to refresh once new data (coils) are received.

Detection target for defective coils: accuracy of flatness prediction of current/future/past coil based on the

error rates.

Detection time: elapsed time from the point of production commencement to the point whereby the product

is characterised as defective.

Response time: elapsed time from the point whereby a product is characterised as defective to the point

whereby production of subsequent [defective] coils is stopped.

Reassignment time: elapsed time from the point whereby a product is characterised as defective to the point

whereby produced coils are reassigned to other uses.

Downtime: the result of a breakdown or simply a machine changeover, downtime is considered one of the

most important KPI metrics to track. When machines are not operating, money is not being made so reducing

downtime is an easy way to increase profitability. Organizations that track downtime typically require

operators to enter a “reason code” via keypad, pushbutton or bar code scanner so that the most common

reasons can be reviewed at a later time

Material utilisation rates: For mass production or products with very expensive materials used, like sheet

metals with special coatings, the material utilization becomes a relevant KPI for the production processes.

Production rate: Machines and processes produce goods at variable rates. When speeds differ, slow rates

typically result in dropped profits while faster speeds affect quality control.

Economic loss: loss of revenue; amount of waste; reduction in revenue arising from quality downgrading of

defective coils; compensation due to reassignment of defective coils; material costs; wage costs; equipment

costs, etc. This will be estimated using current AMIII loss figures and the reduction in waste that could be

achieved given the increase in detection of defective coils.

Reduction in process costs: reduction is process costs resulting from reduction in amounts of waste; decrease

in number of downgraded coils. This will be measured using current AMIII cost figures, reductions based on

early detection of defective coils and changes in down time for the production line.

Target: Display target values for output, rate and quality – used to motivate employees to meet specific

performance targets

Each of these KPIs will demonstrate for AMIII and also potential additional users how the PROTEUS

solution can provide added value via efficiency and resource reduction within the production process. These

KPIs will allow a demonstration of where tools like PROTEUS provide added value, where they need to be

further improved and how they might generate value for potential, additional customers as part of the

production of data for the exploitation work within the project.

3.3 KPIs for AMIII data analysis, visualisation and

visual analytics platform

The next set of KPIs are focused on the intersection between the data analytics and the scenario / use case

requirements presented in Sections 1 and 2 above.

3.3.1 Data extraction-related KPIs

Data quality: As measured by the number or percentage of missing data/null values, outlier values, and other

abnormal values. These should be logged as this could affect the estimation of the mean and variance as well

as highlighting this in the data interpretation as a potential bias. Data quality for Time series, HSM, Flatness

(missing data/null rate, outlier rate). The benchmarks relevant to this KPI may be extracted from the

HOBBIT Data Extraction (unstructured data) and Data Extraction from Sensors benchmarks.


687691 of 88

Validation of data relevant to flatness: (Identify the most relevant variables, features, patterns, trends, and

characteristic)

Database connectivity: ability to access and query historical data stored on the CouchBase server. Ability to

connect to other databases.

Processing data: Batch layer (processing historical data, retrieve all data needed for model improvement) and

speed layer (processing data in real-time). This is for validating the technical concept of using a batch and a

speed layer.

Data transfer: from CouchBase cluster to Visual Analytics Platform

3.3.2 Data analysis-related KPIs

Online average and variance calculation (representing "safety zone"): Here, the accuracy of the estimation is

an important factor. This KPI will measure the mean and variance tracking based on forgetting factors, i.e.,

how much history will be using for the estimation of mean and variance. Also, as an increase or decrease in

mean/variance can have different consequences. Another factor could be to look at the tracking performance

as the mean/variance increase (attack time) or when the mean/variance decreases (release time). Online

average and variance calculation (representing "safety zone")

Accuracy of flatness: Accuracy of the prediction models based on real-time variables such as tension and

temperature and the flatness of previous coils.

Speed and accuracy of model parameter update: Accuracy of the online learning based on real flatness values

and automated feedback from the system.

3.3.3 Data visualization-related KPIs

Data visualization response time: Measurement of the time from making queries to receiving visual

information from the system. This could include various scenarios that: e.g., validates the data throughput or

the database design; handling of stream data with different sampling rate. The HOBBIT Question answering

benchmark may provide a useful guide here.

3.3.4 KPIs relevant to the PROTEUS specific components

Data Storage:

HDFS fault tolerance

HDFS throughput performance

Read/write speed for request

Apache Kafka (data ingestion) KPIs:

Rate of data ingestion and data throughput

Fault-tolerance

o Out-of-order data rates

o Late-arriving data

Load balancing

Read/write speed and size of messages

Latency time

CouchBase (data storage) KPIs:

Map/reduce data retrieval performance

Latency time

Latency variance against changes in workloads

Apache Hive (data exploitation) KPIs:

Latency time


687691 of 88

Read/write speed and size of distributed datasets

Speed of response to queries

Apache Flink (core platform) KPIs:

Latency and throughput

Fault-tolerance

o Out-of-order data rates

o Late-arriving data

Latency variance against changes in workloads

PROTEUS Hybrid engine KPIs:

Hybrid performance vs non-hybrid performance

Fault-tolerance of assimilation of streamed data and batch processed historic data

Lara:

Parallelisation performance vs. serial performance

PEACH:

Low latency responses and fault-tolerance capabilities

Fault-tolerance

Each of these KPIs will enable the project to evaluate how well the technical elements of the PROTEUS

solution are functioning, for example how well the data extraction mechanisms are functioning, how well the

online learning algorithms are functioning and the extent to which they are fit for purpose, etc. It will also

enable the project to evaluate the extent to which the components of the system are functioning adequately

and are fit for purpose. These measurements will enable the project to demonstrate improvements in the

functioning of each iteration of the PROTEUS solution as the project develops. This will demonstrate added

value for AMIII as well as demonstrate the progress in data analytics within the industrial scenario. Further

KPIs will be added as the project progresses – for example KPIs related to end users’ evaluation of the

visualisation tool, based on the development of the solution and the project overall.

3.4 Big data KPIs

The final set of KPIs will enable PROTEUS to demonstrate that the project solution represents

improvements in data analytics in general. This is the most challenging part of the development of the KPIs

because of the relative immaturity of the field. However, as noted above, the project will create links with

other projects and initiatives in order to identify KPIs and benchmarks relevant to big data processes across

contexts and tools. These KPIs will include indicators of system performance and the extent to which the

tools can handle large, fast moving heterogeneous data.

Volume (data at rest): Volume of data able to be processed with the PROTEUS toolset within a specified

period of time.

Velocity (data in motion): Refers to how quickly streaming data is able to be produced and how fast the data

must be processed to meet the need or demand. Measured in milliseconds to seconds in relation to particular

data volumes.

Veracity (data in doubt): This refers to uncertainty due to data inconsistency and incompleteness,

ambiguities, latency, deception, model approximations, accuracy, quality, truthfulness or trustworthiness.

The data extraction related KPIs will enable the project to measure this and brainstorm ways to further

improve this metric.


687691 of 88

Variety: Ability to analyse both batch and streaming data together and produce visualization tools adequately

representing the variety of data relevant to the decision-making process. The HOBBIT Linking benchmark

may be a valuable resource to set a KPI for this.

Uptime: Availability of services, resources and computing environment in real time. PROTEUS intends to

maximise this figure and produce improvements during each iteration of the system.

Bandwidth: Available bandwidth for transmission of data. This will be measured using a ratio of bandwidth

available to volume of data processed.

Performance and capacity: elimination of bottlenecks, tuned applications, and sufficient memory.

Value (data for co-creation and deep learning): Identification of high-value data for the processing,

assessment of the relative importance of different complex data from distributed locations and optimisation

of the data processing with this information. Big data with deep analytics means greater insight and better

decisions, something that every organization needs.

Taken together, these KPIs will enable the consortium to consider the extent to which PRTOEUS represents

progress beyond the state of the art with respect to data analytics capabilities more generally. These KPIs

specifically focus on the characteristics that define big data, including volume, variety, velocity, veracity,

etc. which will provide an indication of how well the PROTEUS solution represents value in large scale

predictive analytics, assuming there are appropriate and sufficient examples against which PROTEUS can be

evaluated. Barring that, these KPIs will be used to measure progress with respect to the PROTEUS solution

during the course of the project and to act as an exemplar of KPIs and benchmarks that could be used to

measure subsequent solutions. As such, this activity will enable project partners to participate in and feed

directly into standard setting in big data space more generally.

3.5 Summary

This section has provided a snapshot illustration of the consortium’s current consideration of KPIs and

benchmarks that will be used in the impact assessment for PROTEUS given the development stage of the

project and the solution. Specifically, the snapshot represents a space where the first prototype is being

finalised, but before it has been made available for demonstration and analysis within the consortium. As

such, the KPIs should be considered a second-generation of potentially relevant KPIs rather than a final list

of variables to be measured. Nevertheless, the KPIs outlined here are a significant step in providing material

that the consortium should agree on in measuring, assessing and validating the contribution made by the

PROTEUS project and the PROTEUS solution in developing big data capacity within European industry and

in Europe more generally. The consortium will continue to evaluate and finalize the list of KPIs both

immediately after the release of the first prototype and also as the project develops into its second period.


687691 of 88

4 Conclusions

The current deliverable has detailed the software implementation and setup of the first prototype in

PROTEUS that has been developed to cover the objectives pursued by different ArcelorMittal use cases.

These use cases are devoted to cover some alarming issues and pattern discovery aiming to detect and

understand the appearance of flatness defects.

In line with the original objectives, this prototype has focused on bringing together, integrating and testing

all of the technology components as available by PROTEUS at the current time, especially including the

Flink-based hybrid engine and a number of features provided by SOLMA (our Scalable Online Machine

Learning library), but even also adding some visualisation features in the form of a user-oriented dashboard

(not originally planned for this prototype).

Besides integrating the technology components produced by PROTEUS as a result of the research WPs

(WP3, WP4 and WP5), the work performed in this task and documented in this deliverable also included the

implementation of specific components that are required for the recreation of the operational environment,

following the actual data production workflow and timeline, as a simulation of the actual production

environment in the industrial context (i.e. in the form of configurable data producers, tailored to the specific

performance of the real scenario).

Finally, some Key Performance Indicators (KPIs) have been defined as the ways to measure to what extent

the objectives, both business and technical objectives, have been guaranteed and fulfilled with the current

technology.


687691 of 88

5 References

[1] Han, R. and Lu, X. On big data benchmarking. in Big Data Benchmarks, Performance Optimization,

and Emerging Hardware SE - 1, vol. 8807, J. Zhan, R. Han, and C. Weng, Eds. Springer

International Publishing, 2014, pp. 3–18.

[2] Chen, Y. Alspaugh, S. and Katz, R. 2012. Interactive analytical processing in big data streams: A

cross-industry study of MapReduce workloads. Proceedings of the VLDB endowment.

[3] Wang, L., Zhan, J., Luo, C., Zhu, Y., et al. 2014. Bigdatabench: A big data benchmarking suite from

Internet services. IEEE International Symposium On High Performance Computer Architecture

(HPCA-2014), February 15-19, 2014, Orlando, Florida, USA

[4] Behrens, B.A. and Lau, P., 2008. Key performance indicators for sheet metal forming processes.

Production Engineering, 2(1), pp.73-78.


687691 of 88

Annex A Coil data formats for streaming data

A.1 Variables measured on 1 Dimension

Field Type

Position X Double

Coil Identifier Integer

Variable String

Value Double

timeStamp String

A.2 Variables measured on 2 Dimensions

Field Type

PositionX Double

PositionY Double

Coil Identifier Integer

Variable String

Value Double

timeStamp String

A.3 HSM data

Variable Name Variable type Variable Name Variable type Variable Name Variable type

V01 factor V2498 double V4995 double

V02 double V2499 double V4996 double




V06 double V2503 double V5000 factor











V017 factor V2514 double V5011 factor


687691 of 88









V026 double V2523 factor V5020 factor

V027 double V2524 factor V5021 double











































687691 of 88




























V096 factor V2593 factor V5090 factor







V0103 double V2600 factor V5097 date







V0110 double V2607 double V5104 date








V0118 factor V2615 factor V5112 date



687691 of 88




V0123 factor V2620 factor V5117 double

















































687691 of 88






















V0192 date V2689 factor V5186 double

V0193 date V2690 factor V5187 double



























