WP 7 - Regola TASK 7.4 : Pilot Application Validation D 7.4 Functional and Non-functional Evaluation...

WP 7 - Regola

TASK 7.4 : Pilot Application Validation

D 7.4 Functional and Non-functionalEvaluation Criteria for C2-SENSE

Pilot Applications

WP 7.4 – Pilot application validation

In this task, the testing environment for the C2-SENSE pilot application, which will be

developed on top of the integrated C2-SENSE components, will be constructed. The

assessment criteria will be defined as test case scenarios. The required tests that will be

performed on the developed pilot application for the thoroughly examination of the system

will be defined. The realized system for the pilot application scenarios will be tested as a

whole in the test environment by the end users. The tests will be in parallel with the

deployment of the pilot application. The feedback from the users will be collected through

questionnaires which will be provided to the system developers to enhance the

functionalities of the C2-SENSE final product.

WP 7.4 - concept

• Verification : technical test works. The product is produced right.

• Validation: User approve the functions. It’s the right product

• Acceptance test: is the final test before a delivery of oa product or a service

Test case: How? Why? Wich purpose?

WP 7.4 - concept

VALIDATION ACCEPTANCE TEST

EU Research, Not a product

technical aspects difficult to evaluate

C2-SENSE VALIDATION

This is the challenge

WP 7.4 - Evaluation Team

• Domain and IT Expert (REGOLA)

• User operator ( Regione Puglia, variuous org.)

• Technical expert of C2-SENSE components (Lutech, SRDC, SAGEM)

WP 7.4 - Object of Evaluation

• FUNCTIONAL REQUIREMENT: what C2-SENSE DO

• NON FUNCTIONAL REQUIREMENT: (simplifying) How C2-SENSE do it? It’s a good solution?

WP 7.4 - Evaluation Process

InspecionAnalysis

QuestionnaireSCORE REPORT

EVALUATION

PILOT APPLICATION EXECUTION

Function evaluationNon functional evaluation

WP 7.4 - Functional Evaluation

FUNCTIONAL EVALUATION

WP 7.4 – Functional evaluation

CFC system

National wheather service

CFD system

Regional civil protection service: SOIR

Dewetra

SOIR SystemCP Aletring

System

CFD Sensor Monitoring

System

Regional civil protection service: CFD

Volunteer Organization

VOL System

National operation room system

Department of Civil Protection

Prov GIS

Province

POAR alerting system

Press Office Apulia Region

POAR system

CCS System

Prefecture

Fire Army System

Fire Army

Army System

Army

Citizen App

Citizen

GIS

GIS

GIS

GIS Healthcare

Information System

hospital

MED system

Medical Emergency Dep

COC System

COC (Municipality)

COC Aletring SystemGIS

GIS

Architecture of the pilot application

WP 7.4 – Functional evaluation


The idea is: To evaluate C2-SENSE component indirectly (where possible) examining the effects on the user operation on the pilot applicatoin

WP 7.4 - Functional Evaluation - the scopes

Scope4.1.2.1 Situation Reporting4.1.2.2 Mission Plan4.1.2.3 Scheduling4.1.2.4 Resource Management4.1.2.5 Alert4.1.2.6 Hospital Communication4.1.2.7 Tracking of Citizens4.1.2.8 Sensor Management4.1.2.9 Enterprise User Authentication and Authorization (EUAA)4.1.2.10 Audit Trail and Node Authentication (ATNA)4.1.2.11 Emergency Situation Map4.1.2.12 Situation Analysis4.1.2.13 Permission

• Action Scope are the ‘functional area of investigation’ of the Functional Validation process

• The idea is to identify the ‘integration profile’ as Scope of investigation.


Objective Evaluation(the system work as

espected)

Subjective Evaluation(operator approve the behaviour of the solution)

Global Evaluation

WP 7.4 - Functional Objective Evaluation

Objective EvaluationN.

Scenario Referenc

eDay Hour Step Case code scope

Institution / Agency /

Actor

Initiator System

Target systemAction simulated test modality evaluation Data structure/data type annotation Open Point to Deploy

1 0 0 AM0000 01 D0-AM0000-01 configuration SOIR SOIR System CP alerting system

preparation of the enviroment. CP receive alerting data form municipalicies

Idea: minimun 10 municipality, 2 prefectures. Several channel

• configure CP alerting system with alerting escalation plan of interested stackeolder (municipality, prefectures and organizations)• Query CP alerting system to check configuration settings

CP alerting system is configured Common NOWTICE instance?

2 0 0 AM0000 02 D0-AM0000-02 configuration Municipality COC System COC alerting system

preparation of the enviroment. COC configure massive alert for the population

Idea: minimun 10 municipality, 2 prefectures. Several channel

• configure COC alerting system with alerting escalation plan of interested stackeolder (volunteer organization)• Query COC alerting system to check configuration settings

COC alerting system is configured Common NOWTICE instance?

3 0 0 AM0000 03 D0-AM0000-03Resource

ManagementMunicipality COC System SOIR System

Communication is necessary because SOIR can involve volunteer services directly

• From SOIR System Send a list of possibly involved Volunteer organization to SOIR System

The data is sent and user receive feedback of delivery Wich system uses SOIR? Simulation UI?

4 1 1 PM0330 01 D1-PM0330-01Situation Reporting

National wheather

serviceCFC system

all interested system, mostly CFD

- makes weather forecast (Forecasts are published on a web site)- notify to other system that a new forecast is available.

human polling on national weather service website. The department pubblish data that CFD operator elaborates. This step will be executed by uman operation only. Then a bullettin is sent to se collaboration enviroment (weather forecast pdf)

The document is sent and CFD receive feedback of delivery in order to be sure the message was received by all the actor connected

Wich system uses CFD? Simulation UI?

5 1 1 PM0330 02 D1-PM0330-02 NONE CFD NONE NONE

• CFD personnel ask to add a new sensor in a specific place. So they define the type and location• Contact specialists to install new sensor• When the sensor is ready and working network sends data to CFD system

It is a step necessary to go further. This line in the sheet it is just a reminder. Sensor have to be installed defining: geografic coordinate, and sensor type, activation time

6 1 1 PM0330 03 D1-PM0330-03Sensor

ManagementCFD Sensor

CFD - Sensor data DBOTHER SYSTEM

The sensor starts to send value sensor data arrive to C2Sense collaboration enviroment and they are delivered to all interested party that are:- mapping tool: display position of the sensor and last value- CFD Sensor data DB

to define CFD tool to simuate this scenario

7 1 1 PM0330 04 D1-PM0330-04Sensor

ManagementCFD

CFD Sensor Monitoring

System (C2Sense)

CFD - Sensor data DB

• check sensor network each 30mins Using a tool (to define) CFD operator read last sensor data Sensor data have to be reguralrly updated to define CFD tool to simuate this scenario

8 1 1 PM0330 05 D1-PM0330-05Sensor

ManagementCFD CFD System

CFD Sensor Monitoring System

• query the Sensor Monitoring system about current data of the deployed sensor• the sensor reply to the request in realtime

Using a tool (to define) CFD query the sensor Last sensor data have to be dispayed on the screen to define CFD tool to simuate this scenario

9 2 1 PM0330 06 D1-PM0330-06Situation Reporting

CFD Dewetra CFC System

• Share bullettin with CFC Network The department pubblish data that CFD operator elaborates. This step will be executed by uman operation only. Then a bullettin is sent to se collaboration enviroment (weather forecast pdf)

The document is sent and CFD receive feedback of delivery in order to be sure the message was received by all the actor connected

to define CFD tool to simuate this scenario

10 2 1 PM0330 08 D1-PM0330-08 NONE CFD CFD System all interested system• publishes bulletin of regional criticity (operator of cfd does that) It is a step necessary to go further. This line in the sheet it is

just a reminder. Sensor have to be installed defining: geografic coordinate, and sensor type, activation time

Human operation of CFD operators its website. Nothing to evaluate

11 3 1 PM0330 07 D1-PM0330-07 AlertRegional

Responsible

Regional Responsible (president of

region)

CP alerting system

• Regional responsible receive the document from CFD system (case D1-PM0330-07) , declare alert state and contact CP alert system to alerts stackeholder• It send an alerting order message to CP alerting system

The regional responsible activate an alert escalation to the configured municipality. 'Regional Responsible' has not a system. From a 'Profile test interface' the initiator send a message to 'alert stackeholder'. The message can contain PDF document, text of wheater phenomena, criticity level foreach alert zone This 'profile test interface' needs to show te response answer from CP alerting system

The activation is enabled on the CP activation system and the responsible recive, on its ui, a feedback of the result

to define wich tool to simuate this scenario. The regional responsible have to receive the documentation (email?)

12 3 1 PM0330 08 D1-PM0330-08 Alert SOIR SOIR System CP alerting system

•Share alerting message to stakeholder previously configured.• Messages will be send via fax, sms, email to the defined stakeholder• After 30 minutes send a delivery report to SOIR system to highlight about not delivered messages

From a 'Profile test interface' the initiator send a message to 'alert stackeholder'. The message can contain PDF document, text of wheater phenomena, criticity level foreach alert zone This 'profile test interface' needs to show te response answer from CP alerting system

The execution of the alerting plan will happen with CP Alerting system (NowTice)

To define SOIR tool to initiate the alerting

13 3 1 PM0330 09 D1-PM0330-09 Alert SOIRCP alerting

systemSOIR System

CP alerting system produce a delivery report to the initiator Automatic. The alerting system perform this operation automatically

SOIR receive the alerting report. From nowtice To define SOIR tool to initiate the alerting

14 6 2 PM0400 01 D2-PM0400-01Alert

(Notification)SOIR SOIR System CP alerting system

• Send a message to CP Alerting system in order to inform people of Local Authority involved of bad wheather condition with the suggestion to apply the civil protection plan. • 20 minutes maximun to alert all local authorities

From a 'Profile test interface' the initiator send a message to 'alert stackeholder'. If possible the message must carry the emergency procedure aproved by Civil Protection. Alerting system can perform this operation only with selected media (eg email, fax…)

The execution of the alerting plan will happen with CP Alerting system (NowTice)

To define SOIR tool to initiate the alerting

High Level test case

SCORE organized per ‘Action scope’


Score are assigend according a pre-defined schema

score meaning0 Function not covered by the solution at all.

1Function covered minimally, but is not sufficient and cannot be accepted in the real world.

2

Function covered, but some critical/important issue was detected (Eg: cost too high, bad performance…). The IA can accept the issue temporarly with the promise of a fix.

3Function covered and it is coherent with the reference. Some issue was identified, but a user can accept the solution.

4Function covered, it is coherent with the reference and no issue was identified.


Report contain annotation and a

radar chart

REPORT


FUNCTIONAL SUBJECTIVE EVALUATION

WP 7.4 - Functional Subjective Evaluation

The idea is: TO ASK to the operator a subjective evaluation of the soluiton, using a pre-defined questionnaire organaized by scope

BUT C2-SENSE is not a product: for this reason the interdisciplinary team is mandatory, in order to explain user to asnwer!!

Regola (Domain and IT expert) will drive the survey.

The scopes are the same of the functional objective evaluation


Example of subjective evaluation survey

Scope Subjective evaluation criteria

4.1.2.1 Situation Reporting

- the solution handles all the information you need?- the solution reduce the operator’s work load?- the solution decreases your response time in emergency management?- the solution decreases the loss of information?- overall impression from 0 to 4.?

4.1.2.2 Mission Plan

- the solution handles all the information you need?- the solution reduces the operator’s work load?- the solution decreases your response time in emergency management?- the solution decreases the loss of information?- overall impression from 0 to 4? annotation: could be useful to verify with operator if C2-SENSE helps to share procedures or protocols among actors.


TO EVERY QUESTION WILL BE ASSIGNED A SCORE USING A PRE-DEFINED SCHEMA!

score meaning0 I totally disagree - or – very bad 1 I disagree – or - bad2 Neutral 3 I approve – or - good 4 I totally approve – or – very good



radar chart

REPORT


FUNCTIONAL GLOBAL EVALUATION


Objective Evaluation(the system work as

espected)

Subjective Evaluation(operator approve the behaviour of the solution)

Global Evaluation

WP 7.4 - NON Functional Evaluation

NON FUNCTIONAL EVALUATION

WP 7 - NON Functional Requirement

Non Functional Requirement:AccessibilityAudit and controlAvailability (see service level agreement)BackupCapacity, current and forecastCertificationComplianceConfiguration managementDependency on other partiesDeploymentDocumentationDisaster recoveryEfficiency (resource consumption for given load)Effectiveness (resulting performance in relation to effort)Emotional factors (like fun or absorbing or has "Wow! Factor")Environmental protectionEscrowExploitabilityExtensibility (adding features, and carry-forward of customizations at next major version upgrade)Failure managementFault tolerance (e.g. Operational System Monitoring, Measuring, and Management)Legal and licensing issues or patent-infringement-avoidabilityInteroperabilityMaintainability

ModifiabilityNetwork topologyOpen sourceOperabilityPerformance / response time (performance engineering)Platform compatibilityPricePrivacyPortabilityQuality (e.g. faults discovered, faults delivered, fault removal efficacy)Recovery / recoverability (e.g. mean time to recovery - MTTR)Reliability (e.g. mean time between failures - MTBF, or availability)ReportingResilienceResource constraints Response timeReusabilityRobustnessSafety or Factor of safetyScalability (horizontal, vertical)SecuritySoftware, tools, standards etc. CompatibilityStabilitySupportabilityTestabilityUsability by target user communityUser Friendliness


ISO 25000 series give a working framework on NON Functional

Requirement


Intrinsic Qualities

Functional SuitabilityPerformance Efficiency

CompatibilityUsability

ReliabilitySecurity

MaintainabilityPortability

Usage Qualities

Effectiveness

EfficiencySatisfaction

Freedom from Risk (Safety)Context Coverage (Usability Scope)

External Qualities

Service CostVendor Risk Mitigation

Product Risk Mitigation


Was defined a detailed list of NON funcional requirement

Most of them will be evaluated thinking at the POTENTIAL C2SENSE final product

deployed in production mode.


EXAMPLEIntrinsic quality – Reliability – Recoverability

Definition : The degree to which, in the event of an interruption or a failure, a product or system can recover the data directly affected and re-establish the desired state of the system

will be evaluated: Recovery Time Objective (RTO): It’s the time to recover full funcions after disarter. It

must tend to 0 in Emergency management Recovery Point Objective (RPO): Max time from data producing and data backup. It

must be near to 0 in Emergency management High availability architecture ready The evaluation will be performed on all C2-SENSE components deployable in production mode on a possible production architecture (not that usesd in pilot application)


Evaluation -> SCORE- 0 : NFR not covered by the solution at all - 1 : NFR covered minimally but is not sufficient and cannot be accepted in the real

world- 2: NFR covered but some important issue was detected (Eg: cost too high, bad

performance, etc.). The IA can accept the issue temporarily with the promise of a fix.

- 3 : NFR covered and it is coherent with the reference. Some issue was identified but a user can accept the solution

- 4 : NFR covered, it is coherent with the reference and no issue was identified. Annotation: in order to identify issue it’s important to mix : IA operator’s point of view Technical point of view State of the art of IT solutions



radar chart

REPORTIntrinsic Functional Suitability

Intrinsic Performance Efficiency

Intrinsic Compatibility

Intrinsic Usability

Intrinsic Reliability

Intrinsic Security

Intrinsic MaintainabilityIntrinsic Portability

Usage Effectiveness

Usage Efficiency

Usage Satisfaction

Usage Context Coverage

Usage Service Experience

External Service Cost

0

2

4

NFR Score

Issue

WP 7.4 - ISSUE

Pilot application need to be well defined technically

WP 7.4 - ISSUE

Pilot application seems to be to complex and less effective Scope Defined

test note4.1.2.1 Situation Reporting 35 too much4.1.2.2 Mission Plan 10 4.1.2.3 Scheduling 1 4.1.2.4 Resource Management 14 4.1.2.5 Alert 13 4.1.2.6 Hospital Communication 2 4.1.2.7 Tracking of Citizens 5 4.1.2.8 Sensor Management 3 4.1.2.9 EUAA 0 correct- too technical4.1.2.10 Audit Trail and Node Authentication (ATNA) 0 correct- too technical4.1.2.11 Emergency Situation Map 5 4.1.2.12 Situation Analysis 7 4.1.2.13 Permission 0 configuration 2 correct- too technical

WP 7.4 - ISSUE

NON FUNCTIONAL REQUIREMENT MUST TO BE EVALUATED BY OTHER PARTNER

WP 7.4 - ISSUE

Thank you for your attention

WP 7 - Regola TASK 7.4 : Pilot Application Validation D 7.4 Functional and Non-functional Evaluation...

Documents

Transcript of WP 7 - Regola TASK 7.4 : Pilot Application Validation D 7.4 Functional and Non-functional Evaluation...