AUTOMATISIERUNG IM IOT-TESTLAB - ASQF · 2020-01-22 · Akzent 1 Akzent 2 Akzent 3 Akzent 4 Akzent...
40
Michael Wagner, Sascha Kretzschmann, Axel Rennoch ASQF Quality Day, Berlin, November 29, 2017 AUTOMATISIERUNG IM IOT-TESTLAB
Transcript of AUTOMATISIERUNG IM IOT-TESTLAB - ASQF · 2020-01-22 · Akzent 1 Akzent 2 Akzent 3 Akzent 4 Akzent...
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
Michael Wagner, Sascha Kretzschmann, Axel Rennoch
ASQF Quality Day, Berlin, November 29, 2017
AUTOMATISIERUNG IM IOT-TESTLAB
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
2
BERLIN CENTER FOR DIGITAL TRANSFORMATION
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
3
• IoT test objects, goals, and configuration
• IoT test automation
– IoT test language TTCN-3
– IoT-Testware project
• IoT-T testlab
• IoT-Quality Engineering outlook
AGENDA
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
Where are we?
INTRODUCTION
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
5
• Mirai botnet, October 2016:
botnet using insecure configured IoT-devices
attack causes blackout and disruption
(e.g. Amazon, Netflix, Twitter, Github)
• Attack to Deutsche Telecom Router, November 2016:
1.200.000 router briefly stopped
vulnerable router subject to be integrated to Mirai botnet
• Wannacry, cyber attack on steel mill in Germany, …
• Recently WLAN WPA2 (KRACK: Key Reinstallation Attack), …
MOTIVATION FOR QUALITY
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
6
TRENDS IN IOT
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
7
Wide portfolio of competences required
Devices (sensors, HW, embedded SW)
Platforms (Cloud, platform domain knowledge)
Applications (SW, dashboard, business logic)
CHALLENGES IN IOT
IoT platforms 360+ worldwide
IoT protocols Rich selection
IP-based
non-IP based
Cellular netw.
Cellular: 4G, NB-IoT, Cat-M1,EC-GSM…[5G]; non-cellular: Wifi, LoRa, Sigfox ,Zigbee, BLE, … connectivity
MQTT MQTT-SN CoAP AMQP3 OPC UA
XMPP HTTP1 Web
Socket2 1|2|3
application IoT application logic
TCP UDP
SMS
DTLS DTLS
IPv4/IPv6(6loWPAN)
transport TLS/SSL
IoT services layer services …
Connectivity options
Throughput
Latency
Power efficiency
Packet size
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
8
TEST COVERAGE BY SDO-S
Cellular netw.
Cellular: 4G, NB-IoT, Cat-M1,EC-GSM…[5G]; non-cellular: Wifi, LoRa, Sigfox ,Zigbee, BLE, …
MQTT MQTT-SN CoAP AMQP3 OPC UA
XMPP HTTP1 Web
Socket2 1|2|3
IoT application logic
TCP UDP
SMS
DTLS DTLS
IPv4/IPv6(6loWPAN)
TLS/SSL
IoT services layer … C
I
P I
C
C C
C Conforman- ce tests
I Interopera- bility tests
P PlugFest/In- teroperability test event
P
C Certification
C
http://www.open
mobilealliance.or
g
T
T Compliance tool
C C
I
C
C
C
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
9
© Fraunhofer FOKUS
QUO VADIS IOT?
Cathedral
OR
Bazaar?
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
10
• IoT devices,
Mikrocontroller (MCU),
Gateways (Bosch XDK, IoT starterkits)
• IoT platforms
RIOT, relayr, Thread, mbed…
service layer (oneM2M, FiWare)
• IoT protocols
Constrained Application Protocol (CoAP)
MQ Telemetry Transport (MQTT)
IoT challenges: complexity, asynchronism, long operation phase, resource constraints
STARTING: TEST OBJECTS
LPWAN
LoRa
oneM2M
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
11
INTEGRATION OF SEVERAL TESTING APPROACHES
IoT Testing
Software Testing
System Testing
Security Testing
Test Automation
Protocol Testing
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
12
• Less resources needed (time and money)
• Avoid human mistakes due to manually testing
• During test development and execution
• Speed-up of regression tests and product time-to-market
TEST AUTOMATION
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
13
MULTIPLE TEST CONFIGURATION (SAMPLES)
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
14
• Toolset (selection of available means)
Protocol tester/monitor (Eclipse Titan, Wireshark)
Test devices (RFID kit, Bluetooth test device)
GUI tester (Selenium, SikuliX, Chrome headless)
Web services tester (soapUI)
…
• Public Testsuites (in preparation)
Application of a standardized notation
Abstract and platform-independent
TESTWARE
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
What do we use?
IOT TEST LANGUAGE
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
16
• TTCN-3 is the Testing and Test Control Notation
• Internationally standardized testing language for formally defining test
scenarios.
• Designed purely for testing
CHALLENGE TEST AUTOMATION
testcase Hello_Bob () {
p.send(“How do you do?“);
alt {
[]p.receive(“Fine!“);
{setverdict( pass )};
[else]
{setverdict( inconc )} //Bob asleep!
} }
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
17
• Distributed testing
• Automatic execution (TTCN-3 -> Java/C++) and logging
• Import and use of external data types (ASN.1, IDL, XML, JSON)
TESTING AND TEST CONTROL NOTATION (TTCN-3)
testcase Hello_Bob () { p.send(“How do you do?“); alt { []p.receive(“Fine!“); {setverdict( pass )}; [else] {setverdict( inconc )} //Bob asleep! } }
SUT
System
Tester
TTCN-3
API / Network
API / Communication
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
18
• One test technology for different tests
Distributed, platform-independent testing
Integrated graphical test development, documentation and analysis
Adaptable, open test environment
• Areas of Testing
Conformance and functional testing
Interoperability and integration testing
Real-time, performance, load and stress testing
Security testing
Regression testing
• Used for system and product qualification and certification (e.g. handset certification)
DESIGN PRINCIPLES OF TTCN-3
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
19
TTCN-3 HISTORY
TTCN (1992)
• published as ISO standard
• Tree and Tabular Combined Notation
• used for protocol tests: GSM, N-ISDN, B-ISDN
TTCN-2/2++ (1997)
• enhancements by ETSI MTS
• module concept, concurrency
• used for conformance tests
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
20
TTCN-3 HISTORY (CONT.)
TTCN-3 (2000)
• Further development by ETSI MTS
• Testing and Test Control Notation
• Standardised test specifications:
SIP, SCTP, M3UA, IPv6
HiperLan, HiperAccess, WiMAX
3GPP LTE, 3GPP NB-IoT
OMA
TETRA
MOST, AUTOSAR
EUROCONTROL
oneM2M
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
21
IMPLEMENTATION
Test Specification
TE
Systems
Under
Test
+ TTCN-3
Test
Engine
Test System
Systems
Under
Test
Systems
Under
Test
Systems
Under
Test
Communication /
Invocation
TCI Adaptors
TRI Adaptors
Automated Test
Execution and Reporting
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
22
• Test technology address various protocols and interfaces
• Control of real and/or virtual devices
(special hardware-interfaces, simulators)
TTCN-3 FOR IOT
Test cases: loudness, playCDTitle
MOST Bus
Head Unit
Tester
Speaker
Amplifier / Tuner
CD Changer
(SUT)
(SUT) Tester
PLC (SUT)
Elevator
Gateway (SUT)
MQTT
Tester
TTCN-3
Earlier days Today
Test cases: remote control
Tester
TTCN-3
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
What else?
CONTRIBUTION TO IOT TESTING
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
24
THE CONTEXT
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
25
• Supplement to running Eclipse projects
Paho, OM2M, Titan
• New project at Eclipse Foundation:
https://projects.eclipse.org/projects/technology.iottestware
TTCN-3 test suites for CoAP, MQTT, OPC-UA, LoRa?
• Assured licenses for users
• Currently in cooperation with
relayr GmbH, Ericsson, LAAS/CNRS, itemis AG, Spirent Communications,
Easy Global Market, Iskratel/Sintesio, …
THE ECLIPSE PROJECT
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
26
IOT-TESTWARE
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
27
SAMPLE TESTSUITE STRUCTURE: MQTT
Broker as SUT
All mandatory message data fields
Regular and illegal data
(Fixed/variable header, payload)
Protocol features
General
Connect/disconnect (session)
Subscribe/unsubscribe
Immediate publish
Last will and Testament (LWT)
Heartbeats keepAlive values
Topic
Error handling
Client as SUT
…
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
28
TEST DEVELOPMENT SAMPLE: MQTT
TESTZIEL-KATALOG
Test configurations
Test Suite Structure
Test purpose (catalogue)
Test implementation (TTCN-3)
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
29
TTCN-3 REPOSITORY
https://github.com/eclipse/iottestware.git
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
30
MQTT BROKER EVALUATION (JULY 2017)
Broker Version PASS FAIL INCONCLUSIVE
# % # % # %
Mosquitto 1.4.14 40 88,89% 3 6,67% 2 4,44%
VerneMQ 1.1.0 39 86,67% 3 6,67% 3 6,67%
HiveMQ broker.hivemq.org 39 86,67% 4 8,89% 2 4,44%
EMQ 2.0 36 80,00% 7 15,56% 2 4,44%
lannister ? 31 68,89% 12 26,67% 2 4,44%
ActiveMQ 5.14.5 31 68,89% 12 26,67% 2 4,44%
RSMB ? 26 57,78% 17 37,78% 2 4,44%
RabbitMQ 3.5.7 21 46,67% 24 53,33% 0 0,00%
Mosca 2.5.1 19 42,22% 24 53,33% 2 4,44%
Moquette 0.10 16 35,56% 29 64,44% 0 0,00%
HBMQTT 0.9 15 33,33% 30 66,67% 0 0,00%
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
31
THE TEST EXECUTION TOOL
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
32
ECLIPSE TITAN PROJECT
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
33
• Vulnerability scanner:
in particular for web applications, zero-day/fuzzing, consideration of data
bases, traffic/network analyser, program code scanner
• Penetration tester, e.g. “SQL injection”
• Intrusion detection tools
• Load test/Scalability
• Further utilities: Model-based testing (UML testing profile) and risk modelling
TESTWARE: SECURITY
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
34
• Focus on open source test tools (Eclipse)
• Support of test suites configuration
• Providing several end devices
• Remote test service (online)
Future certification
„light weight“ selection of criteria
„self certification“ if tests are successful
consideration of operational phase
use of standardized test purpose catalogs (in preparation at ETSI)
TESTLAB (TESTING AND CERTIFICATION)
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
35
PROJECT
WWW.IOT-T.DE
standardization
methods
customer
(requirements and
applications)
test and certification
pilots
new requirements
test and evaluation IoT-Testware
All project partners
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
What comes next?
SUMMARY AND OUTLOOK
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
37
Advanced + mature testing technology:
(open source) community:
Standardized basis (for certification):
APPROACH
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
38
New Working Group (TST) will develop IoT test catalogues and specifications (not covered elsewhere).
The types of testing include conformance, interoperability, security and performance testing.
The initial technical focus will be:
IoT network layer
(communication protocols, node connectivity, edge computing etc.),
IoT layer (data accumulation and aggregation),
Application layer (interfaces, business processes etc.).
ETSI TC MTS
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
39
Advanced testing technology:
(Open source) IoT-Testware (code):
Standardized IoT test purposes:
Certified Professionals for IoT
APPROACH
Akzent 1
Akzent 2
Akzent 3
Akzent 4
Akzent 5
Akzent 6
Hinter-
grund /
Text 2
41
Axel Rennoch, [email protected], phone +49 30 3463-7344
CONTACTS
Thank you for your attention!
https://www.fokus.fraunhofer.de/sqc
