.lusoftware verification & validationVVS
Automatic Generation of System Test Cases from Use Case Specifications
Chunhui Wang, Fabrizio Pastore, Arda Goknil, Lionel Briand, Zohaib Iqbal
University of Luxembourg, !Luxembourg!
{chunhui.wang,fabrizio.pastore,arda.goknil,lionel.briand, muhammad.zohaib}@uni.lu
ISSTA’15, Baltimore, July 17th, 2015
Problem
• Verifying the compliance of a system with its requirements in a cost-effective way
• Traceability between requirements and system test cases – This cannot be manual
• E.g., automotive industry which must comply to ISO 26262
2
Objectives
• Automatically generate test cases from requirements
• Capture and create traceability information between test cases and requirements
3
Context Matters
• Requirements are captured through use cases
• Use cases are used to communicate with customers and the system test team
• Complete and precise behavioral models are not an option: too expensive
4
Strategy
• Analyzable use case specifications
• Automatically extract test model from the use case specifications
• Minimize modeling
• No behavioral modeling
5
Elicit Use Cases 1
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
RUCM Use Cases
7
RUCM Use Case Example
8
Use Case Name: Identify Occupancy Status Actors: AirbagControlUnit Precondition: The system has been initialized . . .
"Basic Flow 1. The seat SENDS occupancy status TO the system. 2. INCLUDE USE CASE Classify occupancy status. 3. The system VALIDATES THAT the occupant class for airbag control is valid. 4. The system SENDS the occupant class for airbag control TO AirbagControlUnit.
Specific Alternative Flow RFS 3 1. IF the occupant class for airbag control is not valid THEN 2. The system SENDS the previous occupant class for airbag control TO … 4. ENDIF
Postcondition: The occupant class for airbag control has been sent.
Postcondition: The previous occupant class for airbag control has been sent.
[Yue et al. TOSEM’13]
Elicit Use Cases 1
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
RUCM Use Cases
Model the Domain 2
Domain Model
9
1
System
AirbagControl Sensor
OccupantStatus
- OccupantClassForAirbagControl - OccupantClassForSeatBeltReminder
1
1 1
1 1
1
1..*
Elicit Use Cases 1
Model the Domain 2
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
RUCM Use Cases
Generate Scenarios and Inputs
6
ERRORS ARE ABSENT
TEMPERATURE IS LOW
STATUS IS VALID
Identify Constraints 4
Constraint descriptions
Evaluate Consistency
3
Domain Model
Based on Natural Language Processing
10
11
Basic Flow
1. The seat SENDS occupancy status TO the system.
2. INCLUDE USE CASE Classify occupancy status.
3. The system VALIDATES THAT
the occupant class for airbag control is valid and
the occupant class for seat belt reminder is valid.
4. The system SENDS the occupant class for airbag control TO AirbagControlUnit.
5. The system SENDS the occupant class for seat belt reminder TO SeatBeltControlUnit.
6. The System Waits for next execution cycle.
Postcondition: The occupant class for airbag control and the occupant class for seat belt reminder have been sent.
INPUT STEP
INCLUDE STEP
CONDITIONAL STEP
OUTPUT STEP
OUTPUT STEP
INTERNAL STEP
POSTCONDITION
DOMAIN ENTITY
CONSTRAINT
CONSTRAINT
DOMAIN ENTITY
DOMAIN ENTITY
ERRORS ARE ABSENT
TEMPERATURE IS LOW
STATUS IS VALID
Identify Constraints 4
Generate Scenarios and Inputs
6
Elicit Use Cases 1
Constraint descriptions
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
Model the Domain 2
12
RUCM Use Cases
Evaluate Consistency
3
Domain Model
Missing Entities
Evaluate Model Consistency
Occupant Class for Airbag Control
Occupant Class for Seat Belt Reminder
Domain Entities
Airbag Control Classification Filter Sensor
Tagged Use Case
Evaluate Model Consistency
14
Occupant Class for Airbag Control
Occupant Class for Seat Belt Reminder
Domain Entities
AirbagControl
System
Sensor
OccupantStatus
- OccupantClassForAirbagControl - OccupantClassForSeatBeltReminder
Airbag Control Classification Filter Sensor
ClassificationFilter
Tagged Use Case
1 1
1 1
1 1
1
1..*
1 1
Generate Scenarios and Inputs
6
Elicit Use Cases 1
Missing Entities
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
Model the Domain 2
Evaluate Consistency
3
Domain Model
15
ERRORS ARE ABSENT
TEMPERATURE IS LOW
STATUS IS VALID
Identify Constraints 4
Constraint descriptions
RUCM Use Cases
Errors.size() == 0 Status != null
t > 0 && t < 50
Specify Constraints 5
OCL constraints
Identify Constraints
16
The occupant class for airbag control is valid
Voltage error is detected
List of Constraints
The occupant class for seat belt reminder is valid
Tagged Use Case
Voltage error is detected
List of Constraint Descriptions
The occupant class for seat belt reminder is valid
Tagged Use Case
Engineers Specify Constraints
System.allInstances() ->forAll(b | b.occupancyStatus.classForAirbag <> Status::Error)
Software Engineers specify corresponding OCL constraints
The occupant class for airbag control is valid
17
Errors.size() == 0 Status != null
t > 0 && t < 50
Elicit Use Cases 1
Missing Entities
Specify Constraints 5
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
Model the Domain 2
Evaluate Consistency
3
Domain Model
ERRORS ARE ABSENT
TEMPERATURE IS LOW
STATUS IS VALID
Identify Constraints 4
Constraint descriptions
18
RUCM Use Cases
Generate Scenarios and Inputs
6
Object Diagrams
Test Scenarios
OCL constraints
19
Basic Flow
1. The seat SENDS occupancy status TO the system.
2. INCLUDE USE CASE Classify occupancy status.
3. The system VALIDATES THAT
the occupant class for airbag control is valid and
the occupant class for seat belt reminder is valid.
4. The system SENDS the occupant class for airbag control TO AirbagControlUnit.
5. The system SENDS the occupant class for seat belt reminder TO SeatBeltControlUnit.
6. The System Waits for next execution cycle.
Postcondition: The occupant class for airbag control and the occupant class for seat belt reminder have been sent.
INPUT STEP
INCLUDE STEP
CONDITIONAL STEP
OUTPUT STEP
OUTPUT STEP
INTERNAL STEP
POSTCONDITION
DOMAIN ENTITY
CONSTRAINT
CONSTRAINT
DOMAIN ENTITY
DOMAIN ENTITY
INCLUDE STEP
CONDITIONAL STEP
OUTPUT STEP
OUTPUT STEP
INTERNAL STEP
POSTCONDITION 20
Basic Flow
1. The seat SENDS occupancy status TO the system.
2. INCLUDE USE CASE Classify occupancy status.
3. The system VALIDATES THAT
the occupant class for airbag control is valid and
the occupant class for seat belt reminder is valid.
4. The system SENDS the occupant class for airbag control TO AirbagControlUnit.
5. The system SENDS the occupant class for seat belt reminder TO SeatBeltControlUnit.
6. The System Waits for next execution cycle.
Postcondition: The occupant class for airbag control and the occupant class for seat belt reminder have been sent.
UseCaseStart
INPUT STEP DOMAIN ENTITY
CONSTRAINT
CONSTRAINT
DOMAIN ENTITY
DOMAIN ENTITY
Precondition
CONDITIONAL STEP
OUTPUT STEP
OUTPUT STEP
INTERNAL STEP
POSTCONDITION 21
Basic Flow
1. The seat SENDS occupancy status TO the system.
2. INCLUDE USE CASE Classify occupancy status.
3. The system VALIDATES THAT
the occupant class for airbag control is valid and
the occupant class for seat belt reminder is valid.
4. The system SENDS the occupant class for airbag control TO AirbagControlUnit.
5. The system SENDS the occupant class for seat belt reminder TO SeatBeltControlUnit.
6. The System Waits for next execution cycle.
Postcondition: The occupant class for airbag control and the occupant class for seat belt reminder have been sent.
UseCaseStart
Input
INCLUDE STEP
DOMAIN ENTITY
CONSTRAINT
CONSTRAINT
DOMAIN ENTITY
DOMAIN ENTITY
Precondition
OccupancyStatus DomainEntity
DOMAIN ENTITY
22
UseCaseStart
Input
Include
Condition
Output
Exit
Condition
Exit Output
Exit
OccupancyStatus DomainEntity
System.initialized = true
OCL Constraint
AirbagControlUnit.AllI…
OCL Constraint
… OCL Constraint
… DomainEntity
OCL Constraint OccupancyStatus.ClassForSeat <> Error AND OccupancyStatus.ClassForAirbag <> Error
23
UseCaseStart
Input
Include
Condition
Output
Exit
Condition
Exit Output
Exit
System.initialized = true AND OccupancyStatus.ClassForSeat <> Error AND OccupancyStatus.ClassForAirbag <> Error
Path condition:
OCL Constraint OccupancyStatus.ClassForSeat <> Error AND OccupancyStatus.ClassForAirbag <> Error OccupancyStatus
DomainEntity
System.initialized = true
OCL Constraint
AirbagControlUnit.AllI…
OCL Constraint
OCL Path Condition: System.allInstances() -> forAll(b | b.initialized = true ) AND OccupancyStatus.allInstances()
-> forAll(o| o.ClassForSeat <> Status:: Error) AND OccupancyStatus.allInstances()
-> forAll(o| o.ClassForAirbag <> Status::Error)
24
System.initialized = true AND OccupancyStatus.ClassForSeat <> Error AND OccupancyStatus.ClassForAirbag <> Error
Path condition:
OCL Solver
:System Initialized = true Version = 3
:OccupancyStatus ClassForSeat = Adult ClassForAirbag = Adult
Input Values OccupancyStatus.ClassForAirbag = Adult OccupancyStatus.ClassForSeat = Adult System.initialized = true
Object Diagram
1
System
AirbagControl Sensor
OccupantStatus - ClassForAirbag - ClassForSeat 1
1 1
1 1
1
1..*
Domain Model:
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
ERRORS ARE ABSENT
TEMPERATURE IS LOW
STATUS IS VALID
Identify Constraints 4
Constraint descriptions
Errors.size() == 0 Status != null
t > 0 && t < 50
Generate Scenarios and Inputs
6
Elicit Use Cases 1
Missing Entities
Specify Constraints 5
OCL constraints
Model the Domain 2
Evaluate Consistency
3
Domain Model RUCM Use Cases
Generate Test Cases
7
Test Cases Object
Diagrams Test
Scenarios 25
26
UseCaseStart
Input
Include
Condition
Output
Exit
Test Case
occupancyStatus.ClassForAirbag = Adult
OccupancyStatus DomainEntity
Scenario Under Test
System.initialized = true
<INPUT>
<CHECK>
<INPUT>
AirbagControlUnit.AllInstances() -> forAll(a|a.status = a.bodySense.occupancyStatus)
System.initialized = true
OCL Constraint
Input Values occupancyStatus.ClassForAirbag = Adult occupancyStatus.ClassForSeat = Adult System.initialized = true
occupancyStatus.ClassForSeat = Adult AirbagControlUnit.AllI…
OCL Constraint
Abstract Test Case
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
ERRORS ARE ABSENT
TEMPERATURE IS LOW
STATUS IS VALID
Identify Constraints 4
Constraint descriptions
Errors.size() == 0 Status != null
t > 0 && t < 50
Generate Scenarios and Inputs
6
Elicit Use Cases 1
Missing Entities
Specify Constraints 5
OCL constraints
Model the Domain 2
Evaluate Consistency
3
Domain Model RUCM Use Cases
Generate Test Cases
7
Test Cases Object
Diagrams Test
Scenarios 27
Mapping Table
Generation of Executable Test Cases
28
ResetPower (Time = INIT_TIME)
SetBus (Chanel = RELAY_CLASS, Status = CLASS1)
ReadAndCheckBus ( D0 = OCCUPIED, D1 = OCCUPIED)
Abstract test steps Resulting Test Driver Functions
Matching Patterns Function Name Parameters
<INPUT> Classfor* = Adult SetBus
<INPUT> System.initialized = true ResetPower
<CHECK> AirBagControlUnit.allInstant… ReadAndCheckBus
Chanel = RELAY_CLASS Status = CLASS1 Time = INIT_TIME
Mapping Table:
occupancyStatus.ClassForAirbag = Adult System.initialized = true
<INPUT>
<CHECK>
<INPUT>
AirbagControlUnit.AllInstances() -> forAll(a|a.status = a.bodySense.occupancyStatus)
occupancyStatus.ClassForSeat = Adult
Abstract Test Case:
D0=OCCUPIED …
Executable Test Case:
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND
ERRORS ARE ABSENT
TEMPERATURE IS LOW
STATUS IS VALID
Identify Constraints 4
Constraint descriptions
Errors.size() == 0 Status != null
t > 0 && t < 50
Generate Scenarios and Inputs
6
Elicit Use Cases 1
Missing Entities
Specify Constraints 5
OCL constraints
Model the Domain 2
Evaluate Consistency
3
Domain Model RUCM Use Cases
Generate Test Cases
7
Test Cases Object
Diagrams Test
Scenarios 29
Mapping Table
Case Study • BodySense, embedded system for detecting occupancy
status in a car
• Evaluation:
• Cost of additional modelling
• Effectiveness in terms of covered scenarios compared to current practice at IEE
30
Complexity of the Case Study
31
Use Case Steps Use Case Flows
UC1 50 8 UC2 44 13 UC3 35 8 UC4 59 11 UC5 30 8 UC6 25 6
Costs of Additional Modeling
32
Use Case Steps Use Case Flows
OCL Constraints
UC1 50 8 9 UC2 44 13 7 UC3 35 8 8 UC4 59 11 12 UC5 30 8 5 UC6 25 6 12
5 to 10 minutes to write each constraints
Effectiveness: scenarios covered
33
0
5
10
15
20
25
30
35
40
UC1 UC2 UC3 UC4 UC5 UC6 Scenarios Covered By Engineer Scenarios Covered By UMTG
100%
100%
100%
100% 100% 100%
81%
77% 100%
86%
50% 67%
It is hard for engineers to capture all the possible scenarios involving error conditions.
Problem
• Requirements-driven test automation
• Traceability between requirements and system test cases – This cannot be manual
• E.g., automotive industry which must comply to ISO 26262
34
ERRORS ARE ABSENT
TEMPERATURE IS LOW STATUS IS VALID
Errors.size() == 0 Status �= null ��> 0 && t < 50
Identify Constraints 4
Generate Scenarios and Inputs
6 Generate Test Cases
7
Evaluate Consistency
3
Elicit Use Cases 1
Model the Domain 2
Specify Constraints 5
Domain Model
Missing Entities
Constraint descriptions OCL constraints
THE ACTOR SEND �HE SYSTEM VALI �HE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND �HE SYSTEM VALI �HE SYSTEM DIS THE ACTOR SEND
THE ACTOR SEND �HE SYSTEM VALI �HE SYSTEM DIS THE ACTOR SEND
RUCM Use Cases
���� ���� ����
Test Cases Object
Diagrams Test
Scenarios 0
5
10
15
20
25
30
35
40
UC1 UC2 UC3 UC4 UC5 UC6 Scenarios Covered By Engineer Scenarios Covered By UMTG
100%
100%
100%
100% 100% 100%
81%
77% 100%
86%
50% 67%
Acceptable modelling effort Full scenario coverage
35
NLP Pipeline
Tokenizer UMTG Gazetteer
Part of Speech Tagger
UMTG Transducers
The seat SENDS occupancy status TO the system.
36
NLP Pipeline
Tokenizer UMTG Gazetteer
Part of Speech Tagger
UMTG Transducers
The seat SENDS occupancy status TO the system.
ACTOR SEND NOUN PHRASE SYSTEM TO
Top Related