NBAP message construction using QuickCheck - Andreas Granberg and Daniel Jernberg LITH-IDA...

8/4/2019 NBAP message construction using QuickCheck - Andreas Granberg and Daniel Jernberg LITH-IDA 2007-06-18

1/74

NBAP message construction

using QuickCheck

Andrea s Granberg and

Daniel Jernberg

Master Thesis Projec t,

Kista , 2006, 2007


2/74

NBAP message construction

using QuickCheck

Andrea s Granberg and

Daniel Jernberg

LITH-IDA

2007-06-18

Master Thesis Projec t,

Supervised by Graham Crowe.

Examined by Mariam KamkarKista , 2006, 2007

This thesis presents the work do ne by the twounde rgrad uate students Daniel Jernberg andAnd rea s Granberg a t Eric sson Tra ffic and Fea tureSW. The thesis projec t wa s to integ ra te Quick-Chec k as an autom ate d tool for softwa re verific a-tion into the existing test framewo rk. This thesispresents the purpose, method s, choices ma deand the results of this wo rk.


3/74

i Abstract

ABSTRACT

Traffic and Feature SW is a dep artm ent based in Kista. At this dep artm ent the m ainprocessor software, or MPSW in short, in Ericssons Radio Base Stations is tested p riorto integr ation o f new releases. Traffic and Feature SW, also called MPSW in this th esis,works closely with ano ther d epar tment of Ericsson called RBS System I&V which teststhe same software but for complete RBS nodes. MPSW uses black- and greyboxscripted t esting in regression suites tha t are executed on a daily basis. These regressionsuites are separated into different subsets of functionality that maps to the capabilitiesof the Radio Base Station software. The authors of this thesis has performed an imple-men tation of automa ted test cases for a subset of the Radio Base Station software usingan au tomated software tool called Q uickCheck. These test cases were su ccessfully inte-grated into the test framework and were able to find several issues with the main proc-essor software and its subsystems in the Radio Base Station. The commissioner of this

thesis have plans to further integr ate the QuickCheck tool into the test fram ewor k, pos-sibly autom ating test cases for several subsets of the Radio Base Station software. Theauthor s have therefore analysed and pu t forth metrics that compares the testing of theRadio Base Station software using QuickCheck with the conventional regression testcases. These metrics covers areas such as the cost related to and the inherent capabili-ties of QuickCheck. The evaluation of these metrics was performed by the au thors togive the commissioner decisive information. These evaluations showed that Quick-Check was able to generate comp lex message stuctures in complex sequences. Quick-Check was also able to shrink both the content of these messages and the length of thesequences of messages to be able to provide a m inimal coun terexamp le when a faultwas discovered. The only metric that QuickCheck failed to suppor t was to inherit func-tionality to support the handling of statistics from executions.


4/74

Sammanfattning ii

SAMMANFATTNING

Traffic and Featu re SW r en avdelnin g av Ericsson som r baserad i Kista. P dennaavdelning testas mjukvaran p hu vud processorn och d ess subsystem i Ericssons radio-basstationer innan nya releaser av mjukvaror fr dessa. Avdelningens arbete koordine-ras med en annan avdelning i Kista som kallas RBS System I&V som jobbar medliknande testning men mot riktiga radiobasstationer. Traffic and Feature SW, ven kal-lad MPSW i denna rapp ort, jobbar m ed regressionstestning i en blandning av svartl-demilj och grldemilj. Dessa regressionssviter krs varje natt mot ny mjukvara iradiobasstationerna frn vilka resultat analyseras av utvecklarna av testfallen dagenefter. Dessa regressionssviter r ind elade i mind re sviter baserat p vilken delmngd avfunktionaliteten i radiobasstationsmjukvaran som testas. Frfattarna av denna exa-mensarbetersrapport har under detta arbete genomfrt en implementation av automa-tiserade testfall fr en delmngd av funktionaliteten i radiobasstationen genom att

anvnd a ett nytt verktyg fr autom atiserad testning vid nam n Qu ickCheck. Dessa test-fall integrerades in i testramverket p ett nskvrt stt och lyckades hitta flertaletintressanta frgetecken med mjukvaran i hu vud processorn och dess delsystem i rad io-basstationerna. Eftersom uppdragsgivaren har fortsatta planer p att anvnda Quick-Check som testverktyg i sitt testramverk i strre u tstrckning har frfattarna venanalyserat och postulerat metriker fr att ge mjlighet att jmfra testfall av denna typmed testfall av den konventionella regressionstestningstypen. Dessa metriker tckeromrden ssom kostnader och duglighet hos testverktyget. En utvrdering av dessametriker genomfrdes ocks av frfattarna fr att ge upp dragsgivaren beslutsstd-

jand e information om utfallet av dessa m etriker fr QuickCheck. Dessa evalueringarvisade att QuickCheck ger mjligheten att generera komplexa meddelandestrukturersvl som kom plexa sekvenser av sdana m edd elanden. QuickCheck har ocks funk-tionalitet fr att krymp a sdana med deland en och sekvenser till minimala m otexemp el

nr ett fel upp tckts. Den enda m etriken som Qu ickCheck ej lyckades u pp fylla var attge anvndaren funktionalitet vad gller mjligheten att kunna hantera statistik frnkrningar.


5/74

iii Acknowledgements

ACKNOWLEDGEMENTS

We would like to show our appreciation to the following people that made this the-sis work p ossible and that hav e helped u s in so man y ways.

We w ould like to thank the executives, Erik Backlun d, Mike Williams an d RogerHolmberg, at the departm ent for giving us the opp ortun ity to perform this thesis work.

We wou ld also like to thank our comp any sup ervisor Graham Crowe for his exper-tise in the area as well as his effort to give us all the valuable information n eeded to beable to perform the thesis work.

We would also like to show our appreciation to our examiner, Mariam Kamkar, forher supp ort and p rofessional approach.

We would also like to thank John Hughes and Thomas Arts for their lectures onQuickCheck, their interest in our wor k and th e great sup port they hav e provided for usdu ring the thesis work.

We wou ld also like to thank all the emp loyees at the d epartm ent. Their accomm o-dating m anners and professionalism as w ell as their interesting and funny discussionshave helped u s maintain a healthy app roach to the work being performed.

Finally we would like to show our appreciation towards our friends and familiesfor them putting up with us d uring these intense months.

Andreas Granberg, Kista, 2007Daniel Jernberg, Kista, 2007


6/74

Terms and definitions iv

TERMS AND DEFINITION S

3GPP, 3:rd generation partnership program: A collaboration agreement that bringstogether a num ber of telecomm unications standards bodies to provide standardsand specifications.

ASN1, Abstract syntax notation one: A standard notation in specifications thatdescribes data structures for representation of encoding, decoding an d transmittingdata.

ATM , Asynchronous Transfer Mode: The data link layer protocol describing end-to-end logical connections where fixed cells are transferred.

BP, Board Processor: The processor on each interface board in the RBS responsible forhand ling the board signalling.

Bp, The Bp interface: The interface on which the Main processor communicates withthe BP on th e interface board s.

CCM, Cell configuration Management: The comm on and ded icated p rocedu res speci-fied into a fun ctionality suite by 3GPP for Cell Configuration Man agement.

CDMA, Code division multiple access: The method of transmission in the WCDMAnetwork u sing shared channels.

CR, Change Request: The Change Requests w ritten in respon se to issues discoveredwith the Radio Base Station software or the test framework .

CBR, Capability Report: A report extracted from the m odel to reflect a capability of

the SUT. CRN C, Controlling Radio Network Controller: The mode of the Radio Network Con-

troller in wh ich it is responsible for maintain ing control of connections.

CTCM, Common Transport Channel M anagement: The common and dedicated proce-dures specified into a functionality suite by 3GPP for common transport channelmanagement.

EP, Elementary Procedure: The basic NBAP procedu res for calls against the SUT.

EUL, Enhanced Uplink: The new feature of WCDMA that provides an enhancedup link for faster end user p erformance.

HSDPA, High Speed Data Packet A ccess: The new stand ardized evolution of WCDMAthat w ill enable downlink speeds of up to 14 Mbps.

IE, In formation Element: The most elementary information elements in an NBAP EPmessage.

IFHO, In ter Frequency Handover: A handover of a UE communication between differ-ent RAN frequencies.

Iub, The Iub interface: The interface defined by the NBAP pro tocol wh ich conn ects theNode B and the RNC logically.

MBMS, Mobile Broadcast/M ulticast Service: The RAN feature regarding broadcastingand mu lticasting to UEs.

MO, Managed Object: The Managed Objects in the SUT used to reflect the internalstate of the SUT.

MP, Main Processor: The Main Processor in the RBS wh ose software we are verifying.


7/74

v Terms and definitions

MPSW, Main Processor Software: The software for the Main Processor in th e RBS,wh ich the imp lementation attempts to test.

NBAP, Node B Application Protocol: The protocol which the RNC adheres to in its

communication with the RBS. Node B: The 3GPP name for the base station in the WCDMA RAN architecture.

PDU, Elementary Procedure Definition: The defin ition in the N BAP specification of anEPs conten t.

RAN, Radio Access Network: The telecomm un ications netw ork architecture betweenthe hand set and the core network.

RBS, Radio Base Station: The Radio Base Station is the unit in the RAN on the RadioNetwork Layer that performs the main connection handling with the UE. The RadioBase Station is contro lled by th e RNC.

RLM, Radio Link Management: The 3GPP subset of functionality for h and ling radiolinks.

RN C, Radio Network Controller: The Rad io Network Controller is the unit in the RANresponsible for controlling the RBSs in th e RAN an d providing connection to th ecore network.

SRNC, Signalling Radio Network Controller: The mode of the Radio Networ k Control-ler in which it is responsible for the transmission between handsets and the corenetwork.

SUT, System under test.

TR, Trouble Report: The Trouble Reports written in response to found issues with the

Radio Base Station software or the test framework.

TSG, Technical Specification Group: The group s in 3GPP responsible for su ggesting,maintaining technical specifications for 3G and GSM networks.

UE, User Equipment: The user equipm ent in the RAN, usually a cell phone.

UTRAN, Universal Terrestrial Radio Access Network: The communications networkcommonly know n as 3G.

Uu, The Uu protocol: The protocol used by the RBS to tran smit and receive da ta froma User Equipm ent.


8/74


9/74

vii Table of Contents

5.7 MODULARISATION CHOICES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.8 SIGNAL VERIFICATION AND MODIFICATION . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.9 STATISTICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.10 GENERATION-TIME AN D RUN -TIME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.11 FINDING METRICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6 IMPLEMENTATION

6.1 INITIAL DEVELOPMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6.2 MODULARISATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6.3 GENERATORS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

6.4 GROUPING O F FUNCTIONALITY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

6.5 EVOLUTION OF STATE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

6.6 SYMBOLIC CALLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

6.7 RETURNING PROPER VALUES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

6.8 STATISTICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

6.9 SIGNAL VERIFICATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6.10 INTEGRATION INTO FRAMEWORK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6.11 ISSUES DISCOVERED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

7 METRICS

7.1 COST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

7.2 CAPABILITY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

8 EVALUATION

8.1 COST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

8.2 CAPABILITY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

9 CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

APPENDICIES

APPENDIX A LITERATURE LIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

APPENDIX B NBAP ASN1 SPECIFICATION EXAMPLES. . . . . . . . . . . . . . . 55

APPENDIX C EXAMPLE OF SHRINKING. . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

APPENDIX D INTEGRATION EXCERPTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

APPENDIX E ASN1 GENERATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60


10/74

List of Tables viii

LIST OF TABLES

TABLE 4-1: EXAMPLE DATABASE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

TABLE 8-1: ESTIMATES FOR MBMS IMPLEMENTATION . . . . . . . . . . . . .46

TABLE E-1: ASN1 GENERATORS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60


11/74

ix List of Figures

LIST OF FIGURES

FIGURE 2-1: THE WCDMA ARCHITECTURE. . . . . . . . . . . . . . . . . . . . . . . 4FIGURE 2-2: RNC ROLES IN THE ARCHITECTURE . . . . . . . . . . . . . . . . . . . 5

FIGURE 2-3: HAN DOVER BETWEEN CELLS. . . . . . . . . . . . . . . . . . . . . . . . . 6

FIGURE 2-4: SOFT AND SOFTER HANDOVER . . . . . . . . . . . . . . . . . . . . . . . 6

FIGURE 2-5: TEST SETUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

FIGURE 4-1: BASIC CONTROL FLOW OF THE STATE MACH INE . . . . . . . . . 21

FIGURE 6-1: CODE BASE EVOLUTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

FIGURE 6-2: MODULE USAGE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-


12/74

Outline x

OUTLINE

In the first chapter the reader will be introduced to the assignment that this thesis isbased u pon, the task as p resented by the comm issioner, the pu rpose, goals as well asthe limitations.

In the following chap ter the read er will be introd uced to th e basics concepts of Uni-versal Terrestrial Radio Access Netw ork for Wideban d Cod e Division Multiple Accessand especially the su bset of the Radio Base Station fun ctionality that the th esis work isbased on. This chapter will also describe the stand ardized interfaces in play and howthese have come to affect the thesis work.

In the third chapter the read er will be introduced to th e general concepts of testingthat are necessary to be able to comprehend the subsequent chapters that discusses andrelates the method ologies used in this thesis work to common testing m ethods.

In the fourth chapter a fairly detailed description of QuickCheck will follow in

wh ich the reader w ill be presented w ith enough information abou t the software tool tobe able to comp rehend the choices being m ade w hile imp lementing the test cases forthe thesis work.

The fifth chap ter contains an an alysis of the problems that w e were presented withdu ring the thesis work. We will in d etail describe the problems and will argue wh y thechosen solution is the p referred on e.

In the next chapter the reader will be able to follow our implementation of theissues described in the analysis chapter. The reader will also be able to understand theincremental growth of the code base for the implementation and the decisions beingmad e with regards to that implementation.

In the seventh chapter w e will describe the metrics pu t forth that are needed to beable to compare testing using QuickCheck with conventional regression testing. Wewill in th is chap ter argue w hether or not these m etrics are of value or not.

In the eight chapter we will follow up with an evaluation of these metrics and, inthe ninth and final chapter, the reader will be able to discover our conclusions aboutthe thesis work and the end result.


13/74

1 1. Introduction

1 INTRODUCTION

This chap ter will introdu ce the reader to th e thesis work assignm ent. What the pu r-pose of the assignment was, the main task as presented by the commissioner, the limi-tations, the goals as well as the deliverables w ill be described in d etail.

1.1 PURPOSEComissioner of this thesis work is Traffic and Feature SW, a division of Ericsson,

based in Kista, Sweden . This division of Ericsson is responsible for the d esign, systemi-sation, integra tion an d verification of th e software in Ericssons RBSs. This division isseparated into two sm aller depar tments, one in Kista and one in Ume wh ich w orks inparallel to systemise, test and verify the software for the RBS. These two departmentsalso works closely w ith an other d epartm ent called RBS System I&V which p erforms

similar testing but on actual machines and not in the simulated environment that theaforementioned department does.

As of today the testing of the MPSW at Traffic and Feature SW is performed usingscripted regression test case suites that are run during nights at the department. Thenature of the test environment w ill further be explained in a later section of the thesiscalled Test Setup on page 12. In this regression testing p rocess the d esign of test casesis based on use cases and functional specifications wh ich often covers test cases in aone-to-one fashion without mu ch code reuse wh ich m akes the scripted regression testsuites quite repetitive and h ard to m aintain.

By using QuickCheck as a tool to automatically generate the elementary proce-du res, or EPs in short, from the Node B application protocol, N BAP in short, andsequen ces of NBAP EPs it might be possible to better mod el the system under test and

be able to autom ate a greater d eal of the testing being don e today. By this, QuickCheckmight be able to provide a richer variety of parameterization of these procedures, intheory increasing the possibility of find ing faults that were not possible to find , pr ior tousing QuickCheck.

Traffic and Feature SW is therefore interested in a evaluation of QuickCheck toreceive more information about how effective and cost efficient an integration of thistool can be in th eir test environm ent.

1.2 TASKWe were given the assignment to implement test cases for a subset of the MPSW

functionality using Qu ickCheck and to integrate the resulting test cases into the test

framework. By doing so the thesis work would prepare for a possible deployment ofQuickCheck in the testing framework as w ell as provide best p ractice methods fordealing with issues that might occur d uring such a dep loyment.

As part of the task we w ere to identify metrics to supp ort the comparison betweentest cases implemented using QuickCheck and the scripted test cases used in theregression suites. The thesis would result in an evaluation of these metrics whichshould be able to provide the commissioner w ith enough information to be able tomake an informed decision about w hether or not to go forth with further integration ofQu ickCheck as a tool for testing several subsets of the RBS software.

1.3 LIMITATIONS

The thesis work included imp lement ing test cases using QuickCheck as a tool in thescope that was n ecessary to be able to provide a basis for the best practice docum entthat w as to be provid ed as a deliverable of the thesis work. The best practice report ismore thorough ly described on page 2. This mean s that the subset of the functionality ofthe RBS that is the basis for the implem entation u sing QuickCheck in no way needed to


14/74

1. Introduction 2

be covered fully by our implementation. Only that the implementation as such coveredenough of the functionality to be able to provide the metrics as well as providingenough knowledge about test case development using QuickCheck to be able to pro-du ce the best pr actice document. This also meant th at it was not implied that further

existing functionality that could be covered by test cases written using QuickCheckneeded to be covered.

1.4 GOALSThis subsection describes the goals that w ere established as part of the thesis guid e-

lines. These goals have been established in coordination between us, the companysup ervisor and the company section sup ervisor. The goals are that we, after completionof the thesis work:

Have delivered a best practice repor t that is sufficiently extensive to give Ericsson aprop er guid e for transititioning their test-environment toward s using Qu ickCheck

for testing su bsets of the MPSW functionality.

Should be able to observe whether the result of test cases implemented u sing Quick-Check have equal or greater test coverage than earlier test cases not u sing Quick-Check.

Have gained enough kn owledge about QuickCheck to be able to integrate it in thetest suites given by the sup ervisor.

1.5 DELIVERABLESApart from this thesis, the work also resulted in a n um ber of deliverables, internal

to Traffic and Feature SW, these w ill be described below.

1.5.1 BEST PRACTICE DOCUMENTThe analysis section of the thesis describes p articular im portan t issues wh en u sing

QuickCheck to implement automated test cases for complex system testing. Several ofthe solutions to these problems and other commonly occurring issues have been for-malised into a best practice document that w e will provide the commissioner aftercompletion of the thesis work. These solutions also provides the reader with codeexamp les and use cases that can help him or her in their work when using Qu ickCheck.

1.5.2 IMPLEMENTATIONThe result of the implementation being done during the course of this thesis is in

part included in this thesis to provide important complementary code for the discus-sions. Apart from th is, all code will be delivered to the comm issioner after the comple-tion of the thesis work.

For confidentiality reasons the full imp lementation will not be includ ed but theimplementation excerpts and intellectual property included is sufficient to cover allaspects of the implementation d iscussed in the su bsequent sections.


15/74

3 2. Context of application

2 CON TEXT OF APPLICATION

This chapter introduces the reader to the technologies and concepts that are rele-vant for this thesis work. The reader will also be made aware of the larger picture, thatis, the bigger system around wh ich the rath er focused su bset of functionality this thesiswork take place.

2.1 INTRODUCTION TO WCDMAUpon trying to und erstand the basis and implications of the thesis work it is of

interest to identify the areas of telecomm un ications that th is thesis w ork correlate toand is depend ent u pon. Therefore this chapter focuses on the architectural asp ects ofWCDMA from a larger perspective after wh ich w e will narrow d own u pon w hich sub-set of the telecommu nications systems the thesis work is related to as well as the stand -

ards established for describing this subset.

2.1.1 3GPPThe 3GPP, short for third generation partnership program, is a collaboration agree-

ment that was met december 1998 [15]. 3GPPs initial focus was to bring together anu mber of telecommu nications standard s bodies and their m ain focus w as to functionas the WCDMA specification body w hich meant that they shou ld w ork to provide glo-bally available reports and techn ical specifications for the th ird Gener ation Mobile Sys-tem. These were based on th e evolved Global System for Mobile Comm un ication, orGSM in short, core networks and the radio access technologies that they support (i.e.,Universal Terrestrial Radio Access (UTRA) both Frequency Division Dup lex (FDD)

and Time Division Duplex (TDD) modes). The 3GPP committees are divided into sev-eral group s of wh ich th e TSG group s are of most im portan ce for th is thesis. The TSGgroup s have as their responsibility to prepare, approve an d maintain the sp ecificationsfor the 3G and GSM n etwork functionality. The technical specifications provid ed by3GPP standardizes the interfaces used in telecommunications products which there-fore have a large im pact on all aspects of Ericssons telecomm un ications system s. Eric-sson have officials that are p art of the im pact on these specifications and also internalTSG groups to determine Ericssons position on planned specifications as well as deter-mining the internal imp act.


16/74

2. Context of application 4

2.1.2 BASIC WCDMA ARCHITECTUREAs we can see in the picture below, the WCDMA Radio Access Network is con-

nected to the Core network to provide radio connections to the User Equipment [11].

The Radio N etwork Controllers, or RNCs in short, are responsible for controlling theRadio Base Stations, or RBSs in short, w hich han dles th e connections to the h and set.The RNCs communicates with the RBSs using the Iub interface and with each otherusing the Iur interface. In th e WCDMA network RNCs are connected to th e Core net-work using the Iu inter face. Fur therm ore, the Uu rad io interface is used by th e RBSs totransmit and receive information from the User Equipment, or UE in short.

Figure 2-1: The WCDMA architecture

CORE NETWORK

WCDMA RAN

RNC RNC

RBS RBS RBS RBS

Uu

IubIub

Iur

Iu Iu


17/74


2.1.3 RADIO NETWORK CONTROLLERThe RNCs as nodes in the RAN h ave tw o d istinct roles, to control or to serve [11].

The serving RNCs are the RNCs that are responsible of connecting the subscribers

handsets to the core network, thus providing the radio connectivity to the core net-work. The controlling RNCs takes on the role to control a number of cells in the RANand their associated RBSs. In this case which is the most imp ortan t "mod e" for our th e-sis the RNCs maintains the resources in the netw ork and controls the requests and con-firms being sent betw een the serving RNCs and the controlling RNCs in the n etwork.This is the type of operation that provides the possibility to perform softer handover ofhandsets between different cells in the RAN.

Figure 2-2: RN C roles in the architecture

2.1.4 RADIO BASE STATIONThe RBS is the node in the RAN that is respon sible for handling th e rad io transm is-

sion to and from the h and set. RBSs come in hu nd reds of different configuration typ esranging from the smallest indoor micro RBS 3303 that provides micro cells in the net-work to the largest outdoor macro RBS 3206 wh ich provid es high coverage and capac-ity at the radio access level [12]. The transmission of the RBSs is performed using theUu p rotocol over which different typ es of connections can be established and rem oveddep ending on a lot of factors such as the han dsets location, subscriber need s, num berof users connected and more. This is closely related with the concept of Radio AccessBearer typ es, or RAB types in short, wh ich will be described in a following section. TheRBS is controlled by its controlling RNC using the Iub interface on w hich the RNC canissue elemen tary p rocedu res to maintain just about everything in the RBS, this includ esfor example power control, connections to handsets, cell control and several otherareas of functionality. One Rad io Base Station can control one or m ore cells in the net-work.

CORE NETWORK

WCDMA RAN

Controlling RNC Serving RNC

RBS RBS RBS RBS

Uu

IubIub

Iur

Iu Iu


18/74


2.1.5 H ANDOVERHandovers are what makes a mobile telephony network mobile. A handover is

required when the UE is moving between cells while communicating as depicted in

figu re 2-3.In the seamless network d ifferent type of han dovers are at p lay [11]. One such type

of handover is called inter system h and over wh ere the hand overs can be from GSM toWCDMA depending on the load of the system or the other way around. This type ofhand overs are regulated u sing service differentiation an d load control based on p oli-cies, load control mechanisms or based on the type of subscription of the subscribers.

Figure 2-3: Handover between cells

2.1.5.1 SOFT AND SOFTER H ANDO VER

Soft and softer hand over enab les the UE to be connected to mu ltiple cells and possi-bly to multiple base stations [11]. This enables the UE to travel through the networkform one cell to another w ithout an y ap parent loss of connection. While soft handover

mean s a transition of the UE from one cell in one base station to another cell in anotherbase station, softer hand over takes p lace w ithin the different cells of one single basestation as dep icted in figu re 2-4. To be able to perform a soft or softer hand over th e cellsare required to use the same frequency.

Figure 2-4: Soft and softer handover

RBS RBSCell1 Cell2

RNC

RBS RBS RBS


19/74


2.1.5.2 INTER FREQUENCY HANDOVER

Inter frequency han dover, or IFHO in short, is required w hen a h and set is movingout of coverage of one RAN frequency and needs continuation of service on an other

RAN frequen cy, which means tha t the cells have d ifferent WCDMA carrier frequencies[11]. This means that a new connection will be set up in the new cell allowing for con-tinuation of commu nication as in th e soft- and softer-hand over cases.

2.1.6 CHANNEL TYPESIn the WCDMA network there exist different types of transmission channels for

which the subscriber can transmit information depending on the type of informationand the content type the subscriber needs to transmit [11]. The tw o basic typ es of trans-mission chann els are called comm on chann els and ded icated chann els. The dedicatedchannel is used w hen th e subscriber is currently tran smitting high loads of data su chas in a voice conversation or when down loading d ata from a w eb page w hereas thecommon channel is used when transmitting lower loads of information. There existschann el type switching for this as well in which the subscriber is switched betw een thetwo typ es of chann els depend ing on typ e of information being transm itted. This is oflesser imp ortan ce for this thesis but the subject of chan nels will be men tioned in a su b-sequen t chapters in this thesis. The ded icated chan nels are of particular interest for thisthesis since the thesis work are par t of testing th e establishment and removal of suchchannels between th e han dsets and RBSs.

2.1.7 RADIO ACCESS BEARER TYPESOne imp ortant service of the WCDMA netw ork is its RAB types that carry the sub-

scriber information betw een the han dsets and the core network. A RAB type is a p re-defined set of services which determines the properties of the uplink and downlink set-

tings of the connection on the Uu p rotocol as well as prop erties related to th e ded icatedchannels. This includes information abou t the p roperties of the transport settings forthe up link and down link for the rad io link as w ell as several important parametersconcerning the ded icated chann els to be used for that RAB type.

There is as of today four different RAB types for the WCDMA n etwor k of which allare of imp ortance for the th esis work [11]:

Conversational: A Conversational RAB that is used for voice telephony

Streaming: A streaming RAB that is used for streaming data such as video

Interactive: A interactive type of RAB that is used for web surfing and similar activ-ities of the end user

Background: A background type of RAB that is used for transferring files in thebackground of the other RAB types.

All of these types can be combined in several predefined combinations to p rovideeven richer service configuration of the radio access bearers. We could for examplehave a Conversational Speech + 2 * Interactive Background + 1 * Streaming Unkn ownRABtype that combines three of the above listed types which would be a configurationfor an end user using voice and video telephony while for example transferring amu sic file to the same or another end user.


20/74


2.2 IUB SPECIFICATIONAs we mentioned in the previous chapter about "Introduction to WCDMA" the

3GPP TSG group s have worked together to form specifications for the architecture of

current and future telecommunications networks. Each telecommunications develop-ment company that are part of the agreement implements their architecture basedupon these specifications. This contributes to the standardization of the industry atlarge. What this means is that the architecture previously described in Ericssons casehave its functionality based upon several of these 3GPP technical specifications. Whatwe m ean by "based up on" is that Ericsson have internal TSG group s that d ecides howto interpret the specification by looking at the larger picture and determine w hat p artsof the specifications that are of interest for th em. The Iub interface that we mentionedin the previous section is of much importance for this thesis work and it is thereforeimportan t to consider the implications of 3GPP specification on this part of the archi-tecture.

The 3GPP specification for the Iub interface is called the NBAP specification an d it

specifies the rad io network layer signalling to be used for the Control Plane over theIub interface. This sounds rather complex but what it means is that the specificationspecifies the pro tocol to be used by the RNCs to control the RBSs over th e Iub inter face.

2.2.1 NBAP SPECIFICATIONThe N BAP sp ecification is quite an extensive specification, about 850 p ages long

which is separated into several high level sections.

2.2.1.1 NBAP ELEMENTARY PROCEDURES

The first section of the NBAP specification specifies the NBAP Elementary proce-du res, or EPs in short, which are the basic procedu res that can be invoked by the con-

trolling RNC toward s the N ode B.Each EP is determined by an initiating message called a request an d possibly tw o

associated m essages. The first associated m essage is called th e response m essage whichgives the calle information abou t the results of the outcome of the requ est. The secondassociated message is called the failure message which in case the request failed givesthe calle detailed information about the causes of failures for the request.

The EPs are separated into comm on procedures and ded icated procedu res [9]. Thedifference between the two types of procedures is that a common procedure is a proce-du re that the RNC can invoke that is not dep endent on a p revious comm on procedurebeing called that have initiated a comm un ication context. What hap pens after a com-pletion of an invocation of a common procedure that creates a communication contextis that the communication context is established in the Node B for a specific communi-

cation w ith a UE. This commu nication context is on th e RNC side d etermined by theCRNC communication context ID and on the Node B side determined by a NodeBcomm un ication context ID. It is this ID that is used to invoke d edicated p rocedu res thatdeals with resources established within such a communication context. For each ofthese procedures, a description of the successful outcomes, unsuccessful outcomes andthe possible fail causes for the procedures are described.

2.2.1.2 ABSTRACT SYNTAX NOTATION ONE SPECIFICATION

The next p art of the N BAP sp ecification is the ASN1 sp ecification of the possiblecontent of NBAP messages.

The ASN1 specification of NBAP m essages is specified in three layers.The first layer is what is called th e Elemen tary Proced ure Defin ition specification, or

PDU specification in short, which specifies the content of the NBAP message for eachof the NBAP EPs. This implies an ordering of the Information Elements, or IEs inshort, in the message and presents information about whether or not an IE is consid-ered m and atory or optional. When considering the examp le of the ASN1 specificationfor RadioLinkAdditionRequestFDD on page 55 we can see that th e EP message being


21/74


defined is called RadioLinkAdditionRequestFDD and that the top-level PDU is asequence of tw o IE containers, nam ely the protocolIEs-container and the protocolEx-tensions-container of w hich th e protocolExtensions-container is considered optional.For each of these IE-containers it is specified w hich typ e it is, for examp le the protoco-

lIE-container is to be of the typ e called Rad ioLinkAd ditionRequestFDD-IEs.The second layer is what is called the IE specification which specifies the IEs or IE

containers that can be par t of the NBAP message PDUs or in IE containers themselves.An IE contain er is a container for IEs, that is represented a s a sequence. There are manytypes of IE-containers such as protocol-single-IE-container, private-IE-container andmany m ore.

The containers are specified similar to the PDUs in that it implies an order ing of theIEs and the specification specifies presence informat ion for each IE. As in the exam pleof the ASN1 specification for RadioLinkAdditionRequestFDD on page 55 we can seethat the PDU definition contains a ProtocolIE-Single-Container called RL-Informa-tionItemIE-RLAdd itionRqstFDD w hich is specified using an ID and a d efinition as asequen ce of single IEs.

The IEs themselves are the most basic elements of an N BAP EP message an d are inthe specification defined as ordered elements with an identifier and a value. An IE ele-ment can have different presence depending on the context of application. It can bespecified as man datory, it can be specified as optional or it can be specified as condi-tional which means that its presence is depend ent on th e presence or value of anotherIE in the sam e context of app lication.

The value of an IE can be of d ifferent types such as integer, enu merated , octet string,bit string or a special constan t value reference called ASN1_NOVALUE that is the ASN1represen tation of a null value. If we look at the example of the IE Definition for Diver-sityControlField on page 56, we can see that DiversityControlField, which is an IEthat is part of the IE-container mentioned in the previous p aragrap h, is defined as anenum eration meaning th at the DiversityControlField can have one of the values in theenum eration, that is, either m ay, must or mu st-not.

The third layer of the NBAP ASN1 specification contains common definitions, con-stant values for IE names, EP constant codes and other common information.

2.2.2 APPLICATION OF NBAP SPECIFICATIONThe previous section described in general what is included in the NBAP specifica-

tion. How this is used in th e thesis work w ill be described in a later section of the thesisbut another question begs to be answered and that is how the NBAP specification isused by the different development companies of telecommunications systems. Whatone has to consider is that all development companies that are part of 3GPP have beenrepresented in th e process of developing an d maintaining the specification. All thesecompanies are free to perform th eir own internal app lication given th e specifications

and their pu rpose is to maintain the m ulti vendor op erability.

2.3 RADIO LINK MANAGEMENTAs we men tioned in the section on NBAP Elementary Procedures on page 8 in the

NBAP specification the EPs are divided into common procedures that creates comm u-nication contexts and dedicated procedures that manages these communication con-texts. These procedures are by Ericsson d ivided into several subsets of fun ctionalitythat Ericsson have decided are relevant with each other. This can be based upon local-ity of functionality or wh ether it is related to d ifferent p arts of the h ardw are. One ofthese collections of common and dedicated procedures is called DRL.

DRL is the internal name for the subset of the RBS functionality related to control-

ling rad io links toward s the user equipm ent in the RAN. This fun ctionality has manyimplications on several areas of the RAN, ranging from the controlling and servingRNCs down to the actual user handsets. This thesis focuses on the 3GPP RLM subsetof the N BAP fun ctionality, and since Ericssons division of the N BAP fun ctionality into


22/74


DRL that includ es other functionality such as compressed m ode and ded icated p owermeasurement as well we have decided to follow the group name RLM as set by the3GPP standard .

The elementary procedures that are related to RLM can basically be divided into

two sections. The first part is the collection of EPs that are specified as comm on and th esecond part is the collection of EPs that are specified as dedicated. For more informa-tion abou t these EPs, see NBAP Elementary Procedures on p age 8.

The EPs that are par t of RLM of wh ich are of impor tance for this thesis are the fol-lowing:

RadioLinkSetupRequest/ Response/ Failure

RadioLinkDeletionRequest/ Response

RadioLinkAdd itionRequest/ Response/ Failure

RadioLinkReconfigurationRequest/ Response/ Failure

RadioLinkReconfigurationPrepare/ Ready/ Comm it/ Cancel

2.3.1 RADIOLINKSETUPREQUESTFDDThe RadioLinkSetupRequestFDD is the EP in the NBAP specification that estab-

lishes a communication context between the CRNC and the Node B for associationwith one or more radio links between the serving RNC and the handset. This EP estab-lishes one or m ore Dedicated Ch annels, or DCHs in short, on one or m ore radio linksand can also includ e a H igh Speed Down link Shared Chan nel, or H S-DSCH in sh ort,for one radio link and Enhanced Dedicated Channels, or E-DCHs in short, on one ormore rad io links.

2.3.2 RADIOLINKDELETIONREQUESTThe RadioLinkDeletionRequest is the EP in the N BAP specification that releases one

or more resources associated with one or more radio links in a communication contextfor a handset.

2.3.3 RADIOLINKADDITIONREQUESTFDDThe RadioLinkAdditionRequestFDD is the EP in the N BAP specification that estab-

lishes the necessary resources for one or more rad io links in an existing commu nicationcontext toward s a han dset, effectively add ing one or m ore radio links to the commu ni-cation context.

2.3.4 RADIOLINKRECONFIGURATIONREQUESTFDDThe RadioLinkReconfigu rationRequ estFDD is the EP in the N BAP specification that

allocates resources for the modification of one or more radio links in a communicationcontext.

2.3.5 RADIOLINKRECONFIGURATIONPREPAREThe RadioLinkReconfigur ationPrepare is the EP in th e N BAP sp ecification that is

used to p repare the Nod e B to allocate the necessary resources needed for modificationof one or more r adio links in the commu nication context towards th e UE.


23/74


2.3.6 RADIOLINKRECONFIGURATIONCOMMITThe RadioLinkReconfigurationComm it is the EP in th e N BAP sp ecification that is

used to switch the radio link or radio links to the newly configured resources in the

commu nication context toward s the UE.

2.3.7 RADIOLINKRECONFIGURATIONCANCELThe RadioLinkReconfigurationCancel is the EP in the NBAP specification that is

used to cancel a previously prepared reconfiguration and releasing the allocatedresources in the comm un ication context tow ards th e UE.

2.4 ERLANGThis section introd uces the reader to the concepts of Erlang tha t is need ed to be able

to comprehend the following chapters of this thesis.

2.4.1 RECORDSErlang records are a way for the programmer to store a fixed number of elements

[7]. It has nam ed fields and similar to structs in the C program ming langu age. Erlangrecords can be used by the programmer to ensure cross module consistency of datastructures by using record defin itions in a comm on include file for the mod ules usingthese data types. A record definition in Erlang is declared with the record name, itsnam ed fields and possible default values such as:

-record(name, {field1=Value1, field2, field3=[]}}.

Accessing record fields is a simple task. If the record is bound to a variable as in:

MyRecord = #name{field1=1, field2=3}.

the record field s can be accessed by :

MyRecord#name.field1



If the record is matched by a clause, the field valu es can be bou nd by local variablesas in:

myFunction(#name{field1=BoundVariable1, field2=BoundVariable2}).

The fields of records n ot having a d efault value defined will be initialised with theatom undefinedwh ich is the value that field3 wou ld have got in the record in thisexample.

2.4.2 ERLANG TUPLESA tup le in Erlang is a comp ound data structure w ith a fixed nu mber of terms where

any term can be of any type [7]. The Erlang representation of a tuple is on the form{term1, term2,...,termn} where each element in the tuple is called an element.The num ber of elements in the tu ple is said to be the size of the tuple.

2.4.3 ERLANG LIST COMPREHENSIONSMany p rogramm ing languages includ ing Erlang p rovides a functionality in the lan-

guage that in Erlang is called list comprehensions [7]. This functionality provides anotation for the programmer that enables him or her to generate elements on a list con-


24/74


ditionally. List comp rehensions as langu age constructions are analogou s to set compre-hensions in Zermelo-Frankel set theory and are called ZF expressions in Mirand a. Theyare analogous to the setof an d findall predicates in Prolog [7].

In Erlang a list comp rehension is written w ith the following syntax:

[Expr || Qualifier1,...,QualifierN] where Expr is an arbitrary expres-sion, and each Qualifier is either a generator or a filter. This means that the expres-sion to be evaluated can cond itionally be evaluated d epend ing on the qu alifiers to theright han d of the list comp rehension. Thu s the language construction provides the p ro-gramm er with both th e ability to determine wh at to generate, or if to generate anythingat all, depend ing on conditions external to the construction of the list. This languageconstruction is nothing that can not be implemented using functions together with sim-ilar langu age constructions bu t it p rovides good readability an d traceability since wecan provide functions for the q ualifiers that traces the generation.

A generator for a list comp rehension in Erlang is written as:

Pattern


25/74


The SUT shown in figure 2-5 consists of the MPSW that is running on the RBS.Additional to this software are the stubs taking care of the fun ctionality that we are nottesting in this setup, such as stubs for emulating other hardware in the RBS and thesoftware ru nning on this hardw are as well as stubs for the interfaces. The most impor -

tant one is LOST, this is the stub for the emulation of the subsystem signalling. Thehardware signals are sent over the Bp interface. In this test setup the LOST stub isresponsible for receiving the signals usually sent to th e subsystems and to send backprop er responses to the MP. The w ay that th e LOST stub is implemented today p osessome p roblems for some of th e aspects of testing in a greybox mann er. The signallingperformed by LOST pre-define the content of the response signals that are to be sentfrom the subsystems back to the MPSW. In some use case scenarios where the contentof these signals need to be mod ified to achieve the prop er behaviou r of the MPSW onecan not rely on LOST since this functionality is not implemented. These modificationsneed to be performed by the test case developer by hand after a short-circuit of theLOST stub .

The Iub interface is the interface w e are u sing to invoke the elementary procedures

described in Radio Link Management on p age 9. The Mub interface is another inter-face that can be u sed to read man aged ob ject in the SUT to verify results of invocations.This contributes to th e greybox testing m ethod described in the following chapter.


26/74

3. Testing and test frameworks 14

3 TESTING AND TEST FRAMEWORKS

The testing of complex system software is often a tedious task and the effort toimp lement test cases for these comp lex systems will in man y cases requ ire a substantialamount of effort equal or greater than the effort needed to design these systems [4].There are many d ifferent methodologies and a spects of software testing that are relatedto this thesis work. These methods will be described in this chapter as well as otheraspects of testing that will be important for subsequent chapters. If not specified, allconcepts described in this chapter refers to system testing.

3.1 FUNDAMENTALS OF TESTINGAs is und erstood, softwares w ith no d efects are a utop ia [1]. In practice, it is imp os-

sible to ensure th at even a small software ap plication is free of faults. This is based on

several facts. For one, computer systems are complex entities with complex design.Secondly, the development processes in place can always fail and as always, whenhu man beings are involved, faults can h app en. As a result of this, add ing the fact thatthe resources in development projects available for testing often are quite limited, per-forming adequate testing becomes a substantial problem. This means that test casedevelopers not longer can focus solely on the test case development but also need tofocus on improving the efficiency of the testing p rocess to enable them to d iscover asmany defects as possible in the limited time frame set for testing. Ultimately, the timeframe for testing must eventually end when the given system is acceptable for itsintended purp ose.

Software testing as such can on a high level be sorted into tw o fund amental p ara-digms [2]. The first is called scripted testing an d th e other is called exploratory testing.

In the scripted testing paradigm, the scripted testing is a sequential paradigm wherethe examination of requirements, followed by the design and documentation of testcases is followed by the execution an d examinat ion of the resu lts of execution. The sec-ond paradigm, called exploratory testing is in essence orthogonal to scripted testing inwhich the emphasize is on simultaneous learning of the system, test design and testexecution. The test case developer following this paradigm designs and executes testswh ile learning the behaviour of the system.

3.1.1 STOPPIN G STRATEGIESAs we m entioned above, the time frame set for testing m ust eventu ally end . This is

one of the most difficult questions when it comes to testing since there is no way to

know that the last found fault was the last remaining in the system.Many organisations incorporates different strategies for testing and the stop strate-gies for testing can be very different [2][4]. Economics, of course, dictates that testingeventually mu st stop bu t the completion criterias typically u sed in practice can d iffersubstan tially. Many test case develop ers often adh eres to the first category in wh ich theexecutives at the comp any tells the test case developers w hen it is time to release thesoftware, often after the scheduled time for testing expires. In other cases the h alt oftesting and releasing of the software can be dependent on the rate of found faults,when the marginal cost for finding a "new" defect exceeds the expected loss from thatdefect or when the test cases in test suites completes successfully.

Stopp ing w hen the schedu led amou nt of time for testing expires can be qu ite a use-less criterion since it can be met by doing exactly nothing [4]. Stopping when the testcases completes successfully is also useless since this stop criterion does not measure

the qu ality of the test cases being involved. The second criterion is also coun terp roduc-tive since this criterion might give the test case d evelopers an incentive to w rite testcases that h ave a low probability of encountering faults.


27/74

15 3. Testing and test frameworks

3.2 POSITIVE AND NEGATIVE TESTINGPositive and negative testing are two complementary testing techniques [1].In the p rocess of positive testing it is verified th rough th e test cases that the system

conforms to its stated specification, that is, that the system fulfils its contract. In thisprocess the positive test cases are derived from the system specification documents.

In the process of negative testing, test cases are designed to outside of the scope ofthe stated specifications. In this p rocess, the test cases are designed to show that thesystem does not do what the system specification declares that it should, or to showthat th e specification of the system is incomp lete or faulty.

3.3 WHITEBOX TESTINGWhitebox testing refers to tests where the system under test is completely transpar-

ent for the test case developer. This means that th e test cases can be wr itten based onthe internal path s, structure and implementation of the system to completely cover all

aspects of the functionality tha t are economically viable to test, un der this testing cate-gory such method s as control flow testing and data flow testing come into play [2].

Whitebox testing can ap ply at all levels of system testing and in all situations thetesting methodology implies that the test case developer has great knowledge aboutthe implementation [4][1].

3.4 BLACKBOX TESTINGBlackbox testing refers to testing when the system under test is completely opaque

for the testing code. This means that the test code can not make any assum ptions of theimplementation of the system under test and must solely resort to the functional speci-fications and or functional descriptions that are ava ilable for the system. In these cases

the testing is often referred to testing App lication Programm ing Interfaces, thu s testingat the functional level. Most blackbox test cases focuses on a single function and tests itwith "mid dle-of-the-road"-values. Such tests are highly credible and easy to evaluatebut not p articularly powerful [14]. In such cases, such methods as equivalence classtesting and boun dary v alue testing come into play [2].

3.5 TESTING WITH SHADES OF GREYMany systems are semi-transparent wh ich m eans that the system exposes some of

its functionality externally [2]. Testing such systems is often referred to as testing w ithshades of grey. This means that test code can utilize some of these semi-transparentimplementation details in the design of blackbox tests by "opening" the box to look atinternal details of the system to allow for more efficient blackbox testing.

3.6 SYSTEM SPECIFICATIONSFunctional specifications are a key part of many software projects and they came

into existence when the waterfall process mod el of software developm ent became fash-ionable in the early 70s [2]. The functional specification provide the basis for func-tional testing and the specification as such often describes an external view of an objector a procedure indicating the options by which a service could be invoked. The testcase d eveloper uses this specification to w rite test cases from a blackbox testing per-spective. Besides the fact that it provid es them w ith the ability to imp lement test cases

in parallel with the implementation of the system it introdu ces them to w hat the sys-tem really is [6].


28/74

3. Testing and test frameworks 16

3.7 SPECIFICATION BASED TESTINGSpecification based testing focuses on validating the functionality against a specifi-

cation of the system [14]. These tests are highly motivated since many companies takes

their specifications quite seriously and therefore the verification of such specificationsbecomes importan t. This is du e to the fact that since the specifications often are par ts ofa contract, conforming to those specifications becomes very im portant. O ne can d rawparallels to other products such as cars or mobile phones which must adhere to theiradvertisements or else the customers could feel betrayed.

Specification based testing can often lead to groups inclining towards focusing nar-rowly on what is written in the specification [14]. To them, designing test case suitesthat includes an unambiguous and relevant test case for each claim made in the specifi-cation becomes the definition of a good t est case suite.

3.8 AUTOMATED TESTING

The use of automated test tools provides the possibility to run tests un attended aswell as the possibility to run more tests than would otherwise be possible. The greatestbenefit comes from the inattentiveness which is closely related to regression testingdescribed in the following section.

Most testing tools for test automation creates tests by recording behaviour of theSUT [1]. This recorded beh aviour can then be rep layed on a later version of the systemallowing the behaviour to be compared between the two versions. Differences inbehaviour can p oint out p lausible problem areas. Other tools provide the p ossibility totest the functional requirements of the SUT, one such tool is QuickCheck which is thefocus of this thesis. There are also tools that test non functional aspects such as per-formance and stress testing. John Watkins argues that automated testing tools,although h aving man y potential benefits, are not app ropriate for all testing tasks and

that they hav e to und ergo a thorough evaluation before being introdu ced. One imp lica-tion of this is that the tool must be integrated with the existing process and will notreplace it.

3.9 REGRESSION TESTINGRegression testing is focused on designing test cases that are h ighly reusable with

the intent of reusing the test cases whenever the system under test changes. When con-sidering regression tests the focus is on reu se and prop er docum entation.

As John Watkins m entions, "regression testing is a p articularly candid ate for sup -port by means of an automated testing tool, especially if frequent new builds andreleases of the system und er test are anticipated " [1].

A p roblem with regression tests are that one case might hav e been highly credibleand pow erful at the time of creation, but as time goes by, and th e test case have passedmany times it is not likely that the test case will find any new defects the next time [14].Thus, most of the time regression tests carry little information value.

For regression suite test cases the documentation aspect becomes very important[3]. These cases have to evolve together with the software it is testing and down the linesomeone else might become responsible for these suites and need to be able to quicklyunderstand them.

3.9.1 REGRESSION TESTING TODAYThe regression testing at MPSW today is mostly performed by the employees being

given functional descriptions and functional specifications from which they derive testcases that covers the use case scenarios in the specifications and descriptions for agiven version o f the software in the RBS. These are then gathered into test suites basedon the functionality being tested. The test suites are re-compiled after updates and runon special RBS nodes during the night. The results are gathered by different log capa-


29/74

17 3. Testing and test frameworks

bilities into a web based fram ewor k that allows the test case developers to analyse pos-sible faults and gather statistics from the runs. The framework surrounding theseregression suites are based on a behavioural model that provides initialize and cleanupcallback functions that ascertains that there are no d epend encies between the ind ivid-

ua l test cases as well as between the test suites.


30/74

4. QuickCheck 18

4 QUICKCHECK

This chapter will introduce the reader to the software testing tool called Quick-Check that is the focus of this thesis. The chapter w ill describe the fund amental con-cepts of QuickCheck as well as some intricate details. After reading this chapter thereader will have gathered enough information to be able to follow the QuickCheck-related decisions being made to solve the problems found during the initial analysis aswell as du ring implementation.

QuickCheck is a software testing tool produced by a recently founded companycalled Quviq. The QuickCheck software is the result of many years of development bythe two founders of Quviq namely professor John Hughes and associate professor Tho-mas Ar ts. The tool is a testing software that provides th e ability to test software in anentirely different way than previous tools. QuickCheck allows the user to write testspecifications that includes test properties that ought to hold for the system u nd er test

as well as generators that provide the test case data. QuickCheck gives the test casedevelopers a second a lternative approach to testing their system. Instead o f wr iting testcases that m aps to use-cases in a one to one fashion, QuickCheck enables an au toma-tion of the test case generation and execution. With QuickCheck the user can write thespecifications by specifying several properties that ought to hold for d ifferent asp ectsof the system u nd er test as well as generators that generates a mu ltitud e of the behav-iours that the system m ight inhabit. The outcome of these generators can be checkedagainst the p roper ties that ough t to hold. The imp ortan t difference to notice here is thatQuickCheck enables the test case developer to specify the expected behaviour of thesystem in a concise way that is not spread out in a thousand different test cases butinstead is captu red in a sm all amount of testing code.

To make concise and readable specifications QuickCheck uses qualities from func-tional program ming languages. Examp les of such qualities w ill be visible in th e code

presented th roughou t this thesis. The advan tages of writing specifications instead of anumber of small scripted test cases include that less code is required for testing whichtherefore leads to better readability and maintainability.

No t only does Qu ickCheck prov ide the ability to test in a many-to-one fashion bu t italso provides a qu ick way of analysing errors in the software by shrinking th e sequenceof commands leading to the failure to a minimal sequence to guide the user towardsthe real cause for the failing test case.

4.1 PROPERTIESIn the QuickCheck domain, a property is a logical description encapsulated in a

function. These properties provides a means to specify which behaviours the SUTshould inhabit. Lets start with a simple example. Assume we are the providers of alibrary for the Erlang programming language, lets call it ListIT. In this library we haveimplemented our own list structure that behaves as any lists would do in functionalprog ram ming langu ages. In the docum entation of ListIT we specify all the library fun c-tions for operating on lists. For examp le we could state that:

listIT:delete(elem, list): Deleting an elemen t from a list removes the ele-men t from that list, if the elemen t is on it.

That wou ld be a reasonable docum entation of a library d elete function on a list. Buthow would we test it. One approach could be to generate a sufficient amount of testcases, perhaps using the boundary principle and try to delete an element from the

emp ty list, doing the sam e with a slightly larger list and perhap s deleting a empty ele-ment an d so on. What we d o in QuickCheck is to write a property that says wh at oughtto hold for this list structure in the context of deletion of an element from a list. InQuickCheck w e w rite:


31/74

19 4. QuickCheck

property_listIT_delete()->

?FORALL(I, int(),

?FORALL(List, list(int()),

not lists:member(I,listIT:delete(I, List)))).

So this is a property in QuickCheck. Basically it reads: "For all instances I that are ofthe type that is generated by the int generator and for all lists that are generated byth e list generator using the int generator it should hold that if you delete an ele-ment from th e list, that element shou ld no longer be a m ember of that list.

So a prop erty is a logical description of what ou ght to hold in the system und er test.The property u ses the generators, int an d list in th is case, to generate test cases fortesting this prop erty. These generators will be explained in the following chapter. Aproperty is tested in QuickCheck by generating 100 test cases from the generators andapplying the property on these cases. There are a lot of issues to deal with for the user,since among other aspects, the user is responsible for defining generators that inhabitsa good data distribution.

4.2 GENERATORSIn Qu ickCheck all testing is d riven by the generators that p rovides controlled ran-

dom test data. The problem with totally random generation is that it in many casesresults in p oor test data distribution. Examples where rand omly generated d ata alwaysor almost always would lead to invalid, not good or in the worst case failing test casesare easy to discover. One such examp le could be that, given a nam e, the system shouldreturn th e persons age provided th at the nam e is in the following database:

As you u nderstand the chance of hitting any of the names with a rand om lengthstring with random characters is rather small, even when running millions of tests,which does not test the functionality of the system. It would be better to be able to forexample say:

"Choose either one of the names Erik Fransson, Rickard Gran feld t, Johan Holmqvistor Patrik Hglu nd or possibly choose from a list of any valid nam e or, in for example

one case out of seven, choose a ran dom string."QuickCheck provides just this, a powerful tool to generate random test data with

any complexity with full control.So wh at is a QuickCheck generator? A generator p rovides means to rand omly gen-

erate data for the test case as well as providing possibilities to have built-in shrinkingbehaviour. This is further described in the section about Shrinking on page 23.

The QuickCheck generator d efines a set of values for test data and a probab ility dis-tribution over that set. The test data generators are d efined by the test case developerusing basic generators provided by QuickCheck. The generators in QuickCheck areencapsulated in symbolic functions such as the lambda functions in LISP or the funconcept in Erlang, which provides an important fact. The return value of a generator isnot a value but an encapsulated test data generator that at run-time will provide the

user w ith an actual value. More on this later on.As an examp le of a built-in generator we could look at the int generator.The int generator return s a test d ata generator for sm all integers. Another built-in

generator is the list generator that given an element generator returns a randomlength list of elements from the given generator.

Table 4-1: Example database

Name Age

Erik Fransson 24Rickard Gran feld t 25

Johan H olm qvist 27

Patrik Hglund 22


32/74

4. QuickCheck 20

As another example of generators, lets assume that we are testing a system thatmaintains a h uge d atabase of records of persons that live in a city. If we w ant to testsuch a system we would want to create a generator for generating names of persons.But since no such generator is provided with QuickCheck we would have to create our

own . What we could do is to u se a built-in generator that r and omly selects an elementfrom a list, such as elements. We could then use a d ictionary or a h ard coded struc-ture of names to p rovide us w ith the raw data to choose from. Lets assume a d iction-ary filled w ith names. Then our generator w ould be as follows:

name()->

elements(dict:getAllNames()).

What th is code d oes is to specify a user d efined generator called nam e that as thereturn value returns a new generator that uses the elements generator to encapsulatea test data generator choosing rand om elements from a list of nam es, provided by thedict:getAllNames() function.

4.3 STATE MACHINESIf random sequences of test data is generated, more sequences can be tested than

can possibly be w ritten by h and , hopefully leading to better test coverage. With auto-ma ted testing of such comm and sequen ces it is of imp ortan ce to mainta in control of theexecution of sequences and the generation of test data. If the SUT have side effectsrelating to an internal state, the execution of sequences must also reflect the effects ofsuch changes. In Qu ickCheck this is performed by u sing a state machine. The Quick-Check state machine has a predefined behavioural model w hich is made up of anu mber of requ ired callback functions that defines a specified control flow for the statemachine wh ich will be described sh ortly. A QuickCheck state machine also conta ins aninternal state, determined by the user, that gets modified when running the state

machine. This together with the intern al QuickCheck control algorithm s is wh at consti-tutes side effect testing using QuickCheck.During w hat w e call generation-time the command sequence for the state machine

run is generated. This starts w ith a call to initial_state which sets up the initialstate by for example getting information from the SUT. Then the command functionselects a call either directly or in some cases depending on the current internal state ofthe state machine. This call is, together with the state passed to the preconditionfunction to determine whether the call should be in the sequence or not, if so thenext_state function is called w hich d etermines what state the state machine shouldbe in when comm and is called the next time. After each generation-time run, the gener-ated sequence is run in what we call run-time. In this scenario the QuickCheck statemachine verifies the calls in the postcondition function again st the specification forthe SUT. If the verification is correct, the state will be upd ated to reflect the new state of

the SUT. All of the calls genera ted from th e command function d uring g eneration-timeare symbolic and on the form {call, Module, Function, [Args]} and the valuereturned from the call is of course also a symbolic value in the form of a tu ple {var,Value}. This will be further described in a subsequent section.


33/74

21 4. QuickCheck

Figure 4-1: Basic control flow of the state machine

The behavioural model for the QuickCheck state machine uses the following prede-fined callback functions:

The initial_state function initializes the state machine w ith a p redefined startstate, before each generation and execution of a command sequence. This can beseen in figure 4-1 above as the small box above the internal QuickCheck control.This function can be related to similar functions in conventional regression testingwhere a common fun ction is called p rior to each test case to initialize and verify thateach test case will be independent. The inital_state function in QuickCheckcan be used by the test case developer to set values in the state and for examplegather information needed from the SUT to reflect the SUTs initial state. The

initial_state function is called prior to the generation of command s d uringrun-time in the same way as during generation-time for each run of the statemachine. This means that each run of the state machine will start in a predefinedinitial state similar to conventional testing as w e m entioned above.

The command function is the generator for the test comm and sequen ce. Each time itis called, a new com mand is added to the sequen ce for evaluation. As we can see infigure 4-1, the command generator is called du ring generation-time after theinitial_state function has been per formed and is called every time a new com-man d is to be add ed to the test comm and sequence. This means that one comm andis chosen from the list of commands specified in the command generator. The com-mand generator as such can be as comp lex as the user w ant it to be, ranging fromonly generating a single comm and for the test comm and sequence to possibly gen-erate calls to test each fu nctionality in the en tire SUT. The command generator canuse the state, enabling the generation of tests depend ing on the sequence that havepreviously been generated. What this means is that in some cases we might havedep endencies between th e calls to generate, such that for example w e need to haveexecuted a setup of an object in th e SUT before a deletion of that object can be exe-cuted. This can be done by inspecting the internal state of the state m achine to

QuickCheck

Command sequence generation

next_state/3

precondition/2

property/0

initial_state/0

command/1

Command sequence execution

next_state/3

postcondition/3

Command invocation


34/74

4. QuickCheck 22

determine what to generate. Command is also responsible for controlling the lengthof the generated command sequences. This is don e by having the command genera-tor return a stop atom w hen the sequences should end .

The precondition function is a validation callback function. What this basicallymeans is that th e function return s a boolean value after d etermining the p recondi-tions for its arguments. As we can se in figure 4-1 this function is present duringgeneration-time as a fun ction following the command generator in the generation-time scenario. The function is used during generation-time to decide whether or notto include a command in the test command sequence by returning the booleanvalue oftrue for comm and s that are to be includ ed in the test comm and sequenceand returning the boolean value offalse for commands that are to be excluded.Precondition can therefore be used to filter out comm and s containing know n p rob-lems. This allows th e test case developer to continu e testing even though the SUTcontains faults that w ould otherwise cause the prop erty to fail.

The next_state function is used to up date th e state of the state machine. As wecan see in figure 4-1 the next_state function is what follows the preconditionfunction in the generation-time scenario and what follows the postconditionfunction in the run-time scenario. During generation-time the next_state func-tion is used to update the state information symbolically. What this means is thatthe result of a command du ring generation-time w ill get a symbolic representationin QuickCheck and this representa tion can be stored symbolically in the state of thestate machine for use of subsequent comman ds in the test comm and sequence dur-ing generation-time. During run-time the same information will be stored in thestate of the state machine bu t instead of the symbolic representation the informationused w ill now be the actual run-time values of what have been generated d uringgeneration-time.

The postcondition function is used du ring ru n-time as we can see in figure 4-1.It is the function following the execution of the commands generated during gener-ation-time. The postcondition function is used during run-time to determinethat the result of an executed comman d in the test comm and sequence is the resultthat is expected with respect to the specification of the SUT. The levels of verifica-tion can vary depending on the way of testing. In blackbox testing, as described onpage 15, the only verification that can and should be performed is to verify that th eanswer from the SUT is as specified given the command being executed and theinternal state. In blackbox testing with shades of grey as d escribed on page 15 thetest case d eveloper might have th e possibility to verify a variety of aspects of theSUT. In p ositive testing, that is testing against the sp ecification w ith th e intent ofintroducing the specified faults in the test command sequence, the postcondi-tion function can and should also be used to confirm th at calls that we expect tofail also will do just that d ur ing run-time. This is done by u tilizing fun ctions to ana-lyse the command s. These functions d etermines the cause of failure w hich can beused in comparison aga inst the resu lting fail cause of the failure from the SUT. Thisis of course again d epend ent on the level of transparency in the test environm ent.

4.3.1 GENERATION -TIME VALUESAs w e m entioned above, the command generator that generates the sequence of

command s have an abstract representation of the result of the comm and s. The repre-

sentation is by an Erlang tuple as described on page 11. The representation is as fol-lows:

{set, {var, X}, {call, ?MODULE, function, [args]}}

The representation of comm and sequences is an Erlang list of these kinds of tu ples


35/74

23 4. QuickCheck

wh ere each tu ple represents a comm and . What the tup le represents is a "prom ise" thatdu ring generation-time the value of{var, X} will be set to the result of the symboliccall. This can, as will be show n later, be used to get inform ation on th e resulting va lueof calls during the run-time scenario of the state machine.

4.4 SHRINKINGSometimes the test command sequence that is generated leads to a failing test. This

is of course what th e test case developer is looking for. When this hap pen s, the test casedeveloper kn ows th at either of these four statements are th e likely cause of the failure:

1. The QuickCheck state machine is failing in mod elling the specification of the SUT

2. The specification of the SUT is faulty

3. The test environment has somehow failed to give QuickCheck and the connectionwith th e SUT a proper test environmen t

4. There is a fault in the SUT.

This might look like a long list, but it is actually quite powerful combined withwh at is called shrinking.

A QuickCheck command sequence can range from one to several thousands ofcommand s. Trying to figure ou t wh ich elements in the comman ds led to th e failure inthe case of a longer sequence can be very time consum ing. What Qu ickCheck tries todo is to autom atically shrink the sequence and or the test data in general to a m inimalfailing case. What this means is that QuickCheck tries to run the test commandsequence over and over again, either trying to shorten the sequence of commands orremoving parts of the information in the command that does not affect the failure ofthe test case. The latter could include lowering integer values in command elements,

trying d ifferent combinations of atom-values contained in generators an d m any m orealternat ives. These two strategies are an imp ressive part of the tool which can give verygood results.

4.4.1 EXAMPLE OF SHRINKINGAs an example of this functionality we consider this real world example taken from

a course given by the foun ders of Qu ickCheck [18]. In this examp le we are testing th eErlang function day_of_the_week in the calendar module for the Open TelecomPlatform, or OTP in short, library to determ ine that the day_of_the_week function isfollowing its specification. For more information abou t this specification, see Docu-mentation of day_of_the_week on page 57. In the specifica tion for the

day_of_the_week function and the calendar m odu le from december 2006 we can seethat it is specified th at the fun ction of arity one takes as its argum ent a tu ple of threeinteger values. First a year w hich is not to be abbreviated, wh ich m eans that w e mu stuse absolu te values, since the int generator has a probability distribution between -30and 30 on average. For the m onth an d d ay we only generate values that according tothe specification should be valid, that is, values between one and 31 for the day andbetween 1 and 12 for the mon th. We now w rite the prop erty to follow th e specificationof the day_of_the_week function, calling th e p roperty prop_day_of_the_week .For the implementation of this property, see Property for day_of_the_week onpage 57. The property basically states that, for all years that are positive integers, and,for all month s that are in between the first and last month, and , for all days that are inbetween the first day of the month and the last day it should h old in the calendar m od-ule for the function day_of_the_week that it return s the day of the week as a num ber

between one and seven.We then run this property in QuickCheck and after 31 tests, it fails. QuickCheck

prints out that it failed on an if-clause in the m odule calendar in the function,date_to_gregorian_days with the argu men t 3. It is reasonable to imagine that it is


36/74

4. QuickCheck 24

at this point w e as test case developers wou ld u sually have to acknowledge a fault andstart repor ting it or start looking in the source code to find w hat w ent wrong. But th is iswh en QuickChecks shrinking functionality comes into th e p icture. Qu ickCheck nowstarts to rerun the test cases trying to minimize the generated values to find a minimal

represen tation of the fault. In this case by trying to low er the generated v alues if possi-ble. As we can see in the outp ut from Qu ickCheck it shrinks two times to find a mini-mal test case that still causes the SUT to fail. Now we can see that the shrinking haslowered the value of the day to find that the date 1003-02-29 causes the calendar mod-ule and its day_of_the_week function to fail. This might have been expected sincewe know that february only have 28 days, or 29 on a leap year which apparently theyear 1003 did not seem to h ave. The first d ate that caused th is failure m ight or m ightnot have directed the test case developer towards the failure but the strength of shrink-ing not only lies in this fact but also in the fact that the test case developer can see thedifferences between the original test case that caused th e failure and the min imized testcase that still causes that failure which sometimes can be the deciding factor when try-ing to find faults.

What one should notice is that this is a very easy example of shrinking that onlyshows to a small extent the p ossibilities. If we looked at the specification of the OTPtoday w e wou ld see that they have add ed one little row of information that tells us thatthe date supp lied to the day_of_the_week function has to be a valid d ate [8].

As a theoretical example of the shrinking algorithm when it comes to side effecttesting

NBAP message construction using QuickCheck - Andreas Granberg and Daniel Jernberg LITH-IDA...

Documents

Transcript of NBAP message construction using QuickCheck - Andreas Granberg and Daniel Jernberg LITH-IDA...