Quality of Service Report: KwaZulu-Natal Province 2015 ... · Quality of Service Report:...
Transcript of Quality of Service Report: KwaZulu-Natal Province 2015 ... · Quality of Service Report:...
0 | P a g e
Independent Communications Authority of South Africa Pinmill Farm, 164 Katherine Street, Sandton
Private Bag X10002, Sandton, 2146
Quality of Service Report:
KwaZulu-Natal Province
2015/2016 Quarter 4
1 | P a g e
Table of Contents
1 Executive Summary .............................................................................................................................................. 2
1.1 Introduction ................................................................................................................................................. 2
1.2 Drive Test Route Plan .................................................................................................................................. 3
2 Methodology ........................................................................................................................................................ 4
2.1 Drive Tests Equipment ................................................................................................................................. 4
2.2 Test Plan ...................................................................................................................................................... 4
2.3 Computational Methodology ...................................................................................................................... 6
3 Results and Analysis ............................................................................................................................................. 8
3.1 Parameter Targets ....................................................................................................................................... 8
3.2 Results .......................................................................................................................................................... 8
3.2.1 Accessibility Measurement Results ............................................................................................................. 8
3.2.2 Retainability Measurements Results ......................................................................................................... 10
3.3 KwaZulu-Natal Average Measurements Results ....................................................................................... 11
3.3.1 Average Accessibility measurement data results ...................................................................................... 11
3.3.2 Average Retainability measurement data results for KwaZulu-Natal Province ........................................ 12
3.3.3 Serving Technology Distribution ................................................................................................................ 13
4 Conclusion .......................................................................................................................................................... 13
5 Appendix A (KwaZulu-Natal coverage maps) ..................................................................................................... 15
2 | P a g e
1 Executive Summary
The Authority conducted Quality of Service (QoS) measurements on the networks of South
African mobile operators; Cell C, MTN, Telkom and Vodacom. The measurements were
performed to monitor performance of voice services being offered by the operators in
KwaZulu-Natal province. The measurements were carried out during February 2016 and
covered a total distance of 2574 kilometres of drive testing.
The purpose of collecting QoS measurements was to analyse and monitor the service quality
experienced by the end user of voice services offered by South Africa’s mobile operators
around targeted areas. Measurements were conducted in typical circumstances where
mobile service usage is likely to occur such as major towns, townships, farm areas, rural
areas and major road arteries as per population densities, economic and tourism activities
including historical complaints. The sampled areas includes Cato Ridge, Pietermaritzburg,
Howick, Mpophomeni and Richmond.
A vehicle equipped with TEM Symphony measurement tool was used to collect data, in order
to measure two main Key Performance Indicators (KPIs) which are Retainability and
Accessibility. The Drop Call Rate (DCR) parameter is used to measure Retainability and the
Call Setup Success Rate (CSSR) parameter measures Accessibility. According to the End-
User and Subscriber Service Charter regulations of 2016, DCR should be less than 3% and
CSSR should be greater than 98%.
In terms of the average results of the areas covered MTN and Telkom met the accessibility
target, while Vodacom and Cell C did not meet the accessibility target. Vodacom is the only
operator that met the average retainability target and all other operators (Telkom, MTN and
Telkom) did not met the average retainability target.
1.1 Introduction
The Authority conducted Quality of Service (QoS) monitoring of the GSM voice telephony
service being offered by Cell-C, MTN, Vodacom and Telkom within the KwaZulu-Natal
Province.
The KwaZulu-Natal is located in the south-east tip of South Africa bordering the Indian
Ocean. It also borders on the Eastern Cape, Free State and Mpumalanga provinces, as well
as Lesotho, Swaziland and Mozambique. It covers an area of 94 361 km² and has a
population of approximately 10 267 300. It is the third-smallest province in South Africa by
surface area but has the second largest population size after Gauteng.1
1 http://www.localgovernment.co.za/provinces/view/1/eastern-cape
3 | P a g e
Figure 1: Map of KwaZulu-Natal Province
QoS is defined as the collective effect of service performance that determines the degree of
satisfaction of a user of the service. QoS provides an indication of what the customer
experiences when using the mobile network and is evaluated in terms of Retainability and
Accessibility. Retainability is defined as the ability for a call to stay connected through to a
normal call tear-down process, without abnormally disconnecting from the cell site that
caries the call2. Accessibility is defined as the percentage of the number of times a user is
rejected due to the unavailability of system resources when attempting to place a call3.
The KPI parameters used to measure Retainability and Accessibility are Drop Call Rate (DCR)
and Call Setup Success Rate (CSSR), respectively.
The test methodology adopted is the drive test sampling methodology which provides a
snapshot view of the mobile operator’s quality of service and provides a realistic picture of
network performance from a user’s point of view. It must be noted that drive test results do
not represent the mobile service provider's overall network performance. It is based on the
specified routes, at a particular time of day when the measurements were carried out using
a particular type of handset. All the test phones were allowed to select the network freely.
1.2 Drive Test Route Plan
The KwaZulu-Natal is divided into one metropolitan municipality, ten district municipalities
and further subdivided into 50 (fifty) local municipalities. The areas within eThekwini
Metropolitan Municipality and Umgungundlovu District Municipalities were covered during
the drive tests. The areas covered include farming areas, villages, rural towns, major towns, tourist’s
2 Annexure B.2 of ETSI EG 202 057-3 3 GSM 12.04, ETS 300 615.
4 | P a g e
areas and connecting roads with the neighbouring towns and villages. All the measurements
were performed in the areas where all the operators claim to have network coverage. In
areas where Cell C and Telkom Mobile did not have own coverage they were allowed to roam
on their respective roaming partner.
Table 1: Drive Test routes
Test Type District/Metropolitan Municipality Route Name Dates
Accessibility
and
Retainability
eThekwini Metropolitan Cato Ridge 01/02/2016
16/02/2016
Umgungundlovu and Sisonke Pietermaritzburg
04/02/2016
18/02/2016
Richmond
02/02/2016
19/02/2016
Mpophomeni
05/02/2016
20/02/2016
Howick
03/02/2016
16/02/2016
2 Methodology
A drive test methodology was used to collect samples. TEMS Symphony benchmarking tool
was used to collect the data. This method provided an external indicative snapshot of an
operator’s network performance from the user’s point of view on the selected routes. The
drive test results represent a snapshot of the mobile service provider's network performance,
based on the specified routes during when the measurements were carried out.
2.1 Drive Tests Equipment
The Drive Testing methodology is basically collecting measurement data with drive test
equipment which comprises of: TEMS Symphony 7.7 with Samsung Galaxy Note 4 (SM-
N910F), laptop with Probe Controller 1.9 installed. However, drive tests only provide data
on the downlink side of the communication, it provides a perspective to the service provider
about what’s happening from a subscriber’s point of view.
2.2 Test Plan
A test plan is displayed on Table 2 below:
5 | P a g e
Table 2: Test plan and configurations
Configuration Explanation
Antennas
Tests were carried out from a moving vehicle with roof mounted
antennas. The antennas were arranged in a well-defined fixed way with
a minimum distance to each other, reducing RF-influence to an
acceptable level in an equal radio environment. The coupling loss
between two mobile handsets was min 40.5 dB4.
Band The bands tested were GSM (900 and 1800) and WCDMA (900, 2100)
Call Samples
A minimum of 120 test samples per network operator were tested
except in the areas were the services were limited on most part of the
drive test route. Test drives were planned to ensure, as far as
practicable, that the results adequately reflect the QoS perceived by
customers for the period under review.
The drive test was designed to be representative of the population
relative to the traffic of the network. Measurements were scheduled so
as to reflect accurately the traffic variations over the hours of a day,
and user's behavior.5
Call Type and
Window Call
Long calls and Short calls were used.
Voice Telephony was tested in Mobile Originating Call (MOC)
direction. The following call durations were used:
CD1: 10 seconds for call setup testing;
CD2: 120 seconds for typical tests, default call duration;
Call Window: Call Duration + 30 seconds, (for the setup and release
phases) + 30 seconds (for the minimum pause. Interval), for the
default call duration this results in 180 seconds.6
Equipment
The equipment used for testing was the TEMS Symphony 7.7 using
Samsung Galaxy Note S4 (SM-N910F), Dell Latitude with Probe
Controller 1.9
4 Section 6.3 of ETSI TS 102 250-4 V.1.1.2 (2003-10) 5 Section 4.8 of ETSI EG 202 057-3 V1.1.1 (2005-04) 6 Section 4.2.1 of ETSI TS 102 250-5 V1.2.1 (2005-05)
6 | P a g e
KPI
The measurements focused on the following network parameters:
(i) Dropped Call Rate (DCR);
(ii) Call Set-up Success Rate (CSSR)
Log files
The log files for each test case were stored in different location with
different names. The log files were recorded per network operator.
Mobile terminal
used
The test calls were terminated on each operator’s test platform or IVR
system.
Network tested Cell-C, MTN, Vodacom and Telkom.
Sim card Test sim cards provided by each operator were used during the drive
test.
Vehicle used
Toyota Hilux Double Cab equipped with drive test equipment was used.
All road traffic rules were observed during the drive test. The speed
was maintained at an average of 60km in town and built-up areas and
100km/h on highways respectively.
2.3 Computational Methodology
a) Drop-Call Rate
The measurements of dropped call rate is described by the ratio of successful originated
calls that were found to drop, to the total number of successful originated calls that were
correctly released.
The formula to calculate DCR is shown below:
Table 3: DCR computational formula
b) The Call Setup Success Rate
The Call setup Success Rate refers to the percentage of calls that are successfully set up
and terminated as a percentage of the total call attempts. CSSR excludes dropped calls
Drop Call Rate
Successful Call
Attempts S
Drop Calls D
DCR D/S * 100
7 | P a g e
or calls that experience no network condition, low speech quality calls and calls with long
set up time. The formula to calculate CSSR is shown below7:
Table 4: CSSR computational formula
Call Success Rate
Call Attempts X
Call Success Y
CSSR (Y/X)*100
7 Annexure B.1 Measurement based on networks counters (ETSI EG 202-057-3)
8 | P a g e
3 Results and Analysis
This section provides a summary of the mobile operator’s performance results based on the
drive test route and quality of service parameters.
3.1 Parameter Targets
The Drop Call Rate should be less than 3% as per the End-User and Subscriber Service
charter regulations of 2009. The Call Setup Success Rate should be greater than 95%, which
is not regulated by the End-User and Subscriber Service Charter Regulations of 2009. This
target is based on ITU recommendations E.800 and G.1000.
3.2 Results
3.2.1 Accessibility Measurement Results
The areas covered were: Cato Ridge, Pietermaritzburg, Richmond, Mpophomeni and Howick.
Table 5: Cato Ridge Accessibility Results
Table 6: Pietermaritzburg Accessibility Results
Table 7: Richmond Accessibility Results
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
453 492 499 413 414 439 455 369 867 931 954 782
419 479 476 387 389 426 438 355 808 905 914 742
34 13 23 26 25 13 17 14 59 26 40 40
92.49% 97.36% 95.39% 93.70% 93.96% 97.04% 96.26% 96.21% 93.19% 97.21% 95.81% 94.88%
PHASE 1 PHASE 2 TOTAL
Cato Ridge
Call Attempt
Call Established
Call Setup Failure
Call Setup Success Rate
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
454 452 451 442 441 424 448 433 895 876 899 875
432 403 411 424 428 371 429 419 860 774 840 843
22 49 40 18 13 53 19 14 35 102 59 32
95.15% 89.16% 91.13% 95.93% 97.05% 87.50% 95.76% 96.77% 96.09% 88.36% 93.44% 96.34%
Pietermaritzburg
PHASE 1 PHASE 2 TOTAL
Call Attempt
Call Established
Call Setup Failure
Call Setup Success Rate
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
425 586 547 303 310 406 395 315 735 992 942 618
395 562 522 286 276 380 388 303 671 942 910 589
30 24 25 17 34 26 7 12 64 50 32 29
92.94% 95.90% 95.43% 94.39% 89.03% 93.60% 98.23% 96.19% 91.29% 94.96% 96.60% 95.31%
TOTAL
Richmond
Call Attempt
Call Established
Call Setup Failure
PHASE 1 PHASE 2
Call Setup Success
Rate
9 | P a g e
Table 8: Mpophomeni Accessibility Results
Table 9: Howick Accessibility Results
Figure 2: KwaZulu-Natal Call Setup Success Rate (CSSR)
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
341 348 344 315 424 447 411 338 765 795 755 653
318 315 318 294 402 401 383 322 720 716 701 616
23 33 26 21 22 46 28 16 45 79 54 37
93.26% 90.52% 92.44% 93.33% 94.81% 89.71% 93.19% 95.27% 94.12% 90.06% 92.85% 94.33%
PHASE 1 PHASE 2 TOTAL
Mpophomeni
Call Attempt
Call Established
Call Setup Failure
Call Setup Success Rate
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
298 559 361 555 396 464 497 409 694 1023 858 964
267 525 349 543 355 443 490 378 622 968 839 921
31 34 12 12 41 21 7 31 72 55 19 43
89.60% 93.92% 96.68% 97.84% 89.65% 95.47% 98.59% 92.42% 89.63% 94.62% 97.79% 95.54%
PHASE 1 PHASE 2 TOTAL
Howick
Call Attempt
Call Established
Call Setup Failure
Call Setup Success Rate
Cato Ridge Pietermaritzburg Richmond Mpophomeni Howick
CellC 93.19% 96.09% 91.29% 94.12% 89.63%
Vodacom 97.21% 88.36% 94.96% 90.06% 94.62%
MTN 95.81% 93.44% 96.60% 92.85% 97.79%
Telkom 94.88% 96.34% 94.39% 94.33% 95.54%
Target 95% 95% 95% 95% 95%
82.00%
84.00%
86.00%
88.00%
90.00%
92.00%
94.00%
96.00%
98.00%
100.00%
Cal
l Set
up
Su
cces
s R
ate
(%)
KwaZulu-Natal Call Setup Success Rate
10 | P a g e
3.2.2 Retainability Measurements Results
The areas covered were; Cato Ridge, Pietermaritzburg, Richmond, Mpophomeni and Howick.
Table 10: Cato Ridge Retainability Results
Table 11: Pietermaritzburg Retainability Results
Table 12: Richmond Retainability Results
Table 13: Mpophomeni Retainability Results
Table 14: Howick Retainability Results
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
134 148 150 81 123 132 135 112 257 280 285 193
127 140 147 71 120 127 131 102 247 267 278 173
7 0 8 18 13 2 5 26 20 2 13 44
5.51% 0.00% 5.44% 25.35% 10.83% 1.57% 3.82% 25.49% 8.10% 0.75% 4.68% 25.43%
PHASE 1 PHASE 2 TOTAL
Cato Ridge
Call AttemptCall EstablishedDropped Call
Drop Call Rate
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
135 137 140 128 131 130 132 119 266 267 272 247
127 129 137 120 121 123 128 112 248 252 265 232
12 2 5 15 11 2 5 15 23 4 10 30
9.45% 1.55% 3.65% 12.50% 9.09% 1.63% 3.91% 13.39% 9.27% 1.59% 3.77% 12.93%
Pietermaritzburg
PHASE 1 PHASE 2 TOTAL
Call AttemptCall EstablishedDropped Call
Drop Call Rate
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
147 179 160 107 93 118 119 79 240 297 279 186
131 174 153 102 84 114 117 74 215 288 270 176
7 3 5 13 7 5 8 20 14 8 13 33
5.34% 1.72% 3.27% 12.75% 8.33% 4.39% 6.84% 27.03% 6.51% 2.78% 4.81% 18.75%Drop Call Rate
TOTAL
Richmond
Call AttemptCall EstablishedDropped Call
PHASE 1 PHASE 2
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
104 105 105 86 128 136 126 107 232 241 231 193
97 101 102 85 115 127 121 101 212 228 223 186
11 4 6 16 7 1 7 22 18 5 13 38
11.34% 3.96% 5.88% 18.82% 6.09% 0.79% 5.79% 21.78% 8.49% 2.19% 5.83% 20.43%
PHASE 1 PHASE 2 TOTAL
Mpophomeni
Call AttemptCall EstablishedDropped Call
Drop Call Rate
Cell CVodaCom-
SAMTN-SA Telkom Cell C
VodaCom-SA
MTN-SA Telkom Cell CVodaCom-
SAMTN-SA Telkom
148 160 173 123 124 139 147 89 272 299 320 212
133 152 170 99 111 135 142 79 244 287 312 178
15 11 3 20 19 7 2 21 34 18 5 41
11.28% 7.24% 1.76% 20.20% 17.12% 5.19% 1.41% 26.58% 13.93% 6.27% 1.60% 23.03%
PHASE 1 PHASE 2 TOTAL
Howick
Call Attempt
Call Established
Dropped Call
Drop Call Rate
11 | P a g e
Figure 3: KwaZulu-Natal Drop Call Rate (DCR)
3.3 KwaZulu-Natal Average Measurements Results
3.3.1 Average Accessibility measurement data results
The result below shows the average combination of the total Accessibility results in the tested areas (Cato
Ridge, Pietermaritzburg, Richmond, Mpophomeni and Howick).
Table 15: Average Accessibility Measurement Data
Province Network Operator CSSR Target (%)
KwaZulu-Natal Vodacom 93.04% 95%
MTN 95.30% 95%
Cell-C 92.86% 95%
Telkom 95.07% 95%
Figure 4: KwaZulu-Natal Average Call Setup Success Rate
Cato Ridge Pietermaritzburg Richmond Mpophomeni Howick
CellC 8.10% 9.27% 6.51% 8.49% 13.93%
Vodacom 0.75% 1.59% 2.78% 2.19% 6.27%
MTN 4.68% 3.77% 4.81% 5.83% 1.60%
Telkom 25.43% 12.93% 18.75% 20.43% 23.03%
Target 3% 3% 3% 3% 3%
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%C
all D
rop
Rat
e (%
)KwaZulu-Natal Call Drop Rate
92%
92%
93%
93%
94%
94%
95%
95%
96%
Target >= Vodacom MTN Cell-C Telkom
Average CSSR 95% 93.04% 95.30% 92.86% 95.07%
Ave
rage
CSS
R (
%)
KwaZulu-Natal Average Call Setup Success Rate
12 | P a g e
3.3.1.1 Analysis of the Average Accessibility Results: KwaZulu-Natal
Based on the average Accessibility results, MTN and Telkom met the average accessibility target, while
Vodacom and Cell-C did not met the target.
3.3.2 Average Retainability measurement data results for KwaZulu-Natal Province
The results below show the average combination of the total Retainability results in tested areas (Cato
Ridge, Pietermaritzburg, Richmond, Mpophomeni and Howick).
Table 16: Average Retainability Results
Province Network Operator DCR Target (%)
KwaZulu-Natal Vodacom 1.83% 3%
MTN 4.77% 3%
Cell-C 8.09% 3%
Telkom 19.39% 3%
Figure 5: KwaZulu-Natal: Average Call Drop Rate
3.3.2.1 Analysis of the Results
Based on the average Retainability results, Vodacom met the average Retainability target.
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
Target <= Vodacom MTN Cell-C Telkom
DCR 3% 1.83% 4.77% 8.09% 19.39%
Ave
rage
DC
R (
%)
KwaZulu-Natal: Average Call Drop Rate
13 | P a g e
3.3.3 Serving Technology Distribution
Figure 6: KwaZulu-Natal: Serving Technology Distribution
Vodacom, MTN and Telkom voice services were on GSM technology more than 50% of the time, and Cell C voice service was on WCDMA for more than 50% of the time along the
drive test route (See Appendix A: Section 5.2).
4 Conclusion
The QoS monitoring report in KwaZulu-Natal Province focused on the following KPIs,
Call Set-up Success Rate (CSSR) and Drop Call Rate (DCR) in Cato Ridge,
Pietermaritzburg, Richmond, Mpophomeni and Howick.
Table 17: KwaZulu-Natal Cape Province - Results Summary
Area Tested Network Operator
Results
Call Setup Success Rate
Drop Call Rate
Cato Ridge
Vodacom Passed Passed
MTN Passed Failed
Cell-C Failed Failed
Telkom Failed Failed
Pietermaritzburg
Vodacom Failed Passed
MTN Failed Failed
Cell-C Passed Failed
Telkom Passed Failed
14 | P a g e
Richmond
Vodacom Failed Passed
MTN Passed Failed
Cell-C Failed Failed
Telkom Failed Failed
Mpophomeni
Vodacom Failed Passed
MTN Failed Failed
Cell-C Failed Failed
Telkom Failed Failed
Howick
Vodacom Failed Passed
MTN Passed Failed
Cell-C Failed Failed
Telkom Passed Failed
From the results above, it can be deduced that in terms of the two key parameters, Cell-C
achieved the Accessibility target in Pietermaritzburg only but failed the Retainability test in
all tested areas. Vodacom met the Accessibility target in Cato Ridge only and Retainability
test in all tested areas. MTN met the Accessibility test in Cato Ridge, Richmond and Howick,
but failed the Retainability in all tested areas.
All operators failed the Accessibility test in Mpophomeni. MTN, Cell-C and Telkom failed the
Retainability test in all tested areas.
Based on the average results, the Retainability KPIs, which are expressed as Average Drop
call rate (ADCR), Vodacom met the target. The Accessibility KPI, expressed as Average Call
setup success rate (ACSSR), MTN and Telkom met the average accessibility target but
Vodacom and Cell-C did not met the target.
15 | P a g e
5 Appendix A (KwaZulu-Natal coverage maps)
5.1 Signal Levels for Serving Cells
5.1.1 Vodacom
GSM UMTS
Figure 7: KwaZulu-Natal Vodacom GSM and UMTS Signal Levels
5.1.2 MTN
GSM UMTS
Figure 8: KwaZulu-Natal MTN GSM and UMTS Signal Levels
16 | P a g e
5.1.3 Cell-C
GSM UMTS
Figure 9: KwaZulu-Natal Cell C GSM and UMTS Signal Levels
5.1.4 Telkom
GSM UMTS
Figure 10: KwaZulu-Natal Telkom GSM and UMTS Signal Levels
17 | P a g e
5.2 Serving Technology Details
5.2.1 Vodacom
Vodacom MTN
Cell-C Telkom
Figure 11: KwaZulu-Natal Serving Technology Plots
18 | P a g e