Handling kpi's in wcdma

Handling KPIs in WCDMAApproach, KPI reduction, tips&tricks

Slide title minimum 32 pt

(32 pt makes 2 rows

Text and bullet level 1 minimum 24 pt

Bullets level 2-5minimum 20 pt

!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ–—‘’‚“”„†‡•…‰‹›⁄€™−≤≥fifl

ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș

ΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏ

ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№

Do not add objects or text in the footer area

Handling KPIs in WCDMA | Ericsson Internal | EAB-08:046794 Uen, Rev PA26 | 2010-10-01 |

Content

› KPIs - where is the problem?

› KPI handling strategy

› Tips&Tricks on KPIs

› KPI Library

› KPI reduction example


(32 pt makes 2 rows









Challenges around KPIs

It’s not just about KPI values it’s about acceptance, risk management and supporting a strong cash flow.

Ultimate question: Can we reach 99%....Ultimate question: Can we reach 99%....

What to measure

Where to measureExclusions

Entry & Exit criteria's

Statistical confidence

KPI Values

Statistics

Drive test

KPI definitions

Acceptance process Multi-vendor environment

World of variables.......

Cost and risks


(32 pt makes 2 rows









Why are the KPIs now so impWhy are the KPIs now so impoortant? Isn’t that just technical question?rtant? Isn’t that just technical question?

It’s commercial AND technical – it cannot be treated separately:It’s commercial AND technical – it cannot be treated separately:

WARNING: Millions of SEK in contract value may depend on engineer saying yes or no to certain KPI

Penalties related to KPIs Delayed payments Revenue losses Increased and uncontrolled cost Unclear services scope Weakest link in the chain – HW,

SW, services.

Number of KPIs KPI targets Formulas Number of samples Acceptance methods Exclusions/ non Ericsson

COMMERCIALCOMMERCIAL TECHNICALTECHNICAL


(32 pt makes 2 rows









KPIs - Operators point of view

› 3G – a world of new opportunities...

› Lack of knowledge with new 3G technology

› Expect same 3G performance from day one as their mature 10 year old 2G network...

› Faster than ADSL...

The Good... Open for constructive discussion

Focus on experience

Willing to discuss with Ericsson

The Bad... Want all the “promises” of 3G

Include all possible KPIs (play it safe…)

& the Ugly same as “Bad” but with the intent of maximizing penalties

“All” KPIs – impossible targets to meet


(32 pt makes 2 rows









Content




› KPI Library



(32 pt makes 2 rows









KPI handling strategy

Understanding KPI

Handling KPI

Negotiating KPI

• Providing the logic behind classification and reduction process

• Focus on reducing time and effort for acceptance process

• Focus on controllable acceptance process together with KPIs and levels

• Few contractual QoS KPIs: using ITU-T concept and the KPI tree logic

• Understand Commercial and Technical link for KPIs

• Understand Risk, Acceptance Procedure, Payment Terms related to KPI

• Reduce the risk – KPI reduction process (incl. number of KPIs and values)

• Linking services and cost/effort for acceptance with KPIs

• Exclude aspects outside Ericsson scope and create confidence intervals


(32 pt makes 2 rows










Understanding KPI


(32 pt makes 2 rows









AcceptanceRisk & Services

Understanding KPIs - variables

KPIs can differ significantly in risk, effort to measure and services needed

KPIs and targets

KPI definitions (start & stop triggers, entry exit criteria)KPI ValuesKPI Classification

Acceptance process (measurement area, method)Link to payment termsNumber of KPIs in contract

Services related to achieve KPI levelsRisk assessment on KPIs and levelsUnderstanding the cost and effort to perform KPI based acceptance


(32 pt makes 2 rows









› WHAT : Ericsson KPI strategy is based on having a FEW KEY Contractual

QoS KPIs

– Recommended KPI scope (6 KPI)

– Optional KPI scope (10-15 KPI) – space for negotiations

› WHERE: Define Measurement areas and acceptance split

– Golden Cluster (Recommended and Optional KPI)

– Other Clusters (Recommended mostly and some Optional KPI)

› HOW: Measurements methods, tools, duration and exclusions

– Drive tests ; Static spots; Statistics

– Define statistical confidence intervals

– Define exclusions (not under Ericsson control)

Understanding KPIs - approach


(32 pt makes 2 rows










Handling KPI


(32 pt makes 2 rows









Propose Ericsson KPI strategyPropose Ericsson KPI strategy• Recommended KPIs

• Ericsson Acceptance Procedure

• Ericsson Recommended Values

• Ericsson preferred services

• Ericsson payment terms

Handling KPIs - scenariosTwo main strategies on handling KPIs

Depending on operator KPI requirement

Many K

PIs

Get closer to Ericsson KPI strategyGet closer to Ericsson KPI strategy• Perform KPI reduction process

• Identify risks (commercial and technical)

• Define Acceptance Procedure

• Negotiate KPIs

• Services based on KPI list and values

No

KPIs


(32 pt makes 2 rows









Ericsson KPI Strategy

› Alligned with ITU-T and ETSI Specifications

› Same strategy applicable for any technology (GSM, WCDMA, LTE..)

› All KPIs classified into Quality of Service and Network Performance

› Quality of Service (QoS) KPIs = Contractual KPIs

KPI Strategy: Few KEY Contractual QoS KPI


(32 pt makes 2 rows









Availablity Performance

Ericsson KPI StrategyQuality of Service by ITU-T

Quality of ServiceQuality of Service (QoS)(QoS)

Service Integrity

Service Retainability

Service Accessibilty

QoSQoS

Network PerformanceNetwork Performance

Resources & Facilities

ReliabilityPerformance

MaintainabiltyPerformance

MaintenanceSupport

PerformanceTransmissionPerformance

Design& Dimensioning

Features & Functionalities

Performance Monitoring

(Nodes, System)

O&M(Node, system

Availability)SLAs

Interfaces, Transmission and Transport

Ericsson interpretation (red) of ITU-T


(32 pt makes 2 rows










Ericsson KPI StrategyMain KPI Classification


Service Integrity



QoSQoS





MaintenanceSupport


Design& Dimensioning

Features & Functionalities

Performance Monitoring

(Nodes, System)

O&M(Node, system

Availability)SLAs

Interfaces, Transmission and Transport

QoS – Contractual KPIsQoS – Contractual KPIs

• Recommended & Optional Contract KPIsRecommended & Optional Contract KPIs• End User Service related KPIs End User Service related KPIs


• Recommended & Optional Contract KPIsRecommended & Optional Contract KPIs• End User Service related KPIs End User Service related KPIs

Network Performance KPIsNetwork Performance KPIs

• Not Recommended as contract KPIs Not Recommended as contract KPIs • Network related KPIs (Design, Features, O&M..)Network related KPIs (Design, Features, O&M..)


(32 pt makes 2 rows










Ericsson KPI StrategyKPI Interaction (KPI tree)


Service Integrity



QoSQoS





MaintenanceSupport


RecommendedRecommended

OptionalOptional

Not Not RecommendedRecommended




(32 pt makes 2 rows










Ericsson KPI Strategy - exampleKPI Interaction (KPI tree)




QoSQoS





TransmissionPerformance

Call Completion Success Rate (Speech)

Call Setup Success Rate

Drop Call Rate

RRC Setup Success Rate

RecommendedRecommended

OptionalOptional

Not Not RecommendedRecommended



RAB SetupSuccess Rate

Soft HandoverSuccess Rate

Softer Handover Success Rate


(32 pt makes 2 rows









Contractual Classification

› RECOMMENDED RECOMMENDED – Main QoS KPIs for most used services

like (Voice, Video, Packet services..)

› OPTIONAL OPTIONAL – QoS KPIs that can be replaced by Recommended ones

› NOT RECOMMENDEDNOT RECOMMENDED– Network Performance KPIs

– Other services QoS KPIs (like CS57 completion)

– Other “risky” KPIs for Ericsson (like MOS)


(32 pt makes 2 rows









Ericsson KPI strategyWCDMA QoS KPI

› Main KPI Strategy: Few Contractual QoS KPIs : 6 Recommended KPIs

› If needed combination of Recommended and Optional can be used in Golden

Cluster

Goal: Minimize number of contractual KPIs

No Recommended KPI Network/User Impact

1 Call Completion Success Rate Speech (CCSR Speech) High

2 Call Completion Success Rate CS64 (CCSR Video) High

3 Call Completion Success Rate PS (CCSR PS) High

4 PS Interactive R99 User Throughput (User Throughput R99) High

5 PS Interactive HSDPA User Throughput (User Throughput HSDPA) High

6 PS Interactive EUL User Throughput (User Throughput EUL) High


(32 pt makes 2 rows









› Almost all Optional KPIs can be derived from Recommended KPIs

Optional KPIs give flexibility and “kills” doubts that Ericsson is running away from commitments

Ericsson KPI strategyWCDMA QoS KPI

No Optional KPI Network/User Impact

1 Call Setup Success Rate Speech (CSSR Speech) High

2 Drop Call Rate Speech (DCR Speech) High

3 Call Setup Success Rate CS64 (CSSR CS64) High

4 Drop Call Rate CS64 (DCR CS64) High

5 Call Setup Success Rate PS (CSSR PS) High

6 Drop Call Rate PS (DCR PS) High

7 Call Setup Time Speech (CST Speech) Medium

8 BLER Quality Speech Uplink (BLER Speech) Medium

9 Call Setup Time CS64 (CST CS64) Medium

10 BLER Quality CS64 Uplink (BLER CS64) Medium

11 Call Setup Success Rate PS R99 (CSSR PS R99) Medium

12 Call Setup Success Rate PS HSDPA (CSSR PS HSDPA) Medium

13 Call Setup Success Rate PS EUL (CSSR PS EUL) Medium

14 Drop Call Rate PS R99 (DCR PS R99) Medium

15 Drop Call Rate PS HSDPA (DCR PS HSDPA) Medium

16 Drop Call Rate PS EUL (DCR PS EUL) Medium

17 Minutes per Drop PS Interactive (MPD PS) Medium

18 IRAT Handover Success Rate Speech (IRAT Speech) Medium

19 IRAT Cell Change Success Rate for PS Interactive (IRATCC PS) Medium

20 BLER Quality PS Uplink (BLER PS) Medium

21 PDP Context Activation Time PS (PDP Act Time PS) Medium

22 PS Interactive R99 User Throughput UL (User Throughput R99 UL) Medium


(32 pt makes 2 rows









Handling KPIs - Main KPI classification

› Recommendation: Risk analysis (contractual)– Recommended – in contract

– Optional – medium risk, can be replaced by recommended KPI

– Non recommended – KPIs with large risk

› Network / User Impact : Relevance and Importance

– High– Medium – Low


(32 pt makes 2 rows









KPI reduction process in 4 steps

Process

Output

STEP 1:STEP 1: KPI classification (QoS and Network Performance) & risk analysis

STEP 2:STEP 2: KPI relevance/importance separation

STEP 3:STEP 3: KPI selection per acceptance area

STEP 4:STEP 4: Values and effort analysis

Risk assessment

Reduced KPI list for negotiations with reviewed values

Related services per different scenario

Acceptance scope and effort estimation


(32 pt makes 2 rows









OUTPUTOUTPUTPROCESSPROCESS

STEP 2STEP 2RFQRFQ STEP 1STEP 1 STEP 3STEP 3

Unclassified list of K

PIs

List of KPIs

Classification and Risk

Re

co

mm

-en

de

dO

ptio

na

lN

ot

Re

co

mm

-en

de

d

Relevance separation

Hig

hM

ediu

mL

ow

Golden Cluster

OtherClustersPAC,FAC

Network (Counters)

LAB Static

(Once)

STEP 4STEP 4

Rec

om

me

nd

ed

Op

tion

al

Hig

hH

igh

& M

ed

ium

Hig

h

Hig

h

Me

diu

m &

Lo

w

Ac

ce

pta

nc

eP

roc

ed

ure

& D

ura

tion

Related

Services

Try to avoidTry to avoid Try to avoid or minimize measurement areaTry to avoid or minimize measurement area

98%+/-2%99%+/-1%

.

.

.97%+/-2%

.

.

.96%+/-1%.

Risk

An

alisys

KPI Values&

ConfidenceIntervals

Red

uced

KP

Is List


(32 pt makes 2 rows










Negotiating KPI


(32 pt makes 2 rows









Negotiations – Setting goals

Our goalsOur goals1. Fully defined and controllable acceptance process and achievable set of few main KPIs

2. To understand the risks to achieve the KPIs and how they relate to the solution offered

3. Full understanding of services needed to achieve the KPIs and ensuring payment for them

4. Full control of cash flow (when and how) – more payment milestones

5. To show that we have experience in KPIs and know what is needed to give best result

Operator’sOperator’s goal goalss1. Sometimes, simply to understand how WCDMA works

2. To delay the payments and to reduce the price

3. All the KPIs linked to one payment milestone (usually not achievable or delayed)

4. To have separate discussions on KPIs and services

5. Maximize the number of KPIs (shift the performance risks to vendor), normalized offers and lowest price


(32 pt makes 2 rows









Negotiations – how to do it

OpeningOpening

– Try to avoid the feeling that we are running away from something from the very beginning

– Move out from “defensive mode” –show experience and gain respect

Main message : Not all the KPIs are the same!Main message : Not all the KPIs are the same! – Preparing for the classifications – introduce the Golden Cluster concept

– Start with: “According to ITU-T E800. there are three main groups of KPI…”

– Continue gently by saying that there are Radio and non Radio related KPIs

– Example : Drop call rate and Round Trip Time

– Present technical classification with understandable logic:

› KPIs with different impact to network performance/user perception – Example: Drop call rate and Call setup time

– Show three levels of classification (High,Medium, Low) .....

to lay foundation for for the reduction process

Closing : With Accountability comes ResponsibilityClosing : With Accountability comes Responsibility– Show the Link between KPI commitments with services

– Show how Service scope and effort increase if KPI target is tougher

– Show how target for the same KPI can be different if Drive Test or Stats are used


(32 pt makes 2 rows









Negotiations – values&effort

› Discuss Values after you have agreed on KPIs:Discuss Values after you have agreed on KPIs:– Use the KPI library– Do not commit to Best in Class values for new network– Propose lower targets if necessary– Note: The same KPI can have different value if measured with DT instead of Stats

› Define confidence IntervalsDefine confidence Intervals– Confidence intervals are linked with number of samples– Conf intervals are mathematical calculation and are not related to network performance!!– When setting targets always use confidence intervals

Example:Example: Call Setup Success Rate : 98%

With 200 samples (60 sec call or 3.5 hours of Drive Test) – Interval is 98% +/-2%

With 700 samples (60 sec call or 12 hours of Drive Test) – Interval is 98%+/-1%

Result:Result: With 200 samples, and a result of 96% - it’s in the target range and the KPI is met!

› Acceptance Method:Acceptance Method:– Drive test based acceptance : easier to achieve KPI level, but more effort to measure

– Statistics based acceptance : harder to achieve KPI levels, but very easy to measure


(32 pt makes 2 rows









Negotiations – “good to know”

› ““Show some steel”Show some steel”– Usually list of KPIs is copied from somewhere else:

– Find a tricky KPI and ask them why they want to measure that

– Let them know how many hours of drive test is necessary to drive single KPI

– Show them the “time and effort” just for the measurements – are they ready to pay for it?

› Classification may be more important than reduction! Why?Classification may be more important than reduction! Why?– Giving the possibility to avoid a “All or nothing” scenario

– Example : Conditional acceptance if we meet 90% High; 70% Medium and 50% Low

– Contractual KPIs (main KPIs) are not Optimization KPIs (many KPIs)

– Example:

– 1 contractual KPI: Call Drop Rate

– 3 Optimization KPIs: Drop due to SHO; Drop due to UL Sync, Drop due to missing neighbour

› What to AvoidWhat to Avoid

– Deep technical arguing about the KPIs with the customer (example: engineer eager to understand how WCDMA

works) without any decision making power

– Best in Class targets for newly deployed network, without optimization services

– Numerous measurements in Acceptance: redriving will not make network perform better


(32 pt makes 2 rows









Content




› KPI Library



(32 pt makes 2 rows









Tips&Tricks - how to answer tricky questions

› Q: You are responsible for the KPIs but we’ll do the design and you may review it› A: OK, but we do not propose solutions to problems found in the design review (tilts, more sites), we will point out areas

where KPIs will not be applicable

› Q: All other vendors said yes to all KPIs, why you want to discuss it?› A: Based on our experience and according to the ITU-T recommendations, not all the KPIs have equal importance to

network performance and relevance to end user perspective, so they shouldn’t be treated in the same way. Also, some of the KPIs are not influenced by radio conditions, so why should they be measured in every cluster drive test.. Example (relevance) : Drop call rate (very relevant and important) and Intra MSC interruption time 200ms (not relevant and less important)Example (non radio related) : Round trip time – we can measure it once

On the other hand numerous drive tests will just increase time and effort needed for the acceptance and will not improve the network performance.

› Q: What is the point of Golden Cluster Acceptance?› A: The point of Golden Cluster Acceptance is to show that Ericsson equipment, features and functionalities are working in

a field environment according to the wanted levels.

After we have proven this, there is no need to repeat all the measurements again in every cluster, but just to focus on KPIs related to the different radio environment and mobility.


(32 pt makes 2 rows









› Q: We want to have Drop call or Call setup success rate equal or better than in GSM.› A: It’s hard to expect that newly deployed network (without all the sites in operation) will work like already existing, fully built,

dense and well optimized network.

To get values above 99.x% percent, not only tuning but detailed optimization is needed (the same like in GSM). So let’s have

different targets for acceptance after tuning and higher targets for acceptance after optimization .

Also, we should be aware that since there is no traffic, data will be collected with drive test, without the possibility to achieve

even similar amount of samples like with statistics in GSM, meaning if we want to achieve 0.3% of drop call we will have to

collect minimum 1300 samples, which is 22 hours of drive test for single KPI. This is very time and effort consuming. Are you

ready to pay for this massive measurements?



(32 pt makes 2 rows









› Q: Your processes are OK, but still we want to verify all the KPIs in any place we choose

› A: Of course that you may verify it, but for the acceptance purpose we will have to agree on the

drive route, mutually and in advance. During the drive route discussions you can highlight which

areas and specific places you would like to verify.

› Q: Let’s talk about KPIs and not optimization– that’s different topic for separate discussion

› A: Almost in all the cases KPIs and their values are directly linked with various optimization tasks.

KPIs are verifying the performance that can be improved with optimization services. So if we discuss

KPIs and targets we have to discuss what type of services is necessary to achieve those KPIs.

On the other hand if you are more interested into different WCDMA functionalities, we can have a

separate workshop on that topic.



(32 pt makes 2 rows










› Q: What do you mean by confidence Intervals? Can you achieve 99% or not?

› A: Sure we can. But the question is how to prove it. Any kind of measurement targets are given by the certain range

and not by a concrete number. There are two reasons for this (both of a mathematical nature): First is called standard

deviation of measurements and the second one is the number of collected samples. The bigger the number of samples is,

the smaller the confidence interval is.

Certain number of samples can give certain proximity (confidence interval) to the wanted target:

Confidence intervals are just statistical calculation of needed samples and are not related to the network performance!


(32 pt makes 2 rows









Tips&Tricks - Values

› Choosing the right KPI value – what to considerChoosing the right KPI value – what to consider

– Is this theoretical / feature value or field measurement value

– What will be the measurement method (Drive test or statistics)

– How many samples is necessary to achieve it

– How close this value is to E/// Best in Class or Average

› DifDiffferences between the same KPI - drive test vs. statisticserences between the same KPI - drive test vs. statisticsExample: R99 throughput

– Drive test value will give about 250 kbps, but Statistics will give 20 kbps!!

› DifDiffferences between the same KPI – Best in Class vs. averageerences between the same KPI – Best in Class vs. averageExample: Call Completion Success Rate for Video

– Best in Class Value : 98.7%

– Average value : 94.6%


(32 pt makes 2 rows









Tips&Tricks - Drive Test vs Stats

› DRIVE TEST - DRIVE TEST - easier to achieve targets, harder to measure

– Controlled calling sequence, equipment and timing

– Locations determined by drive route – sample within coverage area

– Covers only “untypical subscriber” only outdoors and driving

– Costly and time consuming to perform in all areas.

– Limited statistical significance

– Some KPI can only be measured by drive test (active measurement) e.g. ping times

› STATISTICS -STATISTICS - easier to measure, harder to achieve targets

– Covers all areas where subscribers call, does not register those outside coverage area

– Will never give 100% success rate, even if tuning is ‘perfect’

– Network outages and disturbances will influence results negatively

– Has statistical significance once traffic has become sufficient

– Relatively cheap to measure, but requires careful analysis

– Does not cover all KPIs


(32 pt makes 2 rows









Content




› KPI Library



(32 pt makes 2 rows









The KPI Library

› One pager KPI library

› Contractual and Technical Information

› Acceptance information

› Performance Related services

› Targets and Implications:– Ericsson Controlled Environment (BNET)

– Best In Class (NetQB)

– Average (NetQB)

– Initially Tuned (field experiences)

› BUGS and BNET Approved

INTERNAL DOCUMENT!!INTERNAL DOCUMENT!!


(32 pt makes 2 rows









The KPI Library

x.y.z. KPIX: ReccomendationY: Network/User ImpactZ: Area

Recommendation

1. Recommended2. Optional3. Not Recommended

Area1. Completion2. Accessibility3. Retainability4. Integrity5. Mobility6. Other


(32 pt makes 2 rows









The KPI Library

High level formula

Used in Drive tests

Detailed formula

Used in Statistics


(32 pt makes 2 rows









The KPI Library

Network/User Impact

1. High2. Medium3. Low

Contractual

Used in risk analysisUsed in reduction process


(32 pt makes 2 rows









The KPI Library

Acceptance definition

Where and how to measure KPIExact Trigger points

Effort analysis

Exact duration of acceptanceConfidence Intervals definition


(32 pt makes 2 rows









The KPI Library

Targets

E/// controlled based on inputs from BNETBiC and Average: NetQB BUGSPre launched : Field experiences BUGS

Services

Design and performance improvement related services as prerequisites for KPI target fullfillment


(32 pt makes 2 rows









Content




› KPI Library



(32 pt makes 2 rows









KPI reduction process in 4 steps

Process

Output

STEP 1:STEP 1: KPI classification & risk analysis

STEP 2:STEP 2: KPI relevance/importance separation

STEP 3:STEP 3: KPI selection per acceptance area

STEP 4:STEP 4: Values and effort analysis

Risk assessment

Reduced KPI list for negotiations with reviewed values

Related services per different scenario

Acceptance scope and effort estimation


(32 pt makes 2 rows









KPI example: 23 KPIs

KPI Description

Time

Criteria (1)

Success Criteria

CS Video Telephony Call Setup Elapsed Time < 10.0 s Average

CS Video Telephony Call Setup Success Rate > 98%

CS Voice 3G-2G Hard HO Success Rate (including inter-MSC HO) > 98.5%

CS Voice Call Completion Success Rate >=98%

CS Voice Call Dropped Call Rate (Cell average) including RF and HO Failures < 5%

CS Voice Call Dropped Call Rate (Cluster average) including RF and HO Failures < 1.3%CS Voice Call Setup Elapsed Time < 8.0 s Average

CS Voice Call Setup Success Rate > 99%

CS Voice listening qualityEqual or better

than GSM

Long Call Video Telephony > 15 min

Long Session PS > 60 min

PS 64/384 DCH/DCH ping round trip delay 100 Byte packet < 150 ms > 95%

PS 64/384 PDP Context Activation Success Rate including Attach > 99%

PS Session Completion Success Rate Downlink (FTP, >5Mbyte file) > 98%

PS Throughput Downlink (FTP, >5Mbyte mean user data rate) > 200 kbps > 90%

PS UMTS to GPRS Cell-Reselection Data Interruption < 9 s Average

Softer HO Success Rate > 97%

Transparent CS Data UMTS-UMTS Call Setup Success Rate > 99%

Voice Call Setup Elapsed Time PSTN-UMTS MTEqual or Less

than GSM Average

Voice Call Setup Elapsed Time PSTN-UMTS MT with PDP context active < 6.0 s > 95%

Voice Call Setup Elapsed Time UMTS-PSTN MOEqual or Less

than GSM Average

Voice UMTS to GSM Handover intra-MSC interruption time < 300 ms > 95%

Voice UMTS-PSTN MT Call Setup Success Rate > 98%

› Unclassified 23 KPIs in contract to be measured everywhere...


(32 pt makes 2 rows









Step 1: KPI Classification&Risk Analysis

Try to Avoid

Recommendation KPI Description

Time

Criteria (1)

Success Criteria Risk

1 CS Voice Call Completion Success Rate >=98% High

1 PS Session Completion Success Rate Downlink (FTP, >5Mbyte file) > 98% High

1 PS Throughput Downlink (FTP, >5Mbyte mean user data rate) > 200 kbps > 90% Medium

1 CS Voice 3G-2G Hard HO Success Rate (including inter-MSC HO) > 98.5% Medium

2 CS Video Telephony Call Setup Success Rate > 98% Medium

2 CS Voice Call Dropped Call Rate (Cluster average) including RF and HO Failures < 1.3% Medium

2 CS Voice Call Setup Success Rate > 99% Medium

2 CS Video Telephony Call Setup Elapsed Time < 10.0 s Average Low

2 CS Voice Call Setup Elapsed Time < 8.0 s Average Low

2 PS 64/384 PDP Context Activation Success Rate including Attach > 99% High

2 Voice Call Setup Elapsed Time UMTS-PSTN MOEqual or Less

than GSM Average High

2 Voice UMTS-PSTN MT Call Setup Success Rate > 98% Low

2 CS Voice Call Dropped Call Rate (Cell average) including RF and HO Failures < 5% Low

2 Long Call Video Telephony > 15 min Low

2 Long Session PS > 60 min High

2 PS 64/384 DCH/DCH ping round trip delay 100 Byte packet < 150 ms > 95% High

2 PS UMTS to GPRS Cell-Reselection Data Interruption < 9 s Average High

2 Softer HO Success Rate > 97% Low

3 Voice Call Setup Elapsed Time PSTN-UMTS MT with PDP context active < 6.0 s > 95% Medium

3 Transparent CS Data UMTS-UMTS Call Setup Success Rate > 99% High

3 Voice Call Setup Elapsed Time PSTN-UMTS MTEqual or Less

than GSM Average High

3 CS Voice listening qualityEqual or better

than GSM High

3 Voice UMTS to GSM Handover intra-MSC interruption time < 300 ms > 95% High

Re

c.

Op

tion

alN

on

rec

om

me

nd

ed


(32 pt makes 2 rows









Recommendation Relevance Area KPI Description

Time

Criteria (1)

Success Criteria

1 1 1 CS Voice Call Completion Success Rate >=98%

1 1 1 PS Session Completion Success Rate Downlink (FTP, >5Mbyte file) > 98%

1 1 4 PS Throughput Downlink (FTP, >5Mbyte mean user data rate) > 200 kbps > 90%

1 1 5 CS Voice 3G-2G Hard HO Success Rate (including inter-MSC HO) > 98.5%

2 1 2 CS Video Telephony Call Setup Success Rate > 98%

2 1 2 CS Voice Call Dropped Call Rate (Cluster average) including RF and HO Failures < 1.3%

2 1 2 CS Voice Call Setup Success Rate > 99%

2 2 2 CS Video Telephony Call Setup Elapsed Time < 10.0 s Average

2 2 2 CS Voice Call Setup Elapsed Time < 8.0 s Average

2 2 2 PS 64/384 PDP Context Activation Success Rate including Attach > 99%

2 2 3 Long Call Video Telephony > 15 min

2 2 3 Long Session PS > 60 min

2 3 2 Voice Call Setup Elapsed Time UMTS-PSTN MOEqual or Less

than GSM Average

2 3 2 Voice UMTS-PSTN MT Call Setup Success Rate > 98%

2 3 3 CS Voice Call Dropped Call Rate (Cell average) including RF and HO Failures < 5%

2 3 4 PS 64/384 DCH/DCH ping round trip delay 100 Byte packet < 150 ms > 95%

2 3 5 PS UMTS to GPRS Cell-Reselection Data Interruption < 9 s Average

2 3 5 Softer HO Success Rate > 97%

Step 2: Relevance separationH

igh

Med

ium

Lo

w

Try to avoid or minimize measurement area


(32 pt makes 2 rows









Area KPI Description LAB/StaticGolden Cluster

Other Cluster

Network (Counters)

Completion CS Voice Call Completion Success Rate x x xCompletion PS Session Completion Success Rate Downlink (FTP, >5Mbyte file) x x x

Integrity PS Throughput Downlink (FTP, >5Mbyte mean user data rate) x x xMobility CS Voice 3G-2G Hard HO Success Rate (including inter-MSC HO) x x x

Accessibility CS Video Telephony Call Setup Success Rate x x xRetainability CS Voice Call Dropped Call Rate (Cluster average) including RF and HO Failures x x xAccessibility CS Voice Call Setup Success Rate x x xAccessibility CS Video Telephony Call Setup Elapsed Time xAccessibility CS Voice Call Setup Elapsed Time xAccessibility PS 64/384 PDP Context Activation Success Rate including Attach x

Retainability Long Call Video Telephony xRetainability Long Session PS x

Step 3 : Measurement Area

› Spreading the Risk :Spreading the Risk :– Golden Cluster 10 KPI

– Other Cluster 7 KPI

– Statistics (Counters) 7 KPI

– LAB/Static 2 KPIs

10 KPI10 KPI 7 KPI7 KPI2 KPI2 KPI 7 KPI7 KPI

Reduction: from 23 to 12 Contractual KPIsReduction: from 23 to 12 Contractual KPIs


(32 pt makes 2 rows









Step4: Values and effort (based on Operator’s values)

9 days of drive tests just to collect minimum amount of samples

Area KPI Description TargetCall

DurationConfidence

IntervalNumber of Samples

Drive test

hoursDrive tests

Completion CS Voice Call Completion Success Rate >=98% 120 2% 200 7 1

Completion PS Session Completion Success Rate Downlink (FTP, >5Mbyte file) > 98% 120 2% 200 7 1

Integrity PS Throughput Downlink (FTP, >5Mbyte mean user data rate) > 200 kbps 120 200 7 1

Mobility CS Voice 3G-2G Hard HO Success Rate (including inter-MSC HO) > 98.5% 360 1.5% 260 26 2

Accessibility CS Video Telephony Call Setup Success Rate > 98% 30 1% 700 7 3

Retainability CS Voice Call Dropped Call Rate (Cluster average) including RF and HO Failures < 1.3% 120 1% 450 15 4

Accessibility CS Voice Call Setup Success Rate > 99% 30 0.7% 800 7 3

Accessibility CS Video Telephony Call Setup Elapsed Time < 10.0 s 30 700 7 3

Accessibility CS Voice Call Setup Elapsed Time < 8.0 s 30 800 7 3

Accessibility PS 64/384 PDP Context Activation Success Rate including Attach > 99% 60 1% 400 7 5

Retainability Long Call Video Telephony > 15 min 930 first time right 4

Retainability Long Session PS > 60 min 3650 first time right 5

Number of Days to collect samples (1 Drive test team with 2 UE) 9 Days


(32 pt makes 2 rows









Area KPI Description TargetE///

ProposalCall

DurationConfidence

IntervalNumber of Samples

Drive test

hoursDrive tests

Completion CS Voice Call Completion Success Rate >=98% >=98% 120 2% 200 7 1

Completion PS Session Completion Success Rate Downlink (FTP, >5Mbyte file) > 98% >=98% 120 2% 200 7 1

Integrity PS Throughput Downlink (FTP, >5Mbyte mean user data rate) > 200 kbps > 200 kbps 120 200 7 1

Mobility CS Voice 3G-2G Hard HO Success Rate (including inter-MSC HO) > 98.5% > 98% 360 2% 190 20 2

Accessibility CS Video Telephony Call Setup Success Rate > 98% > 98% 30 1% 700 7 3

Retainability CS Voice Call Dropped Call Rate (Cluster average) including RF and HO Failures < 1.3% < 2% 120 2% 450 7 4

Accessibility CS Voice Call Setup Success Rate > 99% >=99% 30 0.7% 800 7 3

Accessibility CS Video Telephony Call Setup Elapsed Time < 10.0 s < 10.0 s 30 700 7 3

Accessibility CS Voice Call Setup Elapsed Time < 8.0 s < 8.0 s 30 800 7 3

Accessibility PS 64/384 PDP Context Activation Success Rate including Attach > 99% >=99% 60 1% 400 7 static

Retainability Long Call Video Telephony > 15 min 930 first time right 4

Retainability Long Session PS > 60 min 3650 first time right static

Number of Days to collect samples (1 Drive test team with 2 UE) 6 Days

Step4: Values and effort (our proposal)

Minimal target adjustment - lowering risk and minimizing measurement collection

30% less measurements!


(32 pt makes 2 rows









Handling kpi's in wcdma

Technology

Transcript of Handling kpi's in wcdma