Litmus: Robust Assessment of Changes in Cellular Networks

© 2013 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks contained herein are the property of their respective owners.

Litmus: Robust Assessment of Changes in Cellular Networks

Ajay Mahimkar, Zihui Ge, Jennifer Yates, Chris Hristov*,

Vincent Cordaro*, Shane Smith*, Jing Xu*, Mark Stockert*

AT&T Labs – Research * AT&T Mobility Services

ACM CoNEXT 2013, Santa Barbara, CA

1


Cellular network changes

2

Network changes and assessment Software upgrades, configuration changes, … How does it impact user perception of service quality?

Voice and data connection attempts (Accessibility) Successful termination of ongoing calls (Retainability) Data throughput, Voice Erlangs, …

Extensive testing in labs before deployment in the field However, no lab can fully replicate scale, complexity and

diversity of large-scale operational networks

First Field Application (FFA) Before rolling out the change network-wide, conduct small

scale testing in operational network

Radio Network

Controller

Cell Tower

Circuit switched core

Packet switched core

Voice Data

FFA


Impact Assessment of FFA

3

Go/no-go decision for a wide-scale roll-out

Analyze the performance impact of FFA

FFA pre/post impact analysis of service performance Compare service performance after FFA with that of before If FFA is successfully trialed and shows expected performance

impacts, then it can be rolled out network-wide Go/no-go decision is crucial Challenges: external factors can make assessment difficult

FFA Change

FFA Change

FFA Change

Performance Impacts

Degradation

Improvement

No change


Dependency on external factors

4

Configuration change accidentally co-occurs with strong winds that negatively impacted service performance

Service performance in cellular networks is influenced by several external factors Weather (heavy rainfall introduces obstruction for radio signals) Terrain (Mountains/flat surfaces/tall buildings have different propagation properties) User population densities and mobility patterns Seasonal changes (foliage or leaves budding) Traffic pattern changes (holidays, major events or trade shows) Other network events (outages or maintenance activities in other parts of network)

Unnecessary roll-back of change without knowledge of impact of strong winds



5

Degradations in Voice Accessibility across multiple RNCs due to severe storms and damaging hail during a tornado

Assessment of changes at RNCs would be difficult because of weather impact

Yearly seasonality in Voice Retainability for UMTS cell towers due to foliage

Assessment of changes would be difficult because of seasonal changes



6

Improvements in Voice Retainability across a majority of cell towers due to software upgrade at an upstream RNC

Assessment of changes at cell towers would be difficult because of upstream RNC changes

Dramatic traffic pattern change during holidays induces significant changes in Voice Retainability

Assessment of changes would be difficult because of traffic pattern changesPre/post impact analysis of FFA changes needs to account for

the overshadowing effects of external factors


Litmus idea – study/control comparison

7

Compare performance between study and control group Study group – network elements where change is implemented Control group – network elements without the changeIntuition Performance at geographically nearby elements is correlated External factor influences performance at both study and control A performance impacting change at study will change the

dependency between study and control Challenges Unrelated performance changes in a small number of control

group member Poor selection of control groupLitmus Solution Robust spatial regression algorithm Domain knowledge guided control group selection

Radio Network Controller (RNC)

Cell Tower

User Equipments

Circuit switched core

Packet switched core

VoiceData

FFA

Study Control


Litmus comparison to related work

8

Study-group only analysis Mercury [SIGCOMM’10], PRISM [CoNEXT’11], Spectroscope [NSDI’11], … Does not account for impact of unrelated external factors A/B testing – also known as split testing, control/treatment Popular in web domains for data driven decision making [KDD’07,’12] Web users randomly exposed to the two variants of experiment Why doesn’t it apply in our context?

Tight coupling between experiment and assessment Control group might be subject to other network events such as changes or unplanned outages

Difference in Differences (DiD) Compare mean/median difference between study and control before and after the change Why doesn’t it apply in our context?

Contamination of forecast due to poor selection of control group Sensitivity to performance changes in a small number of control group

Control (A)Study (B)

Serving users at lake

Serving users at business

location

Feedback


Robust spatial regression

9

Before change After changeStudy

Sam

pled

Con

trol

Study

Sam

pled

Con

trol

f

Regression coefficientsf f

Forecast study

Forecast study

Delta Robust rank-order tests

Delta

Output: Degradation/Improvement/No change

Multiple iterations of forecast difference comparison increases the robustness to a few bad members in the control group

Repeat the procedure for m

ultiple iterations


Domain knowledge guided control group selection

10

Guidelines for control group selection Subject to same external factors as the study group Share similar properties with study group such as geographical proximity or configurationControl group size Not too large: difficult to capture similar impact due to external factor Not too small: loose benefits of robustness in spatial regression analysis Attributes for selection Geographical distance using latitude/longitude and zip-code Topological structure of the cellular network Configuration settings such as software version, or equipment model Predicates to select control group Uni-variate – single attribute (for e.g., LTE cell towers within the same zip-code) Multi-variate – combination of attributes (for e.g., UMTS cell towers with same RNC and same OS)


Litmus evaluation

11

Evaluation conducted using data collected from operational cellular networks Lack of complete ground truth makes evaluation extremely challenging Two-step methodology

A-priori known changes and assessment by Engineering & Ops Manually conducted before through visual inspection & analysis

Synthetic injection of changes in performance time-series at cell towers Compare Litmus with Difference in Differences (DiD) and study-group only analysis Accuracy computation

Result summary Litmus outperformed study-group only analysis because of robustness to external factors Litmus outperformed DiD because of robustness to a small number bad members in control group

Thorough and exhaustive evaluation

Real examples

EXPECTATION Improvement Degradation No impact

Improvement True positive False negative False negative

Degradation False negative True positive False negative

No impact False positive False positive True negative

ALGORITHM OUTCOME


Evaluation results

12

Evaluation using known assessments

Precision Recall True Negative Accuracy0

20

40

60

80

100

120

Study Group Only

Difference in Dif -ferences

Litmus Robust Spatial Re-gression

Precision Recall True Negative Accuracy0

20

40

60

80

100

120

Study Group Only

Difference in Dif -ferences

Litmus Robust Spatial Regression

Compared to study group only analysis and DiD, Litmus is robust to external factors and accurately conducts the impact assessment

Evaluation using synthetic injection

Precision = TP / (TP + FP) Recall = TP / (TP + FN)

True Negative = TN / (TN + FP) Accuracy = (TP + TN) / (TP + TN + FP + FN)

Litmus outperforms DiD due to zero false

negatives

Study group only analysis has poor

accuracy due to high FP and FN


13

Litmus operational experiencesLitmus is being heavily used for FFA impact assessment in production cellular networks Pre/post impact analysis across a wide variety of performance metrics Outcome is used for a go or no-go decision for wide-scale deployment of FFA change

Change type Location Impact Expectation

Impact Assessment by Litmus

External factor Go/no-go decision

Reduce start-up times for data sessions


No degradation in voice

Degradation in voice None

Configuration changes

Mobile Switching Center (MSC) – Voice switch

Improvement in voice

No improvement Foliage

SON load balancing and neighbor discovery

Cell Towers Improvement in call connection

Improvement Hurricane Sandy

Improve cell change success rates


Improvement in call retention

No improvement Traffic pattern changes due to holiday


Impact of SON during hurricane Sandy

14

SON (Self Optimizing Network) features were being trialed on some cell towers SON Capabilities: automated load balancing,

neighbor discovery and self-configuration Key question: How did SON perform during

hurricane Sandy? This question cannot be answered without

comparison to control group Control group has to be within the Sandy-

impacted region Both study and control group were impacted

due to Sandy; however study group did better than control

The recovery on study group was also faster than on control group

SON did a good job ! SON features were rolled-out network-wide


FFA to improve cell change success rate

15

FFA change applied at a few RNCs Expectation: Improvement in data retainability Study-group *only* analysis would have led to

improvement inference and recommendation made for nation-wide roll-out

After comparing to control, Litmus identified that improvement was really due to holidays

Traffic pattern changes induced improvements in data retainability across both study & control

FFA change thus was not inducing performance improvements

Decision was made not to roll-out based on Litmus results


16

Conclusions and Future Work

Litmus – an automated tool for robust assessment of changes in cellular networks Carefully accounts for external factors such as foliage, weather, holidays, or network events New spatial regression algorithm for robust performance comparison of study versus control Domain knowledge guided control group selection Outperforms study-group only analysis and Difference in Differences (DiD)

Operational Experiences Litmus is being used successfully in go/no-go decisions for wide-scale deployment of changes Considerably improved the assessment accuracy and analysis time

Future Work Continue to improve methodology for control group selection Apply to other networks and services such as clouds, data centers Extend Litmus to device specific monitoring – e.g., Apple iPhone, Samsung Galaxy or Nokia Lumia


17

Thank You !Questions ?

Litmus: Robust Assessment of Changes in Cellular Networks

Documents

Transcript of Litmus: Robust Assessment of Changes in Cellular Networks