2020 Benchmarking Framework · 2020-04-07 · Public Preface. The umlaut Benchmarking Framework is...
Transcript of 2020 Benchmarking Framework · 2020-04-07 · Public Preface. The umlaut Benchmarking Framework is...
Public
Public
2020 Benchmarking Framework
Overview PresentationDocument Version V1.0, April 2020 7-Apr-20
Public
Preface. The umlaut Benchmarking Framework is based on a unified measurement method for true international comparability combined with the umlaut-Score approach.
2020 umlaut scoring framework 2
Key Features:⚫ Unified Methodology and Score Scheme, globally used in a wide range of activities, both public and non-public
→ International comparability
⚫ User-centric scoring and test cases plus crowd-based assessment
→ end customer relevance
⚫ Full drill-down capability
→ supports engineering and optimization work
⚫ Yearly methodology evolution in dialogue with network operators
→ Transparency, up to date
⚫ Common basis for public benchmarks and independent network certification, rooted in ETSI TR 103 559 recommendations
→ ideal PR tool
⚫ 5G capable setup
→ supporting network evolution
7-Apr-20
Public
Rationale. Measuring performance means collecting many different performance indicators – how to stay focused and easily communicate results?
2020 umlaut scoring framework 3
Measuring Network Performance⚫ Historically, benchmarking results look at a vast multitude of
indicators, easily missing the important aspects
The umlaut Score as a management Tool⚫ The umlaut score aggregates the complete set of KPIs into a
single score figure, preserving the drilldown capability
⚫ It features implicit weighting of environment, services and KPIs and comparability on national and international level
Distracting details
Tons of (K)PIs
What isimportant?
unique performance
profile
Benchmark
History
Learning from best-
in-class
Manageable targets
Priorities
0 1000
7-Apr-20
Public
umlaut Score Principles-Building Blocks. The umlaut score delivers a robust scale to assess network quality and performance. It is coupled with a unified measurement methodology for true international comparability.
2020 umlaut scoring framework 4
umlaut-Score: 1000 PtsOverallScore
Modular Score Approach
NT Data NT Voice UET
City Town Road All
QualifierKPIs
Differen-tiator KPIs
CrowdKPIs
Unified Measurement Methodology
- Measurement Systems- Test cases and Settings- Device capabilities- Geographical Mix- KPIs, post processing- Mapping functions
ServiceModules
ConsistentReporting
- C-level Views- Mgmt. Level
Breakdowns- Drilldown to KPI-Level- Cross-Project Reports
Area Level
KPILevel
7-Apr-20
Public
umlaut Score Principles-Building Blocks ii. For drive and walk testing, each service session is characterized by a Qualifier KPI and one or several Differentiator KPIs.
2020 umlaut scoring framework 5
Qualifier KPIs
Model minimum requirements that have to be met to make the experience of the mobile network satisfactory.
Examples:Voice: Percentage of successful and sustainable callsData: Percentage of sessions faster than 500kbps
Qualifiers are worth:55% of all the points
Differen-tiator KPIs
Differentiator KPIs describe additional aspects of those services that fulfil the qualifier criteria
Examples:Voice: (Avg. & P90) Call Setup Time, (Avg. & P10)
Speech Quality, Data availabilityBrowsing: Avg. Session Duration5MB File: Avg. Session Duration, 10 and 90 Percentile of
Throughput distribution
Differentiators are worth:45% of all the points
For every Service usage, a simple Success / Failure Distinction is made
7-Apr-20
Public
Concept Overview. Characterized by (i) defined weighting between environments and services (ii) fixed maximum score (iii) modular extensions (iv) qualifiers and differentiators.
2020 umlaut scoring framework 6
PropertiesKPIs are grouped into⚫ Qualifiers – achievement of min. requirements⚫ Differentiators – meeting state-of the art targetsPoints in both groups are determined by mapping functions for each individual KPI, yearly updatedThe Baseline umlaut-Score covers a default combination of different environments:⚫ Drive test Voice and Data⚫ Cities, Towns, RoadsAnd is optionally extended by⚫ Walk test⚫ Indoor city, public transport, commuter rail, railways⚫ Crowd-based scoreThe core umlaut-Score covers a default combination of services. Services may be added or dropped from the score on a yearly basis
Advantages⚫ Fixed maximum score allows historic comparison: „A network with 900
points in 2020 has the same performance in relation to the available state of the art as a network with the same points in 2014“
⚫ Fixed categories make the scoring invariant to the detailed campaign planning and the number of samples collected in each category
⚫ Threshold values and mapping functions for qualifiers are stable for min. 1 year (unless change technically required)
Requirements⚫ Threshold values and mapping functions for differentiators still require
regular adaptation to state of the art → yearly update process in place
⚫ Measurements satisfying the scoring requirements have to address a minimum set of scenarios and services → addressed by umlaut‘s best practice approach
7-Apr-20
Public
Test and crowd assessment methodology.Test cases and modules need to be included in the campaign scope.
7
Core RequirementsThe red and blue sets of test cases represent the Network Test.The green modules represent the User Experience Test (UET) component based on crowdsourced data.
M2M Voice
4G preferred to
4G preferred UE
Controlled Background Traffic
Web Browsing (static and live)
Reference web page download, HTTP and HTTPS
Popular live web pages download,
HTTP, HTTPS
HTTP/S File Up-/Download
Fixed Size
Fixed Size HTTP, HTTPS File download
Fixed Size HTTP, HTTPS File upload
YouTube
APP
YouTube Video download &
playout
Video-on-Demand (VOD),
Live Stream
HTTP File Up-/Download
Data stream
Fixed Duration multi socket download
Fixed Duration multi socket upload
BroadbandCoverage
Geographic:Broadband Coverage
Excellence (based on quality of
coverage metric)
User-Centric:
Time on Broadband
Latency
/Interactivity
RTTbased capability
for:
Gaming
OTT Voice
Data Speed
Download speed metrics based on
passive observation of real user traffic
Capability for:
Basic Internet
HD Video
UHD Video
Network Test: Customer Experience 4G and 5GNetwork Test:
Capability 4G and 5GCrowd: User Experience Assessment
2020 umlaut scoring framework 7-Apr-20
Public
Score relation and coexistence of different measurement scopes.In different activities performed by umlaut, different score models will coexist, but maximum scores per scenario are harmonized.
2020 umlaut scoring framework 8
480360 360
120 120
160 160 160
160 160100
60
800
600
200
200 200 200 200 200
1000
0
100
200
300
400
500
600
700
800
900
1000
Baseline umlaut-Score 2020 +Crowd
Incl. City Walktest 2020 +Crowd
Incl. City/Train Walktest 2020 +Crowd
Smallcountry Drive 2020 Small Country Drive + Walk2020
Crowd Only 2020
Cities Drivetest Cities Walktest Towns Drivetest Roads Drivetest Train Walktest Overall Drivetest Overall Walktest Crowd
7-Apr-20
Public
In different activities performed by umlaut, different score models will coexist, but maximum scores per scenario are harmonized.
2020 umlaut scoring framework 9
Overview of measurement scenarios with different geographical modules
⚫ All models share 1000 point maximum⚫ 800 achievable points for Network Testing (NT, drive and walk testing)⚫ 200 achievable points for User Experience Testing (UET, crowdsourcing component)
⚫ In case of non-availability of UET, NT points are scaled up to 1000 max⚫ In case of UET only, UET points are scaled up to 1000 max⚫ In all NT categories, the voice / data split 40 / 60 (not shown)⚫ In all NT categories, the weight of individual services and KPIs is the same
Score relation and coexistence of different measurement scopes.
7-Apr-20
Public
Public Benchmarks, Network Certification and Claim Support.The umlaut benchmarking activities with public visibility can be based on dedicated testing and crowd based assessment.
2020 umlaut scoring framework 10
Description of Approachumlaut activities with public visibility are:⚫ Public benchmarks executed in cooperation with a publishing partner⚫ Independent network certification and claim support activities, commissioned by one or more 3rd parties (e.g. mobile network operator,
network vendor, etc.)
The setup, scope and timeline⚫ for any certification and claim support activity is independently chosen and decided by umlaut⚫ for any public benchmarking activity is independently chosen and decided by the publisher and umlaut
The available components for any activity are⚫ Dedicated Network Testing activity for Smartphone voice and data service, consisting of drive test and optional walktest⚫ Crowd based User Experience assessment of coverage, data speeds and latencyPer default, network certification activities are a combination of dedicated testing activity + crowd based assessment
Exceptions may apply in markets where crowd panel size/distribution are not meeting umlaut requirements. The decision will be taken independently by umlaut. In such cases, all score categories can be proportionally expanded to reach a 1000 point maximum score.
7-Apr-20
Public
www.umlaut.com
2020 umlaut score framework concept 114/7/2020
Public2020 umlaut score framework concept 12
Disclaimer
This document and all information contained herein is the sole property of umlaut.No intellectual property rights are granted by the delivery of this document or the disclosure of its content. This document shall not be reproduced or disclosed to a third party without the express written consent of umlaut. This document and its content shall not be used for any purpose other than that for which it is supplied.
4/7/2020