Post on 21-Apr-2018
Taxonomy of xRperformance
requirements and design of meaningful end to end
KPI for xRPresentation for the ETSI STQ Workshop on
Multimedia Quality in Virtual, Augmented or other RealitiesMay 2017
Dr. Wolfgang Balzer
Overview/ScopeVirtual Reality (VR) and Augmented Reality (AR) areon the rise in multiple fields of applications. Both technologies provide new means of datavisualization for e.g. data processing and man-machine interaction and will have an impact on manysectors of the economy.Mobile data access is a key element to project AR and VR solutions to the places it is used, andsufficient quality of user experience will be – perhapsmore than in other fields – a key success factor.
Scope
Categorization framework: xR use cases, functionality, and requirements for mobile packet data networksDesign of a comprehensive set of KPI fromrespective use cases. Interaction between projected evolution ofmobile networks and xR requirements.
Roles of mobile networks in xR• Access to required data, e.g.
• Download maps and other location-dependent data
• Download real-time environmental information (other vehicles)
• Access to non-local processing power, e.g.• Upload camera feeds and other sensor data• Download data-enriched images
• Maintenance• Software updates
• The trivial rest• General connectivity for users while they use xR
services
Chances and risks• Promises and chances
• Have lean, low-cost local devices with minimum energy consumption
• Do everything in the cloud: Optimization of maintenance cost
• Side effects and risks• Vulnerable (single point of failure) • Potentially extreme radio resource
requirements• In particular if the number of parallel users can vary
strongly
Some examples for VR• Gaming
• Multiplayer, e.g. Shooters or MMORG• Single player (interactive adventures/puzzles)
• Educational• Architecture and Engineering (build/explore large objects
such as buildings, ships,…)• Communication (e.g. immersive teleconferencing)• Sales and marketing• Immersive Data Processing
• (bordering AR; think of all the movies where people enter data worlds or just interact with virtual screens projected into the room)
Some examples of AR• Telemedicine: Guidance in diagnostics or surgery• Logistics: Help humans or robots find items in
warehouses• Maintenance: Assist local repair technicians
(interactive guidance)• Shopfinder, sightseeing assistance• Gaming (Ingress, Pokemon To Go,…) • Interior Design (How will furniture look in your home)
• “Machine AR” in Automotive: providing information about local traffic environment
Are there new QoS/QoEdimensions in xR?• Well-known
• Audio quality• Video quality
• New aspects:• Immersion („credibility“ of xR effects)• Physiological effects (Impact on
physical well-being such as balance, nausea)
xR QoS is not only a function ofcarrier network performance
• QoE result will depend on carrier network performance but translated by differences in implementation and equipment
• data transfer efficiency, transfer strategies, use of equipmentproperties,…
User QoE/QoSxRservice/function
Implementation
Equipment
Infrastructure performance
Carrier networkperformance
Impact of network functionality degradations on functionality or QoE
• VR (from a small survey with VR developers)• Delays > 10 ms can already cause substantial
degradation in QoE (e.g. nausea), Delays > 50 ms make such services practically unworkable
• Data rates below requirements cause visual impairments (actual effects will depend on implementation)
• Degree and shape of dependency is a design parameter (from “graceful degradation” to “hard failure”) if ideal zone is left
xR related bitpipecharacteristics
xR related bitpipecharacteristics
Data Volume MB
Latency From To Data rate RemarkAR processing, minimal local resources < 10 ms 0,001 1
3D FHD or 4K Video data rates
AR or VR fast environment< 1000 ms 3 60
max 450 MBpsburst typical 1 s of video
AR or VR basic environment 1..100 s 3 60Ca. 5 MBps
20 high‐res images
Algorithm or SW updatebackground 10 1000
typical high‐end gaming machine major SW update
Main 5G promises and questions• High data rates
• Load distribution within a given area?• Low latency
• Edge computing – can AR or VR engines be provided near RAN elements? Which other processes (maintenance, security) need to be provided?
• Where will xR take place? Is this consistent with projected coverage characteristics?
• Motion profile (stationary/nomadic, slow moving, fast moving; in-house/deep in-house)
QoS and QoE
Regions of QoE• Perfect/no noticeable QoE improvements
if carrier QoS improves further• Acceptable (small but tolerable
impairments; improvement of carrier QoS will improve QoE noticeably)
• Unpleasant (QoE critically degraded)• Unacceptable
Dimensions of QoE• Degree of impairment• Duration of impairment• Frequency of impairment• Proposed: Averaging of a single indicator will
not work well enough• Short-time strong impairments may be more
tolerable than extended medium or small impairments
• Exact relationship should be expected to depend on the actual use case
KPI families (“pyramid of needs” model)
• Primary• Can the service be accessed at all?• Service disruption rates• Critical range of service-dependent subjective
QoE indicators• Secondary
• Comfort range of QoE indicators• Set-up time (multiple if access is multi-phase)
From use case to KPIUse Case:- Goal (implicitly also providing criteria for
success/failure)- Related flow of activity
- Operational steps- Expected responses Sequence of events as seen from the user’s
perspective- Identify meaningful QoS KPI- Define mapping to QoE
Role of QoS KPI
• Define/identify business critical user perception
• QoE can serve as an instrument to optimally use resources
• Direct investment funds to areas where they produce the highest customer satisfaction
• Provide a framework for diagnostics• Not meaningful for all stakeholders
Specific KPI for VR
• Mostly subjective aspects (human perception)
• General audiovisual QoE (sound, image)• Immersion
• Image artifacts• Responsivity
• Physical well-being• Balance disturbance (due to imperfect
match of images/head motion etc.)
Specific KPI for AR
• Both data and human-experienced related aspects
• General audiovisual QoE (sound, image)• Immersion
• Image artifacts• Responsivity• Match between camera image and
additional information
How to measure xR KPI• Direct (human experience - MOS)
• Similar to video or audio telephony – rate the quality while using the service
• Unclear: Will there be perceptive models such as POLQA or PEVQ?
• Expected: Parametric models to predict xR MOS• Using generic “bit pipe” properties• Probably use specifically designed stimuli• Calibrate models with data from human MOS lab
or field tests• Can crowdsourcing play a role (e.g. embed QoE
into products?)
Upfront xR Testing Strategies; assumptionsApproach: Assume as little as possible about internal working of the service
• Chance: Save some effort (taking shortcuts)
• Risk: Partial blindness• Internal workings can
change anytime
• Assume (within reasonable limits) as little as possible
• Robustness against functional changes of system under test