SITE Roadmap 2016 and Beyond...LTE-A Testing using Cat.6 Smartphones SITE 3.0 offers the possibility...

SITE Roadmap 2016 and BeyondHighlights from the SITE Roadmap

Goce Talaganov

Senior Product Manager SITE

Testing is our Competence

Agenda

Video Quality for Adaptive Bitrate Streaming

SITE Virtualisation Of Backend Components

LTE Cat.6 Smart Probe

Enhanced OTT App Testing

GUI Test Case Setup and Trace Analyzer

Web Application Testing

Smartphone Interactive Recorder

Page 2



IPTV and OTT Video Streaming Ecosystem

Mo

bile

/Fix

ed O

per

ato

rs

Managed Network

STB

Single screenMPEG-TS over RTP/

UDP/IPMulticast

IPTV

Managed Network / Best effort network

ABR

Multi-screenAdaptive Bit Rate Technology, e.g. HLS,

DASH, HSSUnicast

Internet

Adaptive Bit Rate Technology, e.g. HLS,

DASH, HSSUnicast

Multi-screen

ABR

Best effort network

OT

T S

erv

ice

s


720p

480p

360p

ABR Mechanism

M3U8 file is an index that lets the client know which streams and which segments are available at any given time

The device automatically selects the most appropriate stream from the primary manifest file given bandwidth and CPU constraints, downloads the segment, and appends it to the playback buffer

Manifest File

HTTP Video Server/Cache

720p

480p

360p

720p

Encoder

Video Input

Manifest M3u8.Index


IneoQuest Company Introduction

#1 in Video Stream Analytics Solutions

Monitor + Analyse + Optimise Video

End-to-end visibility into every subscriber, channel, ad and viewing experience

Adaptive

Linear

Hybrid

Multi-network

Multi-protocol

Multi-screen

Multi-device

650+ global customers

100+ Multiscreen/OTT Solutions installed

Media Delivery Index (MDI) – RFC #4445A QoS metric for monitoring Traditional Linear Broadcast TV

VeriStreamA QoS metric for monitoring Adaptive Streaming Video


ABR Video Quality testing in SITE available for HLS (75% market share) and MPEG-DASH over different access technologies, e.g. LTE, 3G, WLAN/LAN/DSL

MEPG2-TS Encryption for HLS supported

Testing ABR Streaming with SITE*

Encoder Segmenter Origin

ABR STREAMS: HLS/SS/DASH/HDS

Mobile Core

Fixed ISP Core

3G/LTE

WLAN/LAN

Video

3G/LTETest Interface

LAN/WLANTest Interface

*Available in SITE 3.1.2Demo already available via Technical Sales


Important QoS KPIs

Full support of all KPIs for HLS (75% market share) DASH, MSS and HDS supported without Video MOS

Number of Chunks Downloaded

Number of Manifest Files Downloaded

Video MOS

Number of video chunks that were downloaded

VeriStream as reported from IneoQuest min/max/avgVeriStream

Resolution

Number of requested manifest files

Only applicable to HLS, with or without encryption

The requested resolution of the stream

KPI Description

Throughput Calculate min/max/avg throughput from the PCAP file

Alarm Count As reported from the alarm_count column from IQ results summary

VideoStart Play Duration The time that it takes for the buffer to load and start playing the video

VideoInteruptions Number of interruptions or stallings e.g. buffer empty or rebuffing


LU6 HW

Detailed Architecture...

VM2: IQ ABR Active Client

VM1: SITE SW Image

2G 3G 4G DSLLAN

1.Config 2.Start 4.Results3.Stream

SITE CU

KPIs: HTTP based, VeriStream 1-5 ,MOS QoE 1-5, TCP errors

Encoder Segmenter Origin

ABR STREAMS: HLS/SS/DASH/HDS

Mobile Core

Fixed ISP Core

3G/LTE

WLAN/LAN

Video

3G/LTETest Interface

LAN/WLANTest Interface

SITE will incorporate IneoQuest‘s Active Client which supports ABR streamingand their video streaming analysis engine. Hence the ABR client can use SITE‘s interfaces.

HLS

Service Authentication and URL Stream

Fetching


ABR Video Test Case

Test Cases Supported Protocols Supported Interfaces

HTTP_Video_Streaming HLS (with encryption) DASH* MSSS* HDS*

3G LTE LAN

*Within scope of the product but not included in the prototype phase


YouTube Test Case on Smartphones

MOS-B values are calculated based on a bit-stream analysis of TCP throughput, packet loss and other metrics

The video itself is not decrypted

The existing Vqmon (version 4.1) licenses can be used, without any additional updates required

SITE 3.1 can now test HTTPS encrypted YouTube video streaming on smartphones

*Available in SITE 3.1.1


Netflix Testing with SITE

Internet

3G/LTE/WiFi/LAN/DSL

Netflix Testing test case can be executed on any 3G, LTE, WiFi, LAN or DSL SITE interface, hence testing via various access technologies is possible

The test case automates an HTML5 capable browser in order to navigate to the Netflix URL, logs in to a Netflix account and starts playing the first video from the Netflix billboard, ensuring Netflix availability of the service

Once the Netflix native player is being initiated in the browser, the test case activates the diagnostic mode of the player in order to capture the measurements for the QoE/QoS KPIs while the video is streaming


Test Case Steps

The test case launches Fire Fox v43 browser and navigates to the requested Netflix URL, that is taken from the test case parameters

As a next step the test case automatically logs into Netflix using the credentials that were provided as parameters in the test case

The test case always activates the first profile on the left

Once the profile is chosen the Netflix VoD library is displayed, here the test case will always play the first video of the billboard, in this case that is “The office”


Test Case Steps Continued

Once the Movie is activated to play, the native Netflix player is being loaded in the browser and the video starts playing, at this stage the test case activates the diagnostic mode of the player

The test case has three options on where to start the video to play: from beginning, middle or close to the end

Once the video is being streamed for the parametrisedduration, the test case “signs-out” and starts the post processing of QoE KPIs


Test Case KPIs

KPI Description

UserAgent* Netflix Advertised User Agent

MovieID* Netflix Unique ID of the movie

Position* Position of the video when finished e.g. 2 (for finishing on the second minute)

Duration* Overall Duration of the video e.g. 115Minutes

Buffering State* The health of the buffer e.g. normal if there is no re buffering needed

Buffering State ETA* Estimated Time of Arrival for the rebuffing process

Rendering State* Can be playing, buffering, stoped

Playing Bitrate Audio* The bitrate of the playing audio e.g. 64kbps

Playing Bitrate Video* The bitrate of the playing video e.g. 1750kbps

Playing Resolution* Resolution of the video that was played e.g. 720x480

Buffering Bitrate Audio* The upcoming audio bitrate to be buffered e.g. in case of ABR activated going from lower

to higher resolution 96kbps

Buffering Bitrate Video* The upcoming video bitrate to be buffered e.g. in case of ABR activated going from lower

to higher resolution 3000kbps

Buffer Size in Bytes Audio* Audio Buffer size represented in bytes

Buffer Size in Bytes Video* Video Buffer size represented in Bytes

Buffer Size in Seconds Audio* Audio Buffer size represented in Seconds

Buffer Size in Seconds Video* Video Buffer size represented in Seconds

*Measurement taken from the native diagnostic log of the Netflix player


Test Case KPIs Continued

KPI Description

Current CDN* Content Distribution Network that was used for streaming the video e.g. Akamai

Audio Track Language* Language of the audio e.g. EN for English

Audio Track id* The unique Netflix id of the audio track

Audio Track Channels* How many audio channels were active 1 or 2

Audio Track Codec* CODEC used for the audio track

Video Track Codec* CODEC used for the video track

Text Track Language* The language of the subtitle

Framerate* Frame rate of the video e.g. 24fps

Total Number of Frames Played* Total number of frames that were played and rendered

Total Dropped Frames* Total number of dropped frames while the video was playing

Latency* Latency from Video Player to Netflix Video Server

Throughput* Measured achievable throughput from the video player to the Netflix Video Server

Max Sustainable Video Bitrate* The video Bitrate that can be utilized for video playing

DFR 384* Number of dropped frames while the video was playing in 384 resolution




Netflix Login Duration From pressing login button to loading of the Netflix video content page

Netflix Start to Play Movie Duration From pressing the “play” button on the Netflix page to the moment that the player

became visible as an element in the page

Played Movie Duration How long was the movie played, from start to stop

*Measurement taken from the native diagnostic log of the Netflix player


Outlook for Video Quality Testing in SITE

• One of the most flexible methods for Video Quality Testing is HDMI-based video quality testing

• HDMI-based testing provides detailed KPIs using “pixel-level” analysis from decoded video frames

• HDMI-based video testing is agnostic of input source (STB, tablet, laptop) –highly flexible

• An example use case for HDMI-based video quality testing

is IPTv

CLU5

STBHDMI

Video Test ProbeIR


Mo

bile

/Fix

ed O

per

ato

rs

Managed Network

STB

Single screenMPEG-TS over RTP/

UDP/IPMulticast

IPTV

Managed Network / Best effort network

ABR

Multi-screenAdaptive Bit Rate Technology, e.g. HLS,

DASH, HSSUnicast

Internet

Adaptive Bit Rate Technology, e.g. HLS,

DASH, HSSUnicast

Multi-screen

ABR

Best effort network

OT

T S

erv

ice

s

Outlook for Video Quality Testing Solution Overview


Agenda








Page 18



Virtualisation of SITE Central Components

SITE Central Components can be provided as VM’s:

CU, CUX, UI, VPN, KPI

Improves flexibility for customers that have compatible virtualisationinfrastructure

Customer is responsible for providing the required resources, i.e. CPU, RAM, Storage

SITE UI

SITE CU

SITE VPN Server

SITE KPI Server

Virtualization Platform

CUVM

UIVM

KPIVM

VPNVM


Agenda





GUI Test Case Setup and Trace Analyser



Page 20



LTE-A Testing using Cat.6 Smartphones

SITE 3.0 offers the possibility to test LTE-A Cat.6 networks using the fully

integrated Samsung Galaxy S5 (SM-G901F) Cat. 6 in the Hybrid CLU5

Throughput measurements can be performed using the SpeedTest app or

HTTP and FTP tests even on S6

Troubleshooting and analysis is possible using L3 Traces

Carrier Aggregation KPIs derived from RRC

based on traces:

CA (SCell) configured at RRC level, 1/0

Frequency of the second carrier

BW of the second carrier

Ca configuration duration at RRC level


Nomadic 300 and New LTE-A Cat.6 Wireless Modules*

The new Nomadic 300 is capable of measuring LTE-A Cat.6 speeds

with Carrier Aggregation

New LTE-A Cat.6 wireless module supported in a revised Nomadic HW

LTE-A Cat.6 interfaces for the Nomadic 300 comes in two variants:

Carrier Aggregation KPIs derived from RRC:





MC7430 MC7455

LTE bands: B1, B3, B5, B7, B8, B18, B19, B21, B28, B38-B41WCDMA bands: B1, B2, B3, B4, B5, B8Supports LTE Advanced and VoLTE

LTE bands:B1-B5, B7, B12, B13, B20, B25, B26, B29, B30, B41WCDMA bands: B1, B5, B6, B8, B9, B19Supports LTE Advanced and VoLTE

Future-proof hardware design for supporting the higher LTE UE categories

*Trials are available from July 1st (based on 3.1.0 Release)Order Lead Time of 9 weeks (shipped from SIGOS) for POs that arrive in JulyProduct will be shipped with SITE 3.1.1


SmartProbe LTE-A Testing with Cat.6 Wireless Modules*

An alternative testing approach using Cat.6 capable Wireless Modules

The modules will be integrated in a new shuttle, that occupies two slots in a CLU5 or LU6

The shuttle will include an SBC, to provide local processing and offload the CPU of the LU

Carrier Aggregation KPIs derived from RRC will be provided:





*Planned for Q4 2016

USB3 SBC

LTE Cat.6 Shuttle and Interface


Agenda








Page 24



Introduction – Smartphone QoE Testing

Complementary to the end-to-end QoS testing which relies on robust wireless modules, SIGOS offers QoE testing using commercial devices such as smartphones

Smartphone QoE Testing with SITE comes in two shapes:

Automated, periodic testing for baseline reporting

Smartphone interactive for ad hoc verification of services

SITE 3.1 supports Samsung Galaxy S5, S6* and S7* smartphones via the Hybrid CLU5



Hybrid CLU5 Features

Robust hardware chassis for QoS and QoE testing with increased resiliency via virtualisation of the CLU

Full support for monitoring network services in 2G/3G/4G such as Voice, SMS, Video and Data Speed, as well as OTT App monitoring and smartphone interactive testing

Capable of hosting two Smartphones and two wireless modules

Out-of-band power cycling of the smartphone via battery emulation (Samsung Galaxy S5) and via mechanical fingers (Samsung Galaxy S6* and S7*)

Leverages existing SITE feature licenses and test cases

Flexible Benchmarking enabled with smartphone SIM Multiplexing*Available in SITE 3.1.1


Hybrid CLU5 - S7 Integration*

Out-of-band control via mechanical fingers, for easier maintenance and support

SIMMUX enabled

USB connector for communication with SITE and powering of the device

Audio connector for Voice Quality

SITE 3.1 comes with support of Samsung Galaxy S7 in the Hybrid CLU



Hybrid CLU5 - S6 Integration*

SIMMUX enabled

Out-of-band control via mechanical fingers, for easier maintenance and support

USB connector for communication with SITE and powering of the device

Audio connector for Voice Quality

SITE 3.1 comes with support of Samsung Galaxy S6 in the Hybrid CLU



Hybrid CLU5 - S6 and S7 Integration


To reflect the changes in our product, we also created a new naming convention and package structure

Apps and scenarios available for testing, starting with SITE 3.1 include: Facebook – newsfeed access, status change, image sharing

Instagram – image sharing

Viber – Viber to Viber messaging, Viber to Viber calls, Viber to/from 2G/3G calls

Twitter – Twitter direct messaging

DropBox* - access a folder, upload/download/delete a file

Skype* enhancement* - Skype text messaging


Smartphone

OTT Apps

Skype

YouTube

WhatsApp

SpeedTest

Viber

Facebook

Twitter

Instagram

SnapChat

DropBox

Network Services

VoiceQuality

CSFB

SMS

Data

Joyn

Interactive

SITE Interactive

SITE 3.1 delivers a big enhancement to SITE’s OTT App testing capabilities



Automated End2End Test Cases for QoE

IMS VoiceCall VideoCall InstantMessaging

VoLTE to VoLTE •

ViLTE to ViLTE •

RCS_MSRP to RCS_MSRP

•

SMSoIP to SMSoIP •

3G to 3G •

OTT

Skype to Skype • • •

Viber to Viber • • •

WhatsApp to WhatsApp

• •


ViLTE VoWiFi VoLTE RCS-MSRP SMSoIPWhatspp

IMWhatsApp

VoiceSkype IM

SkypeVoice

SkypeVideo

Viber IM Viber Voice 3G Voice

Sucess (%) 99,2 98,9 99,1 99,4 99,8 99 94 99 95 94 99,5 93,5 98,9

DropedCalls (number) 10,752 14,784 12,096 8,064 2,688 13,44 80,64 13,44 67,2 80,64 6,72 87,36 14,784

80

82

84

86

88

90

92

94

96

98

100

Service Success Rate (%)

OTT versus IMS - Service Availability

IMS OTTs


QoE Call Setup versus Time to Ring

1,20

1,80

1,25

1,85

3,41

5,89

1,50

2,60

2,18

3,62

5,61

6,32

VoLTE

VoWiFi

WhatsApp

Skype

Viber

3G-Legacy

CallSetup vs. Time to Ring Durations

Overall Call Setup Duration (s) Time to Ring Duration (s)


Message Delivery Quality of Experience

0,43

0,63

0,91

1,06

1,44

5,3

RCS-MSRP

SMSoIP

Whatspp

Skype

Viber

3G

Instant Message Delivery

Message E2E duration (s)


QoE Voice Quality of Experience

2,9

3,1

3,2

3,6

3,8

22

36

48

13

Viber

WhatsApp

Skype

3G-Legacy (AMR)

VoLTE (AMR-WB)

Average P.863 POLQA MOS vs. Average Bitrate

Average MOS Average Kbps


Agenda


SITE Virtualization Of Backend Components






Page 36



New SITE UI – Phase 2

New web-UI features include:

Test Setup, Editing and Scheduling

Trace Analyzer

Improved web-reporting view

All other functionality will be available in the existing Java UI

Functionality will be gradually added to the web UI in future releases

Note: The screenshots in the following slides are not the final release and have been taken from a QA system

With SITE 3.2, the second phase of the new user interface is launched


Test Setup

Test Setup can be selected in the main navigation

Java UI, Dashboards available in the Tools Panel


Test Setup – Continued

Column browser for navigation

Action bar for creating, editing, scheduling selected test cases

A list of favourite test case can be created for easy access and a history of the last actions is maintained


Test Setup – Test Categories

Test Case categories organized by service, which make it easier for user to find the appropriate test case

Users can select test cases required for a particular campaign and proceed afterwards to parametrising them


Test Setup Parametrization

Redesigned input fields, drop-down menus and tooltips

Automatic filtering of selection as the user types

Simple and advanced configuration modes


Trace Analyzer

Reduced number of tabs, maintaining only the most relevant information for users

Improved design, which makes it easier to visualize the information

MSC and Table combined into one view. Filtering is done directly in the MSC


Report Overview

Report charts presented as thumbnails. Users can view/edit an existing chart or create a new one.


Chart Configuration

Chart is always visible during configuration

Same concept of report, chart and KPI configuration options


Alarm Details

Information is shown in place for the selected alarm

Alarm properties displayed directly in the alarming view

Alarm history and alarm chart shown in place also


Agenda








Page 46



Web Application Testing via WebBrowsing 2.0

As web pages and web applications increase in complexity and provide more functionality, testing requirements become also more complex

Real user transactions, e.g. filling out forms, adding items to cart need to be replicated in order to test the functionality of the web application

SITE currently offers the possibility to test pre-recorded web transactions on Firefox browsers (Chrome, Opera will be added) and to verify if the actions were performed successfully

The recorded scripts replicate all user interactions, e.g. clicking various elements on the web applications, filling of forms


WebBrowsing 2.0 Evolution

In order to provide more flexibility, SITE will offer the option for users to record their own scripts

Recording will be performed using an actual browser

The recorded script will be imported in SITE for performing periodic testing on the particular scenario


Agenda








Page 49



Interactive GUI in a Browser

Remote screen view, handset can be controlled with mouse gestures and keyboard input

Control section buttons for playing a preset audio file on the handset, upload a preset audio to be played, record the audio session and close the audio session.

Audio streaming player, for the handset audio output

Browser


Dialing between two Smartphones on one Screen


Simultaneous NAS and RRC Analysis on single Screen


PCAP Analysis

1. Get the PCAP file for my current SITE Interactive session

2. Download and open the PCAP file


Outlook on Smartphone Recorder

Page 54



SITE QoE Outlook- Smartphone Testing andMonitoring Architecture

Virtual Server or Bare Metal

SITE UI

SITE CU

SITE SIMMUX

SITE VPN Server

Smartphone Access and

Monitoring Server


Example: Access Smartphone Testing via SITE GUI

SMARTPHONE STUDIO

After SITE login a user candownload theSmartphone Studio via dedicated link


Example: Smartphone Interactive And Visual Scripting

The user can Interact withall devices as an Interctivesession and it can createMonitoring Script


Example: QoE Testing and Monitoring on Smartphones

RecordScript name: speedtest

Interactive

Accessing the SITE Interactive Session

ScheduleA new testcase called Smartphone Monitoring will be created that takes the following parameters as input:

-script name (speedtest)-Smartphone Type (SGS7)-Smartphone Location(s) (Germany*)-SIM

ResultsAfter each test run an XML result file will be createdby the ensamble server that will be imported in a new Smartphone KPI DB in SITE. Hence reporting and analysis can be seen via SITE Reporting


QoE Smartphone Testing - Features

Android Flagship Models: Samsung S7, HTC, LG5, Sony, Nexus etc.

New Device Integration: 1-2 weeks

Interactive Sessions with up to 4 Smartphones

Visual Scripting for Monitoring for 1 or 2 sided testcases

Any App Monitoring (Skype, whatsapp, Viber, myVodafone, myATT, facebook, instagram)

Any Browser Transaction Monitoring (chrome)

Any Mobile Service Monitoring (voice call, sms, SupServ)

SIMMUX support

QoE KPIs as percieved by the user on the actual Smartphone

QoE SITE Reporting

SITE for QoS & QoE Testing

2016 2017Apr Jun Aug Oct Dec Apr Jun Aug Oct Dec 2018

SITE Roadmap Note: This roadmap is subject to change without notice

CSFB on Core (S1 and Iu-cs) GSM A Interface Hybrid Solution (SS7+SIGTRAN) CAT IQ Interface Support G.722 HD Voice Support for Mini VoIP Probe

Release Independent Featuresfrom SITE 2.7

Features:

Video Quality Testing package, Phase 1: QoS testing for HTTP based Adaptive Bit Rate Streaming protocols such as: HLS, DASH, MSS, and HDS.

Enhanced OTT App Testing on HCLU: Viber, Skype, Instagram, Twitter, Facebook, SnapChat, DropBox

Virtualization of SITE Central Components (CU, CUX, UI, VPN, KPI): Capability to deploy SITE central components as VMs on compatible infrastructure

SMSoWiFi: QoS testing for SMS sent over WLAN access

Hardware/Interfaces:

Nomadic Probe for testing Cat.6 LTE Advanced networks, supporting speeds up to 300Mbps

Samsung Galaxy S6/S7 support in the Hybrid CLU

Maintenance:

NEW GUI: Test Setup and Trace Analyzer: Test case overview, parametrisation and scheduling. Result analysis using a redesign trace analyzer

Support for Enhanced Voice Services (EVS) Codec for VoLTE test cases

3.1 3.2 3.3 and OutlookFeatures:

Web Application Testing Phase 1: Software package for recording and replaying user interactions with web pages or web applications

High Performance KPI server: 3x performance improvement, increased capacity

Netflix OTT Video Quality Testing

Hardware/Interfaces:

Smart Probe for testing Cat.6 LTE Advanced networks and above

Maintenance: NEW GUI Reporting and Alarming IuCS interface support for WB-NB and

NB-WB calls Video Quality Testing package Phase 2:

QoS testing for HTTP based Adaptive Bit Rate Streaming protocols such as: HLS, DASH, MSS, and HDS

Outlook:

BigData Analyzer for Identifying User Profiles

Passive APP-based Probe

SRVCC on Air interfaces Package

CSFB Air Interface (Voice Quality capable)

DIAMETER Routing Agent - Use Case: LTE Roaming Radar

Dedicated M2M Interface

IPTV Package

IoT Service Testing

Test Case Flow Editor

iPhone Support on SITE Interactive

SIM Tool Kit Support

SRVCC on Core Interfaces Package

Features:

eMBMS: QoS testing for LTE Broadcast services

OTT Video Monitoring

Video Quality Testing for Set-Top-Box via HDMI

Hybrid MLU: On the move testing using Smartphones

Network Functions Virtualization Testing

Support GSMA Embedded SIM (eSIM)

WiFi Roaming offload (Passpoint2.0)

Smartphone Interactive Recording Phase 1: Software solution to record and replay actions performed during Smartphone Interactive

Maintenance:

Web Application Testing Phase 2

SITE Interactive Recording Phase 2

SITE for QoS & QoE Testing


Thank you

[email protected]

SITE Roadmap 2016 and Beyond...LTE-A Testing using Cat.6 Smartphones SITE 3.0 offers the possibility...

Documents

Transcript of SITE Roadmap 2016 and Beyond...LTE-A Testing using Cat.6 Smartphones SITE 3.0 offers the possibility...