R2 Test strategy

Test strategy

Testing is still a key challenge for OPNFV

All the projects must manage their test strategy (unit, fonctional, security, performance)

Several specific test projects have been validated by TSC- Define testcases- Perform tests not covered by a single project- Create tooling- Study Performance end to end

R1

In R1:- BGS managed its sanity tests- Functest created and managed 4 test suites- Octopus was in charge to run Functest at the end of a fresh installation- Pharos was in charge of providing documentation on hardware and testbeds

R2

- More testcases (vIMS, ?)- Better coverage- Detect missing test domains- Analytics system

Pharos

Common

VPerf YardstickQTIP Transformers

Octo

pus

Functest

In R1Note: to be discussedCommon could be managed by Functest (avoid creation of new project..)

DB- Traffic profile

- Results

…

Overview of the different projects

Test projects

Testbed

Project Interactions

Test projects

Pharos (federation of testbeds)

Octopus (ContinuousIntegration)Test are run on

Tests are integrated in CI

Pharos exposesAvailable labs

Octopus triggers build/deploy/tests on pharos labs

Artifact

Consumes results generated by CI Push results

Is not a test project

Is a testbed management project– Develop a tool to dynamically retrieve configuration (hardware,

network, tooling)– Manage history of test campaigns– Create Guides

– HW Requirements– Best practices

– Create a tool to manage testbed resource– Create a Dashboard (describing the available testbeds, their

capabilities,…)

Pharos

Test projectsProject Scope R1 R2 deliverablesCommon Common functions for testing project

- definition of default configuration for tools

- definition of format for output files- Collection of test profiles/payloads,..- coverage - Collection of results + DB + Analytics- Test Dashboard

- Prescription (format, configuration,..)- Test Dashboard

– API for Pharos– Test coverage– testcase search engine- Analytics tooling through API for any test project- traffic profile definition and collection

Functest(Morgan Richomme)

- new functional tests- integration of upstream suites (R1)- update of existing suites

- Tempest + rally suite- ODL suite- vPing- 4 suites integrated in CI

- vIMS testcase- vSBC?- new testcases in upstream suite- Move from Tempest smoke to full?

Vperf(Mike Lynch)

Vswitch perf testing Dev & tests done but not integrated in CI for v1

New testsIntegration in CI

Yardstick(Ana Cunha)

In -VM testingToolingTest suites

- Methodology for Infrastructure verification from VNF perspective, aligned with ETSI-NFV TST001- Generic test cases for Compute, Storage and Networking areas- OPNFV Infrastructure verification from VTC and SFC perspective- Mobility Traffic Profile as Test Stimuli

Qtip(Wenjing Chu)

Transformers(Michael Wiegers)

Note: For R2 Functest (Functional & Common Testing) will manage common part

Test Coverage by component

Component OPNFV project External tool

OpenStack Functest Tempest, Rally

ODL Functest/ODL Robot

KVM Yardstick

OVF Yardstick

VNF Functest (vPing, vIMS, vXXX)Yardstick

Spirent/Ixia tooling

Ceph Functest?, Yardstick

Test Coverage by domain

Domain OPNFV project Comments

Security

Installation BGS

OAM

ETSI domains- vEPC- vIMS- Mobile base stations- Fixed Access Network- vHGW- vCDNs- Network Functions

Virtualisation IaaS- Virtual Network

Function ‘aaS- Virtual Network PaaS- VNF Forwarding Graphs

FunctestYardstick

Is it possible to automate/integrate some of the 36 ETSI PoCs into OPNFV?

OpenStack Telco WG testcases

https://wiki.openstack.org/wiki/TelcoWorkingGroup#Use_Case_Definition

Common needs: Big data / analytics

Problem: Thanks to CI and test projects we will get lots of results logs coming from different testbeds (pharos federation including different hardware, configuration), the idea is to perform some analytics on these tool sets to study the influence of hardware on performance

The tool could include:- API for collecting the results of all the tests (from the artifact)- A way to associate a toolset and a configuration (description needed

from Pharos)- A NoSQL DB to store the results- Analytic tooling - Analytics script- Display and share results

Reference Testcases

ProblemEach OPNFV subproject is responsible for its test strategy. Specific test projects have been created. It is difficult to get the big picture of the testcases by checking project by project. List of testcases shall be automatically generated. It shall be possible to search somewhere if testcase has already been run, in which project. It would also be interesting to have an history of the testcases (testcase X from project Y is run oin testbed Z1,Z2, ….)

The task will thus mean:- Automation of production of a document collecting all the

testcases- Keywords + search engine system: goal know if testcase

already considered, which suite, how to automate it,..- Template for test case documentation

Questions