Bringing collaborative test to life an example of community effort
-
Upload
comarch -
Category
Technology
-
view
771 -
download
0
Transcript of Bringing collaborative test to life an example of community effort
1
Bringing collaborative test to life:An example of community effort
Fred BlesserTechnology Manager – Symbian Foundation
Mariusz LasekBusiness Development Director – Comarch
Collaborative test
Collaborative Test consists of having the community contributing to the
validation and testing of the Symbian Foundation asset, with specific
focus on:
Feature development
Hardening
These tests
Shall be run on vanilla kits delivered by the Symbian Foundation to the
community
No differentiation
Shall be easily reproducible, without the need of specific equipment/tools,
except the ones available through the Symbian Foundation or open source
community
No discrimination
Test results shall be available for all to see and compare
2
Test working group update
Key current responsibilities:
Define the Symbian^3 test plan
Define the Symbian^4 test plan
Help the Release Council define the milestone acceptance criteria (proposals, etc)
Feature Complete
Stable
Participate in the validation and testing of Symbian^3
New members
Atelier
Symbian DevCo
Comarch
3
5
Comarch established in 1993
Founder & CEO Prof. Janusz Filipiak
Comarch is the largest Polish Software House
Quoted on the Warsaw Stock Exchange (Public company)
since 1999
Headquartered in Kraków, Poland
Well diversified in order to embed business stability and security
Developing products and rendering services for
Telecommunications, Banking & Finance, Industry &Utilities, Public
Sector, Trade and Small & Medium Enterprises
Comarch Intro
6
Independent software vendor selling
proprietary products, solutions and services
on the global market
Strong direction towards matured markets of
Western Europe and the USA
Offices located in Europe, Americas and
APAC
Market Capitalization – 225 M€
Customers on 5 continents in more than 30 countries,
with over 3000 successfully completed projects
Comarch Today
7
Customer Mobile Services Application Development
Platform Development
Customization and Variant Creation
User Experience Design
Customer Mobile Applications Comarch Drafter
Comarch VOM (Targeted Adds)
Comarch Mobile Insurance
Comarch Mobile Banking
Customers OEM
ODM
Telco operators
Banking
Public Sector
Retailers
8
Comarch Contribution
Events and Marketing
World Mobile Congress in Barcelona
Towel Day 2010 in Turku, Helsinki
Educating
Training Contribution
Bug Fixes
Code Contribution
Headset Simulator
Collaborative Test
Database
9
Bluetooth Headsets Simulator
Business case
Existing client – Mobile Services for OEMs
Code contribution
Services for new clients based on contributed solution
Consulting
Customization
Simulator controlled remotely by API and UI
Test automation support
Modular architecture
New profiles support
New platforms support
11
Bluetooth Headsets Simulator
Headsets Simulator
Package: /btservices/bluetoothengine/
/*
* Component Name: Headset Simulator
* Author: Comarch S.A.
* Version: 1.0
* Copyright (c) 2010 Comarch S.A.
* /
Code size: 30 kloc
Effort: 9 Man-Months
Current Status: Architectural Review
Symbian^3 system test plan
System test cases have been contributed by:
These test cases have now been prioritized: Priority 1: 32 test cases
Non-concurrent test cases; to be run with each kit
Priority 2: 152 test cases
All priority 1 test cases, plus some more, providing sufficient feature coverage
Will be part of FC criteria
Priority 3: 468 test cases
All non-concurrent test cases, plus some key concurrent ones
Will be part of Stable criteria
Priority 4: remaining test cases
To be run ad-hoc
13
Proposal for Symbian^3 stable milestone
Take the latest Symbian^3 PDK (3.0.3)
The community will run the tests on this vanilla PDK according to the test plan defined in previous slide
All priority 2 system tests cases
These results will constitute an intermediate milestone between the Feature Complete milestone and the Stable milestone
The Test Working Group will then have to analyze these results, and propose to the Release Council one of the acceptance criteria for the Stable milestone
All test results will be stored on the new tool contributed by Comarch: the Collaborative Test Database (CTD)
14
15
Collaborative Test Database (CTD) goes live during SEE’2010 event
New Symbian Package under Tools domain
Grzegorz Wachocki – Package Owner from Comarch
Project is hosted at Symbian.org
Source Code available
Contribution welcome
Defects managed by Symbian’s Bug Tracking
Manual available at Symbian’s Wiki
http://developer.symbian.org/wiki/CTD_User_Guide
Collaborative Test Database
16
CTD has been established to record and present the Symbian quality
Various PDK releases
Various test targets
CTD is to be used by community members
Community verifies and correct test results
Check-in / Check-out mechanism in order to improve community efficiency
Integration with Bug Tracking system
Collaborative Test Database
17
Every test result is confirmed by at least two members (users)
Single authority is not possible – cross-checking improves reliability of results
Once the test is check-out for the testing, the whole community is aware of that fact
Avoid overlapping in testing effort
Check-in of the final result doesn’t mean that it’s fixed for ever
Any user can change the verdict at any time – results always up to date & reflect the actual platform quality
Results and summaries available for everybody
Collaborative Test Database
18
Similarities to Test Management Tools
Recording Test Specifications– list of Test Cases
Grouping Test Cases based on the Domain & Sub-domain
Multimedia
Telephony
Kernel…
Verifying status of the each Test Case
Draft
Published
Un-published
Collaborative Test Database
19
What makes the CTD Different?
Designed for the purpose of collaboration -> use by the ecosystem in contradiction to regular test management tools
Build from scratches to reflect certain workflow typical for the Symbian community
Ad-hoc testing instead of test plans and sessions – test when you need and when you can
Open for everybody
The tool is open source
Results are freely available
Collaborative Test Database
“1 person-day per month” Each test working group member should if possible allow one
person to run tests for one day, each month, in order to gather results
Not mandatory
... but each member can offer to contribute more
Effort project-managed through the Foundation, then transition to CTD for:
Organize the test case list
Dispatch test cases to avoid redundant results
Follow-up on execution and bugs raised
Gather and collate test results
Follow-up with the bug squad
This is an effort that is being asked jointly and regardless of company size
But this effort is small when compared to the benefit it can bring to the Symbian platform and the Symbian Community
22
23
CTD will continue to evolve with new features in the pipeline
Collaborative test is a concept that can be applied to any open source community
Open information sharing
Provides standard test specifications and tools
CTD offers a centralized collection point for results
Hopefully this is bound to continue in the new world organization as announced this week
WHAT NEXT???