Project quality (and test process) metrics

Project quality and test process metrics

Zbigniew Moćkun, Tomasz Rękawek

Client/partner logo

Who are we?

Agenda

• Introduction to metrics• Code quality metrics

− Metrics− Sonar as example

• Project quality and testing process metrics− Metrics− Example quality report

What is it about?

Metrics

Client/partner logo

Why do we use metrics?

• Project overview• Control a process• Control risks• Force the use of good practices• Project audits• Projects / methodologies / ... comparison

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore

I realized that metrics...

Encourage me to ask questions

Be careful with metrics!!

Processes

People

Type of measurements

• Trend• KPI (Key Performance Indicator)

− Lower threshold− Higher threshold− Value

• How to set correct KPI?

Not only about numbers

Qualitative vs. Quantitative- case study

Level of details

• You cann’t measure everything• Choose the most important metrics• Monitor and ask questions • Looking for details when you found a problem

Metrics gathering

• Analysis of live data (on request)• Analisis based on daily values (can be run as cron job)• Data vary in time manner• Froze data for analysis

Visualize metrics

• Use charts if possible• Use colors as status (green, amber, red)• Add description• Remember about recommendation (your feeling)\

− Subjective assessment is important too

Quality is everywhere

Project Quality metrics

Client/partner logo

Code quaility metrics are covered by Tomek Rękawek presentation which can be found here:

http://www.slideshare.net/TomaszRkawek/code-metrics

Code quality metrics

Client/partner logo

QA vs. code quality metrics

• Part of assurance

• Define KPI with Technical Lead

• Understand what you measure and why

• Discuss trends, exception with TL

• Motivate developers to keep high coding standards

Project quality metrics

Client/partner logo

Why do we test?

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test

Testing = measuring

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test

Source: wikipedia

Dev – QA cooperation

Metric by actions

One measurement, two metrics

• Acceptance rate / Internal Quality− Acceptance actions/all qa actions

− All QA actions = accept or reject

− Measure quality of software send to QA

• QA velocity− Sum off all actions

− Divide by number of QA Engineers assigned to project

− QA velocity depend on software quality that comes to qa

− Why testing takes so long?

Test Case Execution metrics

• Planned test cases vs. written test case (Preparation, UAT)• Number / percentage of executed test cases (Acceptance, regression, UAT)• Number of passed/failed test cases (Acceptance, regression, UAT)• Test Case metrics mean nothing

Requirement coverage

• Requirement coverage− How many requirements have test cases?

− How many requirements were tested?

− How many requirements passed?

− Can we measure coverage if we use exploratory approach?

• Requirement traceability− Defect – test case - requirement

Bugs metrics – S curve

• S Curve

• Open against Resolved

• Wages − Blocker: 10

− Critical: 8

− Major: 6

• No Major or higher rule− 5 Trivials = 1 Minor

− 5 Minors = 1 Major , ...

External quality

External Quality

• All found bugs = internal + external

• Shouldn’t be higher than 0.15 (15%)

• Sometimes divided on two parts− Found during UAT (Used Acceptance Test phase)

− Found during Live (on production)

• Sometimes used wages

• Compared with internal shows our testing process quality

The importance of metrics

• Example A− Story comes to QA

− QA found 1 major issue and reject (1 day)

− Dev fix it (1 day)

− QA test it again, 1 major issue found and reject(1 day)

− QA accept (1 day)

− Sum: 5 days, 2, rejects, 2 issues only

• Example B− Story comes to QA

− QA found 1 critical, 1 major and 3 minor issues and reject story (1 day)

− QA test it again and accept (1 day)

− Sum: 4 days, 1 reject, 5 issues

• Acceptance rate against raised issues metric

Bug Taxonomy

• Phases− Project: Discovery, Development, UAT, Live

• Test Case lifecycle− Requirements/Documentation, Test Scenario, Test Execution

• Risk areas

• Application modules/components

• Functional/non-functional

• Product specific areas− CMS: author, publish, frontend, backend

Defect Density - components

Defect Density - Acceptance vs. Regression (automation vs. manual)

Defect DensityApplication specific

Defect Density – won’t fixDo we test in right way?

Automation - coverage

• Defects found (manual vs. automated)

• Coverage− Manual vs. automated test cases

− High priority

− Application specific Author vs. publish

Other metrics

• Performance monitoring trend as example

• Client side grades (webpagetest.org as example)

• QA queue

• Agile metrics− Lead Time (Open to Resolved time)

− Idle Time (time spend in queues)

Qualitative metrics

• Client feeling− Surveys

− Demo

− Just try to talk

• Tester / team subjective opinion− Recommendation

− Feeling

− Usability

Agile vs. Waterfall

• Quality metrics are independent against:− Methodology

− Approach

• Two main test approach can be used in both− Exploratory vs. scripting

• Agile put more emphasis on qualitative metrics

Example report

References

• http://www.kaner.com/pdfs/BugTaxonomies.pdf

• ISTQB Advance Test Manager Syllabus− http://www.istqb.org/downloads/viewcategory/46.html

• Michael Bolton blog post− http://www.developsense.com/blog/2009/01/meaningful-metrics/

• Douglas Hoffman− http://www.softwarequalitymethods.com/Papers/DarkMets%20Pape

Poznań Testing and Quality Group

• Local group

• Next meeting: 28th of November− Bogdan Bereza-Jarociński: Quality in Agile – quality at all

− Jakub Bryl: How to organize security testing

Project quality (and test process) metrics

Technology

Transcript of Project quality (and test process) metrics

Video Quality Metrics

SFGH Inpatient Dashboard: Key Operational and Quality Metrics 10... · SFGH Inpatient Dashboard: Key Operational and Quality Metrics NOTES/DEFINITIONS Operational and Quality metrics

Quality Metrics - Arts Council England...1. Chapter One: Introduction 14 1.1 The Foundations of the Quality Metrics National Test 14 1.2 The importance of co-production and an arts

Schedule Quality Assessment Metrics

Software Quality Metrics Overview - Cal Polyusers.csc.calpoly.edu/~djanzen/setopics/metrics/SoftwareMetrics.pdf · Software quality metrics are a subset of software metrics that focus

Software Test Metrics QA

Quality Analysis with Metrics

Full-Reference Image Quality Metrics: Classification and ... · 2 Classification of Image Quality Metrics 4 2.1 Existing Classification of Image Quality Metrics 4 2.2 Proposal

Machine Translation Quality Metrics

METRICS AND TEST PROCEDURES FOR DATA …arizona.openrepository.com/arizona/bitstream/10150/596445/1/ITC...METRICS AND TEST PROCEDURES FOR DATA QUALITY ESTIMATION IN THE AERONAUTICAL

Quality Metrics Payment - Colorado

Quality Process KPIs Metrics

Software Quality Metrics III. Software Quality Metrics The subset of metrics that focus on quality Software quality metrics can be divided into: End-product.

Requirements for Data Quality Metrics for Data Qualit… · Additional Key Words and Phrases: Data quality, Data quality assessment, Data quality metrics, Require-ments for metrics

Quality Metrics in Pharmaceuticals

Agile code quality metrics

Quality Metrics - Culture Counts...1.1 The Foundations of the Quality Metrics National Test 14 1.2 The importance of co-production and an arts sector-led initiative 14 1.3 The Key

Sw quality metrics

Software Metrics/Quality Metrics Software “Quality” Metrics: –Product –Pre-Release and Post-Release Process –Project Data Collection Product Characteristics.

Software Quality and Metrics