Thursday, October 7, 2010

Effective use of Continuous Integration by QA

Ashwini Malthankar

Lead Consultant QA

ThoughtWorks

Agenda

Problem Statements

About Continuous Integration

How does Continuous Integration help resolve problem

Case study on large scale distributed project

Open source / commercial tools

Q & A

Scenario for Thought Exercise

Given it’s a Thursday today when we have Demo

to customer then what is the state of a Tester?

We have to showcase new facebook feature.

Problem Statements Identifying good build for Demo, Release or Regression testing

Testing on various environments like the Grid shown below

OS App

Servers Browsers

Versions DB

Windows WebSphere Firefox ? Oracle

Linux Weblogic IE ? MySql

mac JBoss Safari ? Teradata

Why is it important to solve

Constant availability of a "current" build for testing, demo, or release

purpose and to avoiding last-minute chaos at release time.

About CI ?

http://twistcruise:8153/cruise/tab/pipeline

High level Overview of Build Server

Pipeline

Stages

Jobs

Jobs

Jobs

Jobs

Pipeline in the box

Requirement

Scope

Document

+ Test cases

Design

Test Plan +

Test case

Coding

System

Testing

Product

Build

Functional

Automation

Test Suites

Test

framework

BackEnd

Testing

API level

testing Defects Cycle

Reports Reports

R

e

p

o

r

t

s

Unit

Testing

Continuous Build

Integration

Code

Coverage

Project/Product Development Process Flow

Requirement

Usecases

+ Test cases

MSL

Test Plan +

Test case

System

Integration

Testing

Ruby on Rails

RSpec

Postgress

Stub

available

for Testing

Defects Cycle

Reports Reports

R

e

p

o

r

t

s

RCov

UI Automation /

Cucumber

github

Mingle

Go

Development Tools usage Flow

How does Continuous Integration help to resolve current problem

• MATs /Smoke tests, SRGs, LRGs running it continuously to quickly Identify Demo, Release candidate build on any environment.

• frequent code check-in pushes developers to create modular, less complex code

• metrics generated from automated testing and Continuous Integration

Give visual representation of CI Schedules

Performance Builds

QA Builds

Platform Specific Builds

Q & A

Project details

“largest distributed agile project“

for every check in, a build is triggered, installers deployed continuously

New release every 6 weeks.

260 Build & Deployment Servers

3 active development branches, 'branch per release pattern', SVN model.

Application has Service Oriented Architecture, having multiple tiers and is Web-based

15 Independent Installers to deploy on Dev, QA, UAT, Performance, Staging, Production environments

Case study on large scale distributed Project

Database

Services(15)

Channels channels

Third Party

Apps

channels

Deployment Architecture & Environments

SVN

Build and Installer

Compile

Unit test

Integration Test

Smoke test Publish

Build test

Iteration -0

Iteration – 1

Build test

Build and Installer

Compile

Unit test

Build and Installer

Compile

Unit test

SVN

Build and Installer

Compile

Unit test

Smoke Test Publish

Last good

Last

good

Last good

Iteration - 1

SVN

Build&Installer

Compile

Unit test

Integration

CodeQaulity

Smoke

Verification Showcase QA/validation

UAT Performance Staging

Build Cycle and Flow

Dev-feedback(28

build in parallel)

Generate installer

Install the installer

Test the installer Publish the

installer

Get the installer Install the application

Verify the installer

Publish the installer

Get the installer Install the application

Verify the installer

Publish the installer

Trunk Build

Test Box Deployment

Showcase env Deploy

Build Layers

Continuous Integration things to watch

• Initial setup time required

• Well-developed test-suite required to achieve automated testing advantages

• Large-scale re-factoring can be troublesome due to continuously changing code base

• Hardware costs for build machines can be significant since CI like production makes more sense

• Final feedback is only with system in production

What more can be Done

• Every build with Performance test execution to know the performance of the application per check-in.

• Code health check apart from Code Coverage which we can be integrated in the build with tools like Code City. It can give the code complexity metrics to know how complex the code is getting per check-in.

• Continuous Integration Continuous Deployments Continuous Regression Continuous Performance Continuous Software Health

What are other commercial tools available in the market

AnthillPro — continuous integration server by Urbancode

Apache Continuum — continuous integration server

Bamboo — commercial continuous integration server by Atlassian Software

Go — Commercial agile build and release management software- ThoughtWorks

Hudson — MIT-licensed, written in Java, runs in servlet container

TeamCity — commercial continuous-integration server by JetBrains with free professional edition

Q & A