Functional Continuous Integration with Selenium and Hudson

CONFIDENTIAL Copyright © 2010 Constant Contact Inc. 1Copyright © 2010 Constant Contact Inc.

Functional Continuous IntegrationSeleniumRC and Hudson

January 25, 2011

David Jellison: Director, Quality EngineeringCollin Riley: Principal Quality Engineer


Agenda

About USContinuous Integration (CI) Concepts

■ Continuous Integration (CI) Concept■ Functional Continuous Integration (FCI) Extends the

concept■ SeleniumRC is part of the capability for an FCI strategy■ Continuous Integration Servers make CI easy

Constant Contact FCI Approach■ Hudson is a rich dashboard to orchestrate traditional CI &

FCI■ How Constant Contact uses Hudson and SeleniumRC for

FCI ■ What other pieces are in the FCI framework at Constant

Contact

Q&A


Speakers

David Jellison: Director, Quality EngineeringDavid has 18 years experience in test and development

management, and over the last 4.5 years he led agile development and test organizations with .Net, Rails, and Java stacks in SaaS organizations. David is passionate about test leadership and efficient software development practices. He is consistently has his hands in technological and agile practice continuous improvement.

Collin Riley: Principal Quality EngineerCollin is the architect of Constant Contact's use of Selenium

RC in our Java automation framework utilizing Hudson. This framework integrates with Quality Center, uses page objects, and other scalable features. Collin has 8 years Quality Engineering and automation experience and has been using Selenium RC for over two years.


Who is Constant Contact? (CTCT)

Constant Contact Corporate■ Helps small businesses, associations, and nonprofits

connect with their customers, clients, and members. (>400,000 subscribers)

■ Launched in 1998, Constant Contact champions the needs of small organizations and provides them with an easy and affordable way to build successful, lasting customer relationships.

Constant Contact Engineering■ ~200 people, ~16 Agile teams (½ product & ½ services)■ ~30 in Quality Engineering, 3:1 Dev:QE■ SaaS (Software as a Service), Java/JBoss/CentOS/DB2

stack, live up-time deployments to production■ Practice ScrumBan (blend of Scrum release management

and KanBan work item management)■ Use Java/Groovy test scripts with SeleniumRC and Java

libraries for functional tests

CONFIDENTIAL Copyright © 2010 Constant Contact Inc. 5CONFIDENTIAL Copyright © 2010 Constant Contact Inc.

CI Concepts


Assumptions

Familiar with the following:■ Selenium Remote Control (SeleniumRC)■ Agile Practices (what Scrum and eXtreme Programming

are)■ Source Code Repository (e.g. ClearCase, CVS, SVN, Git,

etc.)■ Integrated Development Environment (IDE - e.g. Eclipse)■ Infrastructure Monitoring (e.g. Nagios)■ xUnit test runner family (e.g. JUnit, NUnit, Test:Unit etc.)


Continuous Integration

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” – Martin Fowler (martinfowler.com)

Approach (Developer Focused)■ Developer creates unit tests along with new and re-

factored code■ Developer builds and runs unit tests locally often■ Developer often commits code to a source code

repository that is auto-built, tested, and packaged on a CI server

Benefits■ Significantly reduce integration problems■ Allows teams to develop cohesive software more rapidly

Note: The term ‘Continuous Integration’ was introduced as one of the 12 practices of eXtreme Programming.


Continuous Delivery

Deployment Automation■ Theme of reducing cycle time between idea and usable

software■ Culture of frequent reliable deliveries■ Configuration management for OS, Server, and

Application settings■ Manifest description of each environment

■ Machines and their allocation of applications■ Database and resource permissions■ Load balancer configuration

Test Environments■ Represent configuration of production (manifest for each)■ Rely on manifest and deployment automation for

changes■ Dedicated CI test environment to validate deployment

automation with each change


Functional Continuous Integration (FCI)

Prerequisites■ Continuous Integration (CI) practices■ Continuous Delivery (Deployment Automation) practices■ Dedicated CI test environment (minimum configuration)

CI Functional Tests■ Deployment tests that prove a system is properly

configured■ Application smoke tests that prove an application is

testable■ Extend the culture of break-the-build urgency

■ Passing tests can be delivered to test environments or production

■ Failing tests require immediate attention to resolving problems

Benefits■ Robust deployments to stable test environments■ Frequent deliveries to production■ Increased release confidence, lowering business risk


Challenges of scaling CI and FCI

Multiple Branches■ Rebasing often (merge back from trunk)■ Merge back to trunk as often as feasible

Multiple repositories■ Package integration installers often (automated

packaging)■ Package library includes and dependencies carefully

Many services and applications■ Distributed application architecture■ Clearly defined interfaces (with versioning)■ Test fixtures (test components in isolation)■ Health check requests for infrastructure monitoring (e.g.

Nagios)■ Deploy and functional test often■ Require functional regression tests with new features


Agile Testing

Collaborating with Developers■ Agile Testers collaborate with Developers■ Developers participate in test planning■ Testers participate in feature design■ Testers test as early as code changes are available

■ Domain expertise■ Traditional and exploratory testing■ Develop automated test cases

■ Developers work with testers to keep page objects consistent with application pages as code changes are delivered

■ Developers run regression tests prior to check-in■ Regression tests consistently pass with automated test

runs


Develop & Re-factor Applicatio

n Code

Develop Functional Tests

Deploy to

Sandbox

Run Regression Tests

With passing tests, commit changes

Agile Testing

■ QE & Dev work together to create, re-factor, and update functional regression tests

■ Dev deploys to sandbox system

■ QE runs tests against the sandbox

■ Dev updates the application code for problems found – no defects written

■ When all tests are complete and working, Dev commits changes

FCI


Constant Contact


Functional Test Needs

Test Case Manager (Quality Center)■ Representation of what is automated & automation runs

Test Runner (SeleniumRC / Junit / Page Objects)■ Browser & OS Testing■ Scalability

Run Manager (Hudson)■ Push-button runs■ Parallel test runs■ Run reporting and artifacts

Drill-in Run Reports (internal reporting application)■ Reporting Application leveraging Quality Center■ Aggregate run reports■ Dynamic charts


Choosing Hudson for FCI Dashboard

Familiar Java-based server■ Hudson was already CI server choice by Release

Engineering

Job flexibility■ Can configure job parameters, run externally, run a

collection of other jobs, produce artifacts, permission levels

Job Queuing■ Can create slave pools to queue many jobs to cycle

through the target slave pool

Job Agents■ Control over agent machines

Iframe views■ Can create dashboard applications on a report server and

integrate into Hudson views.

Support many plug-ins to CI tools


Test Automation Stack


FCI Flow – Authoring Tests

Quality Center■ Add Test Case(s) / Test Plan

Test Templates■ Pull Templates from QC into the Java Framework

Write the Test Code ■ Page Object classes■ Helper methods■ Web Class■ Reporting Class

Test locally■ Once Pass, mark in QC for Hudson


FCI Flow – Before Running in Hudson

Build the Java Project in HudsonPush to Hudson slaves

■ Java Project Jars■ Custom Firefox profile■ Host file entries

Restart Selenium Server Instances on SlavesPublish Hudson Jobs

■ Talk to QC and Push Jobs to Hudson■ Test Plans automated

■ Selenium based■ Webservice based

■ Run Types


FCI Flow – Running with Hudson

Launch a Run■ Hudson talks to QC for what to run■ Hudson adds Testplans to the build queue■ n Number of slaves pull from the queue in parallel

Test Plans report back■ To Hudson

■ Red or Green■ Archive html report with screenshots■ Junit output

■ To QC■ Full results for TestCases Status with framework Run Id■ Detail info for each step of the TestCase


Running Tests

Suite 01

Pool of Win Machines

Pool of Apple MachinesSmoke Daily

Suite 02

Suite 03

Suite 04

Build Test Jar

Environment 1

Repository

QC


Reporting

View Reporting■ Via a simple Java web application integrated into Hudson■ View Aggregate reporting for all TestPlans in a Run

■ Graphing■ Trends, Pies

■ Drill in grid based reports■ Compare run over past runs

Act on Reporting■ After Failure is investigated

■ Mark Failure as Known■ Defect■ Script Issue■ Configuration Issue


Run Results

■ Wiki regression status charts■ Email drill-in grid report


Drill into run comparison grid & detail report


Q&A


http://ConstantContact.com/Careers

Quality EngineeringSenior Quality Engineer - Event Marketing - Waltham, MASenior Quality Engineer - Contact Management - Waltham, MASenior Quality Engineer - Scalable Infrastructure - Waltham, MASenior Quality Engineer - Web Services - Waltham, MA

Software DevelopmentPrincipal Software Engineer - Online Survey - Waltham, MAPrincipal Software Engineer - Website - Waltham, MAPrincipal Software Engineer, Social Media - San Francisco, CASenior Software Engineer - Event Marketing - Waltham, MASenior Software Engineer - San Francisco, CASenior Software Engineer - Waltham, MASenior Software Engineer - Waltham, MASenior Software Engineer - Waltham, MASoftware Development Manager - Waltham, MASoftware Engineer - Waltham, MASoftware Engineer - Social Media - San Francisco, CA

Functional Continuous Integration with Selenium and Hudson

Technology

Transcript of Functional Continuous Integration with Selenium and Hudson