Automated Testing Environment by Bugzilla, Testopia and Jenkins

Chin Pin Chang04/07/2014

1

2

Outline

• Bugzilla• Testopia• Jenkins • Automated Testing Environment (Hypervisor

Testing as an example)– Automated Testing Process– Example

• References

3

Bugzilla Introduction

• Bugzilla is a Defect Tracking System and it is server software designed to help you manage software development

4

Bugzilla Features

• Excellent security to protect confidentiality• Optimized database structure for increased

performance and scalability• Integrated email capabilities• Comprehensive permissions system• Editable user profiles and comprehensive email

preferences• Proven under fire as Mozilla's bug tracking

system

5

Bugzilla Requirements

• Hardware– CPU : 3 GHz– RAM : 4 GB or more recommended– Hard Disk : 50 GB of free space is large enough

• Operating System : Including Windows, Linux, and Mac OS X

• Software– Perl : 5.12– Database Server : Bugzilla supports MySQL 5.5,

PostgreSQL 8.3, Oracle and SQLite– Web Server : Apache 2.2

6

MySQL Installation

Step 1. Update

Step 2. Install MySQL

Step 3. Set the password

7

MySQL Installation (cont.)

Step 4. Login MySQL– user– password

8

MySQL Configuration

Step 5. Create an user for Bugzilla

Step 6. List users

9

MySQL Configuration (cont.)

Step 7. Create database for Bugzilla

10

Apache Configuration

Step 1. Install Apache2

Step 2. Modify Apache2 configuration file

11

Bugzilla Installation

Step 1. Download Bugzilla : http://www.bugzilla.org/

Step 2. Decompress the file to /var/www/

Step 3. Modify folder name and switch to /var/www/bugzilla

12

Bugzilla Installation (cont.)

Step 4. Execute checksetup.pl

13

Bugzilla Installation (cont.)

Step 5. Install module

14

Bugzilla Installation (cont.)

Step 6. Re-run checksetup.pl

Step 7. Need to modify localconfig

15

Bugzilla Installation (cont.)

Step 8. Modify Bugzilla configuration

Step 9. Modify permission

16

Bugzilla Installation (cont.)

Step 10. Re-run checksetup.pl

Step 11. Set up user information

17

Bugzilla Installation (cont.)

Step 12. Modify Apache configuration and restart Apache2

18

Bugzilla Installation (cont.)

Step 13. Installation complete

19

Testopia Introduction

• Testopia is a test case management extension for Bugzilla

• Testopia Requirements– Bugzilla– MySQL or PostgreSQL– Mozilla compatible browser– Additional Perl Modules

• Testopia Architecture– Test Plans– Test Cases– Test Runs

20

Relationship between Bugzilla and Testopia

• Since Testopia is an extension to Bugzilla, it relies on many of the same objects that Bugzilla uses to track bugs

• Testopia is comprised of several objects that are interdependent, so they make managing the testing procedure possible

21

Relationship between Bugzilla and Testopia (cont.)

• Component– A Bugzilla component – An attribute of a product

22

Relationship between Bugzilla and Testopia (cont.)

• Product– The product name belong to Bugzilla

23

• Milestone– A Bugzilla object– Testopia builds are associated with milestones

Relationship between Bugzilla and Testopia (cont.)

24

• Test Plan– At the top of the Testopia hierarchy are Test Plans. Before you can do anything else in

Testopia, you need a Test Plan – Test plan will serve as the storage point for all related Test Cases and Test Runs and it will

act as the Dashboard for your testing

Relationship between Bugzilla and Testopia (cont.)

25

Relationship between Bugzilla and Testopia (cont.)

• Test Case– Test Cases are the heart of all testing – Test Cases spell out what steps to take while running a test and what outcomes to expect – Test Cases are associated with one or more Test Plans and with zero or more Test Runs

26

Relationship between Bugzilla and Testopia (cont.)

• Test Run– Once you have defined a set of Test Cases, you are ready to run through those tests in a

Test Run – Each run is associated with a single Test Plan and can consist of any number of Test Cases

from that plan

27

Relationship between Bugzilla and Testopia (cont.)

• Build– Software development is usually an iterative affair. In Testopia, each iteration is called a build– In software testing, a string denoting the compiled results of a period of development– Builds are often associated with milestones of the project. This is reflected in the build's

relationship to Bugzilla's target milestones object

28

Relationship between Bugzilla and Testopia (cont.)

• Case Run– A Test Case-Run is the record of how a particular Test Case fared in a particular run for a given

Build in a given Environment

29

Relationship between Bugzilla and Testopia (cont.)

• Category– A property of a product that is used to classify Test Cases

30

Relationship between Bugzilla and Testopia (cont.)

• Environment– A list of the surrounding conditions that a test run is performed in

31

Testing Process of Testopia

The testing process in Testopia is as follows : Step 1. Create a Product and multiple ComponentsStep 2. Create a Test PlanStep 3. Create Test CasesStep 4. Create a Build

32

Testing Process of Testopia (cont.)

The testing process in Testopia is as follows : Step 5. Create EnvironmentStep 6. Create a Test RunStep 7. Execute the Test RunStep 8. Produce a test report on Dashboard

33

Testopia Installation

Step 1. Download Testopia– http://www.mozilla.org/projects/testopia/

Step 2. Decompress the file to Bugzilla folderStep 3. Run checksetup.pl and install module

34

Access to Testopia

Step 4. Re-run checksetup.pl

Step 5. Installation complete

35

Jenkins Introduction

• Jenkins is an open source continuous integration tool written in Java

• Jenkins provides a web-based user interface

36

Jenkins Features

• Easy installation• Easy configuration• Change set support• E-mail integration• Test reporting• Plugin support

37

Jenkins Installation

Step 2. Access to Jenkins– http://localhost:8080

Step 1. Install Jenkins

38

Install Testopia plugin on Jenkins

39

Integrate Jenkins with Testopia

• This plug-in integrates Jenkins with Testopia and generates reports on automated test execution

• With this plug-in you can manage your tests in Testopia, schedule and control in Jenkins

• It reads TAP test report formats, used to update Testopia test case's executions

40

Integrate Jenkins with Testopia(cont.)

• Testopia configuration section

Auto Test Process

Test Controller

Ubuntu 13.10

Bugzilla 4.4

Testopia

Jenkins 1.5

Shell scripts (expect)Shell scripts (expect)

Shell scripts (expect)

Step 1. Retrieve Test Case from Testopia

41

1.

Step 2. Invoke corresponding script to initialize the test environment

2. Auto testing environment for ITRI ARM hypervisor :

Auto Test Process (cont.)

Test Controller

Ubuntu 13.10

Bugzilla 4.4

Testopia

Jenkins 1.5

Shell scripts (expect)Shell scripts (expect)

Shell scripts (expect)

Ubuntu 12.04 LTS

ITRI ARM Hypervisor

Guest VM

Guest VM

Guest VM

Test Target

Step 3. Initialize the test environmentStep 4. Launch necessary Guest VMs

42

3.

4.

Auto Test Process (cont.)

Test Controller

Ubuntu 13.10

Bugzilla 4.4

Testopia

Jenkins 1.5

Shell scripts (expect)Shell scripts (expect)

Shell scripts (expect)

Ubuntu 12.04 LTS

ITRI ARM Hypervisor

Guest VM

Guest VM

Guest VM

Test Target

Step 5. Execute testing tasks

43

5.

Auto Test Process (cont.)

Test Controller

Ubuntu 13.10

Bugzilla 4.4

Testopia

Jenkins 1.5

Shell scripts (expect)Shell scripts (expect)

Shell scripts (expect)

Ubuntu 12.04 LTS

ITRI ARM Hypervisor

Guest VM

Guest VM

Guest VM

Test Target

Step 6. Generate and return test result (by TAP, Test Anything Protocol)

44

6.

Auto Test Process (cont.)

Test Controller

Ubuntu 13.10

Bugzilla 4.4

Testopia

Jenkins 1.5

Shell scripts (expect)Shell scripts (expect)

Shell scripts (expect)

Step 7. Return test result to Jenkins

45

7.

Auto Test Process (cont.)

Test Controller

Ubuntu 13.10

Bugzilla 4.4

Testopia

Jenkins 1.5

Shell scripts (expect)Shell scripts (expect)

Shell scripts (expect)

Step 8. Write corresponding result to Test Case in Testopia based on TAP

46

8.

47

Auto Test Example

Step 1. Add user on Bugzilla to execute auto testing

48

The process to create a Test Plan

• To create a new Test Plan in Testopia – Step 1. Click the "New Plan" link in the Bugzilla footer– Step 2. Enter a name for this Test Plan– Step 3. Select a product from the product list– Step 4. Select a type for this plan – Step 5. Select a product version– Step 6. Type or paste your plan document into the Plan Document Editor– Step 7. Click the "Submit" button

49

Step 2. Create a new Test Plan

• Plan Name – Test Plan name

50

Step 2. Create a new Test Plan (cont.)

• Product – Bugzilla product name

51

Step 2. Create a new Test Plan (cont.)

• Product Type– The testing type for this Test Plan

52

Step 2. Create a new Test Plan (cont.)

• Product Version– The Bugzilla product version

53

Step 2. Create a new Test Plan (cont.)

• Plan Document– This is document that spells out the type of testing and testing methods used

for this plan

54

The process to create a Test Run

• To create a new Test Run in Testopia – Step 1. Click "Create a New Test Run" link on your Test Plan or the New Run link in the footer– Step 2. Select which Test Cases to include – Step 3. Enter a Summary for this Test Run– Step 4. Select a build from the Build list or type the name for a new one – Step 5. Select an environment from the Environment list – Step 6. Click the "Submit" button

55

Step 3. Create a new Test Run

• Product Version– The Bugzilla product version

56

Step 3. Create a new Test Run (cont.)

• Run Manager– The user in charge of this Test Run

57

Step 3. Create a new Test Run (cont.)

• Target Completion Rate– The expect completion rate of target

58

Step 3. Create a new Test Run (cont.)

• Build– The default product build

59

Step 3. Create a new Test Run (cont.)

• Environment– The environment in use for this Test Run

60

Step 3. Create a new Test Run (cont.)

• Target Pass Rate– The expect pass rate of target

61

Step 3. Create a new Test Run (cont.)

• Summary– A short description of this Test Run

62

Step 3. Create a new Test Run (cont.)

• Notes– This field is a place to make notes and observations about this run

63

The process to create a Test Case

• To create a new Test Case in Testopia – Step 1. Click "Create a New Test Case" on the plan page or

New Case from the footer– Step 2. Enter a short description of your Test Case in the Summary field– Step 3. Select a Category– Step 4. Add a default tester – Step 5. List the steps for testing in the Action field – Step 6. Provide the expected outcomes in the Expected Results field– Step 7. Click the "Submit" button

64

Step 4. Create a new Test Case

• Summary– A short description of the Test Case

65

Step 4. Create a new Test Case (cont.)

• Default Tester– The person is assigned to this Test Case

66

Step 4. Create a new Test Case (cont.)

• Alias– A unique string that uses to identify test result

67

Step 4. Create a new Test Case (cont.)

• Priority – The level of testing

68

Step 4. Create a new Test Case (cont.)

• Category– The product category

69

Step 4. Create a new Test Case (cont.)

• Estimated Time– The estimated time that this Test Case should take to complete

70

Step 4. Create a new Test Case (cont.)

• Bugs– You can attach bugs to your Test Cases

71

Step 4. Create a new Test Case (cont.)

• Blocks– If this Test Case prevents others from being run their ID numbers will appear

here

72

Step 4. Create a new Test Case (cont.)

• Status : Test Case status – PROPOSED : This test case is a work in progress and has not been reviewed for accuracy. It is therefore not ready to be included in test runs– CONFIRMED : This test case has passed review and is ready to be included in future test runs. Only test cases with this status can be included in new test runs– DISABLED : This test case is no longer applicable to current testing

73

Step 4. Create a new Test Case (cont.)

• Add Tags– A user defined string used to classify Test Cases

74

Step 4. Create a new Test Case (cont.)

• Requirements– The requirement number or URL to a document containing the requirement

this Test Case is designed to test

75

Step 4. Create a new Test Case (cont.)

• Automated – Automatic Test Cases are run by a script

76

Step 4. Create a new Test Case (cont.)

• Scripts – The name of the script that runs this Test Case

77

Step 4. Create a new Test Case (cont.)

• Arguments – The specific arguments sent to the script

78

Step 4. Create a new Test Case (cont.)

• Add to Run– Entering a run ID here will include this Test Case in a Test Run if it is not

already included

79

Step 4. Create a new Test Case (cont.)

• Depends On – If this Test Case requires other Test Cases be run before this one, their case

numbers should appear here

80

Step 4. Create a new Test Case (cont.)

• Action – This field lists the steps of testing

81

Step 4. Create a new Test Case (cont.)

• Expected Results– This details the expected outcomes of a test

82

Process to Launch Test Script

Configure

1. Clean Environment2. Check Image3. Launch Test Script

Start Test Script1. 3.

2.

1. Login to Host2. Start VM3. Ping Host4. Turn off VM5. Return Result

Testopia

Contents :　1. Parameters2. Functions

4.

1 . Retrieve Test Script

83

Set up a new project

Step 5. Create a project on Jenkins to invoke Test Script

84

Step 6. Set Testopia configuration section

• Testopia Version– The version name of Testopia

85

Step 6. Set Testopia configuration section (cont.)

• Test Run ID– The Test Run ID of Testopia

86

Step 6. Set Testopia configuration section (cont.)

• Single Test Build Steps– The command to execute by single step

87

Step 6. Set Testopia configuration section (cont.)

• Iterative Test Build Steps– The parameters of Testopia are used to execute by test cases

88

Step 7. Set Result configuration section

• Test Anything Protocol (TAP)– TAP is a simple text-based interface between testing

modules in a test harness– It allows individual tests to communicate test results

to the testing harness

89

TAP File Example

• The path of the file

• Success contents

• Fail contents

90

Launch The Project

Step 8. Start this project on Jenkins

91

Jenkins Console

Step 9. Monitor the shell scripts

92

Jenkins Result

93

Return Result to Testopia

94

Question

• Mail:walkerchang@iii.org.tw

95

References

• http://www.bugzilla.org/• https://developer.mozilla.org/en-US/docs/Mozilla/

Bugzilla/Testopia#Requirements• http://blog.codylab.com/testcase-management-

using-testopia/• https://wiki.jenkins-ci.org/display/JENKINS/

Testopia+Plugin• http://testanything.org/• https://wiki.jenkins-ci.org/display/JENKINS/

Meet+Jenkins