Making software testing greener

Making software testing greenerSkill Level: Introductory

Nageswararao V. Gokavarapu ([email protected])Software Test SpecialistIBM

11 Aug 2009

Recently there has been an increased emphasis on the effective utilization ofresources. Testing plays a key role in any software development lifecycle. Hence,effective utilization of resources in testing becomes the prerogative for any productteam. Read about the effective utilization of resources available for testing on bothhardware and software, thus leading to more efficient testing.

Introduction

There are two software testing methods, black-box testing and white-box testing,which are designed to cater to the Functional Requirement Specification (FRS) andSystem Requirement Specifications (SRS). SRS contains the complete descriptionand behavior of all the functions of software to be developed. FRS contains theexpected outputs for list of inputs for each function of the software.

In white-box testing, the test engineer has access to the internal data structures andalgorithms.

Black-box testing treats the software as a "black box," without any knowledge ofinternal implementation and the test engineer will perform tests on the software as acustomer would use the product. System testing is a method of black-box testingand is conducted on a complete, integrated system to judge the software compliancein the context of a FRS and SRS. The different types of testing that should beconsidered during product testing are:

• Installation testing

• Build verification testing

Making software testing greener© Copyright IBM Corporation 2009. All rights reserved. of 18

mailto:[email protected]

http://www.ibm.com/legal/copytrade.shtml

• Functionality verification testing

• Usability testing

• GUI software testing

• Security testing

• Load testing

• Stress testing

• Performance testing

• Scalability testing

• Regression testing

Each of these testing methods ascertains the software product quality in differentfunctional areas.

Virtualization is a modern approach for enhancing a system's ability to completetasks. Virtualization can be adopted in different areas in the computing world likehardware virtualization, operating system virtualization, network and storagevirtualization, presentation virtualization, and application virtualization. Operatingsystem-level virtualization is a server virtualization method where the kernel of anoperating system allows for multiple isolated user-space instances. Each of theseinstances may look and feel like a real server from the point of view of its end user.Each of these instances are isolated and the kernel often provides resourcemanagement features to limit resources for every instance. This article address howto adopt operating system virtualization methods in different types of testing andthereby minimizing the utilization of resources in terms of hardware, software, andtest engineers.

Types of software testing

The following are some examples of testing that will be performed during the testingphase of a software product.

Installation test

In this test, the test engineer installs and sets up the software. The testing processmay involve full, partial, or upgrades to the installation/removal of the product. Thesimplest installation approach is to run an install program, sometimes calledPackage Software. The test engineer should perform every possible configurationwith every mode of installation and should receive a better quality of testing so thatthe software can be released to customers with confidence.

developerWorks® ibm.com/developerWorks

Making software testing greener of 18 © Copyright IBM Corporation 2009. All rights reserved.


Build Verification Test (BVT)

This test involves the basic functional tests on the product to make sure the codechanges by the development team in that particular build will not affect or regressany pre-existing features in the product. Once this test is passed successfully andthe build is certified, then the functionality of the product is tested by the othermethods of testing. As part of globalization, the BVT will be executed in supportedlocales of the software product that is available.

Functional Verification Test (FVT)

The purpose of this test is to check that a product conforms to its functionalspecification. As part of FVT, the functionality of the product is tested for all itssupported features and then certifies the product readiness for the system test.

In Globalization Verification Test (GVT), the FVT is also tested in all supportedlocales to make sure the product functionality is consistent.

GUI software test

The GUI test tests the software that supports the graphical user interface to ensure itmeets the required specifications in FRS. Most products deliver system functionalityto the clients using a GUI. The test designer must take care that the test suite coversall the functionality of the product with GUI, and also ensure that the suite fullyexercises the additional functionality provided by the GUI. The GUI test also needsto be done in all supported locales and the test engineer needs to verify thestandards defined for different objects in GUI, for different locales.

Load test

In load test, multiple users access the program concurrently to simulate theexpected usage of a software program. To simulate the load, vendor-acquiredsoftware may be used. As many users access the application or productconcurrently, the CPU usage, network or disk I/O operations, and disk spacerequirements are greater for these type of tests.

Stress test

Stress test refers to tests that put a greater emphasis on robustness, availability, anderror handling under a heavy load on the product. The main goal of these tests is toensure that the software does not crash in conditions of insufficient computationalresources such as memory or disk space. It also determines the stability of theproduct by testing beyond normal operational capacity. As the product is tested inheavy load conditions, no other testing can be done on the server.

Regression test

ibm.com/developerWorks developerWorks®



This test reveals the software regressions. Regressions are determined whenever afunctionality that was previously working correctly stops working as intended withnew test run. Typically, regressions occur as an unintended consequence ofprogram changes during a bug fix. Regression testing include re-running previouslyrun tests and checking whether the previously fixed issues have occurred.Regression testing is used to determine the correctness and quality of the product.

The different types of tests discussed here require different environments to run thetest. Some of the test types require less CPU and disk space, whereas some of thetest types requires high CPU usage, I/O operations, and disk usage. So, theresource utilization of a particular test type depends on the sequence of testsperformed in that test type. Adopting operating system virtualization for all test typesis not possible because of dependencies. The following section describes how tochoose the test area.

Choosing the test area

The following general questionnaire helps in identifying the test area to useoperating system virtualization technologies to minimize system and humanresources.

General questionnaire for choosing test type:

• How much CPU and memory allocation is required?

• How many processes will run for executing the test?

• How much virtual memory is required for running the test?

• Does the test require any physical devices?

• Will the test require modifying the kernel parameter at runtime?

• Will the test access raw or routing sockets?

• Does the test require mount/dismounting file system?

Inside the operating system virtual instance, some facilities like mounting the filesystem, reading/writing the data to raw sockets, and changing the kernel parametersdynamically are not available. The file system mounting can be done from the globalsystem environment to a particular OS instance. The virtual OS instance can accessthe mounted file systems without any issues, but direct mounting from OS instanceis not possible. Similarly, accessing the physical devices is not possible from virtualinstances. Even though there is a way to access these devices by exporting from theglobal environment, applications that do not support these exported devices will fail.So, based on the resources required for test types, the virtual opersting systemtechniques are adopted for product testing.




Example test areas for consideration

As discussed previously, the product testing involves multiple test types and each ofthe test types needs different hardware, software, and network resources. Forexample, load test, stress test and performance tests, where the product is testedunder high load and each of these tests uses the same system resources from thesystem. So, these tests should not be run on the same physical server at the sametime. If these tests are running on different virtualized instances on a physical server,you need to be careful when configuring your system for allocating resources forthese operating system-virtualized systems. The load tests require more resources,and all will share from the same physical server, so these benchmark tests shouldnot be performed on operating system-virtualized systems.

For installation test, build verification test, globalization verification test, and GUIsoftware test, you can use the operating system virtual instances. In installation test,the product is tested in all supported modes of installation. Most of the time, theinstallation tests are executed in a freshly installed operating environment. Once theproduct is installed, it will add a set of configurations in the system environment.Even though the product is removed after testing, some of these environments willremain in the system. The subsequent installation on the same machine will use theexisting configurations. Sometimes this will hide the installation issues that can benoticed in a freshly installed operating system environment. But practically speaking,it is not possible to perform each of the installation test scenarios in freshly installedoperating system environments, because it includes additional operating system andthird-party prerequisite software setup in the server. As part of globalization, theinstallation test is done in all supported locales. So, the number of machinesrequired will be more. The operating system virtualization technology helps forinstallation tests. Once the operating system and other third-party prerequisites areinstalled in the physical server, you can make as many as virtual operating systemsinstances and all of these will have the same system operating environment as theglobal physical system. Creating these virtual instances will take less time.

Table 1. Different test areas with effort in conventional methodsTest type Number of servers

requiredAdministrationeffort

Test effort

Install tests 2 4x 12x

BVT 2 4x 12x

FVT 3 6x 24x

GUI tests 2 4x 12x

Table 1 shows different test areas, their server usage, and the administrator effortneeded for the operating system, software setup,and testers effort in configuring andexecuting the test cases where x is the unit of time.




Most of the products support a minimum of three modes of installation modes likeGUI, Console, and Silent. For installation test, if the product is tested in all modeswith all supported locales (assume the product supports 10 locales), then installationmust be tested in 30 various configurations. Similarly, BVT/FVT needs to be run inall locales to certify the product functionality. By using operating system virtualizationtechnologies, the administration and test efforts are reduced, as shown in Table 2.The administration effort is reduced as the number of servers used is less and thetest effort reduced, because operating system virtual instances contain all thesoftware installed in a global system. Hence, there is no need to installvendor-acquired products inside each instance. It also saves power and contributesless heat, thus reducing the investment required to cool the systems.

Table 2. Different test areas with efforts required using OS virtualizationmethodsTest type No of servers

requiredAdministrationeffort

Test effort

Install tests 1 2x 8x

BVT 1 2x 8x

FVT 1 3x 12x

GUI tests 1 2x 4x

Another test area where virtualization methods is used is GVT. As part of the GVT,BVT and FVT execute in all supported locales. If the product is installed in a globalphysical system and the virtual instances are created inside, then there is no need toinstall the product inside each instance. Only some additional configuration needs tobe done to execute these test suites. Even if the product image refreshes in theglobal system, it reflects immediately in all virtual instances. Hence, it reduces thetest efforts.

Operating system-level virtualization technologies

Virtualization is done in different levels and each virtualization technique has adifferent advantage based on the usage. For product testing, hardware andoperating system-level virtualization helps in reducing the number of physicalservers required and time required to set up the servers. In operating systemvirtualization, different isolated instances of the operating system are created and allshare the physical resources available in the system, as shown in Figure 1. Themaximum resource usage for a each instance can be configured at the time ofcreation. This article discusses the different server virtualization methods to be usedin product testing.

Figure 1. Operating system virtualization




There are different operating system virtualization technologies available like AIX®V6.1 workload partitions (WPAR) and Solaris 10 containers.

AIX 6.1 provides a WPAR facility that provides an application-isolated environmentand the resource control for applications. WPAR monitors the processes running ona particular WPAR from the global environment. Different automated test scripts canbe added in the startup profiles of the WPAR. Once the WPARs are started, all theautomated tests are executed on different WPARs simultaneously. Manual tests canalso be executed on each WPAR if needed. Using shared-system WPARconfiguration, there is no need to install the product on each WPAR. The test suiteswill use the product installed in the global environment to run different test areas ofproduct testing. Once the tests are started, Global environment can monitor thestatus logs for the initiated tests. Some of the test areas needs minimum systemresources and these tests can be run at the same time. This saves the resources,mainly in terms of servers and cost.

Solaris Container allows completely isolated virtual servers within a single operatingsystem instance called zones. The zone facility allows sharing the system resources.Each zone behaves as a separate system. The similar virtualization technologies arealso available on other operating systems like HP-UX, Linux and Windows.

WPARs in AIX 6.1

AIX 6.1 supports two types of workload partitions, Application WPAR and SystemWPAR. Application WPAR creates and starts with the application in an isolatedresource environment and automatically removes when the application terminates.System WPAR is a complete virtualized opersting system environment running




multiple services and applications. There are two different types of System WPAR:

• Shared system WPAR

• Non-shared (private) system WPAR.

Each of these behaves as a completely isolated operating environment.

In shared system WPAR, it uses the software installed in the global system and inprivate WPAR it allows for installation of the product inside the WPAR.

1. Create a WPARYou can create System WPAR by doing the following steps:

1. Log in to an AIX 6.1 system as a root user. Note that this is for aglobal system in a WPAR environment.

2. Select the WPAR name that you want to create.

3. If you want to mount the WPAR file systems on a specific mountpoint, create a directory; otherwise, by default WPAR uses the/wpars directory.

4. If you have created a directory, then change the directorypermission to 700 (that is, drwx------).

5. Add the IP address of the WPAR in the /etc/hosts file.For example:

123.456.789.128 txwpar.domain_name txwpar # IP address of WPAR

where:

123.456.789.128 is the IP address of the WPAR.

txwpar is the WPAR name.

6. Run the mkwpar command as shown in Figure 2 to create a sharedsystem WPAR.

Figure 2. Creating shared system WPAR




In the example shown in Figure 2, above, txwpar is the WPAR name.

-r is copy global network name resolution configuration into the workload.This copies the following:

/etc/resolve.conf/etc/hosts/etc/netsvc.conf/etc/irs.conf/etc/networks-N is network settings ( [-N attr=value ...] )address= 123.456.789.12 is the IP addressinterface = en0

To find the interface number, run the ifconfig -a command. For example:

# ifconfig -a en0:flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>inet 123.456.789.126 netmask 0xffffff00 broadcast 123.456.789.255tcp_sendspace 131072 tcp_recvspace 65536netmask=255.255.255.0 is the netmask value

For non-shared (private) WPARs only, you can use the -l option. Thiscreates a private, writable version of /usr and /opt.

Upon successful creation, it shows a successful message as shown inFigure 3. It also updates the network configurations to access the WPARfrom outside.

Figure 3. WPAR creation




2. Start the WPAR:

startwpar -v WPAR_name

where -v is verbose and -m is maintenance mode. Once the WPAR isstarted successfully, as shown in Figure 4, it returns the status as asuccess.

Figure 4. Start WPAR and check the WPAR status

3. Check the status of the WPAR:In Figure 4, the running status of WPAR is known by using lswparcommand, where S means that txwpar is a system WPAR and A meansthat txwpar is active.




4. Log in to the WPAR:

clogin txwpar.

5. Now WPAR is ready to use for testing.As it is a shared system WPAR, it contains all the software installed in theglobal system. Assume that two WPARs are created and WPAR1 FVTand WPAR2 BVT are configured as shown in Figure 5. Both tests canstart in parallel. Once it checks and makes sure that all test caseexecution is completed inside each WPAR, it can start the next test case.The results are written in a test log file inside the WPAR. The applicationrunning on AIX 6.1 system keeps on monitoring these logs and takes thenecessary steps. In case of GVT, the same WPARs can be used to runthe BVT and FVT in different locales by setting the locale one after theanother. As a result, it saves lots of resources like servers, powerconsumption, installation, and configuration of different products on eachserver, test execution time, and server administration.

Figure 5. Product testing with WPAR

Zones in Solaris

The Solaris Zones feature provides the facility to create separate operating systemenvironments for running applications with dedicated resouce allocation like CPUand memory. There are two types of zones, global and non-global. A server onwhich the zones feature is enabled has one global zone and up to 8191 non-globalzones. Non-global zones contains subset of the Solaris operating installed in theglobal zone and some additional packages. Each non-global zone is completely




isolated, has its own package database, and does not share package informationwith the global or other non-global zones.

1. To create zones, you need to have an IP address for each zone that youconfigure.

• Connect to the Solaris 10 system as an administrator.

• Choose the name for the zone.

1. Create a zone using the zonecfg command and follow thesteps as shown in Figure 6.Figure 6. Create a Solaris zone

2. Set the IP address and physical network card interface IDfor the zone created as shown in Figure 7. Get the additionalinformation about the zone by issuing the info command.

2. Enter the ifconfig -a command to get the interface card name.Figure 7. Configure the zone with network interface




3. Once the zone is created successfully, install the zone. This will copy thedifferent softwares installed from the global zone to the newly createdzone. The time taken for installing the zone depends on the number ofpackages installed in the global zone, as shown in Figure 8. Once thezone is installed, it shows the following output: Successfullyinstalled zone <zone name> ".Figure 8. Install the zone once it is created

4. Now you are back to the normal login system prompt. To check the zone




status, run zoneadm list -cv .If the value in the ID column against the zone name is '-', it means thezone is created and installed but not running. Run the boot command, tostart the zone and the ID column against zone name will show '1,' asshown in Figure 9.

Figure 9. List all zones from the global zone

5. For allocating resources to zones, use zonecfg from the global zone andthen specify the resources for the particular zone. If you are creatingmultiple zones, share the resources appropriately, as shown in Figure 10.In this example, zonecfg is allocating 40% CPU to the txzone.Figure 10. Allocating resources to zones

6. Reboot the zone to effect the resources allocated for a particular zone.Using prctl from the global zone, the CPU share for global zone canalso be defined. The following example defines 60% CPU share to theglobal zone.

prctl -r -n zone.cpu-shares -v 60 -i process $$.prctl -i zone global

7. Once the zone is created and resources are allocated, connect to thezone for setting up additional configurations, such as setting up thepassword, DNS name, IP address, Locale, and the like. Connect to the




zone using the zlogin command, as shown in Figure 11. Once you login inside the zone, the global zone is not visible. To verify, run uname -a.To log out from the zone enterthe ~.(tilde dot) command.Figure 11. Login to the zone

8. Now the zone is ready with the required configuration. The newly createdzone is completely isolated from other the zone as well as the globalzone. To the end user, it looks like as a separate Solaris system. TheSolaris zones are very useful for product installation tests, GVT tests, andresource manager tests.Figure 12. Solaris zones

In Figure 12, three zones are configured inside the global zone andresources allocated as 15% CPU share for zone 1 and zone 3. For zone2, a CPU share of 25% is allocated from the global zone. All of these




zones, including the global zone, use the same kernel to access thehardware available in the physical system. There are some testapplications running inside zone 2 and the database servers are alsoconfigured inside this zone. The database clients installed in zone 1 andzone 3 will connect to zone 2 to execute the database-based testapplications.

Conclusion

Adopting virtualization technologies saves resources like servers, installation andconfiguration of different products on each server, test execution time, and alsomanpower. It also saves the administration effort to maintain the servers.

On AIX WPAR, even if one server goes down for maintenance, it is easy to movethe WPARs from one server to another server with minimal effort, using theworkload partition manager.

With all these advantages, adopting virtualization techniques helps the testingenvironment to move closer towards green IT.




Resources

Learn

• The AIX 6.1 Information provides technical details about AIX Operating system.

• See Introduction to Workload Partition Management in IBM AIX Version 6.1 tounderstand AIX 6.1 WPAR operating system virtualization capability.

• Workload Partition Management in IBM AIX Version 6.1 provides information ofenhancements on WPAR with AIX 6.1 TL2.

• The AIX and UNIX developerWorks zone provides a wealth of informationrelating to all aspects of AIX systems administration.

• Stay current with developerWorks technical events and webcasts.

• Browse the technology bookstore for books on these and other technical topics.

Get products and technologies

• Download IBM product evaluation versions or explore the online trials in theIBM SOA Sandbox and get your hands on application development tools andmiddleware products from DB2®, Lotus®, Rational®, Tivoli®, andWebSphere®.

Discuss

• Check out developerWorks blogs and get involved in the developerWorkscommunity.

• Participate in the AIX and UNIX forums:

• AIX Forum

• AIX Forum for developers

• Cluster Systems Management

• IBM Support Assistant Forum

• Performance Tools Forum

• Virtualization Forum

• More AIX and UNIX Forums

About the author

Nageswararao V. Gokavarapu



http://publib.boulder.ibm.com/infocenter/systems/topic/com.ibm.aix.doc/doc/base/aixparent.htm?tocNode=toc:front/front.cmb/0/1/

http://www.redbooks.ibm.com/abstracts/sg247431.html

http://www.redbooks.ibm.com/abstracts/sg247656.html

http://www.ibm.com/developerworks/aix/

http://www.ibm.com/developerworks/offers/techbriefings/?S_TACT=105AGY06&S_CMP=art

http://www.ibm.com/developerworks/apps/SendTo?bookstore=safari

http://www.ibm.com/developerworks/downloads/

http://www.ibm.com/developerworks/downloads/soasandbox/

http://www.ibm.com/developerworks/downloads/soasandbox/

http://www.ibm.com/developerworks/blogs/

http://www.ibm.com/developerworks/community

http://www.ibm.com/developerworks/community

http://www-128.ibm.com/developerworks/forums/dw_forum.jsp?forum=747&cat=72





http://www.ibm.com/developerworks/forums/forum.jspa?forumID=748

http://www-128.ibm.com/developerworks/forums/dw_auforums.jsp


Nageswara Rao is a Post-graduate student from Andhra University and joined IBM in2006 as Software Engineer. He is presently at the India Software Labs, IBM, workingwith the TXSeries test team and has expertise in designing and developing testapplications for middleware like TXSeries on UNIX-like systems. His areas ofinterests are vVirtualization, transaction processing management, and UNIX systems.




Making software testing greener

Documents

Transcript of Making software testing greener