ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to...

28
ARTE The Automatic Regression Test Environment Manual for ARTE version 0.2.0, February 13, 2009 Aleksander Morgado

Transcript of ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to...

Page 1: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

ARTEThe Automatic Regression Test Environment Manual

for ARTE version 0.2.0, February 13, 2009

Aleksander Morgado

Page 2: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

This manual is for ARTE (version 0.2.0, February 13, 2009)Copyright c© 2009. Aleksander Morgado

Permission is granted to copy, distribute and/or modify this document under theterms of the GNU Free Documentation License, Version 1.2 or any later versionpublished by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in thesection entitled “GNU Free Documentation License”.

Page 3: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

i

Table of Contents

1 Overview of ARTE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Reporting Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Copyright . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

3 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33.1 Installation from release tarball . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33.2 Installation from local repository copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33.3 Built-in plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

4 The ARTE Core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44.2 Execution sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54.3 Execution arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64.4 Directory Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74.5 Configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.5.1 The arte.conf framework configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.5.2 The [module].conf suite configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.5.3 The test.conf test configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.6 Execution modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5 The ARTE RTA Test Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.1 The RTAv1 file format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.2 The RTAv2 file format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.3 Compression and Encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125.4 The TCL test script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

6 The ARTE Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156.1 Plugin directories and files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156.2 Importing the functions defined in the ARTE Core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156.3 Exporting the plugin dictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166.4 Exporting the setup and teardown functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Appendix A GNU Free Documentation License . . . . . . . . . . . . . 19

Page 4: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 1: Overview of ARTE 1

1 Overview of ARTE

ARTE is a unit/system testing framework designed to test software modules in GNU/Linux sys-tems. It’s based on the automatic execution of test suites/collections and helps the improvementof Your software with fast detection of regression problems.

1.1 Reporting Bugs

We welcome bug reports and suggestions for any aspect of the ARTE sys-tem, programs, documentation, installation, anything. Please email them [email protected]. You can get the latest version of ARTEfrom SourceForge ARTE archive.

For bug reports, please include enough information for the maintainers to reproduce theproblem. Generally speaking, that means:• the version number of ARTE and the program(s), plugin(s) or manual(s) involved.• hardware and operating system names and versions.• the contents of any input files necessary to reproduce the bug.• a description of the problem and samples of any erroneous output.• any unusual options you gave to configure.• anything else that you think would be helpful.

When in doubt whether something is needed or not, include it. It’s better to include toomuch than to leave out something important.

Patches are most welcome; if possible, please make them with ‘diff -c’, include ‘ChangeLog’entries and follow the existing coding style.

1.2 History

ARTE was originally developed at Panda Security, which kindly published the source code underthe GPLv3+ license. Therefore, ARTE is free software, and can be extended by anyone as longas the rules of the GPLv3+ license are satisfied.

ARTE was written in year 2005 by Aleksander Morgado [email protected] which isalso the current maintainer.

Also helped in development of ARTE:• Pedro Romayor• Pablo Martın• Placido Revilla

1.3 Copyright

The original code of this project (svn revision 2) is fully copyrighted by Panda Security S.L.,which kindly decided to publish and free this source code licensed as GPLv3 (or later) in January2009.

Additional modifications done in the project after svn revision 2 are copyrighted by theauthors of the changes.

Page 5: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 2: Definitions 2

2 Definitions

‘CHECK’ ‘CHECK’ is a Free Software project which provides an easy to use unit testing frame-work for C source code.

‘TCL’ The Tool Command Language, or ‘TCL’, is a very advanced scripting language whichcan be easily integrated with C source code.

‘RTA’ The Regression Test Archive, or ‘RTA’, is the basic unit for testing, which allowsencapsulating in a single file all the needed files needed to execute a single test inthe ARTE framework. Currently, two versions of RTA files are managed in ARTE:version 1 and 2.

‘SUITE’ A ‘SUITE’ is a collection of RTAs prepared to analyze a given software module.

‘PLUGIN’ An ARTE ‘PLUGIN’ is an extension to the basic functionalities of the framework,needed when the basic commands of TCL are not enough to test a given softwaremodule.

Page 6: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 3: Installation 3

3 Installation

3.1 Installation from release tarball

The simplest way to compile this package is:1. ‘cd’ to the directory containing the package’s source code and type ‘./configure --

prefix=/opt/arte’ to configure the package for your system. Running ‘configure’ mighttake a while. While running, it prints some messages telling which features it is check-ing for. It is a good idea to always install ARTE in a completely independent path, like‘/opt/arte’, as a complete layout of directories will be created when installing it.

2. Type ‘make’ to compile the package.3. Type ‘make install’ to install the programs and any data files and documentation. Con-

sider running the previous command as ‘root’ user if you chose a path where standard userscan’t write.

4. You can remove the program binaries and object files from the source code directory bytyping ‘make clean’. To also remove the files that ‘configure’ created (so you can compilethe package for a different kind of computer), type ‘make distclean’.

5. Often, you can also type ‘make uninstall’ to remove the installed files again.

3.2 Installation from local repository copy

If the source code is a local copy of the ARTE subversion repository, You will need to initializesome auto-tools related files by running ‘./bootstrap’. This script will create the ‘configure’file, and You just need to follow the steps from the previous section to finish the installation.

3.3 Built-in plugins

You need to explicitly tell the ‘configure’ script which built-in plugins should be compiled andinstalled in the system.

Run ‘./configure --help’ to get a whole list of options that can be given to the configurescript.

The current built-in plugins can be enabled with the following options:• Run ‘./configure --enable-all’ to enable the compilation and installation of all available

built-in plugins.• Run ‘./configure --enable-generic’ to enable the compilation and installation of the

Generic example plugin.

Page 7: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 4: The ARTE Core 4

4 The ARTE Core

4.1 Architecture

The ARTE Core system (the one common to any installation of ARTE) is divided in severallayers or subsystems:• ‘Configuration Subsystem’• ‘CHECK Engine’• ‘Visualization Subsystem’• ‘ZIP/RTA Subsystem’• ‘TCL Subsystem’• ‘ARTE Plugin’

The following images shows a simple diagram explaining how all these Subsystems are orga-nized in the ARTE Core:

‘Configuration Subsystem’The ‘Configuration Subsystem’ is responsible for reading the input main configu-ration file of the ARTE system (‘arte.conf’). This configuration file will specify,among others, the log level to use or which test suites should be run.

‘CHECK Engine’The ‘CHECK Engine’ will, based on the input configuration loaded by the‘Configuration Subsystem’, initialize in memory all the information neededto run the different test suites. This engine is based on the Check unit-testing

Page 8: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 4: The ARTE Core 5

framework, and it uses its API to setup and run independent CHECK Tests, whichmay be configured to run in different forked processes. The ‘CHECK Engine’ alsocontrols the forced ending of tests due to test timeouts.

‘Visualization Subsystem’The ‘Visualization Subsystem’ takes as input the log of the execution created bythe ‘CHECK Engine’, and the output results XML generated by the Check framework,and will modify those files to match the format specified by the ARTE system.

‘ZIP/RTA Subsystem’The ‘ZIP/RTA Subsystem’ is executed within the context of a given ARTE test.This means that if forking is configured, this Subsystem will always be executedin an independent forked process. The ‘ZIP/RTA Subsystem’ is in charge of de-compressing the RTA test files, and preparing all the needed context files for thetest.

‘TCL Subsystem’The ‘TCL Subsystem’ is the one in charge of executing the specific TCL test scriptstored in the RTA file. The TCL Subsystem will export the generic and commonARTE Commands to the interpreter of the TCL script, as well as those new com-mands created by the specific ARTE Plugin being used, if any.

‘ARTE Plugin’If the standard TCL language is not enough to test your external software module,you may need to extend the commands available in the script interpreter using aspecific ‘ARTE Plugin’. These plugins are easy to use and deploy, and provide anefficient way of preparing ARTE to be able to communicate with any kind of externalsoftware module to be tested.

4.2 Execution sequence

The normal execution of ARTE goes as follows:• ARTE initialization

1. ARTE framework is started after having issued the ‘arte’ command with the correctarguments.

2. The input arguments are parsed and the initial framework configuration is setup.3. The ‘Configuration Subsystem’ reads the ‘arte.conf’ configuration file to know,

among others, which are the test suites to run.4. The ‘Configuration Subsystem’ reads the suite-specific configuration file.5. The ‘CHECK Engine’ creates de list of valid test suites to run (one suite per module to

be tested, usually).6. The ‘CHECK Engine’ creates de list of tests to execute in each test suite.7. The ‘CHECK Engine’ starts running the tests in each suite, one by one, creating a

CHECK Test environment (forked processed if necessary) and initiating the per-test‘ZIP/RTA Subsystem’.

• For each RTA test,1. The ‘TCL Subsystem’ decides which RTA should be executed, and triggers its checking

and decompression in the ‘ZIP/RTA Subsystem’.2. The ‘ZIP/RTA Subsystem’ creates a temporary directory with a random name, and

decompresses the RTA file in it.3. The ‘ZIP/RTA Subsystem’ checks the correctness of the RTA file, gets the version num-

ber of the RTA and decides if it can be run in the framework.

Page 9: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 4: The ARTE Core 6

4. The ‘Configuration Subsystem’ reads the specific configuration of the RTA test.5. The ‘TCL Subsystem’ runs the plugin initialization (SETUP) function.6. The ‘TCL Subsystem’ starts the module to be tested (if specified).7. The ‘TCL Subsystem’ gets the module-specific command from the ‘PLUGIN’.8. The ‘TCL Subsystem’ creates the customized TCL interpreter, containing the ARTE

Core commands plus the Plugin commands.9. The ‘TCL Subsystem’ sets the environment variables coming in the suite-specific or

test-specific configuration files.10. The ‘TCL Subsystem’ runs the TCL script for the given RTA file.11. Once the test finishes, the ‘TCL Subsystem’ stops the module to be tested (if specified).12. The ‘TCL Subsystem’ runs the plugin deinitialization (TEARDOWN) function.

• Generating outputs1. The ‘Visualization Subsystem’ modifies the Check-generated XML results, file to

include the link to the XSLT to be used.2. The ‘Visualization Subsystem’ creates the HTML log file, if specified.3. The ‘Visualization Subsystem’ adds an entry to the list of ARTE executions in the

index HTML file.

4.3 Execution arguments

You can get a whole list of arguments to be passed to the arte command calling it with the -hoption:� �$> arte -h

Usage: arte [options]Options:

-h Show this help-v Verbose mode, showing all possible logs-s Silent mode, showing only the errors-l Create a log of all tests in HTML format-m Avoid deleting the temporary directories in /tmp-t Disable the timeout failure of tests-c FILE Specify the path to the ARTE configuration file-d INTEGER Specify the timeout for each test, in seconds-f Avoid forking for each test-r RTAFILE Only execute the specified RTA. It must be in the

’tests’ directory of the suite.-x RTAFILE Decompress in the suite Sandbox the specified RTA-k Create an RTA from the Sandbox, check and execute

The default timeout for each test is 15 seconds, and if more time is needed, the -d INTEGER

argument should be used. With this option, the default timeout for all tests is changed in thecurrent ARTE execution.

With the -x RTAFILE argument, the RTA file given in ‘RTAFILE’ is decompressed (whichmust be in the ‘tests’ directory of the suite) in a new directory called ‘sandbox’ (under thesuite directory). In addition to just decompressing the RTA file, it creates a ‘.rtaname’ file inthe same directory, containing the name of the original RTA file. The contents in the ‘sandbox’

Page 10: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 4: The ARTE Core 7

can then be modified, including the filename stored in the ‘.rtaname’ file to rename the RTAfile that can be then created with the -k argument. This newly created RTA is stored in the‘tests’ directory of the suite, and directly executed.

An standard execution of ARTE could just be:� �$> arte -v -l

4.4 Directory Layout

For a proper organization of the different suites and their contents, a well defined directorylayout is created for each suite when ARTE is installed (when running make install. Thislayout starts in the prefix specified when running ./configure.

‘prefix/bin’The ‘bin’ directory stores both the framework initialization script ‘run_arte.sh’and the executable binary ‘arte’.

‘prefix/etc’The ‘etc’ directory stores the main ARTE configuration file ‘arte.conf’.

‘prefix/suites’The ‘suites’ directory stores the installed suites’ custom layout (see below).

‘prefix/www’The ‘www’ directory stores the XSLT files to be used with the XML files that ARTEgenerates, and is the default directory where the generated HTML and XML filesare stored.

‘prefix/suites/[module]/bin’The suite-specific ‘bin’ directory stores the script to start/stop the external module,if any (‘run_[module].sh’).

‘prefix/suites/[module]/etc’The suite-specific ‘etc’ directory stores the suite-specific configuration file,‘[module].conf’, which is common to all the tests to be executed in that suite.

‘prefix/suites/[module]/lib’The suite-specific ‘lib’ directory stores the suite-specific plugin library(‘lib[module]plugin.so’).

‘prefix/suites/[module]/common’‘common’ is a general-purpose suite-specific directory, where any common needed filein the suite may be stored. Note that this is the place for those files needed by anytest in the suite, not the place for per-test specific files.

‘prefix/suites/[module]/test’The suite-specific ‘test’ directory stores all the RTA files related to the given suite.All the RTAs to be executed must be available in this path.

‘prefix/suites/[module]/sandbox’The suite-specific ‘sandbox’ directory is the workplace for single RTA files. Usingspecific arte arguments, the developer can easily trigger the creation of new RTAtest files, or modify the existing ones.

Page 11: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 4: The ARTE Core 8

4.5 Configuration files

There are three types of ARTE configuration files, and each of them are applicable in well-defineddomains:

‘arte.conf’The global ARTE configuration file, which is applicable in the whole ARTE frame-work (‘prefix/etc/arte.conf’).

‘[module].conf’The suite-specific ARTE configuration file, which is applicable in the [module] suitedomain (‘prefix/suites/[module]/etc/[module].conf’).

‘test.conf’The test-specific ARTE configuration file, which is applicable in the single testexecution domain (‘test.conf’, inside the RTA).

4.5.1 The arte.conf framework configuration

The main ARTE configuration file just needs two configuration parameters:

‘suites (Mandatory)’The suites parameter indicates which are the suites to be run in the system. Notethat the suite name must follow the conventions specified everywhere in this doc-ument as the [module] template (for directory and file names). Only those suitescompiled and installed with ARTE can be run (those specified when running the./configure script.

‘xmloutputdir (Optional)’If specified, the xmloutputdir configuration parameter tells the framework wherethe output XML and HTML files should be placed, instead of using the default‘prefix/www’ path. Note that the given path must exist before calling ARTE, andthe user running it must have R/W permissions on it.

An example of ‘arte.conf’ configuration file is the following:� �# arte.conf# Configuration file for the ARTE testing framework

# Configuration parameters (mandatory)suites = generic, module1, module2# Path to the directory where the XMLs are stored (optional)xmloutputdir="/srv/www/htdocs/data/tests/"

4.5.2 The [module].conf suite configuration

The suite-specific configuration file needs the following four mandatory parameters:

‘[module].name (Mandatory)’The descriptive name of the suite

‘[module].start (Mandatory)’Command to start the external module to be tested. If none needed, you can setthis parameter to none. Also, you can use the reserved %BIN% word in this field toindicate the suite-specific ‘bin’ directory (‘prefix/suites/[module]/bin/’)

Page 12: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 4: The ARTE Core 9

‘[module].stop (Mandatory)’Command to stop the external module to be tested. If none needed, you can setthis parameter to none. Also, you can use the reserved %BIN% word in this field toindicate the suite-specific ‘bin’ directory (‘prefix/suites/[module]/bin/’)

‘[module].runtype (Mandatory)’How the suite should be run. The allowed values in this field are always, manualand once (they are explained in the next section).

In addition to the previous mandatory parameters, you can specify an unlimited number ofglobal variables which will be available in each test of this suite:

‘[module].[VARNAME] (Optional)’Where ‘VARNAME’ will be a valid environment variable that may be used inany TCL test of this suite. As done in the start/stop commands with thereserved %BIN% word, you can specify in any of these environment variablesthe %COMMON% reserved word to indicate the suite-specific ‘common’ directory(‘prefix/suites/[module]/common/’). You must take into account that thetest-specific configuration may override the value of any environment variabledefined in the suite-specific configuration.

An example of ‘[module].conf’ configuration file is the following, the configuration file forthe ‘generic’ module (named ‘generic.conf’).� �# generic.conf# Configuration file for the GENERIC suite in the ARTE testing framework

# Configuration parameters (mandatory)generic.name="This is the GENERIC module, dude"generic.start="%BIN%/start_generic.sh"generic.stop="%BIN%/start_generic.sh"

# Available runtypes (mandatory): always | manual | oncegeneric.runtype="always"

# Environment vars for the TCL interpreter (optional)generic.HELLO="Hello World"generic.INPUTFILE="%COMMON%/inputfile.txt"

4.5.3 The test.conf test configuration

Finally, you need to setup an additional configuration file for each test to be executed. This‘test.conf’ file is stored inside the compressed RTA file.

There is no mandatory parameter in this configuration file, and only environment variablesmay be defined in it (new ones or overriding the ones specified in the suite configuration file).

‘[module].[VARNAME] (Optional)’Where ‘VARNAME’ will be a valid environment variable that may be used in thespecific TCL script of this RTA test. You can specify in any of these environmentvariables the %COMMON% reserved word to indicate the suite-specific ‘common’ directory(‘prefix/suites/[module]/common/’), or the ZIPBASE reserver word to indicatethe directory where the RTA was decompressed (so that test-specific files storedinside the RTA can be referenced from within the TCL script).

Page 13: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 4: The ARTE Core 10

An example of ‘test.conf’ configuration file (to be used in a test of the ‘generic’ module,is the following:� �# test.conf# Configuration file for a specific test in the ARTE testing framework

# New environment vars for the TCL interpretergeneric.TESTDATA="ZIPBASE/testdata.txt"

# Override suite-specific environment vargeneric.HELLO="Hoola Mundo!"

4.6 Execution modes

As said before, ARTE allows three different execution modes (a.k.a ‘runtypes’), dependingon the needs for each module to be analized, or depending on the type of testing to be done.The different runtypes allow to configure how the external module will be started before run-ning the tests, and stopped after having run them (using the provided [module].start and[module].stop commands in the suite-specific configuration file).

‘manual’ In this mode, ARTE will neither start nor stop the external module. The externalmodule to be tested must be already started before calling ARTE with this runtype.This runtype is specially useful when debugging the external module.

‘always’ In this mode, ARTE will start the external tested module before each test, and stopit after each test. In other words, each test is executed in a completely independentexecution of the external module.

‘once’ In this mode, ARTE will start the external module before the first executed testand will stop it after the last one. This runtype is specially useful when performingload-tests or checking the behavior of the external module when multiple tests areexecuted over the same single execution1.

1 When using ARTE with the first external module in Panda Security S.L., it happened sometimes that runningall the tests with ‘once’ runtype gave different outputs than running the all the tests with ‘always’ runtype,due to inter-test problems. For example, running ARTE in ‘once’ runtype, first test was causing a problem inthe external module which was also causing some of the next tests executed to fail, while when executed in‘always’ runtype, the external module was a completely new instance of execution with no corrupted internalstate, so all the tests passed correctly.

Page 14: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 5: The ARTE RTA Test Files 11

5 The ARTE RTA Test Files

The ‘RTA’ files (Regression Test Archive) are the basic test execution units in the ARTEframework. In other words, to correctly execute the ARTE system, at least one RTA file mustbe available.

All the ‘RTA’ files (either ‘RTAv1’ or ‘RTAv2’) must have the .rta extension.

5.1 The RTAv1 file format

‘RTAv1’ files are compressed and encrypted zip files with the following mandatory contents:

‘TCL script (whatever.tcl)’The TCL script to be executed in the RTA test. Note that the filename is notimportant, but it must have the proper tcl extension. Also, only one single filewith tcl extension must be available inside the RTA.

‘test.conf’The test-specific configuration file, to include test-specific environment variables asspecified in section Section 4.5.3 [The test.conf test configuration], page 9.

‘version’ The file specifying the version of this RTA. For ‘RTAv1’, it must contain:� �version 1

In addition to the previous mandatory parameters, the ‘RTAv1’ file may contain any additional

file needed to in the test.

5.2 The RTAv2 file format

The ‘RTAv2’ file format is just an extension of ‘RTAv1’ to include an additional mandatory file(‘description.xml’):

‘TCL script (whatever.tcl)’The TCL script to be executed in the RTA test. Note that the filename is notimportant, but it must have the proper tcl extension. Also, only one single filewith tcl extension must be available inside the RTA.

‘test.conf’The test-specific configuration file, to include test-specific environment variables asspecified in section Section 4.5.3 [The test.conf test configuration], page 9.

‘version’ The file specifying the version of this RTA. For ‘RTAv2’, it must contain:� �version 2

‘description.xml’

This additional file is mandatory in ‘RTAv2’, and contains the description of the testin an easily computer-parseable XML format. The format of the XML file is fixed,and specified by the ‘description-rta.xsd’ XML Schema available in the ‘etc’directory of the source code. Note that an invalid ‘description.xml’ indicates awrong RTA file, and ARTE will reject it, so it is very important to make sure thatthe file validates against the provided XSD.

Page 15: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 5: The ARTE RTA Test Files 12

� �<?xml version="1.0" encoding="UTF-8"?>

<ARTE-TEST xmlns="http://arte/description-rta"><ModuleID>generic</ModuleID><TestGroupID>generic-test-hello</TestGroupID><Author>aleksander</Author><Date>Sun Feb 15 20:59:03 2009</Date><TestDescription>

This test does this and that, to test problemreported in ticket #12345

</TestDescription><ExpectedResult>

The test should pass if whatever thing happened</ExpectedResult>

</ARTE-TEST>

In addition to the previous mandatory parameters, the ‘RTAv2’ file may contain any additional

file needed to in the test.

5.3 Compression and Encryption

Standard ZIP compression is used to generate the ‘RTA’ files. As these files may be sent/receivedby e-mail, they are always encrypted so that any Antivirus/Antispam system can modify itscontents. The unique ‘arte’ password is used to encrypt the files.• Manually creating/modifying an RTA file

To create an ‘RTAv2’ file with the files ‘test.conf’, ‘test.tcl’, ‘version’,‘description.xml’, ‘foo.txt’ and ‘foo2.txt’, the following command may be used:� �$> zip -e test1.rta test.conf \

test.tcl \version \description.xml \foo.txt \foo2.txt

... and type ‘arte’ when it requests a password.To extract any ‘RTA’ file manually, you may use:� �$> unzip -P arte test1.rta

• Using the suite sandbox to create/modify an RTA file

You may also use the -x and -k arguments of the arte command to work with the contentsof an existing RTA in the suite sandbox (see Section 4.3 [Execution arguments], page 6).

• Using the ‘arte_rta.sh’ helperThe ARTE distribution also provides a helper script to work with multiple RTA files, named‘arte_rta.sh’. You can run this script in the directory where the RTA files for a givensuite are stored:Run the following to decompress al RTA files and create ‘.rta.dir’ directories:

Page 16: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 5: The ARTE RTA Test Files 13

� �$> ./arte_rta.sh unzip

Run the following to compress the ‘.rta.dir’ directories again back into correct RTA files:� �$> ./arte_rta.sh zip

5.4 The TCL test script

TCL is a powerful language, at least for the requirements of ARTE. The following examplesshow the basic functionality of the ARTE Core without needeing any extra Plugin to extendthe TCL interpreter’s default dictionary.

1. The following script will just set the title of the Test (to be shown in the results report), printa debug message in the output ARTE log, and make the test fail (with the ARTE-builtinfail command).� �#test1.tcl

# Set test title...settitle "Dummy test which will always fail"

# Print a debug message...debug "And show log..."

# And force the test to failfail "Message shown if failed"

2. If a given test doesn’t have any managed failure point, it will pass, as the one shown here:� �

#test2.tcl

# Set test title...settitle "Dummy test which will always pass"

# Print a debug message...debug "More logs..."

3. The following script uses two other ARTE-builtin commands: failifdifferent and

failifequal, which compare two given strings.

Page 17: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 5: The ARTE RTA Test Files 14

� �#test3.tcl

# Set test title...settitle "Dummy test which will also fail comparing strings"

# Print a debug message...debug "And show another log here..."

# Compare two strings and fail if they are different (they will# be the same here)failifdifferent "str" "str" "Message shown if failed"

# Compare two strings and fail if they are equal (they will# be different here)failifequal "str" "nostr" "Message shown if failed"

# Compare two strings and fail if they are different (they will# be different here, so this will be the failure point)failifdifferent "str" "nostr" "Message shown if failed"

As the interpreter is a complete TCL interpreter, plus some customized commands, you can

use whatever TCL allows you. TCL can treat outputs from commands as strings, and using thebuilt-in failifdifferent or failifequal commands, you can control wether a test correctlypassed or not.

Also, when your test has multiple places where it may fail (like ‘test3.tcl’), you can alwaysadd an extra last string to the command, which will be included in the reports generated byARTE, to know the exact failure point.

If, for whatever reason, you need another new magic command to be built in the TCLinterpreter of ARTE, you can easily implement it creating a new plugin and exporting the new‘PLUGIN Dictionary’ (see Section 6.3 [Exporting the plugin dictionary], page 16)

Page 18: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 6: The ARTE Plugins 15

6 The ARTE Plugins

You can easily create new plugins to be used by the ARTE framework, so that the standardcommands in the TCL language are extended with a new ‘Plugin dictionary’ (new TCLcommands).

The procedure shown here explains how to create a new plugin for ARTE and include it to becompiled along with the Core and the generic plugin. Non built-in plugins may also be createdwithout much problem, as the API of the plugin is fixed and installed with the Core.

6.1 Plugin directories and files

1. Create the following directories in the ARTE source code directory:• ‘src/suites/[module]’, to store the source code of the plugin• ‘etc/tests/[module]/etc/’, to store the suite-specific configuration file.• ‘etc/tests/[module]/bin’, to store the suite-specific script to start/stop the module.

2. In ‘etc/tests/[module]/etc/[module].conf’, create the suite-specific configuration filefollowing the information given in Section 4.5.2 [The [module].conf suite configuration],page 8.

3. In ‘etc/tests/[module]/bin/run_[module].sh’, create the suite-specific script tostart/stop the module if needed.

4. In ‘src/suites/[module]/test_suite_[module].h’, create the suite-specific header fileof the plugin.

5. In ‘src/suites/[module]/test_suite_[module].c’, create the suite-specific source codefile for the plugin.

6.2 Importing the functions defined in the ARTE Core

The plugins need to import some ARTE Core functions needed to indicate that tests failed, orto print debug messages. This import is done using the following structure, defined in ‘arte.h’:� �/** Standard plugin actions */typedef struct{

/** Make the test fail */void (* FAIL) (char *message);/** Make the test fail if str1 != str2 */void (* FAIL_IF_DIFFERENT) (char *str1,

char *str2,char *message);

/** Make the test fail if str1 == str2 */void (* FAIL_IF_EQUAL) (char *str1,

char *str2,char *message);

/** Dump debug message */void (* DEBUG) (arte_log_type_t type,

const char *format,...);

} arte_core_functions_t;

The imported functions are the following:

Page 19: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 6: The ARTE Plugins 16

‘FAIL’ The ‘FAIL’ function is used by the plugin to indicate that the test being executedshould fail. The additional message parameter allows the plugin to include a mes-sage/description string of why the test failed.

‘FAIL_IF_DIFFERENT’The ‘FAIL_IF_DIFFERENT’ function will fail the test if the provided strings aredifferent. The additional message parameter allows the plugin to include a mes-sage/description string of why the test failed.

‘FAIL_IF_EQUAL’The ‘FAIL_IF_EQUAL’ function will fail the test if the provided strings are thesame. The additional message parameter allows the plugin to include a mes-sage/description string of why the test failed.

‘DEBUG’ The ‘DEBUG’ function allows the plugin to include traces in the ARTE Core log file.

The import process of the functions is done when the library is dynamically loaded by theArte Core. In order to provide a correct communication between the Plugin and the Core, thePlugin must export in its API the following:� �/** Function to retrieve the array of arte_command_t from the module */arte_command_t *arte_plugin_get_commands (arte_core_functions_t *common);

The previous function not only allows to import the Core functions in the plugin, but also

exports the plugin dictionary into the ARTE Core (as shown in the next section).

6.3 Exporting the plugin dictionary

As already explained, the main purpose of the Plugin is to create new commands that can beused within the TCL scripts of the ARTE RTAs. These set of new commands exported by aPlugin is known as the ‘PLUGIN dictionary’.

The new commands in a plugin are also named as ‘command-function’, because they arecomposed of a TCL-side (command) and a C-side (function). This can be seen in the arte_command_t structure which defines a given ‘command-function’, available in ‘arte.h’:� �/** Struct to link the TCL command to a given C function */typedef struct{

/** Command token to be used in the TCL scripts */const char *command_token;/** Command function to execute in the C side */int (* command_func) (ClientData clientdata,

Tcl_Interp *interprete,int argc,char **argv);

} arte_command_t;

The Plugin will define an internal NUL-pointer-terminated array of ‘arte_command_t’ struc-

tures, with the exported ‘PLUGIN dictionary’:

Page 20: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 6: The ARTE Plugins 17

� �/** Array with all the specific commands exported by the Generic* Plugin */static arte_command_t _arte_ts_generic_commands[] = {

{ "helloworld", __arte_ts_generic_command_helloworld },{ "byeworld", __arte_ts_generic_command_byeworld },{ NULL, NULL }

};

This array will be returned to the ARTE Core in the pointer returned by the previously

shown arte_plugin_get_commands function.

Once the plugin is integrated at runtime with ARTE, you will be able to use the new com-mands exported in the ‘PLUGIN Dictionary’ directly in the TCL tests:

1. The following script will just set the title of the Test (to be shown in the results report),print a debug message in the output ARTE log, and call the new helloworld command).� �#testplugin1.tcl

# Set test title...settitle "Dummy test trying the new commands"

# Print a debug message...debug "Show a pretty log..."

# It should print Hello World and continuehelloworld

# If any extra parameter is passed, it will failhelloworld Peter

2. And yet another example using both the built-in ARTE Core commands and the new ones

exported by the Plugin.� �#testplugin2.tcl

# Set test title...settitle "Built-in and new commands in action"

# Print a debug message...debug "Show a pretty log..."

# It should fail if the printed string is not the one expectedfailifdifferent [helloworld] "Hello World" "Not a hello world?"

Implementing new commands in the ‘PLUGIN Dictionary’ is a very good idea when you really

need your own C code to perform an operation. You may for example, want to test an HTTPclient library developed in C. For that, you may create new commands in the dictionary whichinternally use the API of the C library.

Page 21: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Chapter 6: The ARTE Plugins 18

6.4 Exporting the setup and teardown functions

In addition to the ‘PLUGIN Dictionary’, the Plugin must export an additional structure withSetup and Teardown functions:� �/** Setup & Teardown functions exported by the ARTE Plugin* into the ARTE Core */typedef struct{

/** Setup test */void (* setup_func) (void);/** Teardown test */void (* teardown_func) (void);/** Setup suite */void (* suite_setup_func) (void);/** Teardown suite */void (* suite_teardown_func) (void);

} arte_setup_functions_t;

These functions allow preparing and controlling the context of each test or suite of tests.

‘setup_func’The ‘setup_func’ will be executed just before starting to load a new test, evenbefore creating the customized TCL interpreter.

‘teardown_func’The ‘teardown_func’ will be executed just after the test stopped its execution (bothif passed or failed).

‘suite_setup_func’The ‘suite_setup_func’ will be executed once before any test of the given suite.

‘suite_teardown_func’The ‘suite_teardown_func’ will be executed once after all the tests of the givensuite have been run.

In order to export the Setup and Teardown functions to the Arte Core, the Plugin mustimplement the following function in its API:� �/** Function to retrieve the array with the module setup and* teardown functions */arte_setup_functions_t *arte_plugin_get_setup_functions(void);

Page 22: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Appendix A: GNU Free Documentation License 19

Appendix A GNU Free Documentation License

Version 1.2, November 2002Copyright c© 2000,2001,2002 Free Software Foundation, Inc.51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA

Everyone is permitted to copy and distribute verbatim copiesof this license document, but changing it is not allowed.

0. PREAMBLEThe purpose of this License is to make a manual, textbook, or other functional and usefuldocument free in the sense of freedom: to assure everyone the effective freedom to copyand redistribute it, with or without modifying it, either commercially or noncommercially.Secondarily, this License preserves for the author and publisher a way to get credit for theirwork, while not being considered responsible for modifications made by others.This License is a kind of “copyleft”, which means that derivative works of the documentmust themselves be free in the same sense. It complements the GNU General Public License,which is a copyleft license designed for free software.We have designed this License in order to use it for manuals for free software, because freesoftware needs free documentation: a free program should come with manuals providing thesame freedoms that the software does. But this License is not limited to software manuals;it can be used for any textual work, regardless of subject matter or whether it is publishedas a printed book. We recommend this License principally for works whose purpose isinstruction or reference.

1. APPLICABILITY AND DEFINITIONSThis License applies to any manual or other work, in any medium, that contains a noticeplaced by the copyright holder saying it can be distributed under the terms of this License.Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use thatwork under the conditions stated herein. The “Document”, below, refers to any such manualor work. Any member of the public is a licensee, and is addressed as “you”. You acceptthe license if you copy, modify or distribute the work in a way requiring permission undercopyright law.A “Modified Version” of the Document means any work containing the Document or aportion of it, either copied verbatim, or with modifications and/or translated into anotherlanguage.A “Secondary Section” is a named appendix or a front-matter section of the Documentthat deals exclusively with the relationship of the publishers or authors of the Documentto the Document’s overall subject (or to related matters) and contains nothing that couldfall directly within that overall subject. (Thus, if the Document is in part a textbook ofmathematics, a Secondary Section may not explain any mathematics.) The relationshipcould be a matter of historical connection with the subject or with related matters, or oflegal, commercial, philosophical, ethical or political position regarding them.The “Invariant Sections” are certain Secondary Sections whose titles are designated, asbeing those of Invariant Sections, in the notice that says that the Document is releasedunder this License. If a section does not fit the above definition of Secondary then it is notallowed to be designated as Invariant. The Document may contain zero Invariant Sections.If the Document does not identify any Invariant Sections then there are none.The “Cover Texts” are certain short passages of text that are listed, as Front-Cover Texts orBack-Cover Texts, in the notice that says that the Document is released under this License.A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25words.

Page 23: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Appendix A: GNU Free Documentation License 20

A “Transparent” copy of the Document means a machine-readable copy, represented in aformat whose specification is available to the general public, that is suitable for revisingthe document straightforwardly with generic text editors or (for images composed of pixels)generic paint programs or (for drawings) some widely available drawing editor, and that issuitable for input to text formatters or for automatic translation to a variety of formatssuitable for input to text formatters. A copy made in an otherwise Transparent file formatwhose markup, or absence of markup, has been arranged to thwart or discourage subsequentmodification by readers is not Transparent. An image format is not Transparent if used forany substantial amount of text. A copy that is not “Transparent” is called “Opaque”.Examples of suitable formats for Transparent copies include plain ascii without markup,Texinfo input format, LaTEX input format, SGML or XML using a publicly available DTD,and standard-conforming simple HTML, PostScript or PDF designed for human modifica-tion. Examples of transparent image formats include PNG, XCF and JPG. Opaque formatsinclude proprietary formats that can be read and edited only by proprietary word proces-sors, SGML or XML for which the DTD and/or processing tools are not generally available,and the machine-generated HTML, PostScript or PDF produced by some word processorsfor output purposes only.The “Title Page” means, for a printed book, the title page itself, plus such following pagesas are needed to hold, legibly, the material this License requires to appear in the title page.For works in formats which do not have any title page as such, “Title Page” means thetext near the most prominent appearance of the work’s title, preceding the beginning of thebody of the text.A section “Entitled XYZ” means a named subunit of the Document whose title eitheris precisely XYZ or contains XYZ in parentheses following text that translates XYZ inanother language. (Here XYZ stands for a specific section name mentioned below, suchas “Acknowledgements”, “Dedications”, “Endorsements”, or “History”.) To “Preserve theTitle” of such a section when you modify the Document means that it remains a section“Entitled XYZ” according to this definition.The Document may include Warranty Disclaimers next to the notice which states thatthis License applies to the Document. These Warranty Disclaimers are considered to beincluded by reference in this License, but only as regards disclaiming warranties: any otherimplication that these Warranty Disclaimers may have is void and has no effect on themeaning of this License.

2. VERBATIM COPYINGYou may copy and distribute the Document in any medium, either commercially or noncom-mercially, provided that this License, the copyright notices, and the license notice sayingthis License applies to the Document are reproduced in all copies, and that you add noother conditions whatsoever to those of this License. You may not use technical measuresto obstruct or control the reading or further copying of the copies you make or distribute.However, you may accept compensation in exchange for copies. If you distribute a largeenough number of copies you must also follow the conditions in section 3.You may also lend copies, under the same conditions stated above, and you may publiclydisplay copies.

3. COPYING IN QUANTITYIf you publish printed copies (or copies in media that commonly have printed covers) of theDocument, numbering more than 100, and the Document’s license notice requires CoverTexts, you must enclose the copies in covers that carry, clearly and legibly, all these CoverTexts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Bothcovers must also clearly and legibly identify you as the publisher of these copies. The frontcover must present the full title with all words of the title equally prominent and visible.

Page 24: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Appendix A: GNU Free Documentation License 21

You may add other material on the covers in addition. Copying with changes limited tothe covers, as long as they preserve the title of the Document and satisfy these conditions,can be treated as verbatim copying in other respects.If the required texts for either cover are too voluminous to fit legibly, you should put thefirst ones listed (as many as fit reasonably) on the actual cover, and continue the rest ontoadjacent pages.If you publish or distribute Opaque copies of the Document numbering more than 100, youmust either include a machine-readable Transparent copy along with each Opaque copy,or state in or with each Opaque copy a computer-network location from which the generalnetwork-using public has access to download using public-standard network protocols acomplete Transparent copy of the Document, free of added material. If you use the latteroption, you must take reasonably prudent steps, when you begin distribution of Opaquecopies in quantity, to ensure that this Transparent copy will remain thus accessible at thestated location until at least one year after the last time you distribute an Opaque copy(directly or through your agents or retailers) of that edition to the public.It is requested, but not required, that you contact the authors of the Document well beforeredistributing any large number of copies, to give them a chance to provide you with anupdated version of the Document.

4. MODIFICATIONSYou may copy and distribute a Modified Version of the Document under the conditionsof sections 2 and 3 above, provided that you release the Modified Version under preciselythis License, with the Modified Version filling the role of the Document, thus licensingdistribution and modification of the Modified Version to whoever possesses a copy of it. Inaddition, you must do these things in the Modified Version:A. Use in the Title Page (and on the covers, if any) a title distinct from that of the

Document, and from those of previous versions (which should, if there were any, belisted in the History section of the Document). You may use the same title as a previousversion if the original publisher of that version gives permission.

B. List on the Title Page, as authors, one or more persons or entities responsible forauthorship of the modifications in the Modified Version, together with at least five ofthe principal authors of the Document (all of its principal authors, if it has fewer thanfive), unless they release you from this requirement.

C. State on the Title page the name of the publisher of the Modified Version, as thepublisher.

D. Preserve all the copyright notices of the Document.E. Add an appropriate copyright notice for your modifications adjacent to the other copy-

right notices.F. Include, immediately after the copyright notices, a license notice giving the public

permission to use the Modified Version under the terms of this License, in the formshown in the Addendum below.

G. Preserve in that license notice the full lists of Invariant Sections and required CoverTexts given in the Document’s license notice.

H. Include an unaltered copy of this License.I. Preserve the section Entitled “History”, Preserve its Title, and add to it an item stating

at least the title, year, new authors, and publisher of the Modified Version as givenon the Title Page. If there is no section Entitled “History” in the Document, createone stating the title, year, authors, and publisher of the Document as given on itsTitle Page, then add an item describing the Modified Version as stated in the previoussentence.

Page 25: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Appendix A: GNU Free Documentation License 22

J. Preserve the network location, if any, given in the Document for public access to aTransparent copy of the Document, and likewise the network locations given in theDocument for previous versions it was based on. These may be placed in the “History”section. You may omit a network location for a work that was published at least fouryears before the Document itself, or if the original publisher of the version it refers togives permission.

K. For any section Entitled “Acknowledgements” or “Dedications”, Preserve the Titleof the section, and preserve in the section all the substance and tone of each of thecontributor acknowledgements and/or dedications given therein.

L. Preserve all the Invariant Sections of the Document, unaltered in their text and in theirtitles. Section numbers or the equivalent are not considered part of the section titles.

M. Delete any section Entitled “Endorsements”. Such a section may not be included inthe Modified Version.

N. Do not retitle any existing section to be Entitled “Endorsements” or to conflict in titlewith any Invariant Section.

O. Preserve any Warranty Disclaimers.

If the Modified Version includes new front-matter sections or appendices that qualify asSecondary Sections and contain no material copied from the Document, you may at youroption designate some or all of these sections as invariant. To do this, add their titles tothe list of Invariant Sections in the Modified Version’s license notice. These titles must bedistinct from any other section titles.

You may add a section Entitled “Endorsements”, provided it contains nothing but endorse-ments of your Modified Version by various parties—for example, statements of peer reviewor that the text has been approved by an organization as the authoritative definition of astandard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of upto 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the ModifiedVersion. Only one passage of Front-Cover Text and one of Back-Cover Text may be addedby (or through arrangements made by) any one entity. If the Document already includesa cover text for the same cover, previously added by you or by arrangement made by thesame entity you are acting on behalf of, you may not add another; but you may replace theold one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission touse their names for publicity for or to assert or imply endorsement of any Modified Version.

5. COMBINING DOCUMENTS

You may combine the Document with other documents released under this License, underthe terms defined in section 4 above for modified versions, provided that you include in thecombination all of the Invariant Sections of all of the original documents, unmodified, andlist them all as Invariant Sections of your combined work in its license notice, and that youpreserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identicalInvariant Sections may be replaced with a single copy. If there are multiple InvariantSections with the same name but different contents, make the title of each such sectionunique by adding at the end of it, in parentheses, the name of the original author orpublisher of that section if known, or else a unique number. Make the same adjustment tothe section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled “History” in the various originaldocuments, forming one section Entitled “History”; likewise combine any sections Entitled

Page 26: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Appendix A: GNU Free Documentation License 23

“Acknowledgements”, and any sections Entitled “Dedications”. You must delete all sectionsEntitled “Endorsements.”

6. COLLECTIONS OF DOCUMENTSYou may make a collection consisting of the Document and other documents released underthis License, and replace the individual copies of this License in the various documents witha single copy that is included in the collection, provided that you follow the rules of thisLicense for verbatim copying of each of the documents in all other respects.You may extract a single document from such a collection, and distribute it individuallyunder this License, provided you insert a copy of this License into the extracted document,and follow this License in all other respects regarding verbatim copying of that document.

7. AGGREGATION WITH INDEPENDENT WORKSA compilation of the Document or its derivatives with other separate and independentdocuments or works, in or on a volume of a storage or distribution medium, is called an“aggregate” if the copyright resulting from the compilation is not used to limit the legalrights of the compilation’s users beyond what the individual works permit. When theDocument is included in an aggregate, this License does not apply to the other works inthe aggregate which are not themselves derivative works of the Document.If the Cover Text requirement of section 3 is applicable to these copies of the Document,then if the Document is less than one half of the entire aggregate, the Document’s CoverTexts may be placed on covers that bracket the Document within the aggregate, or theelectronic equivalent of covers if the Document is in electronic form. Otherwise they mustappear on printed covers that bracket the whole aggregate.

8. TRANSLATIONTranslation is considered a kind of modification, so you may distribute translations of theDocument under the terms of section 4. Replacing Invariant Sections with translationsrequires special permission from their copyright holders, but you may include translationsof some or all Invariant Sections in addition to the original versions of these InvariantSections. You may include a translation of this License, and all the license notices inthe Document, and any Warranty Disclaimers, provided that you also include the originalEnglish version of this License and the original versions of those notices and disclaimers. Incase of a disagreement between the translation and the original version of this License or anotice or disclaimer, the original version will prevail.If a section in the Document is Entitled “Acknowledgements”, “Dedications”, or “History”,the requirement (section 4) to Preserve its Title (section 1) will typically require changingthe actual title.

9. TERMINATIONYou may not copy, modify, sublicense, or distribute the Document except as expresslyprovided for under this License. Any other attempt to copy, modify, sublicense or distributethe Document is void, and will automatically terminate your rights under this License.However, parties who have received copies, or rights, from you under this License will nothave their licenses terminated so long as such parties remain in full compliance.

10. FUTURE REVISIONS OF THIS LICENSEThe Free Software Foundation may publish new, revised versions of the GNU Free Doc-umentation License from time to time. Such new versions will be similar in spirit tothe present version, but may differ in detail to address new problems or concerns. Seehttp://www.gnu.org/copyleft/.Each version of the License is given a distinguishing version number. If the Documentspecifies that a particular numbered version of this License “or any later version” appliesto it, you have the option of following the terms and conditions either of that specified

Page 27: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Appendix A: GNU Free Documentation License 24

version or of any later version that has been published (not as a draft) by the Free SoftwareFoundation. If the Document does not specify a version number of this License, you maychoose any version ever published (not as a draft) by the Free Software Foundation.

Page 28: ARTE - SourceForgearte.sourceforge.net/documentation/arte.pdf · efficient way of preparing ARTE to be able to communicate with any kind of external software module to be tested.

Appendix A: GNU Free Documentation License 25

ADDENDUM: How to use this License for your documents

To use this License in a document you have written, include a copy of the License in the documentand put the following copyright and license notices just after the title page:

Copyright (C) year your name.

Permission is granted to copy, distribute and/or modify this document

under the terms of the GNU Free Documentation License, Version 1.2

or any later version published by the Free Software Foundation;

with no Invariant Sections, no Front-Cover Texts, and no Back-Cover

Texts. A copy of the license is included in the section entitled ‘‘GNU

Free Documentation License’’.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the“with. . .Texts.” line with this:

with the Invariant Sections being list their titles, with

the Front-Cover Texts being list, and with the Back-Cover Texts

being list.

If you have Invariant Sections without Cover Texts, or some other combination of the three,merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasingthese examples in parallel under your choice of free software license, such as the GNU GeneralPublic License, to permit their use in free software.