DRAFT: Source Code Analyzer Tool Assessment Guide

December 6, 2007

Researchers at the National Institute of Standards and Technology have prepared this document. It may represent preliminary research findings and does not necessarily

represent any policy positions of NIST.

Consistency in terms

Tool functionality vs capability vs featuresMeasure?

Can we say ‘required functionality’ --- required by who/what(Or should we say recommended functionality?)

I need to understand the metadata file better and how all the files are linked – named – put together. I want to make sure that there is a way to tell which files go with which metadata which go with which violations.

All examples are for the same thing – we need to point this out.

Add/fix section numbers

TABLE OF CONTENTS

Preface..................................................................................................................4Purpose..........................................................................................................................5Scope.............................................................................................................................5Audience........................................................................................................................5How to Use This Guide..................................................................................................6Structure of this Document.............................................................................................6Related Documents........................................................................................................7

1 Background.....................................................................................................82 Static Source Code Analysis Tool Functionality (How the Tools Work)..........8

2.1 Introduction........................................................................................................82.2 Determining a Tool’s Suitability for VVSG..........................................................9

2.2.1 Source code editors/style checkers...........................................................................92.2.2 Compilers................................................................................................................102.2.3 Bug Checkers and Security Analyzers....................................................................102.2.4 Source Code Understanding Tools.........................................................................10

3 Source Code Analysis Tool Tests.................................................................113.1 Tool Test Case Content...................................................................................113.2 Installing and Using the Tool Test Suite..........................................................12

3.2.1 Installation...............................................................................................................123.2.2 Using the Tests.......................................................................................................13

References..........................................................................................................15Appendix A – Tool Test Suite Listing...................................................................16Appendix B - Test Metadata Example................................................................19Appendix C - A Test Walk-Through....................................................................21Appendix D - A Source Code Test.....................................................................25

3

Preface

The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology (IT).

This document provides guidance to test labs in their use of source code analysis tools to examine and verify voting system source code for conformance to coding conventions defined in VVSG core requirements. Specifically, this guide provides information about source code analysis tools and a set to tool tests to help the test lab determine if the tool is capable of finding coding convention errors. This guide and its associated tool tests is NOT a conformance test suite for determining a voting system’s conformance to the VVSG.

4

1 Intoduction

1.1 Purpose

This guide is designed solely for the purpose of assisting test labs in assessing the capabilities of source code analysis tools that may software-testing laboratories are currently be useding, during or intend to use for determining conformance testing of the to VVSG. In particular, test labs may use software testing tools to analyze the VVSG 2005 or the next VVSG coding convention requirements. These software tools may be commercially available, in the public domain, or developed in-house by the test lab. Although a test lab need not use source code analysis tools, if tools are used, it is important that the test tools work as purported and are capable of finding violations to the coding convention requirements. This guide can help test labs make that determination. The guide provides information about source code analysis tools, identifies features commonly found in the tools, and functionality essential to the tool. It also includes test materials which can be used to measure a tool’s ability to find ???? . Measuring tool capability ....(need to explain what this is or what it provides or why do it)

The goals of this guide include: helping a test lab laboratory determine if source code analysis tools will work for

its intended purpose, providing guidance to tool users in how to properly use the tools, increasing public confidence in a testing lab’oratories ability to use appropriate

tools when testing voting systems software for conformance to the VVSG.

Note, t This is NOT a guide or test suite for determining assessing a voting system’s conformance to the VVSG.

1.2 ScopeThere are many types of tools used to assure the quality of software today. This includes requirements, architecture and design analysis, and static binary analysis tools. Additionally, dynamic testing tools are used to assure that software performs as intended and is free from defects and security vulnerabilities. This tool guide is limited to

5

assessing the capabilities of the most commonly used source code analysis tools, including:

style-checkers compilers bug-checkers/security analyzers source code understanding/metrics generator tools

Details of how these tools work, features and , and capabilities that must be present in the tools and what features are necessary to evaluate them areis described in section 3.

1.3 Audience

The primary audience for this tool guide is EAC accredited test labsing laboratories. Additionally, testing laboratories and consultants performing state certifications of voting systems can benefit from this guideits use. It may also assist voting system developers in their quality assurance (QA) efforts.

1. 4 How to Use This Guide

This guide can be used as both a general reference for the understanding of source code analysis tools and their role in VVSG conformance testing, as well as for step-by-step instructions on how to measure a tool’s capabilities.

At the higher level, this guide provides background information regarding what source code analysis is, and the benefits that automated tools can offer in VVSG conformance evaluations. Additionally, the guide provides an overview of the four types of tools that can be employed and , theirthe fundamental features and capabilities inherent in each type of tool required functionality.

At the lower level (for testing laboratory staff), this guide provides all of the necessary information and test materials (i.e., tests in the form of sample code or code snippets) to assess whether a source code analysis tool is appropriate for can be of use in verifying source code conformance to VVSG coding convention requirements. This The test materials includes documented tool tests with guidance on how to configure and use the tool, and how to run the tool tests.

1.5

Related Documents

6

The U.S. Election Assistance Committee’s Voluntary Voting System Guidelines (VVSG) specify the requirements rules that manufacturers developers of voting systems must implement not violate in developing source code for voting systems. The tool tests in this guide are based upon the coding requirements in the following documents:

VVSG 2005

o Volume I, section 1.6, Conformance Clause o Volume I, section 5.2, Software Requirements o Volume II, section 5, Software Testing

VVSG 2007

o Vol III: Sec 4.5.1.A, Sec 4.5.1.B, Source Code Workmanship Review

Do we need to say something about 2005 having many more coding convention requirements than the next VVSG?

1.6 Structure of this Document

This guide is divided into three sections:

Section 1 is this introduction.

Section 2 provides a general background information on the role of source code analysis tools in the software development process. Each of the

Section 2 introduces the four types of automated source code analysis tools are introduced that can assist testing laboratories in verifying whether source code conforms to coding convention requirements. A short description of each type of tool is provided, along with a list of typical tool features, and required tool capabilities that should be present functionality for effective testing lab use.

Section 3 introduces the tool tests, describes the content of a test case, and provides a step-by-step instructions oin the use of the tool tests.

In addition, supporting reference information is provided in Appendices A and B.

Appendix A provides a complete list of all tool tests, cross-referenced against the tools for which they are applicable, and hyper-linked to the actual tests.

Appendix B provides an end-to-end walk-through of a source code analysis test case execution, using one of the NIST tests, and a source code analysis tool.

7

2 Source Code Analysis ToolsBackground

The use of source code analysis tools can save testing laboratory staff days or weeks of manual source code examination through automated scanning. In the case where the code consists of hundreds of thousands to millions of lines of code, automated source code analysis tools provide a repeatable, unbiased review that can quickly identify some of the common programming flaws that exist in today’s software.

Because of their relative ease of use, scalability for large programs and maturity, source code analysis tools play a fundamental role in examining source code for compliance to coding conventions. Coding conventions are rules created by an individual software developer, a software development company, or a consortium that define and enforce style and structure in the writing of source code. These rules can encompass practices that include source code formatting, commenting and naming conventions, as well as source code modularity, integrity and security. The VVSG 2005 and next VVSG define coding conventions. ESome exxamples of a VVSG coding convention violations include:

lack of source code commenting (give VVSG reference), exceeding 80 characters per line (reference), and exiting a program without providing a message to the user and poor program

modularity (reference),.

Source code analysis tools typically come with support for conventions that are generally common and adopted by the software development community. Additionally, a user can often customize a tool to examine code against unique coding rules.

This document provides guidance to testing laboratories in their use of source code analysis tools to examine and verify voting system source code for conformance to coding conventions defined in VVSG core requirements.

2.1. S

Static Source Code Analysis Tool Functionality (How the Tools Work)

Introduction

Static source code analysis is a generalized term for examining source code for particular properties. The word “static” means that the code being examined is not actually executed (i.e. the code is not “dynamic” and running). The word “analysis” can have

8

many meanings [1]. Analysis can be as simple as searching for particular strings of text in source code, and reporting them to the tool user. Analysis can also be complex, such as searching for bugs and/or security vulnerabilities in source code through dataflow and control flow analysis and property verification, or computing code quality metrics. Many tools limit themselves to performing a single function. General-purpose source code analysis tools typically perform a combination of these functions.

Say something about labs may use a combination of tools

2.2 Determining a Tool’s Suitability for VVSG

The coding requirements specified in the VVSG limit source code analysis primarily to verification of correct coding style (not code correctness, which is addressed in another part of the VVSG). Because of the narrow scope of source code requirements, this document limits its scope to the discussion of four particular classes of tools that are suitable for this purpose:

Source code editors/style checkers Compilers Bug checkers and security checkers Source code understanding tools

To be assist a testing laboratory in determining conformance to VVSG coding conventions, source code editors/style checkers, compilers, and bug checkers and security checkers the first three tools listed above must be capable of:

Identifying one or more violations of the VVSG coding conventions, Correctly identify the file, module name, line number and column location of the

violation in the source code.

TPlease note that the fourth tool type, source code understanding tools, by themselves cannot identify a coding convention violation. This type of tool will always require human analysis. That said, source code understanding tools are particularly useful for identifying coding convention violations that automated analysis tools cannot find, and are therefore included in this guide.

2.1.1 Source code editors/style checkers

These tools focus on syntactic correctness of the source code against a language specification, as well as compliance to particular coding conventions. They identify syntactic, and to a lesser degree than bug/security checkers, semantic programming style problems in source code files. Common tool features include:

9

Highlighting of language keywords, comments and operators, Enforcing indentation/whitespace rules for code and comments, Searching for particular strings in source code using regular expressions. Verifying code syntax against a particular API,. Enforcing naming conventions.

2.1.2 Compilers

As part of compiling source code, these tools report any problems that will prevent compilation (in the form of compiler error messages) or that may produce unexpected runtime results (in the form of compiler warning messages). Compilers do not, as a rule, enforce coding conventions outside of the basic syntactic and semantic rules specified for that particular language, however some compilers (through command-line options) support the enforcement of a small set of programming style rules. A useful compiler feature for enforcing coding conventions is type checking analysis.

2.1.3 Bug Checkers and Security Analyzers

2.1.4These tools identify source code problems of a deeper variety than those found by source code style checkers and compilers. Bug checkers and security analyzer identify bugs and security weaknesses in source code by uUsing program property checking as well as , dataflow and control flow analysis techniques. ; bug checkers and security analyzers identify bugs and security weaknesses in source code.

Bug checkers – Find flaws in source code that may prevent the program from performing as intended (not necessarily a security vulnerability). Some common bugs identified by this class of tool include:

Un-initialized variables, Omitted break statements, Infinite loops, Unreachable code, Null pointer dereferences.

Security checkers – Check for security-related conventions that should not exist in source code. Some common security-related conventions identified by this class of tool include:

Potential buffer overflow Memory leaks Presence of “debug” code in final product

10

Not all bug checkers and security checkers find the same types of problems in source code. Therefore, to use a tool for VVSG conformance testing, it is helpful to verify what a tool identifies “out of the box” (by reading its documentation) prior to purchasing it.

2.1.5 Source Code Understanding Tools

These tools assist a user in understanding the structure of source code by generating a graphical, navigable and searchable representation of the code. They also generate commonly used code quality metrics that provide indicators of how well the code was constructed.. Common tool fFeatures include:

Graphical control-flow mapping and traversal, Graphical data-flow mapping and traversal, Keyword search through code by function, variable name or other property, Source code metrics generation, such as McCabe cyclomatic complexity and lines

of code per module.

These tools serve as an “aid” to human analysis. By themselves, they cannot identify a coding convention violation. However as an aid to human analysis, they can greatly assist in helping the user testing laboratory staff quickly identify areas of code in need of for greater scrutiny, as well as nd help the tool user identify a coding convention violations.

3 Source Code Analysis Tool Test Suites

The NIST Source Code Analysis Tool Test Suite consists of the set of tests and NIST has developed source code tests (and supporting documentation) for for the purpose of determining if a source code analysis tool can identify violations of the VVSG coding conventions. The test suite reflects VVSG coding convention requirements in the following areas: They are written in multiple languages, and cover the following VVSG source code conformance areas:

Integrity – requirements regarding self-modification at run time (reference), Modularity – requirements defining module size and function encapsulation

(reference), Naming conventions – requirements for function and variable naming semantics,

proper use keywords and name scoping (reference), Control conventions – requirements for logic conventions such as loops and case

statements and module entry/exit points (reference),

11

Comment conventions – self-documenting requirements regarding appropriate header files explaining function, I/O and revision history (reference),

Additional coding conventions – requirements that fall outside of the above categories (reference),

For each of these areas, there is at least one tool test. The tool tests are small programs (i.e., source code files) that instantiate violations to specific coding practices, techniques, structure, etc. The tests are coded in one or more programming languages, depending on which language(s) most appropriately represents the targeted coding convention. Source code files are written in C, C++ and Java in this initial set of tests. Other languages will be added as appropriate.

3.1 Tool Test Case Content

The ZIP archive file that contains this document also contains the source code analysis tool test suite.

The tool test suite consists of at least one test for each of the above coding convention violations, in at least one programming language. Source files are written in C, C++ and Java in this initial set of tests. Other languages will be added as appropriate. Each tool test contains at least one source code file and an XML-encoded metadata file that provides all of the necessary information to understand the test and run a tool against it.

The A test case contains an XML metadata file, ; named metadata.xml, that uniquely identifies the set of applicable source code files example file set and provides for identification and traceability back to the VVSG. Thus, each tool test is related to specific VVSG requirements. Specifically, the metadata includes:

Name of test ???? Source code file IDs ????? Test Description -– information for a high-level description view of the purpose

of theis test VVSG requirement identifier (by date, volume and section) - providing

traceability to the VVSG coding requirements Expected Results – details regarding where the location of the actual coding

violation, is located, including:o The source file nameo The line and column number o The module name

Usage suggestions for appropriate tools that could assist in finding this particular coding convention violation, broken down by tool type, and containing recommendations for:

o Tool configurationo Tool extension

12

o Tool Analysis Test Design – low-level details about the test case Test Analysis – a view of how the tool should analyze test code Tool Notes - factors that can influence the ability of the tool to find

the coding convention violation

See Appendix B for an example of To see an example of what this metadata file. looks like, please examine the example in Appendix B.

3.2 Installing and Using the Tool Test Suite3.2.1 Installation

1. The test suite is bundled in the same ZIP file, xxxx.zip archive containing this guide. Unzip the test suite archive (do we have a name??) into a location that is accessible to the tool(s) that you wish to test. The directory structure of the test suite is as follows:

a) The “sca_tool_testsuite” root directory containing this document, a directory named xml containing an xml schema and stylesheet for test suite metadata

b) The “tool tests” sub-directory containing the entire tool test suitec) 6 sub-directories corresponding to the six categories of coding convention

violations listed in section 3. d) Sub-directories, each named for a particular coding convention violation,

containing all the tests (in all programming languages) for that particular coding convention

e) Subdirectories for each particular language that a test is written in (Cc, Cc++ and Jjava)

f) If more than one source code fileexample is needed to verify whether a tool can identify a particular coding convention violation for a particular language, then individual test case directories labeled 01, 02, 03 etc…exist.

For An example: of the directory path containing the third C-language tool test case containing a violation of the array boundary limits coding convention would be:

sca_tool_testsuite/tool_tests/

integrity/ exceeding_array_string_bondaries/

c/ 03/ main.c

13

3.2.2 Using the Tests

The following instructions guide the test suite user through the process of selecting test cases, understanding their meaning, configuring and running a tool against the tests. The basic workflow for using the tests are illustrated below:

3.1.1.1 Select a Tool

Select your a source code analysis tool and determine which that is one of the four tool types described in this guide (source code editor/style-checker, compiler, bug-checker/security analyzer, or understanding/code metrics generator tool) it fits. You can determine which type of tool you have by consulting section 2.2. Note that some tools are “hybrids”, and have features of more than one tool type; for example a code security analyzer may also have features of a code understanding tool or a style checker may also have feature of a bug checker tool. A hybrid tool can be run against a greater number tests in this suite than a “standalone” tool.

3.1.1.2 Select a Test

1. Go to the Source Code Analysis Tool Tests table in Appendix A, Table 12. Click on the hyperlinked coding convention violation name in column 1 that has

an “X” in the column corresponding to the tool that you wish to test. This will take you to the test suite directory containing all of the tests for that particular coding convention violation.

3. Move to the appropriate subdirectory (Cc, Cc++ or Jjava) based upon the programming language the tool can analyze.

4. One or more sub-directories, labeled 01, 02…. etc. will be present in this directory, corresponding to the number of tests for this coding convention violation.

5. Click on any sub-directory icon to change to that directory and examine a particular test.

6. Each test case directory contains an XML metadata file, named metadata.xml and one or more source code test files. Each test file is labeled main.c, main.cpp or main.java (depending upon the programming language of the test).

7. Click on the metadata.xml file icon to display information about the test, including its description, VVSG requirement ID and expected test results. It is best to view the XML metadata using a web browser.

14

Select a Tool Select a Test Configure/Extend Tool Test Tool Analyze Results

3.1.1.3 Configure/Extend the Tool

If the tool does not find a particular coding convention violation “out of the box”, use the metadata.xml file for will also contain tool configuration and/or extension recommendations.

a. Examine the configuration information to learn what additional tool options need to be set to run this test.

b. Examine the extension recommendation if the tool does not support the identification of this coding convention violation “out of the box”.

3.1.1.4 Run the tool

Start the tool and load the test file

a) For tools with a command-line interface Open a shell window on the machine where the candidate tool is

installed, Change the working directory to the location of the test, Invoke the tool, including any required and optional arguments

specified in the test configuration metadata described in 3.2.2.3,. Use the name of the test file main.c, main.cpp or main.java as the

last command line argument,. An analysis report is generally written directly to the console

window, or to a file.

b) For tools with a graphical user interface Invoke the tool from the desktop environment, Provide any configuration options to the tool as specified by the

tool configuration metadata described in 3.2.2.3,. Set the working directory to the location of the test, Open the main test file main.c, main.cpp or main.java as

appropriate, Run the automated analysis (for code understanding tools, perform

the manual analysis described in the “Test Details” portion of metadata.xml file).

3.1.1.5 Analyze Results

Compare the analysis results against the expected test result information provided in the associated metadata.xml file.

15

a. For automated tools, verify that the tool reports the coding convention violation by a semantically equivalent name, file name and line number.

b. For code understanding tools, verify that the tool user was able to identify the coding convention violation with in its correct file, module name, and line number.

References

[1] Chess, West, “Secure Programming with Static Analysis”, Addison-Wesley Software Security Series, 2007

Appendix A – Tool Test Suite Listing

Table 1 below provides a complete list of the VVSG source code convention violations. Each item is hyperlinked to the test suite directory containing the actual tool tests associated with that convention. A corresponding “checked box” indicates which type of tool (or tools) may assist testing labsoratory staff in identifying coding convention the violations.

There are some coding convention violations that no tool can find in source code. Only a visual examination by test laboratorytest lab staff can verify whether there is a violation or not. In such a case, no box is checked.

Convention Violation

StyleChecker/Editor

Compiler Bug Finder /Security Analyzer

Source CodeUnderstanding

16

Use of low-level language

X X

Presence of self-modifying code

X X

Use of DLL X XUse of interpreted code

X X

Exceeding array/string boundaries

X

Improper pointer addressing

X

Improper dynamic memory mgmt

X

Lack of code modularityMissing header comment

X

Missing library header

X

Multiple module entry points

X X

Multiple module exit points

X X

No formal exception handling

X

Unacceptable control structuresDo-while (false) XIntentional exception

X

Inconsistent code and documentationMultiple definitions of same variable

X

Name semantics and mnemonicsImproper name character differentiation

X

Improper use of single character names

X

Improper name scoping

X

17

Improper use of language keywords

X X

No unit purpose comment

X

No other units called comment

X

No input parameter comment

X

No output parameter comment

X

No file and access methods comment

X

No global variables comment

X

No date of creation comment

X

No revision history comment

X

No uniform calling sequences

X

No explicit return value

X X

Does not return “0” for success

X X

Uncorrected error return value

X X

Return statement in macro

X

No default choice for case statements

X X

No vote count (integer) overflow controls

X

No indentation XImproper module size

X X

No generated code delineation

X X

Exceeds 80 column line limit

X

Exceeds maximum executable statements per line limit

X

18

Improper use of executable statements in conditions

X

Uses mixed mode operations

X X

No exit message explanation

X

Clear normal status message

X

Clear error status message

X

Exceeds variable indirection limit

X

Exceeds indentation scope

X

No variable initialization

X

Improper constant definition

X

Exceeds minimum implementation of conditional statement

X

Assert statement is present in code

X X

TABLE 1

Appendix B - Test Metadata Example

ID??? Or NAME?????

Test Description

This test case is a simple example of a buffer overflow that any generalized bug checking or security analysis tool should correctly identify. This test can be used to quickly determine if the tool can in fact identify a simple buffer overflow at all.

Requirement ID: VVSG 2005:Vol 1:5.2.2

19

Expected Test Results

File: path = main.c

Coding Convention Violation: name = exceeding array string boundaries, line = 38, column = 12, module = test

Tool Recommendations:

Bug Checker or Security Analyzer

Extension:

Custom extension for this type of weakness is typically not necessary in today's generalized bug checking and security analysis tools. The tools support finding this coding convention violation, more often referred to as a "buffer overflow" out of the box. That said, there are many different manifestations of buffer overflows in (particularly) C and C++ source code, and no tool claims to catch all of them.

Analysis:

Test Design:

This is an example of a buffer overflow, in which data can be written to a character array beyond its bounds through direct user input from the command line.

Test Analysis:

A tool is expected to recognize that a potentially tainted input string that could exceed the declared length of a character array overwrites stack memory.

Tool Notes:

Buffer overflow reports in bug checkers and source code security analyzers are often "false positive" reports; meaning that no real potential for a buffer overflow exists, but the tool reports that there may be one present in the code. This requires human inspection to verify if in fact a buffer overflow exists. All bug checkers and source code security analysis tools generate false-positive reports, and a tool should not necessarily be dismissed because it generates them. Each laboratory must determine when the percentage of

20

false-positive to true-positive (actual) reported buffer overflows generated by a tool is "excessive", and wastes valuable human resources spent verifying the reports.

Complex coding constructs such as pointer aliasing or inter-procedural control and dataflow can confuse a tool and result in a "false negative" report, indicating that it did not find the buffer overflow. To verify that your tool can find buffer overflow in more complex source code, run the tool against all of the other buffer overflow examples in this directory.

Appendix C - A Step-by-Step Test Walk-Through

Below is a step-by-step walkthrough of a test of a “candidate” source code security analyzer tool that a user may wish to include in his toolbox for verifying conformance to VVSG coding conventions.

1. Testing laboratory staff acquires download a copy of Tool A to assess its potential for use in identifying violations of VVSG coding conventions in source code. The tool documentation claims that it finds many common security flaws in source code, including buffer overflows in both C and C++ programming languages. The tool also provides reports of what it finds, providing the name of the coding flaw, the module containing it, and the file name and line number.

2. The tool user downloads and installs the NIST Source Code Analysis Tool Assessment Guide and accompanying test suite and finds that “exceeding array/string boundaries” is a

21

test that can be used to verify whether the tool will be useful for their VVSG source code conformance testing program.

Convention Violation

StyleChecker/Editor

Compiler Bug Finder /Security Analyzer

Source CodeUnderstanding

Use of low-level language

X X

Presence of self-modifying code

X X

Use of DLL X XUse of interpreted code

X X

Exceeding array/string boundaries

X

Improper pointer addressing

X

Improper dynamic memory mgmt

X

3. Clicking on the “Exceeding array/string boundaries” link in the table, the user navigates down the directory tree to the first test case for detecting this coding convention violation.

22

4. Open the metadata.xml file (using a web browser) to view all of the information concerning this test.

a) Reading the test description, the user understands that the purpose of this test is to verify that the tool can at a minimum identify a simple array overflow

b) Reading the VVSG requirement ID: VVSG 2005:Vol 1:5.2.2, the tool user can go to that document and verify that the test purpose is aligned with the requirement:

…“Where the development environment (programming language and development tools) includes the following features, the software shall provide controls to prevent accidental or deliberate attempts to replace executable code:

Unbounded arrays or strings (includes buffers used to move data)”…

c) The tool user then verifies what is expected from the tool by reading the “expected test results”, and understands that the tool should find a coding convention violation semantically equivalent

23

to the name “exceeding array string boundaries” in the file main.c at line 38, column 12 in the module “test”

6. The tool user examines the “tool recommendations” section of the metadata.xml and finds that there are no required tool configuration measures necessary prior to running the tool against the test because none is specified in the metadata.

7. Likewise, no tool extension measures are necessary because the extension information informs the userhim that the tool generally comes with the capability to find a buffer overflow “out of the box”.

8. The user examines the Analysis information and learns the details of:

a) How the test is designed at the code level through reading the “Test Design” information. In this case, a character array boundary is overwritten through direct user input from the command line.

b) How the tool is expected to perform an analysis through reading the “Test Analysis” information: …“a tool is expected to recognize that stack memory is overwritten by a potentially tainted input string that could exceed the declared length of a character array The tool can at a minimum identify a simple array overflow”

c) Circumstances outside of the test itself that may influence the tool’s ability to identify the coding convention violation are described in the “Tool Notes” information. For this particular test, a class of tool sometimes generates “false positive” results when in fact no coding convention violation exists. Additionally, complex coding structures surrounding the coding convention violation (such as pointer aliasing and inter-procedural control flow) may also influence test results. Most importantly, this section tells the user that the other tests against this coding convention will exercise a tool’s capability to deal with “false positives” and complex code.

9. The user runs the test, using the instructions in section 3.2.2.4 of this guide. The actual results of the a tool test are displayed below:

24

[C:\vote_tool_testing\tool_tests\integrity\exceeding_array_string_bondaries\c\01]

[71D04D5D4B1482E574FF37CB70006991 : high : Buffer Overflow : dataflow ]main.c(42) : <=>

main.c(54) : ->test(0) main.c(53) : <=> (userstr) main.c(48) : ->main(1)

These results indicate that the source code security analysis tool found a “buffer overflow”, a semantic equivalent name to the “exceeds array boundaries” coding convention violation. The tool found the violation at line 42 in the file main.c, at line 42, in the module test.

If you look at the actual source code in the file main.c, you can verify that this is in fact where the array boundary is exceeded at line 42, as annotated with the comment /*BAD*/ on that line.

10. If the test results are suspect, and the tool tester does not feel that the tool was adequately configured and/or extended the tool to run this test, they can revisit the configuration and extension information, possibly revise their testing procedure, and run the test again. The test user can iterate through this step as much as is necessary to satisfy them with the final test result.

25

Appendix D - A Source Code Test(which test is this ? Is this for the example above ? Does this go with the metafile example? We should say this, somewhere earlier in the document.)

#include <stdio.h>#include <string.h>

#define MAXSIZE 40

voidtest(char *str){

char buf[MAXSIZE], *p;

p = buf;while((*p++ = *str++)) /* BAD */

continue;printf("result: %s\n", buf);

}

intmain(int argc, char **argv){

char *userstr;

if(argc > 1) {userstr = argv[1];test(userstr);

}return 0;

}

26