Model-Based Testing made easy with Spec Explorer 2010

Name: Wolfgang GrieskampTitle: Software ArchitectCompany: MicrosoftSession Code: DEV316

Modeling and Model-Based Testing

3

Modeling in Science and Engineering

A model:Is an abstraction of the system from a particular perspectiveSupports investigation, construction and prediction Is not necessarily comprehensiveCan be expressed as a table, graphical diagram, formal notation,etc.

Ubiquitous and Universal

Software Modeling Styles and Notations

4

Software Models

Structure

Class Diagram

Component Diagram

Behavior

Interaction Based

Use Cases

Message Sequence Charts

Traces/Patterns

State Based

Guarded Update Machines (Code)

State Charts

Focus of this session

Model-Based Testing in a Nutshell

Model

Implementation

Expected Outputs (Test Oracle)

Inputs(Test Sequences)

Control

Requirements

Generate

Observe

Feedback

Feedback

Author

Issue

Feedback

Test Cases

Verdict

Feedback

Model-Based Testing with Spec Explorer

Model-Based Testing with Spec Explorer

Explore

Analyze

Generate

Execute

C# Model (or other .Net Language)

Model Graph

Test Suite

VSTT Result

Walkthrough: Modeling and Testing a Web Service

Example: A Simple Web Shop

Web Shop Service

Web Shop Client

Billing

Shipping

AddToCartCheckout

Bill

Ship

System Under Test

Web Shop Service

Web Shop Implementation in WCF [ServiceContract(CallbackContract = typeof(IWebShopCallback))] public interface IWebShop { [OperationContract] // Add item to stock of web shop void AddItem(string itemName, int price); [OperationContract] // Add item to shopping cart of user bool AddToCart(string userName, string itemName);

[OperationContract] // Check out void Checkout(string userName); }

public interface IWebShopCallback { [OperationContract] // Initiate billing, returns true on success bool Bill(string userName, int amount);

[OperationContract(IsOneWay = true)] // Initiate shipping void Ship(string userName, string itemName); }

Understanding the Web ShopWhich of the following traces is valid?

(assuming an initialization of the shop with AddItem(“wine”,10) and AddItem(“cheese”, 5))

1 AddToCart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”,10)/true Ship(“Peter”,”wine”)

2 AddToCart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”,10)/false Ship(“Peter”,”wine”)

3 AddToCart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”,12)/true Ship(“Peter”,”wine”)

4 AddToCart(“Peter”, “wine”) AddToCart(“Peter”,”cheese”) Checkout(“Peter”) Bill(“Peter”,15)/true Ship(“Peter”,”wine”) Ship(“Peter”,”cheese”)

5 AddToCart(“Peter”, “wine”) AddToCart(“Peter”,”cheese”) Checkout(“Peter”) Bill(“Peter”,15)/true Ship(“Peter”,”cheese”) Ship(“Peter”,”wine”)

Modeling Web Shop: Basic Approach

1. Import or declare action vocabulary in configuration script

2. Define state variables in C# model program3. Define state update rules in C# model program4. Define scenarios and generate state transition

system

config Actions{ action static void

TestAdapter.AddItem(string itemName, int price); action static void

TestAdapter.AddToCart(string userName, string itemName); action static void

TestAdapter.Checkout(string userName); action event static void

TestAdapter.Billed(string userName, int amount, bool success); action event static void

TestAdapter.Shipped(string userName, string itemName);}

Modeling Web Shop: Action VocabularyAdapter for the

System-Under-TestReference to test adapter method

Indicates that this action is an event raised by adapter

Modeling WebShop: Define Model State in C#

enum ShopperStatus{ Shopping, CheckedOut, Billed }

class ShopperInfo{ // Current status of this user internal ShopperStatus Status;

// Items on cart so far internal SequenceContainer<string> Items;}

// Foreach user shopping, contains info static MapContainer<string, ShopperInfo> shoppers;

// Foreach item in the shop, contains pricestatic MapContainer<string, int> items;

Modeling WebShop: Defining State Update Rules

[Action(“AddToCart(userName,itemName)”)]static void AddToCart(string userName, string itemName){ Contracts.Requires(items.ContainsKey(itemName));

if (!shoppers.ContainsKey(userName)) shoppers[userName] = new ShopperInfo();

Contracts.Requires(InStatus(userName, ShopperStatus.Shopping)); shoppers[userName].Items.Add(itemName);}

static bool InStatus(string userName, ShopperStatus status){ return shoppers.ContainsKey(userName)

&& shoppers[userName].Status == status;}

Binding rule to action in action vocabulary

Condition under which rule is enabled.

Fires a state update

State update

Modeling WebShop: Define Scenarios

machine MultipleItemScenario() : Config{ AddItem("Bread", 2); AddItem("Cheese", 5); AddItem("Wine", 10) AddToCart("Paul","Wine"); AddToCart("Paul", "Cheese"); (Checkout | Billed | Shipped)*}

machine MultipleItemSlice() : Config { MultipleItemScenario || construct model program from Config}

Combining scenario with C# model program

Or’ing actions in repetition (Note parameters omitted)

Sequence of actions

Modeling WebShop: Exploration of the Scenario

Parameters and control flow undetermined

Modeling WebShop: Exploration of the Model Slice

Choice point for system-under-test

Modeling WebShop: Execution of generated test code

Model expects bill of $2, implementation bills $4

The WebShop Example in ActionDemo

Open Protocol Documentation Testing

Spec Explorer @ Microsoft

“BlueLine” Technical Document Testing Project

222 open protocol specifications tested22,847 pages

66,962 person days (250+ years)Vendor test teams

Hyderabad: 250Beijing: 100

Tool dev center in Beijing (Spec Explorer and other)~10,000 document bugs filed

Comparison Model-Based/Traditional Testing

Vendor teams had (guided) choice which approach to useOut of 222 protocols approx. ½ modeled, ½ traditional Modeling resulted in 42% productivity gain in average (some teams reached more than 300%)

Overall, modeling saved 50 person-years (12,547 days)

No special education background was requiredTeams of junior engineersMostly vendors, fresh out of college

Question & Answer

Spec Explorer on the Web

DevLabs home page and download http://msdn.microsoft.com/devlabsSpec Explorer team blog http://blogs.msdn.com/specexplorerSpec Explorer forum http://social.msdn.com/forums/specexplorer

Wolfgang Grieskamp’s blog http://blogs.msdn.com/wrwg

A Model-Based Testing tool from Microsoft

www.microsoft.com/teched

Sessions On-Demand & Community

http://microsoft.com/technet

Resources for IT Professionals

http://microsoft.com/msdn

Resources for Developers

www.microsoft.com/learning

Microsoft Certification & Training Resources

Resources

Complete an evaluation on CommNet and enter to win an Xbox 360 Elite!

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,

IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.