Windows Protocol Test Suites: Overview and Demo

Diane Larsen

Jiajun Wang

What is a protocol test suite?

Evaluates whether a protocol implementation meets certain interoperability requirements.

Originally developed for in-house testing of Microsoft Open Specifications.

Used to test/verify Windows behavior.

Also used to test 3rd-party implementations.

Our test suites do not cover every protocol requirement, and do not certify an implementation, but can be a useful indication of interoperability.

Windows Test Suite Types

Individual protocols: MS-SMBD, MS-SMB, MS-ADFSPIP

Protocol families: AD, File Sharing (MS-SMB2 and related protocols), Remote Desktop, Kerberos, Branch Cache

Overview documents: MS-ADOD, MS-AZOD

2. Develop protocol test suite (Synthetic Client) according to Open Specification

3. Run against Windows Server to verify Open Specification & OS behavior

4. Run against partner’s Server Implementation to help identify & debug issues

1. Open Specification defines messages, sequences, behaviors

5. Success! Interop between partner’s Server and Windows Client

Example: Client/Server Side Partner Implementation

Open Specification

Windows Client(desktop , surface,

phone etc)

Windows Server

Partner’s Server(storage, AD…)

Protocol Test Suite

Synthetic Client

4

5

Message Over the Wire1

2

3

Test Case 1:1. Establish connection;2. Client.Send(msgA);3. Expect server respond with msgB, and

• Assert (msgB.Field1 == valid1)• Assert (msgB.Field2 == valid2)• …• Assert (msgB.FieldN == validN)

Test Case 2:1. Establish connection;2. Client.Send(Invalid msgA);3. Expect server respond with msgC, and

• Assert (msgC.Status == ERROR)

Protocol Specification:If SMB2 client sends message A to SMB2 server, SMB2 server should respond with message B. If message A is invalid, server should return error message C.

Where are they?

Windows protocol test suites are on GitHub.

Two formats: source code and MSI files

Both formats have been updated in the past week for this event.

Support is available via GitHub (or talk to test team members this week).

https://github.com/Microsoft/WindowsProtocolTestSuites

Required and optional components

Required Software

• Windows 8, Windows Server 2012 (and later)

• Visual Studio 2012• Visual Studio 2013 for MS-SMBD test suite

• Spec Explorer Power Tool • For model-based testing used by most test suites

• Microsoft Protocol Test Framework

• Simplifies configuration, logging, checking

• Based on Visual Studio Unit Test framework

• Test Suite(s)

• Full list per test suite is available on GitHub

Optional Software

• Protocol Test Manager

• Simplifies test suite configuration and execution.

• Detects system capability, selects test cases, displays status.

• Demo in a few minutes!

• Microsoft Message Analyzer

• Capture, display, and analyze protocol messaging traffic.

Other sessions this week

We’ll have a demo next (as part of this session)

After this session, Vivian will talk about open source test suites.

At 3:00 in building 20, Helen will introduce the test suites environment and show how to run a test suite.

Also at 3:00 is a session in this room about Office tests and tools followed by a session about SQL Server test and tools.

Later in the week will be a session about Message Analyzer, the Microsoft network (and more) analysis tool.

Protocol Test ManagerJiajun Wang

Overview of Protocol Test Manager

A unified UI wizard to execute protocol test suites:

• Embed deployment guide

• Detect SUT capability automatically

• Select test cases automatically

• Configure test suite easily

• Run test cases friendly

Options to Execute Test Suites

Option 1: Using batch files bundled with Test Suites installer

Pros: Less dependency.

Cons: Manually modifying the

ptfconfig file is time-consumingand error-prone.

Options to Execute Test Suites

Option 2: Using Visual Studio

Pros: Easy to control every test step and debug.

Cons: Need to install Visual Studio.

Need to download source code and build the test suite yourself.

Need to manually modify ptfconfig file (difficult).

Visual Studio

Options to Execute Test Suites

Option 3 (recommended): Using Protocol Test Manager (PTM)

Pros: Easy to use.

Auto detects your test environment and modifies the ptfconfig file.

Import/Export test profiles to save your current configurations.

Cons: Need to download & install PTM separately.

Protocol Test Manager

Protocol Test Manager

Detect

Detection summary

Filter cases

All configurations

Run test cases

Protocol Test Manager

Detect

Detection summary

Filter cases

All configurations

Run test cases

Protocol Test Manager

Detect

Detection summary

Filter cases

All configurations

Run test cases

Protocol Test Manager

Detect

Detection summary

Filter cases

All configurations

Run test cases

Protocol Test Manager

Detect

Detection summary

Filter cases

All configurations

Run test cases

Command Line Interface

PtmCli.exe -p adfamily.ptm -s -r report.txt

cat report.txtFailed APDS_S1_TC23_NTLM_NETWORK_VALIDATE_A2AF_RESTRICTIONPassed APDS_S3_TC04_DIGEST_DIFF_RESERVED4_SAME_REPLYPassed APDS_S3_TC05_DIGEST_REJECT_ALGTYPE_MD5Passed APDS_S3_TC06_DIGEST_REJECT_ALGTYPE_MD5_ASSUMEDPassed DRSR_DRSBind_UnBind……

Handy to use when doing automatic testing and analysis the results.

Recent Updates to Protocol Test Manager

Added status icons in the result page so that users can more easily track the status of test cases.

Added more error handlers and improved stability.

Demo

Using Protocol Test Manager to run ADFamily test cases.

Thank You!Questions?