MGS Testing

A High Level Overview of Testing in Microsoft Games Studio

Joe Djorgee – Test Lead

Today’s Topics

» Staffing Philosophy» Testing Roles» Basic Org Structure» Specialized Test Labs» On-Boarding Program» Why Quality Test Talent is Important» Working with Partners» Test's Role in the Cycle» Automation

Staffing Philosophy

» Involve the right Testing at the right time:» Emphasize early involvement of Test in product

development… “Test Upstream”» Keep a smaller team of FTE in place longer, ramp up

when RTM is more predictable, hold off on using large Testers resources until end of cycle

» Harness benefits of centralization by sharing technologies and methods, moving resources as needed, focusing training, and removing redundancies

» Minimize overtime and avoid “carrying the standing army” due to late project slips

» Encourage small onsite Testing for external development partners – they act as first line of Testing for BVT’s, milestone drops, and as GTO liaisons

Testing Roles

» Full Time Test Manager» Manages the teams for multiple projects

» Project Test Lead» Responsible for Test efforts for a specific project

» Software Dev Engineer (Shared)» Spend ~85% of time coding/scripting tools/automation for specific

title

» Software Dev Engineer in Test (SDET)» Spend ~25% of time coding/scripting, own specific Test areas and

report results from Test Passes

» Testers (STE/TA)» Creates and executes Test Cases, play through game, runs

specified Test Passes» Can be full time or temporary Testers» Core, Core Assist, Play Testers

» Specialized Labs» PC Compatibility Lab» Multiplayer/Network Lab

Basic Org StructureFull Time Test Manager

Specialized Labs

Project Test Lead

Area Lead

STE/SDET

STE/SDET

STE/SDET

Area Lead

STE/SDET

STE/SDET

STE/SDET

Area Lead

STE/SDET

STE/SDET

STE/SDET

Shared Dev Team Full Time Lead of Temp Testers

Core Temp Testers

Core Assist Testers

Play Testers

Specialized Labs

» PC Configuration (Config) Lab» Testing on the highest possible market share percentage

of computer hardware and OEM systems» Extensive, in depth product Testing under all supported

operating systems against a vast array of hardware driver versions

» Work with other internal teams (Beta, PSS) to track, reproduce and regress all real world bugs

» Extensive coverage on input devices as well (game controllers, carious keyboards, mice, etc.)

» Muliplayer/Networking Labs» Provides a clean lab environment to Test multiplayer

functionality» Scheduled Passes on various connections and hardware» Focuses on technical network configurations and

connection issues

Temporary Tester On-Boarding Program

» Formal on-boarding is provided to all new Testers in their first week on assignment

» The on-boarding program contains a lot of information» Helps our temporary Testers learn the tools and processes required

to effectively Test our titles» How do I do this?» What is this?» Why am I doing this?» What does success mean?

» Areas covered during this process include» An overview of the game development cycle» Bug writing» General Testing principles

» Assessment» Facilitator is present to assist new temporary Testers and assess

skill level» Evaluations/final exam are required at the end of each section

Why Quality Test Talent is Important

» Testing of titles is growing increasingly complex and challenging, and a high performing and proactive team is necessary for predictable RTM’s

» Test teams are doing more than just verifying reliability and functionality

» Higher skilled teams arguably reduce costs in other more expensive functions (i.e. dev)

Working with Partners

» Establish strong working relationship with Developer point of contact(s)» Daily/weekly status» Direct phone/video conferences

» Sync up on project issues by using a single bug database

» Test Pass summaries – Milestones, Performance, Certification

» Provide gameplay and Design feedback » Share Test tools» Acclimate Partner to MS processes

Test's Role in the Cycle

» Testing is an end to end process occurring throughout the product cycle

» Early Test involvement exposes issues early, when they are least costly… “Testing Upstream”

» Test assessment is critical to accurate forecasting, planning and production

» A portion of the Testing complements the efforts of other disciplines - Localization, Play balancing, Usability, Audio, Fun, etc.

Automation

» Automation helps to improve Test coverage and efficiency

» Microsoft has a long history of engineering Test solutions

» Test and Dev teams work to implement hooks to aid Test efforts

Automation

» Infrastructure Examples» Distributed automation frameworks» Build recognition, replication, and

distribution tools» Build creation and compile tools» Coding our own debug hooks» Test Case managers» Test Case generators and optimizers

Automation

» Administrative/Productivity Examples» Game event tracking and coverage

tools» Bug filing and reproduction tools» Automated bug frequency tracking

used to help determine fix priorities» Visual reporting techniques» Consistent Task Management system

for Testers» Resource usage reporting

Automation Examples

Automation Examples

Automation Examples

Questions?