“Software Engineer in Test” in Game De-velopment

How can TERA be verified via test automation?

Sung Min Kim(sungmkim@bluehole.net)

Lead Software Engineer in Test / Bluehole Studio

• I haven’t converted all contents in the original document into this English version yet, so many contents are not shown in this document, compared to the original version due to my laziness

• I don’t have a good English skill, so please understand if wrong sentences or grammars in this document. Thanks in advance for your understanding

• This document is an English version of the original presentation document used at KGC 2014 written in Ko-rean (http://www.slideshare.net/SungminKim12/kgc-2014-software-enginner-in-test-in )

• Hope that this document can help the people who are interested in the test automation development espe-cially for games for developing your test automation system

• Feel free to let me know if you have any questions on the document (Email: smkyhh@gmail.com)

Session 1 (15:50 ~ 16:50)

1. About me2. Software Engineer in Test?

• Test Automation• SET at Bluehole Studio

3. TERA Test Framework• TeraPuppet• TeraTest• Selenium Integration

4. Q & A

Agenda

Session 2 (17:00 ~ 18:00)

1. Test Scenarios2. CI & Build System3. Performance Measurement4. Summary 5. Q & A

Agenda

• LG Electronics : 3 yearsProject: 4 mobile devices

About me

• A programmer who loves new developments ( Email: smkyhh@gmail.com )

Me

• Realtime Worlds, Client Programmer : 1 yearProject: My Worlds (Data Generation)

• Microsoft, SDET and SDE 2 : 5 yearsProject: Office 2010, Office 2013 (SharePoint Designer)Award: Gold Star Stock Awards (2011) with SPD Test Automation

• Bluehole Studio, Lead SET (2013.08 ~ Now) : 1.5 yearsProject: TERA (Test Automation )

Software Engineer in Test

Software Engineer in Test

• General jobs in many global game companies, but likely un-

common in Korea yet– Global game companies like Blizzard, Riot Games, and EA are now hiring

many SETs or similar names like that

– SET @ Blizzard, EA, Google, …

– SET @ Bluehole Studio

– SDET @ Epic Games, Microsoft Game Studios,…

• Programmer on Test Area (TEST ≠ QA)– Test Automation

– Verifying features via programming

– Finding code defects as early as possible at the development stage

– Detailed performance and security measurement

– Issue management on CI system

– Share various info with Devs and QA as a evangelist on test automation area

• A specialist working only on test area as a programmer like

server programmer and client programmer

Software Engineer in Test ?

Software Engineer in Test

• Test automation is a more efficient way if some features can

be verified via computer instead of people– Verification on regression areas

– Measure and verify exact values inside game logics like packet data be-

tween client and server

– Can find performance, security issues, code defects in detail

• Can have totally different directions on test automation system

according to development status, organization, project speci-

fication, and so on* Bluehole Studio

– Only one testable project now: TERA

– Organization: about 150 employees based on TERA

– Game Genre: MMORPG

– The number of countries where TERA is working on live mode : 6 (Korea,

China, Taiwan, Japan, Europe, and US)

– The number of SET at Bluehole Studio: only 1 (me.)

Software Engineer in Test ?

For software?

Software Engineer in Test

• Not possible to verify all features in Games via Test Auto-mation only

– Fun– UX– Complicated multi-player behaviors

• Co-work between developers and QAs is very important

• Need to always consider 1) Development efficiency, 2) Cov-erage improvement, 3) Development Process improvement

– Reduce development costs– Test Efficiency – Sustaining– Better development process

Software Engineer in Test ?

Fun

UX

SET at Bluehole Studio

Software Engineer in Test ?

• s

Efficient Test Automation

• Improve Quality• Easy to understand the system under test• Reduce the risk• Easy to run, and maintain• Easy to create scenarios• Minimum maintenance cost

Current point (TERA)

SET

QA and developers

Not to be like this

My SET History (2013.08 ~)

SET at Bluehole Studio

• What I have done so far alone as a SET

– Create SET and SET’s role at Bluehole studio

– Developing Tera Puppet 1.0 for TERA

– Developing TeraTest for TERA

– Selenium integration system development for Web automation

– Adopt JIRA into Bluehole Studio for better BTS system

– Tool development for test automation for QA

– CI system development for running test automation scenarios all the time

– Running test automation scenarios for various countries using the CI system

– Development Process improvement

– Game Client performance measure

:

SET at BHS

Software Engineer in Test ?

Designing and developing all test automation framework and tools for all games at Bluehole Studio

Directing test automation strategy with QA and Devs for better test automation works

Implement test automation libraries and tools for games

Develop and support unit tests for DEVs

Measure detailed performance based on source codes

Find code defects via security / performance test and related tool developments

Measure code coverage based on server / client source codes to find out exact performance issues in detail

12

Gathering requirements and Planning test automation framework

Designing and developing test automation framework

Executing test cases and supporting QA and developers

Maintenance

TERA vs Office 2010

Game (TERA) Application (MS Office 2010)

When started Live Service Before finalizing specs

Logic flow Asynchronous Synchronous

Verification Source Code + Data Source Code + Data

Target Fun + functionalities Functionalities only

Interaction between users

Need to verify with many players No

SET scale 1.6 years * 1 person More than 10 years * ? people

The number of re-lease countries

6 countries 30 countries

OS Windows 7 or Windows XP All kinds of OS including Mac

Purpose For fun For business

When accident happens

Path + reward Lawsuit (possibly)

Update iteration Fast Slow

SET at Bluehole Studio

TERA Test Framework

In-house vs External Solution

• Of course in-house development is much better than external solutions

• TERA Test framework is a in-house framework working only for TERA

• So, we are developing a test framework on actual product codes after fully under-standing source codes on both client side and server side

• when using external solution

• TERA 의 서버 & 클라이언트 코드를 완전히 이해하고 ,

실제 Product 코드 위에 테스트 자동화 시스템을 구축하자– 실제 Product 코드에서 개발

– 누구나 시나리오를 만들 수 있는 , 시나리오 제작 & 재생 방식의 통합된 프레임웍

– Unit Test 및 다양한 테스트 라이브러리 개발을 통한 추후 확장 용이

– .NET 을 이용하여 개발 생산성을 높이자

– 필요한 경우 공개 라이브러리를 활용하자 (e.g. Selenium)

TERA Test Framework

IN-HOUSE

EXTERNAL SOLUTION

Framework Overview

TERA Test Framework

TeraPuppet(Record & Playback for TERA)

- Scenario creation and run via recording & playback

- Everyone can easily create and review scenarios

- Easy to maintain scenarios- But, need to update and stabi-

lize grammars for new func-tionalities

Selenium Integration(Record & Playback for Web)

- Developed libraries and provided various templates

- Integrated with TeraPuppet- Need to implement more libraries- Useful for creating scenarios based

on Web

TeraTest(C# Library & Script)

- Working based on c# test library- Can use other fluent and strong

C# library- Useful for very complicated scenar-

ios using huge data sheets- Useful for the scenarios difficult to

be created by Tera Puppet- But, need to develop separated

libraries for specific scenarios

Puppeteer (BlueholeTestWeb) - Scenarios management and execution via Web

CI with the test automation (TeraTestAgent)- In-house CI system only for Tera

TERA

TeraPuppet

TERA Test Framework

17

• Recording & playback, using 1) sever-client packet, 2) UI event, 3) key event, 4) mouse event, and 5) custom actions

• XML based scenario

• COM(Component Object Model) server and client to communicate between TeraPuppet and Tera-Client

• May help developers debug issues when check-ing events and packets between server and client

• Need to know various grammars to use all func-tionalities

TeraPuppet

TERA Test Framework

18

DEMO

TeraTest

TERA Test Framework

19

• Control and verify TeraClient with C# scripts and li-braries

• Script based scenarios

• Need to implement Test APIs in the managed world (C#) from the native world (C++)

• Useful when verifying w/ various datasheets and complicated logics

TeraTest

TERA Test Framework

20

• WPF and MVVM to show UIs and C++/CLI to communicate between TeraClient and TeraTest

• May help developers create Unit Tests

• Various scenarios can be implemented though it’s not possible by TeraPuppet

• Under implementation with various libraries– BOT– Game UI verification based on Direct X

TeraTest

TERA Test Framework

21

DEMO

AutoIt vs Selenium

• Bluehole Studio decided to use Selenium because it’s more flexible with C#

TERA Test Framework

AutoIt Selenium

Difficulty - -

Expandability Middle High

Community Middle High

Good• Easy to use• Own libraries• Easy to modify

• Firefox IDE• Expandability!! using C#• Can create scenarios in various languages

(VBA, C#, Autoit..)

Bad• Not easy to integrate with C++ /

C#• Need to learn a language like C#

Selenium Integration

TERA Test Framework

• Selenium is a test automation library for Web

• Because Web is based on HTML and DOM, it’s easy to verify scenarios by automation, other than JavaScript area

• SET developed test automation libraries integrated with Tera-Puppet, and now it’s used on TERA web sites for administra-tors

• QA creates scenarios with the Selenium integration framework

• Because Selenium can run on Tera Puppet, we can create vari-ous complicated scenarios using both selenium and Tera Pup-pet

Selenium Integration

TERA Test Framework

DEMO

Q & A

Test Scenarios

Scenario Status for TERA Test Automation

Test Scenarios

• More than 400 scenarios are now working in 5 countries– Countries: Korea, China, Japan, Europe, US

– Purpose: Build Verification Test, Issue verification test, Release veri-

fication test, content verification test

– Target: Contents at live stage

– Tools: TeraPuppet, TeraTest, Selenium Integration

• QA is generating and sustaining scenarios continuously

• Roles– QA: Scenario generations

– SET: Scenario tool development, and provide sample scenarios and

libraires

• The scenarios can be executed by both manual and CI system TeraTestAgent (CI tool for Test Automation) regularly

FYI, How SET currently works with QA and Devs ?

Test Scenarios

• SET– Design and develop test frameworks, tools, and libraries– Provide scenario samples and templates using the developed libraries– Perform research to acquire new knowledge and provide solutions– Review scenarios created by QA– Share knowledge if needed through studies like C# Study

• QA– Create scenarios, and investigate scenario failures– Log and handle issues happening with the test automation scenarios– Review and verify the test frameworks, and provide valuable feedbacks

• DEV– Use scenario results before DEVs submit codes

– Using Puppet and TeraTest as their debugger tools

Scenario Example (TeraPuppet)

Test Scenarios

Scenario Example (TeraTest)

Test Scenarios

Scenario Example (Selenium)

Test Scenarios

Code Coverage

Coverage Measurement

• A way to know how many features and codes are covered exactly by test scenarios

• Need to measure exact coverages to figure out if the production can be released or not

Code Coverage

Performance Measurement

At Microsoft, measuring and improving code coverage is a key to decide if a product can be

released to market or not

Code Coverage in Game development

Performance Measurement

• Limitation only with Code Coverage– Code + Contents Data

– Client + Server + Contents Data

• Packet Coverage

– How many packets are verified between server and client commu-

nication

– Measuring not only Code Coverage but also packet coverage for

getting exact coverage

• Data Coverage

– Need to find out a way to measure how many contents data are

covered by scenarios Packet Coverage

Are you interested in Software Engineer in Test ?– sungmkim@bluehole.net / smkyhh@gmail.com