Working Well Together: How to Keep High-end Game Development Teams Productive
-
Upload
perforce -
Category
Technology
-
view
1.719 -
download
0
description
Transcript of Working Well Together: How to Keep High-end Game Development Teams Productive
![Page 1: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/1.jpg)
#
Frank CompagnerGuerrilla Games
Working Well Together
![Page 2: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/2.jpg)
#
• Background• Iteration• Teamwork• Stability• Releasing
Overview
Iterate
Launch
How we learned to stop worrying and love the development cycle
![Page 3: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/3.jpg)
#
• Killzone Shadow Fall – PS4 launch title• 200 local users, 150 remote• Head revision: 1.8 TB (1.6 M files)• Depot size: 8 TB• Peak change: 180 GB / week• 60 dedicated build servers• All in Perforce since 2004
Background
![Page 4: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/4.jpg)
#
P4 setup - nowServer
Build Farm (60)
Proxy Proxy Proxy Proxy
Users (200)
Proxy
Proxy
Proxy
Proxy
Cambridge
Guilford
San Diego
ShanghaiAmsterdam
![Page 5: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/5.jpg)
#
P4 setup - soonCommit
Server
Build Farm (60) Users (200)
Proxy
Proxy
Proxy
Cambridge
Guilford
San Diego
ShanghaiAmsterdam
Edge
Server
Edge
Server
Edge
Server
![Page 6: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/6.jpg)
#
P4 scale
Submits GB / week
![Page 7: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/7.jpg)
#
• Monitors all vital stats of server and proxies• Simple Python script, produces simple html• Works for Windows and Linux servers
• Proxies cleaned by another Python script
• Both available in the Perforce Workshop
P4 monitor
![Page 8: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/8.jpg)
#
P4 monitor
![Page 9: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/9.jpg)
#
Iterate, Iterate, Iterate
![Page 10: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/10.jpg)
#
Code
Compile
Link
Test
Creation means Iteration
Programmer
![Page 11: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/11.jpg)
#
Code
Compile
Link
Test
Creation means Iteration
Programmer
IDE and Tools
Distribute
Incremental linker
CodeHotloading
![Page 12: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/12.jpg)
#
Model
ProcessTest
Creation means Iteration
Artist
Hotloading
![Page 13: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/13.jpg)
#
Model
ProcessTest
Creation means Iteration
Artist
Tools
DistributeHotloading
![Page 14: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/14.jpg)
#
Programmer
Rigger
AnimatorDesigner
Tester
Creation means Iteration
Team
![Page 15: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/15.jpg)
#
Lots of Iteration
Programmer
Rigger
Animator
Animator
Designer
Tester
Programmer
Modeler
Programmer
VFX artist
Designer
Audio Engineer
Modeler
Designer
Animator
![Page 16: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/16.jpg)
#
Teamwork
![Page 17: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/17.jpg)
#
• Does not look like this
Games development
![Page 18: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/18.jpg)
#
• It’s more like this
Games development
![Page 19: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/19.jpg)
#
• OK, like this
Games development
![Page 20: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/20.jpg)
#
• Personal branch is fine (sandbox / streams / Git)
• Feature branches sometimes work• Give control over your environment• But create distance to others• Team work requires frequent branch switching
Can branches restore order?
![Page 21: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/21.jpg)
#
• For us, branch switching is always expensive:– Amount of data and change– Unmergeable files– Code-Data dependencies
• Branches add complexity, we already have that• Team effort, we don’t want distance• Almost everybody works on the trunk• So why isn’t it always broken?
TBD - light
![Page 22: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/22.jpg)
#
Stability
![Page 23: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/23.jpg)
#
Programmer
Rigger
AnimatorDesigner
Tester
Creation means Iteration
Team
![Page 24: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/24.jpg)
#
Programmer
Rigger
AnimatorDesigner
Tester
Creation means Iteration
Team
Submit
Submit
Submit
Submit
Submit Sync
Sync
Sync
Sync
Sync
![Page 25: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/25.jpg)
#
Programmer
Rigger
AnimatorDesigner
Tester
Creation needs Testing
Team
Submit
Submit
Submit
Submit
Submit Sync
Sync
Sync
Sync
SyncTest
Test
Test
Test
Test
![Page 26: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/26.jpg)
#
• Test pre submit (user)• Test post submit (builder)• Test in parallel• Test as fast as possible• Still accidents do happen
StabilityUser B
Sync
< 30 minutes
User A
Submit Sync
Build Game SubmitSync
Test Game
…
Build Tools SubmitSync
Test GameSync
Cook DataSync
Sync
Upload
Cook Data Upload
Test
![Page 27: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/27.jpg)
#
…Labels.xml
Build labels
Label 42: change 1200
write
Build machines
Label 41: change 1180
Safe-Sync Tool
read
Label 40: change 1155
Label 39: change 1150
• Each game build creates a label• Stored in Labels.xml file in depot• Builds add their result to the label• Builds can add artifacts to a label
• Nobody gets latest (not safe)• Everybody uses safe-sync tool
![Page 28: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/28.jpg)
#
Safe Sync ToolSelecttests
Labels thatpass tests
Labels thatfail tests
• Custom safe-sync tool shows labels and test results• Use it to sync to a known good label
– This includes artifacts from other builds
Sync to get label
![Page 29: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/29.jpg)
#
Releasing
![Page 30: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/30.jpg)
#
• We branch for every release (often every week)• Nobody works on the release branch• Everybody works on the trunk• Changes are merged up from main to release
Release Branches are great!
main
release
= change
= release
![Page 31: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/31.jpg)
#
Release Manager• Cherry picking rules!• Gives you a lot of control• But also lots of confusion• And mistakes are easy
• Good tools can fix this• Give flexibility• And control
(See whitepaper for details)
![Page 32: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/32.jpg)
#
Conclusion• Iterate as fast as you can• Don’t branch too much• Test everything• Sync should be safe
• Release all the time• Control what you release
Speed
Control
![Page 34: Working Well Together: How to Keep High-end Game Development Teams Productive](https://reader031.fdocuments.net/reader031/viewer/2022013003/5537f4a94a79590a7f8b4648/html5/thumbnails/34.jpg)
#
RESOURCESPerforce Workshop (P4 monitor, cache cleaner):
https://swarm.workshop.perforce.com/files/guest/frank_compagner
ReleaseManager:
http://www.perforce.com/resources/presentations/user-conference-talks/tools-for-smooth-release-cycle
http://www.perforce.com/sites/default/files/guerrilla-games-whitepaper.pdf