If I had to pick a favourite panel from Blizzcon, Blizzard Engineering would be near the top. Hearing how the various programming teams overcome technical challenges to produce games with incredible polish is always interesting, particularly as their passion always comes through in their enthusiasm on-stage.
While there was only the one session this year, the Blizzard Engineering panel at Blizzcon 2018 still had a few interesting tidbits. Compared to last year’s deep tech examination, this year saw more of a focus on engineering culture, quality approach, and delivery methodology. There was a clear demonstration that it’s not just about problem-solving, but collaborating as part of a team that’s becoming increasingly important.
However, if I had to ask for one thing for 2019, it would be to repeat a request for more discussion around architecture. System componentisation came up a few times during the panel, but yet there was never an explanation of why it’s important or what value it provides. Likewise, many of the questions were from hopefuls looking for advice on getting a job at Blizzard, indicating a strong need for a separate careers fair at the event.
Before I get into a summary of the panel, I’d like to give a little attribution to those on-stage:
- Somer Esat (Panel Chair) – Senior Engineering Manager, Overwatch
- Rachelle Davis – Senior Software Engineer, New project. Previously an original member of the Hearthstone team, and has been with Blizzard for 7.5 years
- Bruce Wilkie – Principal Software Engineer, Overwatch. Specialist in game engine and graphics rendering.
- Diane Cochran – Lead Engineer, New project. Previously lead on Blizzard’s approach to Continuous Integration, lead on battle.net desktop app. Worked on authentication, public API, and has been with Blizzard for 8 years.
- Collin Murray – Lead Software Engineer on WoW. Has written code for Diablo, Starcraft, Warcraft III, and every version of WoW. Been with Blizzard for 23 years.
- Sarah Doebler – Technical Lead Battle.net. Started as a GM, has worked on Blizzard.com, the Armoury, and Content platform. Now leads tools team on battle.net, and has been with Blizzard for 14 years.
Start Small & Break Stuff
The panel shared that one of the hardest parts about making a game is ‘finding the fun’ – a core gameplay loop that’s both engaging and entertaining to repeat. At this early discovery stage, a small prototyping team will work in whatever tool works best to help shape that core gameplay. For example, Hearthstone (known internally as Pegasus during those early stages) was prototyped in Flash before eventually switching to Unity. It allowed the team to focus on how the game feels and find the right notes, without having to worry about how it would scale.
Once that core feel is shaped, the conversation gradually shifts towards production. How to make it scale and support potentially millions of players. But the key thing is experimenting, testing, making mistakes and iterating rapidly to hone in on the fun.
From a quality perspective, Blizzard’s approach has evolved over the years in conjunction with shifts in development. Warcraft may have followed a sequential waterfall approach, but most other teams have switched over to agile/scrum. Sometimes (such as with Web and Mobile teams), this is also used with continuous integration and continuous delivery. Either way, the approach embeds QA engineers within small development teams so that requirements or user stories can be tested early and regularly. Some teams even require QA signoff before a user story can be closed, helping to cut down on final test time before deployment.
Like many software development houses, Blizzard uses unit testing to help identify where software breaks. It then uses integration testing to throw simulated game input, randomised input, and so on to make sure things don’t fall apart. Much of this can be automated, but manual QA testing is still used to provide feedback on the experience, look and feel.
Even when a game is out in the wild, Blizzard is still trying to discover crash-causing bugs. In the early days of WoW back in 2004, player-submitted error crash reports were all processed by one person using a Perl script. Over time, that changed into a system called Exceptions, which collects all crash reports submitted by players. It’s able to collect those crash and dump files, categorise and prioritise, and even let developers see the error as if it happened on their own desktop. Patterns emerge, helping the team to focus in on fixing the biggest problems for the most number of users.
Challenges and Conundrums
Alongside their considerable experience, the panelists also shared some specific examples based on their own projects. One example was how the new Overwatch hero, Ashe, was particularly technically challenging. Her Ultimate ability can summon the Omnic robot Bob, who can be buffed and healed separately. Yet both entities (Ashe and Bob) need to fit in the same memory budget as one regular hero in order to maintain performance across a wide range of PCs and consoles.
The solution was to have three quality variants: a high-quality version that’s used during character selection and a lower quality for intros where more stuff is going on. An even lower spec version with more compressed textures is used during gameplay to keep the action smooth. Due to its competitive nature, Overwatch faces a significant technical benchmark – 300fps, multicore support, no lag spikes – which is all very challenging.
WoW has its own set of challenges. Engineering staff grew from 12 in 2004 to almost 75 today, in order to meet the demands of more frequent and larger updates. Tools and processes have been refined, and testing has become more automated. Even so, the game engine carries over 14 years of legacy code in places, resulting in some eccentric behaviour that’s become part of the Warcraft experience. And, as there are no sacred cows, the changing demands of systems can result in systems getting pulled out for a rewrite.
Sometimes the player community uses a service or feature in a completely unexpected way, which also prompts a rewrite or re-engineering. One example is the suite of community APIs, which provide data to a huge range of fansites and community projects. It started with the Warcraft Armory and associated API, where the Web team realised that most of the XML traffic was from fansites and community developers. When the old Armoury was retired, the team wanted to keep the service running while migrating to REST and exposing the same dataset.
From there, the Web team reached out to various game teams and established the first agreements on sharing game data. It’s a service that’s seeing continual development, with the team working on making it even easier for internal projects to author new APIs and bring them to the community of developers.
Tools and Tuition
It’s true that Blizzard builds many internal tools to help with development, but they also use a standard foundation that almost anyone can start. WoW and Overwatch both use Visual Studio 2017, although the latter also uses some SDKs to support console versions. Some teams are starting to use VSCode for C++, and harnessing vi bindings are highly recommended.
Beyond that, the panelists were unanimous in how budding engineers could develop their skills. One is by starting up (and completing) hobbyist projects, publishing clean and reusable code on Github, and using tools and resources to fulfill ideas and solve problems in a self-sufficient manner. But just as important is demonstrating the ability to collaborate with other engineers and the wider team to deliver solutions.
Even for those who don’t have a typical computer science background, there’s still a path into game design. Small, achievable goals help to demonstrate visible progress, whether it’s learning from scratch or adapting to a new language or architecture. Wider principles such as data structures, object-oriented design, and algorithm structures are all transferable.