The Supermassive Beta

How many beta testers does it take to change a lightbulb?

The rather surprising answer is 89.

• Fifty to just carry on as if light doesn’t matter and continue killing swamp rats or whatever.
• Ten to raise a ticket complaining about the complete lack of lightbulb operation and DEMANDING that it be rectified THIS INSTANT.
• Two to proclaim that lightbulbs themselves are overrated, and that when they started playing MMOs all they had was candles. And they jolly well liked it.
• Five to argue that lightbulbs could actually do with a buff
• Six to counter-argue that it’s actually daylight that needs to be buffed, and that lightbulbs themselves are slightly overpowered.
• One to troll the entire discussion by stating that “lightblbs r for lozers lol”
• Fifteen to give up the mission in frustration, going on a textual tirade about the pointlessness of lightbulbs and asking who designs a stupid quest around lightbulbs anyway.

And one level designer who sighs inwardly as he moves the light switch to a more prominent location.

There’s been a few eyebrow-raising incidents lately in beta-circles. Firstly there was the announcement that Blizzard were going to give everyone signing up for the Annual Pass entry into the Mists of Pandaria beta. I’m not sure how many people that’s likely to be, but I reckon it’s easily in the tens of thousands and possibly in the hundreds of thousands.

Not to be outdone, there was also the recent announcement from BioWare that everyone who’d previously signed up for the Star Wars: The Old Republic Beta would get a chance to play at an upcoming weekend test. This invite list is likely to be in the order of millions of players.

There’s a question that immediately crops up.When do you reach critical mass – the point when there are so many testers raising tickets that your QA department implodes under the crushing weight of so much data, creating a testing black hole that threatens to engulf other nearby departments?

There’s also the comparison of testing approaches. The long test where the gamer can relax and bathe in the hot, fresh waters of unseen MMO content, delicately selecting the finest quests from the fruitbowl of offerings direct from the designers. Or the stress test, where you have a horde of rampaging gamers banging on your server doors like an invasion of Justin Bieber fans armed with pneumatic drills and JCBs.

I’m reminded that these games have grown to be truly massive these days, and as a result need massive amounts of testing before they’re released on a massively eager gaming population. It’s a far cry from when getting a beta invite was like receiving the One Ring via email. You’d spend all night playing “my precious beta” before shuffling into work the following morning looking grey and mumbling incoherent babblings about shooting dragons with railguns.

The interesting bit is that testing has moved on – bug reports are a thing of the past and data mining is in full swing. BioWare has been compiling heat maps based on player data – everything from where characters die to what mobs get avoided the most. Tickets and forum posts might tell you what a player thinks, but data collection will tell you what they do. And the more players you get in, the more accurate your heat map becomes.

It’s clear that beta testing is changing to include a great deal more science and statistics. Whether we’ll continue to be wanted for our thoughts and opinions is a little more unclear. Is there a future where beta tests are measured and analysed purely by the lines of generated datapoints?

Keep testing. For Science.