Get the latest Education e-news
  • Beta Testing: Methods And Techniques

    - Joost van Dongen
  • Giving players access to the beta of a new game or new content before it's released is a great way to get feedback and find bugs, allowing you to add that extra bit of polish, balance and quality before the official full release. There are many different ways to give players access to a beta. Which to choose? In this article I'd like to give a comprehensive list of options in today's market and discuss the differences.

    Traditionally bugs in games are found by QA testing companies. However, hiring a QA company to exhaustively test a complex game is very expensive. Many smaller companies don't have the budget to hire QA at all, or can only get a limited amount of QA and can't let QA cover every aspect of the game, let alone doing so repeatedly for every update. However, even if you do have the budget for large amounts of QA testing, that won't give good feedback on whether a new feature is actually fun or balanced. That requires real players, experiencing the content in the wild. So whether you can afford paid QA or not, a beta might still be a good idea.

    There are many aspects to doing a beta. Should everyone get access, or only a limited number of players? Is the beta for a new game that hasn't released yet, or for new content for an existing game? Is the beta also intended to gather additional development funds, or only for testing purposes?

    Another interesting topic is what to actually put in a beta. Should it be all the content, or only a portion so as not to spoil the main release too much? (There's an interesting bit about that in this talk about Diablo 3.) How early should we do a beta? Although these are important questions, to limit the scope of this post I'm going to ignore the content of the beta: today I'm focusing exclusively on how the beta is delivered to customers.

    Over the years at Ronimo we've done a bunch of different approaches to betas. With Awesomenauts and the recently released Swords & Soldiers 2 Shawarmageddon we tried betas before release and for new content, through DLC and through beta branches, limited paid betas and open betas, and more. That means a large portion of this post is based on our own experiences, but since I want this list to be as comprehensive as possible I'll also discuss approaches that we haven't tried ourselves.

    Since consoles offer very few possibilities for betas and since Steam is the biggest and most complete platform on PC, this post mostly lists options in Steam. Some of these will probably be possible in similar ways on competing platforms like GoG or If I missed anything that's fundamentally different on other platforms or if I missed some approaches altogether, then please let me know below in the comments so that I can add them.

    Steam beta branches

    For updates to an already released game

    This is the most common way to do a beta on Steam. When uploading a build you can select in which branch it should go live. This makes it possible to release a build under a 'beta' branch only. Users can then simply right click the game in Steam and select the branch they want, after which Steam will download it and replace the main game with the version from the branch.

    If you want to limit access to the beta you can set a password for the branch. This works fine, but it's a single password for all users, so if you share this password with players there's a good chance that some might share it further with others. For Awesomenauts we got lucky with our community: no players posted the passwords for closed betas publicly online. Undoubtedly some players did share a password with a few friends privately, but that never caused any problems.

    Branches are also great for internal testing purposes. When we want to test a build internally or want to provide a build to QA we also use Steam branches and simply share the password only internally or with the QA company.


comments powered by Disqus