Game Career Guide is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Get the latest Education e-news
 
  • Postmortem - Starcom: Nexus

    [07.21.20]
    - Kevin Lin

  • The Feedback Cycle

    Not every element in the game required as much time as the Planet Problem, but almost every major feature took multiple attempts to get right. So Starcom: Nexus didn't start its first round of beta testing with actual players until early August 2018, after the equivalent of a fulltime year of development.

    Beta testers were randomly selected from the game's mailing list which had been very gradually accumulating subscribers and were about as "in market" as I could expect.  If a significant portion of them didn't like the core gameplay, I had a serious problem.

    Initially, I started with small test rounds of a dozen or so players with the goal of identifying the big problems, gradually increasing as Early Access launch approached.

    Qualitative Feedback

    Qualitative feedback is a player's subjective impressions of the game. It's what they consciously report liking and not liking in the game. It could be given in person, email, responses in discussion forum, etc.

    For the closed beta tests I implemented an in-game system where players could submit feedback any time by pressing F8, which paused the game and brought up a dialogue where they could put in their thoughts along with a rating for how they were enjoying the game. This was an interface I copied from Subnautica. A reminder was subtly, but constantly visible in the lower corner of the screen throughout the closed beta and Early Access, as well as one of the possible tips that could appear during game loads.


    The in-game feedback dialogue

    There are a number of advantages of this system:

    • The ability to give feedback any time allows me to captures impressions that players may forget by the end of their session.
    • The anonymity of the system makes players less likely to feel that they have to soften their criticisms or they must say something good if they're saying something bad.
    • I can easily collate feedback based on the game's version from the database.

    The only significant disadvantage is that it doesn't prompt players to give their gestalts impression of the whole game as an experience. Some players might elect to give it at some point, but many did not. To help with this I encouraged players to submit their impressions via other channels like the Steam forums.

    The qualitative feedback was overall very positive and helpful: players professed to really like the game play and had suggestions on how to make it better.

    Analytics Data

    Player feedback was probably the most important source of data on what needed fixing, but to reinforce and validate that feedback I also collected anonymous event data to ask statistical questions like "how long does it take the median player to finish the game?" or "what percentage of players who start the first mission finish it?"

    It also helped assuage the notion that perhaps players were merely being polite: e.g., having been given access to the beta, they felt obligated to give some kind of positive feedback.

    But combined with analytics there was a strong case that some players were really enjoying the game. In the first limited round of beta tests, half of players stopped playing very early on (within two minutes) for reasons I wasn't immediately able to figure out.  But of those who made it past the first few minutes, almost all finished the available content. While it was conceivable that a player might launch a game and give some quick impressions to be polite, the fact that a good chunk of testers played to the "end" was an encouraging sign.

    A pleasant surprise was that the game was longer than I thought: I could play through the current content in 20 minutes, so I had guessed new players might take 30 to 45 minutes. Real players averaged 90 minutes. With a goal of at least 4 hours of gameplay for Early Access, I wasn't as far away as I feared.

    Observational Data

    Analytics data tells you what players are doing and qualitative feedback tells you what they are feeling. Combined they give you a lot of insight into what is and isn't working. But observation fills in gaps that you might be missing. For example players may wonder aloud what would happen if they did "x." Players who aren't sufficiently engaged may quit in boredom just before something interesting happens.

    As a specific example, I mentioned earlier that a non-trivial percentage of players stopped playing very quickly, within a few minutes. Was the game's performance bad? Did they just not like the genre? Neither the players nor the data were telling me.

    I speculated that they missed the instruction that they should head toward a specific objective, so I made the objective indicator brighter and flashing. This change cut the number of players who failed to make it to their first objective in half, from 13% to 7%.

    It wasn't until I observed a player in a live session that I realized some players thought they were supposed to follow the ship that had given them their orders, and had their eyes fixed on it, ignoring the glowing, flashing arrow in the corner of the screen. Worse, once the player was sufficiently far from the objective, it was removed from the radar.

    I added a relatively minor change to have the first officer gently remind the player, if they continued in the wrong direction. I also extended the radar range for mission objectives. These changes cut the early drop-out rate from 7% to 5%.


    A few players followed the ship that gave their orders instead of the mission beacon.

    These fixes to the first half hour of the game were huge: consider 1000 players starting the game. A 13% drop out at the beginning of the game is an immediate 130 players whose opinion will be mediocre at best. They will likely return it and/or leave a negative review. It's hard to connect minor improvements in the early experience with the overall response to the game, but I strongly suspect they have a more than trivial effect. As an aside, Steam reports the game's median playtime of 9 hours to be "above average" (although they do not say what the average actually is).

    Over the next several months I repeated a loop: release a build, email a batch of fresh testers, gather feedback and analytics, implement the changes that players either asked for or I inferred were needed, create new content, and repeat. By late October things felt good enough that I could launch in 2018 before the Winter Sale, so I announced a release date of December 12.

    Marketing

    I am probably not unique among indie devs in that marketing is not my favorite game dev activity. I'd much rather just spend my time working on the game than try to talk about it at all.

    But all my research has consistently pointed at one conclusion: the success of a game on Steam depends almost entirely on reaching its market before launch. Using Steam review counts as a proxy for sales, review score as a proxy for quality, and pre-launch followers as a proxy for market awareness, the rank correlation between quality and sales is 0.25 (weak) and the correlation between pre-launch awareness and sales is 0.86 (strong).


    The uneven traffic to the game's Steam page prior to launch.

    As mentioned previously, when I decided to commit to Starcom: Nexus as a serious endeavor, I also committed to start taking marketing seriously.

    I started a marketing journal. I made a list of games in the same "space" as mine, either released or upcoming, and read their reviews. I made a list of YouTube streamers and journalists who had covered those games. I tweeted often. I posted videos and images to relevant Reddit subs.

    One aspect of the game that was a problem from a marketing perspective was the lack of clearly identifiable "hooks." As I mentioned earlier, the genre doesn't even really have a name. When prospective players asked in the discussion forum what other game it was like, there was no consistent answer.


    Explosions are easier to show-off than "an abstract sense of mystery"

    The game's best feature was really hard to communicate in a short clip, GIF or marketing blurb: the sense of exploring a mysterious open universe. In a post-launch player survey in which players checked-off their favorite parts of the game, exploration was the most selected. When players talked about their favorite aspects of the game, it was usually a response to something that was hard to convey visually, like discovering that by aligning a pair of mass accelerators, they could be transported to a hidden star system. The event itself was not spectacular-it was the delight in discovering something unexpected.

    I tried to convey the exploration elements by showing planet surveys, alien dialogues and space artifacts, but my trailer ended up focusing mainly on the most eye-catching aspects of game play: enemies exploding in fireworks of particles and pretty space settings.

    In the aforementioned player survey, I also asked how people learned about the game. Here's what players said:

    • 79% saw it on Steam
    • 12% saw a YouTube or Twitch streamer play it
    • 6% played the original Flash game
    • 3% saw it somewhere else

    Because I didn't post the survey until after full launch, it probably underestimates the number of players who learned about it from the original Flash game, since I think they were more likely to have been on the mailing list and bought it in Early Access. But it does underscore the importance of the Steam algorithm working as a multiplier of your marketing efforts.

    Note that I say "multiplier": it has been consistently observed in data and anecdotes that 95% of games which launch with little to no public awareness vanish into obscurity, buried in the avalanche of other titles released that week. Games that achieve sufficient traction are shown more prominently by Steam's algorithm.

Comments

comments powered by Disqus