Skip to main content

Gameplay metrics: game design's best kept secret?

So Laralyn McWilliams just wrote:

And I agree so much with that that I had to write something about it.

When game developers talk about metrics, we typically mean one of three things:

1. F2P metrics, like retention, DAU, ARPPU, etc. This all came out of online marketing, and it's what people associate with "evil f2p".

2. Level design metrics, like how high are cover objects, how high are walls the player can't jump over, etc. This is needed to build level geometry that plays well.

3. Gameplay metrics, which is what Laralyn is talking about, and which seems like the best kept secret in game design or something.

Let's look at some history:

What if I told you Crash Bandicoot was made according to precise metrics about what happens when? There was an interview in Next Gen magazine back in the 90s that explained how they did this. It had a big influence on one of the teams (hi Stéphane) at the company I worked back then. If I remember and understood correctly, this was Mark Cerny's big contribution to the game, Naughty Dog used it in later games as well, and Michael John, who worked with Cerny, used the same method on many other successful games. I recently found my photocopy of the interview - I've not been able to find it online. I should really scan it.

What if I told you that to me the most interesting part of the 1999 post-mortem for Half Life was not the cabal process (remember that?), but this single paragraph:

Toward the middle of the project, once the major elements were in place and the game could be played most of the way through, it became mostly a matter of fine-tuning. To do this, we added basic instrumentation to the game, automatically recording the player's position, health, weapons, time, and any major activities such as saving the game, dying, being hurt, solving a puzzle, fighting a monster, and so on. We then took the results from a number of sessions and graphed them together to find any areas where there were problems. These included areas where the player spent too long without any encounters (boring), too long with too much health (too easy), too long with too little health (too hard), all of which gave us a good idea as to where they were likely to die and which positions would be best for adding goodies.

(While searching my email archives on this topic I just found a 2007 email I wrote to a mailing list that says:

Quantified analysis of gameplay has fascinated me ever since I read the Game Developer post-mortem of Half-Life 1.

And I'm still fascinated over 15 years later.)

What if I told you that Ensemble and Bungie used gameplay metrics since at least 2000 or so? And that Microsoft gave regular presentations on how to do usability testing with metrics at GDC in the early 00s?

What if I told you that Ubisoft was using playthrough analysis tools in 2004 for the multiplayer modes in Splinter Cell?

Naughty Dog, Valve, Ensemble, Bungie, Ubisoft… see a pattern?

This is all pretty much public knowledge. But how come I don't read about this in game design books? How come I don't regularly see Gamasutra articles or conference talks about this? It feels like game design's best kept secret sometimes.

Update: And of course pretty much every MMO has done this since forever! And have given talks about it!