Hello,
Does anyone have any examples of how design considerations and decisions can be driven by in-game analytics?
One example I know is using in-game analytics to calculate the percentage of advertisement clicks to help decide whether a certain advertisement banner should be moved or removed altogether.
Thank you
Another thing you can do, at least during the testing/tweaking phase, is gather analytics on how much time players spend at each “area” of your game. Areas could be levels, or they could be smaller regions within a level.
This will tell you where players are getting stuck, or else where they enjoy hanging out (hopefully you’ll be able to tell the difference from other knowledge about the game). If they’re getting stuck, you may need to make that challenge easier or more obvious in some way. Conversely, if there are areas of the level players pretty much never visit, you can either cut them, make them more obvious, or make them more attractive (with loot, useful shortcut, or whatever).
One of the best thing to do is to literally watch people play your game (without any input from your side). Watch a live-stream for example. You’d be surprised.
One of my professors explained why this is so crucial when it comes to UI design. It’s the elemental problem of the developer and the user looking at the UI from two different perspectives.
The developer sees the features and constructs the UI. The user sees the UI and constructs the features. It might sound weird, but it’s true. When you start up an application for the first time, you create yourself a mental model of “How it’s suppose to work” or how you expect it to work.
Developers tend to react like this:
“OH MY GOD! THE BUTTON IS RIGHT THERE! RIGHT THERE! WHY DOESN’T HE CLICK IT? MY GOOOOOOOD? IS HE STUPID?” (Which explains why it’s required(!) for you to be physically disconnected from the testers.)
The same principle applies to games. As the developer, you know all the solutions and tricks. You can’t look at the game from the perspective of someone new. You simply can’t.
And @JoeStrout 's tip, to collect data like this - even though it’s useful - won’t change that.
Virtually all the success and growth of web/social gaming was built on analytics. Our company’s analytics tools are what led us to where we are today. One of the benefits of web/connected games is the ability do deep analytics. Statistics and sampling are certainly useful, but analytics give you an ongoing, live, 100% sample rate for whatever you are looking at. It is so effective that we have been using it on our console games now as well.
For example, we use it for things like Joe pointed out, player progress. Players actions are recorded when the play a level which is used for sharing battles. That information is also used by the designers to see how levels perform, and if there are problems, where. We can heat map all the players actions and see and discover where people are getting stuck, potential holes, and overall balance the gameplay.
A common application of deep analytics, is AB testing. Where you can test a feature or modification or even tuning by rolling it out to a subset of your player base. This is very common, and very useful. Say you want to introduce a new mechanic, you can creating a few silos of players, (a group with the feature, one with the feature with different variables, and a control group). Run it for a few weeks (or however long), and see what impact it has. This very useful for an established game as often changes, no matter how good, can turn off regular players.
This type of testing really shines when it comes to challenging conventional wisdom. Often we make choices to do things a certain way because it has worked in the past (when to introduce players to features, NUFs, certain flows). If a designer wants to explore a new way, AB testing can allow them to test it against real players without having to compromise the whole game. Occasionally this lead us down new paths, and sometimes it just reinforced previous methods. Sometimes common design choices work differently in different contexts, being able to test multiple choices in the same context is very valuable.
Certainly there are downsides. It can be easy for some designers to rely on it to heavily. To the point where they aren’t actually designing, they are just tossing ideas into a live game and seeing what sticks. Several years back, we had one designer who insisted on AB testing everything. Right down to the color choices in the UI. It got the point where most of the tests we ran were statistically pointless. Also we were running so many tests at once, that they became meaningless. At one point the game had nearly 60 unique shards (versions of the game). Players were all playing different games, the results were nearly useless, and the only thing we were actually doing is annoying our players.
Analytics also provide valuable insights into patterns and trends. We can offer incentives like sales or special features at times of the year that usage is low. Things like that. Overall, they can be very useful and informative.
Few statements are as true as this. Well said.
I do it like this, “I’d like to watch you play my game. After I hand you the device, I’m not going to say anything or answer any questions. I just want to watch. To find areas that need improving.” And then, I do what I say. It’s hard to shrug when they ask you questions, and yet, it will make your game better.
Gigi
Thought I’d add some links to a really good article on heatmaps and an inexpensive Asset Store product (Heat Emitter).
Heatmaps are awesome level design tools. In adventure games, they can identify spots where players are sitting around banging their heads on the wall, stuck on an unclear puzzle. In RPGs, they can identify spots where players are monotonously farming instead of enjoying the story you want to tell. In shooters, everything @JoeStrout said. 
Hey, that’s neat. I notice this asset doesn’t have any reviews — if you have it, I suggest writing them one. It looks like it deserves more attention than it’s gotten so far.
I don’t have it, but I thought it was worth mentioning. It might be fighting an uphill battle for attention against GameAnalytics, which is free but might be overkill if you’re only worried about heatmaps.
Here is another heatmapping tool, which happens to be free:
(Note I haven’t tried any of them)