At Riot, we're "data-informed." Basically, that means we use research tools to gather loads of info about League of Legends, then use it to help us figure out how we'll make changes to League. Data is just one decision-making tool among others like experience, education, or good-old gamer intuition. Being data-informed, says Davin Pavlas from the research team, is about "incorporating data into the decision-making process, rather than having data be the decision-making process."
To get truly data-informed, we have to go far beyond the basics like winrate and pick/ban rate. Rioters on the Insights team—the peeps behind the Clairvoyance blog—rely heavily on three research tools in particular: surveys, analytics, and lab studies.
DO YOU HAVE A BRIEF MOMENT TO ANSWER 6,000 QUESTIONS ABOUT OUR SERVICE?
Surveys are useful for finding out what players want (skins? modes? someone to love?). Sometimes we'll use them to ask super obvious stuff, like "would you like more alternate game modes?" Surprising no one, players often answer that question with "yes pls." The reason we ask obvious questions like these is to understand how players rank potential changes against other things that they might also want.
We've asked players in North America and Korea to choose adjectives that apply to Yasuo.
As the data piles up, we're able to compare and contrast the idea on a scale that ranks it relative to others. Of course, people like new ways to play League. "But how much do they want it in comparison to other stuff?" asks Davin Pavlas. "How much do they want it compared to stuff we know they want really bad, but which we're not making right now?"
We also use them in more off-the-wall ways: for example, we might ask about a change to League we consider extremely controversial even if we'd never make that change. Sometimes we just need to calibrate our "bad idea" detector.
Surveys are most useful in aggregate. When we can look at tens of thousands of responses over a long period of time, we can make better-informed statements about how players value things. Surveys give us a chance to figure out exactly how much players love or hate our ideas. We'd rather make changes that everybody loves (obviously), but even things that the community overwhelmingly loves will still upset some folks.
READY YOUR SPREADSHEETS
Surveys are about what people feel—or, at least, what they say they feel—but analytics lets us gather loads and loads of data about what players are actually doing in-game. A lot of our analytics tools are embedded straight into the game. There's all the broad stuff you'd expect (How many games are people playing per day? What percentage of games include at least one AFK?) but we get pretty specific too: We track how many times per second players in every single region are using Annie's Q ability. Like a high-elo Santa Claus, we see you when you're B-ing, and we know when you've bought a Phage.
Yasuo has one of the steepest learning curves of any champion we've ever released. The above chart shows his average win rate by number of games played, compared to a couple of other beloved champs.
As Rudi Bonaparte, who focuses on analytics, puts it, "analysts convert billions of hours of gameplay into information Rioters can use to make League more awesome." Take, for example, Yasuo. When we released him alongside patch 3.15 back in December 2013, Yas limped onto Summoner's Rift with a 36 percent win rate. In earlier days, we might have over-corrected on Yasuo with significant buffs, but the data told us to only go light with the changes, and to trust several functional bugfixes to carry him towards balance.
Sure enough, the data soon revealed that Yasuo had a steep learning curve, but his win rate did actually climb past the 50% point once players put something like 35 games in with him. In the weeks after Yasuo's release, players began to figure him out, and by patch 4.2 we had to tune the Unforgiven back with minor nerfs.
If we want to understand balance at all, we've got to use analytics. Everyone looks at champ win rates, and while they're important, you have to go deeper to achieve stronger balancing. If nobody is bothering to use a certain skill, we'll revisit the champion to see whether they need a rebalance that makes all of their abilities useful.
PAY NO ATTENTION TO THE RIOTERS BEHIND THE MIRROR
Surveys tell us about how players think about League, and analytics tells us how players play League. Lab studies—where we bring players to Riot's office to observe them playing the game—are a chance for us to put everything we know into context.
The research lab is where players test unreleased champs or features. We also use it to brutally interrogate the champion balance team about changes we don't like.
Labs may bring up visions of lab coats and bubbling test tubes, but they're pretty simple: we bring a player in, they sign a VERY serious NDA, and then they get to play with a new champ or a work-in-progress feature while we watch from behind a two-way mirror in an adjacent room.
Labs let us catch things that are impossible to measure with numbers. One of the first things we realized when we started doing labs is that many new players (even core gamers) had no idea what to do with their hands the first time they play the game. They rest their hands on WASD, or they ignore the keyboard altogether.
"My wife plays with her fingers on ASDF," says Pavlas. "We didn't notice that for two years, and she can't change it now." Nobody says "hey, I don't know where to put my hands" in a survey. It's just not one of those obvious things, but it is easy to fix. We put a picture of where your hands are supposed to go in the tutorial. Problem solved.
The best use of labs is when we present players with new ideas and get immediate feedback on them. Players often react to new champions and features in ways that are surprising to us, so being in the room with them when they first get their hands on it puts us one step ahead.
BEING DATA-INFORMED MEANS THAT WE TRY TO LEARN AS MUCH AS POSSIBLE.
Ultimately, research at Riot is about collaborating with players. There's a lot of y'all hitting the Rift, and we need powerful ways to listen to you. We're passionate about data, and we we work hard to find that balance around data and gut reactions so we can be data-informed.
Our research efforts teach us all sorts of weird things about League of Legends and the people who play it. To keep up with some of the coolest stuff our research teams are learning, check out the insights blog.
