Luck, Skill, and Magic

12 Jun 2018

You don't have to have played for very long to understand, viscerally, how much luck can matter in a game of Magic. You might draw too many lands, draw too few lands, or have your opponent topdeck the one card they need to snatch victory away from you.


Or you might be Patrick Chapin resolving an Ignite Memories:


And for context, this match was worth at minimum $7,000 dollars. Thousands of dollars literally rested on the throw of a die. The contradiction is, although the result of this game came down to pure luck, the participants here are not just skilled Magic players, they are perhaps some of the most skilled players to ever play the game. Gabriel Nassif won player of the year in the 2003-2004 season, and Patrick Chapin is a Hall of Famer. Yet when these two great players faced each other the outcome of this game came down to dumb luck.


This contradiction can be a little distressing to your average player, and it raises the question: Is Magic a game of luck or skill?


The question is a deep one, and the answer is complicated. Undeniably there are elements of both luck and skill in the game of Magic. I can't win if my deck doesn't provide me the lands I need to cast my spells, but mistakes have cost Magic players countless games. Perhaps the real question is: What's the relationship between luck and skill in the game of Magic? How can we answer that? Can we even hope to quantify it?


I believe we can, and to do so we have to understand what "luck" means.




Variance is the term used to describe the luck inherent in a game. In an interview, Aaron Forsythe, Senior Design Director for Magic: The Gathering, talks not only about the variance inherent in Magic, but why it's an important aspect to the game:


What's the balance between skill and luck in Magic: The Gathering?

They're both there, like there are in a lot of great games. In sports like football, one team might be better than the other but anything can happen. It's the same with Magic: The Gathering. In a game like chess where there's effectively no luck, the better player will win every time. But in Magic, stuff can go right for the weaker player, they can get the perfect combination of cards, they can draw the right card they need at the right moment. That suspense and drama makes it really interesting even when one player is a lot better than the other. It sometimes hurts - if you know you're a better player and stuff went their way and they beat you, it can make you a little upset. But skill way, way outperforms luck in the game, there's just a variance there to keep it interesting no matter what.


Variance makes for dramatic moments, unforgettable topdecks and iconic gifs:

As Aaron Forsythe eluded to in his quote (and is witnessing first hand above) variance limits how often a better player can beat a worse player. A game that is pure luck - say a rousing game of Candy Land or Snakes and Ladders - is one in which the winner is decided purely by the luck of the draw or the throw of a die. The "better" player in both of these cases is never going to have more (or less) than 50% odds to win a game.


Contrast that with a hypothetical game of "who's the tallest?" in which participants win if they are taller than their opponent. Here there is zero variance and zero luck but the "better" player is instead 100% guaranteed to win the game. These two sets of "games" land at the extremes of variance - and all real games, Magic included, live somewhere in the middle.


To answer the question "How much does success in Magic depend on luck versus skill?" we can use this as our metric: how often, on average, does the better player win. And to understand what that raw number implies we need to see how the variance of Magic compare with other competitive endeavors. How would we even begin to do this? Let me introduce my good friend Arpad.


Elo Ratings


Arpad Elo was a chess player and physics professor who wanted to devise a rating system for chess players. The goal was to estimate, before the game was played, which player was favored and how heavily favored they were.


Elo's system defines the average player as having an Elo rating of 1600, and it assumes that this average player would be favored to beat a player rated 1500 64% of the time and an even weaker player rated 1400 74% of the time. The predictions are independent of the absolute value of the ranking and only care about the relative difference in ratings - an 1800 rated player is expected to beat a 1600 player just as often as a 1600 player is expected to beat a 1400 player. You can read about the full math behind Elo's system here.


Players ratings are calculated based on wins and losses. Player's ratings go up when they win, with likely victories against comparatively weaker opponents resulting in small bumps in ratings, and unlikely wins against stronger players resulting in bigger rating bumps. Player's ratings go down when defeated in a similar way - expected losses against strong foes only lower ratings a little, unexpected losses against weak foes lower ratings a lot.


For the purpose of understanding the level of variance in Magic, it is useful to think about how games with difference levels of variance work when you starting using the Elo rating systems to rank the players involved. In Chutes and Ladders, players' ratings would stay very close together, the randomly determined wins and losses of the players cancelling out. In "Who's the tallest?" the players' ratings would begin to diverge, the ratings growing ever farther apart as the tallest player keeps winning, and the shortest player keeps losing, no matter how many times the game is played.


These two examples illustrate how the spread of player's Elo ratings reflects the amount of variance in the game - the bigger the spread, the more the results are from "skill" and the less from "variance". In the game of chess, a game with little to no variance, the best players in the world have ratings over 2800. They are expected to lose to an average player less than 1 game in 1000.


This means that for different games if we measured the spread of Elo ratings (using, say, the standard deviation), we could measure how much luck, on average, determines the outcome of a match. Luckily Elo ratings have been applied to many different sports and games - as well as Magic itself.


Before we get comparing, let's discuss one wrinkle. Because Elo ratings are relative, the spread of ratings only tells you how much variance there is in the game among the group of rated players. To illustrate let's imagine taking the six best chess players (or Street Fighter players), giving them a fresh Elo rating of 1600 and dropping them into an arena where they only play against each other. Despite the enormous skill involved in each game, and the low variance of the games overall, Elo ratings would only stray slightly away from average, reflecting the fact that among this group of players the outcome of a match is mostly luck. Each player can win little more than 50% against his opponents, even though if you or I were unlucky enough to be dropped into the same arena we would lose 999 of every 1000 matches we played.


That said, let's look at how Magic compares to other competitive endeavors, for instance pro sports.


Luck in Sports


Fivethirtyeight is a website that specializes in using statistics to look at politics and (more importantly for us) sports. They have used Elo ratings to rank the teams in the NFL, NBA, MLB and the World Cup. Using these ratings we can calculate two quantities - the standard deviation of Elo ratings and a related quantity - how favored is the "better" team in an average match up. We calculate the latter by calculating the average Elo difference between two teams, selected at random from a normal distribution with the calculated standard deviation. The less variance and the less luck involved in each game, the more favored the "better" team is on average.


2014 World Cup

Standard deviation: 130

Average win % for better team: 69.93%


2017 NFL season

Standard deviation: 120

Average win % for better team: 68.55%


2017 NBA season

Standard deviation: 107

Average win % for better team: 66.69%


2017 MLB season

Standard deviation: 41

Average win % for better team: 56.69%


Soccer matches in the world cup came down to luck the least, the better team winning almost 70% of the time. However in Major League Baseball, the better team only won about 57% of the time, and so luck was a more significant contributor to individual wins and losses. So now the question: how does Magic compare?


Luck in Magic


Using the wonderful Mtg Elo project I took two random samples from competitors in Pro Tour Ixalan to see how the spread of Elo ratings for competitors compared (and for those mathematically minded, yes I corrected for the different logistical curves used in these Elo ratings). The results:


Pro Tour Ixalan (Random Sample 1)

Standard deviation: 48

Average win % for better player: 57.77%


Pro Tour Ixalan (Random Sample 2)

Standard deviation: 42

Average win% for better player: 56.81%


What we see from these observations is that an average match between two Pro Tour competitors is slightly less dependent on luck than your average MLB game, but more dependent on luck than a typical NBA, NFL, or World Cup game.


So there you have it, Magic matches are more dependent on luck than football, less than baseball. But wait! Let's remember our Gedankenexperiment where we looked at Elo ratings for only the best six chess players in the world. Although Chess has almost no variance, the implied variance by comparing the Elo ratings of only a roomful of the top players would give the sense that Chess games mostly came down to luck.


Similarly, by looking at only the Pro Tour we are seeing highly skilled players playing incredibly tightly contested games. This, ironically, amplifies the role of luck, since matches at this level are played with little margin for error and every draw step has the potential for swinging a tight match.


A draw step like this perhaps:


If we look at other sources of Magic Elo ratings we see a slightly different story. One such source is Magic the Gathering: Online which tracks Elo ratings for limited and constructed.


Using a data set provided by the appropriately named mtgratingtester I got these numbers for all limited players on MTGO (you can see more limited rating her collected here and similar numbers for constructed ratings here):


MTGO limited ratings

Standard Deviation: 83

Average win % for better player: 63.09%


MTGO is still a comparatively competitive environment full of enfranchised players - but we see that compared to the pro tour, the implied impact of luck is diminished. Even within the different types of queues and leagues on MTGO we see differences in the spread of the competition. For instance taking estimates of my opponent's Elo rating in Friendly Sealed leagues I see the following:


MTGO limited ratings (friendly sealed leagues)

Standard deviation: 112

Average win % for better player: 65.68%


Which is close to the variance numbers you see for professional football, professional basketball, and World Cup soccer. However, there is anecdotal evidence that as you look at a broader swath of Magic players, you find evidence of an even greater spread of Elo ratings, and consequently a lesser contribution of luck in the average Magic match among those player.


Before Planeswalker points, the DCI used Elo ratings just like MTGO still does. And while the top MTGO ratings don't ever go much above 2000 - the competition is too strong to maintain the win rate required - the best paper magic ratings topped out when Nicolai Herzog hit 2351.


Since the DCI is no longer maintaining a database of Elo ratings, we can only do a rough estimate for what the standard deviation might have been. By definition, the mean rating was 1600. If we make an estimate for how uncommon a  2351 rating is we can make a guess at what the standard deviation of the distribution might have been. The more rare we say Herzog's rating was, the more conservative our estimate of the standard deviation. To get a lower bound on the standard deviation let's assume the record Elo rating of 2351 represented a very rare rating from the distribution - only 1 out of 20,000,000 samples would be that high. Not coincidentally, 20,000,000 is the estimate of the total number of Magic players that was provided in 2016 by Hasbro.  Using these rough estimates, we would assume that this is about 5.32 standard deviations out from normal and that the lower bound on the standard deviation for the population is 141 - implying an average win percentage for the better player of 71.42%. Sadly, a better estimate than this very rough one is probably not possible since the DCI switched away from Elo ratings all together.


Good luck, have fun


So what does this all mean? It means that yes, luck plays a major role in Magic, with players at the pro tour relying on lucky breaks in their matches just as much as an average team relies on luck to win a Major League Baseball game. That's not to say the players in both cases aren't enormously skilled, and errors in baseball and punts in Magic can both easily lead to losses.


However, as you look at how Magic is played in the wider world you see that the luck involved in winning a game of magic in the MTGO friendly leagues relies on luck more akin to that of a World Cup soccer game. At your local gaming store, luck seems to plays an even smaller, but ever present, role.


But according to the designers of Magic this element of luck is crucial, because it provides encouragement to new players and leads to exciting and unforgettable comebacks:


Ultimately, success in Magic is dependent on both luck and skill. Skill will pay dividends over the long term, but especially in extremely competitive environments like the Pro Tour, where games are tight and advantages hard to come by, making it to the top 8 is going to take both lucky breaks and the skill to take advantage of them.


I find the subject of luck/variance, and how it intersects with skill, to be a deep and fascinating one. In this article we tried to quantify this relationship in Magic and show it in relationship to other competitive environments. To see another attempt to quantify this relationship I would whole heartedly recommend Frank Karsten's article touching on the subject from a different angle.


As you can see, there are lots of other topics to explore when it comes to variance: how to make decisions when faced with it, how to have evaluate your performance and account for it, and how to consistently win despite it. And maybe I can cover some of those in a future article.


Until then, the most important rule of thumb: if you win, it's probably skill, if you lose, probably luck.

Share on Facebook
Share on Twitter
Please reload