News | Matchmaking Tests Results
Over the past few weeks, we’ve run a series of short matchmaking tests based on various feedback we’ve received. These ranged from making the matchmaker faster, stricter, various combinations in between, and yes, even disabling Skill Based Matchmaking and reverting to the old Grade Based Matchmaking.
We want to start by thanking everyone who took the time to share their thoughts about each of the tests. While we’re able to track things like queue times behind the scenes, your feedback is invaluable when it comes to finding extraordinary cases and for getting an idea of how these changes feel rather than how they perform. Thank you!
We’ve kept the details of the tests a secret until now as to not influence feedback, but since the tests are over, we’re ready to dive in and share some details on what was tested. But first, we need to explain what each of the changes mean.
Fair warning, this is going to be a long and very technical one!
Variables
We have a bunch of settings which we can tweak behind the scenes. Here’s a quick rundown on the ones we changed during the tests.
The first is maximum range. When you search for a match, we try to find other players who are close to your rating (within 50 points). If we aren’t able to do so, the search gradually expands until we’re able to find you a match. By default, this is set to 500 points away from your own rating. This is so you’re guaranteed to get a match after waiting a while, though it may not be the fairest.
The second is expansion time. This is how long it takes to reach the maximum range. By default, this is set to 5 minutes.
Then we have a few switches we can toggle. These are:
Bidirectional matchmaking: Normally once one player has been waiting long enough, other players can be pulled into their match (as long as they’re within the maximum range). Bidirectional matchmaking requires both players to have expanded enough to encompass each other for them to be pulled into the same match. By default, this is off.
If this sounds confusing, don’t worry, it’s complicated. Here’s a little chart to clear things up:
Each player’s actual rating is shown in red, and their current matchmaking search is shown in blue.
- In Example 1, Player A’s search has expanded to cover Player B, but Player B has not yet expanded to cover Player A, so they won’t be matched together.
- In Example 2, both players have expanded enough to cover each other, so they can now be matched together.
- When we use the default (unidirectional) matchmaking, both examples would be eligible for a match.
Grade visibility: This one’s nice and simple. We can switch off grades in the post-match screen since they typically aren’t used for matchmaking. This doesn’t actually have an effect, but we were curious to see how people feel about it and how it affects their perception of matchmaking.
Grade Based Matchmaking: We can switch off Skill Based Matchmaking entirely and instead use Grades for matchmaking. For folks who joined us recently, this is roughly how the matchmaking worked before the introduction of Skill Based Matchmaking.
The Tests
And now, the part you’ve been waiting for: Just what exactly was tested each day? We’ll also share our expectations and how they matched up to the results. Quick disclaimer: We’ll be talking a lot about averages in this section. If this doesn’t line up with your experiences, don’t worry, we’re aware that things can vary depending on a bunch of different factors (time, region, role, rating, etc.).
For the first three tests, we did not expect much of an improvement. These three were meant to test individual variables and see the effect they have on matchmaking. Afterwards, we started to mix these variables to see the impact they would have together.
March 28
Test:
- Expansion time decreased to 1 minute (from 5 minutes)
Expectation: Faster matches, particularly for players in off-peak times, though the quality of matches would be lower.
Results: As expected, wait times dropped significantly across the board (faster than any other day), especially for the players on the extreme ends of the rating spectrum. The quality of matches on average did not change a whole lot, though some players may have noticed a larger gap in skill levels.
Kill rates noticeably changed on the ends of the spectrum, with high rated Killers becoming more deadly and low rated Killers having a harder time. This is likely due to players on the ends of the spectrum being paired with more players from the middle.
March 29
Test:
- Maximum range decreased to 250 (from 500)
Expectation: Slower matches, especially for players with extraordinarily high/low ratings and/or in off-peak hours, but higher quality matches in general.
Results: Like we predicted, wait times did increase for players on either end of the spectrum, although players in the middle did not have a noticeable impact on matchmaking times (on average). Overall, this test only slightly reduced the average rating difference compared to a non-test day, though curiously, players on the far ends of the spectrum may have seen an increase.
March 30
Test:
- Enable bidirectional matchmaking
Expectation: Longer queue times for players in off-peak hours & players at the extreme end of the rating spectrum, but fairer matches for players that get matched quickly.
Results: Longer queue times were an understatement; some players were experiencing exceptionally long queue times (15 minutes+). This is because there weren’t enough players to match them against in their region & skill level at the time they were playing, and players who were ‘close enough’ to their skill level would not get matched with them unless they also were unable to find a match.
This is longer than we’d ever like anyone to wait. To play it safe, we cut this test short and altered it a few hours in. A hard cap was added to switch back to the default unidirectional matchmaking after someone had been waiting for 5 minutes. This cap was used for all future tests where bidirectional matchmaking was used.
Of all the tests, this one reduced the average rating difference the most, cutting it by roughly a third. However, the long queue times make this a fairly unappealing solution.
March 31
Test:
- Maximum range decreased to 250 (from 500), expansion time decreased to 1 minute (from 5 minutes)
Expectation: Players who already get quick matches may notice a slight drop in quality, but players on the far ends may find matches quicker. On the other hand, the lowered range may limit the pool of players to match with on the far ends of the rating spectrum.
Results: Queue times for Survivors were faster across the board, but Killer queue times had no noticeable difference. In terms of kill rates, on the mid-high end of skill groups, kill rates dropped slightly (closer to 50%) while the lower groups remained mostly the same. This test also earned the second biggest reduction in average rating difference of all the tests.
April 4
Test:
- Maximum range decreased to 350 (from 500), expansion time decreased to 1 minute (from 5 minutes), enable bidirectional matchmaking
Expectation: Matchmaking may be slower for some players, but the quality of matches should be improved.
Results: Survivor queue times were faster across the board, while Killer queue times remained mostly the same (with the exception on the far low end of skill ratings). Kill rates, meanwhile, climbed slightly at the highest skill levels and remained mostly unchanged at the lower end. The average rating difference didn’t decrease noticeably with this test.
April 7
Test:
- Maximum range decreased to 350 (from 500), expansion time decreased to 1 minute (from 5 minutes), enable bidirectional matchmaking, grades hidden
Expectation: More or less the same results as the previous test, as all the variables are the same.
Results: We were curious to see what kind of impact hiding grades would have on the feedback we received, since grades are no longer used for matchmaking (though they can give the impression that they’re a measure of skill). For this test, we ran all the same variables, only we hid grades as well. This way we could compare feedback to the previous test to see how grades affected the sentiment.
To our surprise, the overall feedback appeared slightly more positive with grades hidden. When directly comparing opinions from folks who left feedback on both the April 4th & 7th tests, we discovered that the feedback was usually the same if not better when grades were hidden. We opted to leave grades hidden over the weekend.
April 11 - 12
Test:
- Switch to Grade Based Matchmaking
- (April 12th only: Show grades)
These tests were added later as we started to see more speculation about the tests. Occasionally someone would predict that one of the things we were testing was the old matchmaking system, and we figured hey, why not give it a shot?
Expectation: Matches would have a lot more variety in player skill, very similar to the old Rank Based Matchmaking. We also expected to see the average grade difference to fall dramatically.
Result: As we suspected, reports of one-sided matches started flooding in. We can also confirm this on our end; the average kill rates for this day saw the most dramatic change yet, with the highest skill levels rocketing up to roughly 80% (>3 kills per match on average) and plummeting to around 20% (<1 kill per match on average) on the lower end of the spectrum. The average grade difference, however, decreased significantly, as expected.
Now to be clear, this is not quite the same as the old Rank Based Matchmaking. For starters, grades reset all the way back to Bronze I while ranks only reset one colour tier, and it’s not possible to lose grades anymore. We wanted to give it a fair shot, so we specifically set this test as close to the grade reset as possible to give people the largest window to settle into their grades. (This is also why the last day was cut short: We didn’t want to leave the test running over the grade reset since that would have everyone matches against anyone.)
Furthermore, a 1-2 day test is not the same as leaving it on for an extended period of time. Given more time (and the ability to lose grades), we expect that people facing far too difficult opponents would eventually have their grades drop to a place where they’re more comfortable, while folks who are consistently overperforming would climb.
Ultimately, though, this highlights a big issue with using grades (or ranks) for matchmaking: A very experienced player who just hasn’t played a lot lately can very easily be paired against players that are new to the game. As we’ve learned in the past, grades/ranks are a stronger indicator of playtime than skill.
This test was very polarizing, but interestingly, we started to see more positive feedback when we started showing grades again. This indicates that there’s definitely a strong connection between seeing grades in the post match screen and the sentiment about a match.
Learnings
We’ve learned that the strictness of the matchmaking and the quality of the matches is without a doubt linked. Looser conditions had a notable effect on kill rates, opponent variety, and also the feedback we received. While the larger variety in opponents was appreciated by some, there was an opposite reaction from others who were now struggling in tougher matches. Simply put, on the other side of any casual match- where one player can win without trying- is a person who’s having a bad time despite trying their best.
On the other hand, the inverse is also true. Stricter matchmaking brought the kill rates closer to the middle, but many found the matches to be too competitive to enjoy. While folks in the lower groups of skill rating would have fewer unfair matches, the higher end of skill groups ended up in tough matches filled with meta perks & Killers.
These tests overall highlighted two distinct groups:
- High skill players who want more variety in their matches
- Lower-mid skill players who, understandably, don’t want to get destroyed by high skill players
The biggest takeaway here is that going too far either way is never a one-size-fits-all solution. What we’d often notice is how some players loved a particular test because they were seeing more variety in their opponents, but at the same time, those same tests were also frustrating to a lot of people who were suddenly getting matched with people way better than them. Neither person is wrong in this scenario: Each of them just expects something different from matchmaking.
Lastly, we have discovered that hiding grades on the post-match screen definitely influences the feedback we receive. We suspect this is for two reasons: 1) Long time players who may have missed the matchmaking changes could still thinking that grades are used for matchmaking. And 2) Some players still feel like grades are an indicator of skill in spite of not being used for matchmaking. Most importantly, however, we found that there was a lot of support for hiding grades, so this is something we’ll be considering as a permanent change going forward.
Next Steps
With all these learnings in mind, we have a solution that will improve matchmaking for both groups of players. We will be making the following changes shortly:
- Reducing the maximum range
- Reducing the expansion time proportionally (so it’s no faster or slower than before)
- Lowering the effective cap for skill ratings
- Removing grades from the post-match screen for the foreseeable future
Why these changes?
As mentioned in the learnings, we found two dominant groups:
- High skill players who want more variety
- Lower-mid skill players who want even matches
In order to make both groups happy, we first need to isolate them from each other. To do this, we’ll reduce the maximum expansion range. This will help keep players of similar skill levels together, reducing the number of one-sided matches. The majority of players fall in the middle of the spectrum, so there is still plenty of people to match with. Additionally, since it seems like the expansion is happening at a good rate, we’ll be adjusting the expansion time proportionally.
Next, we need to consider how this will affect the high skill group. Reducing the range would mean less variety. Fortunately, we have another variable to tweak that solves this issue perfectly. For the purposes of matchmaking, we have a “cap” for skill ratings. Although your rating can continue to climb beyond this cap, you’ll be treated as if you’re at the cap for the purposes of matchmaking. This is in place to prevent the best players out there from climbing to a point where there’s simply nobody to match with.
By lowering this cap, we’re able to expand the pool of players in high skill matches, leading to more variety in the opponents you face. One match may be very close and competitive, but the next may be a little more laid back. This will also ensure that queue times don’t grow longer at higher skill levels.
Lastly, we’ll be removing grades from the post match screen. When we originally hid grades during the test, we saw a lot of people asking for this to be made permanent. Additionally, long time players who may have missed the news could often think that grades were still used for matchmaking, leading them to feel like the match was less fair than it actually was. These, combined with the generally more positive feedback during the tests while they were disabled, were enough of a reason for us to disable them for the foreseeable future.
For the future, we’re still hard at work implementing various improvements to the matchmaking system. These include making Survivor ratings team based rather than personal (so dying heroically to save everyone else isn’t as punishing) and improvements to backfilling lobbies to prevent players from outside of the acceptable range from being pulled in. These improvements will take longer to implement, but rest assured, work is already underway on them. You can find a full list of changes we’re looking into in our January Developer Update.
Once again, we want to thank everyone who took the time to leave feedback throughout the tests. Understanding not just how the matchmaker performs, but also how it feels is incredibly important for us to make improvements. As always, we’ll be keeping a close eye on the matchmaking system moving forward and making small adjustments as necessary. Please keep the feedback coming!
The Dead by Daylight team