How the MMR works, and why what Patrick said makes absolute sense (from a dev POV)

Options

The recent controversy over Patrick's comments embarrasses me; it just highlights the mob-mentality of the DBD community and I absolutely empathise over why the devs refuse to comment on matters... because too many people screech in response about things they don't understand. And right now it's MMR, like everyone has suddenly become experts in how it works and are therefore qualified to comment on it.

Sadly, this is not the case, and let me explain why.

Warning: this will be slightly complicated, because that's just the nature of how these things are. But I'll do my best to simplify.

Short background: I've been a software dev for 20+ years, been involved in some mmr systems in the past (did some collab with the original MS Trueskill researchers back in the day), and generally deal with s/w that support large amounts of concurrent users.

So... let's start at the basics.

The system is known by multiple names, the two common ones being SBMM and MMR. I'm going to use 'MMR' because the term 'Skill' in SBMM is a loaded term that can be misinterpreted. It doesn't mean skill in the way you would mean in natural use.

The purpose of the MMR is to provide Match-Making... hence the MM in the name. More importantly, it's to provide MM between parties who are considered "equal", so that matches will be "balanced". You'll notice those terms are in quotes, because they don't necessarily mean what you think they mean. They need to be defined.

Before we do that, let's also point out some assumptions the system makes:

1) Both sides are attempting to achieve their overall objectives, i.e. killers are trying to kill, and survivors are trying to survive.

2) There is 1 killer and 4 survivors (to begin with)

3) A desirable match (i.e. what the MM should aim to produce) is one where all players have equal chance of achieving their (diametrically opposed) objectives... either the killer kills, or the survivors survive (on a 1v1 basis, i.e. per survivor... the killer either kills the survivor, or the survivor survives).

Now... you can argue with those assumptions if you like, but that'd be a separate discussion altogether

So, how is does the MMR system match people together?

IT REDUCES YOU TO A NUMBER!

That's literally it. To the MMR system, you are just a number! All your skilled plays, jukes, strategies, etc are distilled down to a single number which represents your "skill" or "level"!

That's the first thing to understand and get over. The system knows-not, and cares-not, about how you played, the incredible saves you pulled off, how you sacrificed yourself to save your team, how you did or didn't tunnel/camp, etc. It just cares about what your "level" is so that it can find suitable players to match you with for the next game. Anything else is just noise.

For those that want to argue whether it should be this way, I would invite you to please design a system... heck, even an Excel spreadsheet with basic formulas, that'd take all the data you want to count and be able to not only interpret it, but also provide some objective mechanism to determine how it should affect your level or how it affects who you're matched with (in an absolute mathematical sense, not some vague description of wants!)

What is the number you'd give to a person who had a 5-gen chase and chased, such that no-one else would get that number from escaping via other means? It can't be done! When you reduce things down to a number, it loses all of this nuance!

Software isn't magic! Computers are not people. They don't understand nuance, or intention. Everything that happens to affect any calculations has to be written/coded to accommodate that!

There's a lot more to be said on this, but for the sake of brevity and sanity, I'll leave it there for now.

So, now you're just a number. Here's another important point to understand. YOUR NUMBER DOESN'T MATTER!

Think about it. If your number was 10, 100, or 1000, what difference does it make? You might be tempted to answer "well, 100 is obviously more than 10", but lets say your peers where in the range of 9-11, or 90-110, etc, it still puts you in the same spot, i.e. smack-bang in the middle.

So I'll say it again... YOUR NUMBER DOESN'T MATTER!

What does matter is where your number stands in relation to everyone else's.

If yours is 10, but the player-base only ranges from 5-15, then you're firmly in the middle. But if the player-base averages around 100, then your number (or level) is severely below average. Your number doesn't matter... where your number fits in relation to everyone else is what matters!

The MMR system produces matches by trying to fill a lobby with players who have similar numbers, i.e. they're within a specified range of each other. That range could be determined by a number of factors, and we also know it increases the longer you're in the queue (so you'll match with more players whose level are further away from yours).

Now, here's what too many people fail to understand. In order to do any match-making... any system has to have some way to rank or equate players so that it can determine what is a good match vs a bad match. How else are you going to compare players with each other without some common comparable criteria between the two? Think about it! You need some objectively measurable indicator that you can ascribe to both survivor and killer in order to compare them with each other to match-make! Otherwise you're literally comparing apples and oranges.

So now you're in a match, and the match plays out. You either survive or die, or as a killer you get kills or not. At the end of the match, THE ONLY THING THAT MATTERS IS HOW MUCH YOUR LEVEL MOVES! Remember, the MMR system doesn't care about your amazing plays, because there's no objective number it can ascribe to it. It just needs to know whether your level should be adjusted, and if so, by how much.

Remember, you and your plays are just being reduced down to a number!

Here's where there's a lot of room for confusion, misinterpretation, and a range of opinions. There are many ways to skin a cat... and there are many ways to determine what that adjustment should be.

Maybe it should take hooks into account, or how many chases won/lost? All great ideas. And I'm sure BHVR have tried these out - we've had a number of MMR tests in the past year or so. I strongly suspect that, at the end of the day, the results they produced weren't as good as what would have been desired.

Or, more likely, they discovered something else... but before I mention it, let me point out another mechanism of the system... THEY'RE DESIGNED TO WORK OVER A RANGE OF MATCHES!

There is literally NO ranking system anywhere in the world that will accurately place any player after a match or two. That's impossible. There's simply not enough data points to determine where you fit in the ranking overall, unless you play enough matches to provide as much.

And the second point... YOUR LEVEL FLUCTUATES!

No-one is the same "skill" all the time. You're either a newb, and your skill increases. Or you've been away for a while and have adapted to the new mechanics etc, and your skill decreases. See, by the time you've played enough matches for the system to know where you rank in comparison to others, bear in mind that their rankings have changed too! Perfection is a moving target, and a good working system is one that can accurate match-make based on the data it has at that time.

You will never have any system that accurately places players exactly where their rank is for all time... the system just tries to get as close to an ideal as possible given the current data!

At the end of the day, if the system produces a match with an even 50/50 result, then it's done it's job. The 4 survivors had 50/50 change of surviving or dying... 50% survived, 50% died, that's an even match! Hence BHVR mentioned back in the day that they consider 2-kills+2-escapes an even match.

Note: as a gamer, I recognise how stupid this seems, because we know that once two survivors go down, the other two have a harder job to survive... unless the deaths are occurring in end game which probably means the killer had a stressful game. I get it. It seems completely counter-intuitive. But the maths checks out... 4 survivors, each with 50/50 chance to win/lose will mean that 2 "win" and 2 "lose". It's mathematically balanced so should be a "good" game, as far as one can predict.

So, back to mechanics of how to reduce the game to an adjustment for your level... it appears BHVR have noticed that a persons average kill/escape ratio appears to correlate with other more complicated mechanics... so rather than fiddle around with the complicated mechanics, they can just take the shortcut and go with the clearer/easier kill/escape mechanic.

And herein is the misunderstood point... Patrick said that it's an indicator of skill, and when you consider that the MMR self-calibrates over a series of matches, it appears to prove an accurate one.

So let's clarify some things bearing this in mind:

A face-camping Bubba is more skillful: No, that's not what he said, nor is what he's suggesting. Firstly, ignore the term "skill", because it's confusing... the devs and players are using the same term to mean two different things. What he's saying is that if the killer is consistently getting kills - howsoever they're getting them - then they should go against "better" survivors who'll be able to resist their strategies and provide a more even match. They're not saying "wow, great plays"... they're saying "fine, we'll match you against better players, now let's see how you do!" The "skill" is in Bubba being able to catch the survivor in the first instance (and then however the rest of how the match plays out... maybe he loses 5-gens 'cos of the camp, maybe altruism gets the team killed). Either way, the Bubba will be adjusted to go against teams better suited (in theory).

Losing a 5-gen chase is low skill: Again, no! If you lost a 5-gen chase and died (which could be for a number of reasons, maybe your team just left you to be one-hooked, etc) then one of two things happened: (i) it was a completely fluke chase against a bad killer which you won't be able to repeat, and you're fine as you are... or (ii) you are that skilled that you can do it again, and over a series of matches your level will increase because you will do it again!

Again, straw-manning the point and expecting the system to produce an accurate result on a match-by-match basis is stupid. It's not going to do that. It's not designed to do that. It cannot do that, because the data fluctuates anyway!

Perfect example is Patrick's hockey example... he mentioned that for hockey players you could rank them by how many shots they take on goal... but if they're winning games (i.e. scoring more than others), then you can already presume they're taking more shots because how else are they getting the goals? You don't need to do the complicated calculations when there's a much simpler indicator that correlates closely enough. For any sport, if you look at their league tables, you can infer the teams (consistently) at the top have the best players... because how else did they get there? You don't need to know the ins and outs of each player, etc.

But of course, the DBD community are just like "waaah... hockey is not dbd, waah!" completely missing the point of what he's saying!

There is also the factor of diminishing returns to consider... the more "fine-tuned" the system is, the more effort it takes to maintain that system, and the less useful (i.e. different) data is produced. Whilst it is technically possible to build the system around hooks, or chases, etc... it requires so much more investment in time/development, the results aren't worth it because they don't produce anything that is fundamentally different from just looking at kills/escapes. So why adopt the harder method?

So... back to the adjustment at the end of the game. Adjusting your level by kills/escapes is very simple, and a reasonable indicator of where you need to be. If you're killing everyone, you need to be moved up to match against players you struggle to kill. If you're escaping every match, you need to move up to face killers that can better catch you and do something about it. And vice-versa.

Stop thinking about the MMR system as an ego-boost, the way some strimmers say they are "max-mmr"... there's no such thing. If the system matches you with players that'll give you a "balanced" match (as per how it was mathematically defined), then the system works as intended.

One other minor interesting point to note: Patrick mentioned the adjustment to a survivor's rank is also based on the order in which they died, because they recognise that a 4v1 death is different to a 3v1 death, etc, so you "lose" less if you died later (in order) rather than earlier.

The system, as a whole, is supposed to self-calibrate. It requires you to play a number of matches, and so long as you play consistently (for the most part), it'll do it's job in matching you with players that'll give you a 50/50 chance of achieving your objective. But bear in mind other player's levels are moving too, so it'll never be perfect. And there are always ways to "cheat" the system by artificially lowering your rank over a series of games (how is the system supposed to know though?)

Again, I've glossed over a lot, and simplified things a lot, to make it easier to understand... not to make it easier for people to provide strawman rebuttals. As a community, I wish we'd try to understand things properly before passing judgement/comment... and not just wildly complaining for the sake of complaining because we don't know what we're talking about.

«134567

Comments

  • Tranquil_Blue
    Tranquil_Blue Member Posts: 335
    Options

    One additional point: they said there's a soft cap for MMR, so there are obviously many players with ratings way above the soft cap. Those players are definitely playing matches at the highest possible MMR, are they not? I agree that no one should be making claims about their own MMR (since it's hidden), but I'm saying that technically there is a highest possible MMR (for matchmaking purposes).

  • zarr
    zarr Member Posts: 898
    edited January 2022
    Options

    I appreciate to hear voices of reason on this. The response to MMR has been baffling, many people think the devs have it backwards but their own reasoning is backwards and fundamentally fails to understand what an MMR system is, as well as riddled with other fallacies and misconceptions.

    And I would go even further than you, and say the system is not only the technically most sound system it should be (which is, as any functioning MMR system, it should be about winning, and clearly-defined, diametrically opposed win conditions, which in DbD completely logically means - and without changing the game fundamentally indeed can only mean - killing versus surviving), but that it does by looking at kill and survival rates in fact also factor in skills more accurately than any other metric.

    As you point out, ultimately it wouldn't even matter if the system had nothing to do with "skill" as it is commonly understood, it would still do its job of giving people matches in which they on average are roughly equally likely to succeed. This is desirable even if the reasons for it would be something other than "skill" factors, since people generally like winning and generally do not like losing, and in DbD specifically, like surviving/killing, not dying/failing to kill. It would still yield more desirable gameplay experiences (more fair/even/"balanced") that lead to less frustration and more satisfaction. But I also am convinced that killing and surviving account for skills more than anything else, and that the skills required to consistently succeed in terms of killing and surviving - i. e. the skills that correlate with killing and surviving - are absolutely the skills that should be selected for. Chase skills are still heavily weighed by these metrics, but macrogame skills that a "number of hooks" metric for example would not nearly as much account for also play a huge role, and they do indeed require a fair bit of mental and mechanical abilities and make for nuanced play and decision-making on both sides, particularly of course because in an MMR system this happens between players that are increasingly more apt at competing in these ways.

    That doesn't mean the system is perfect (the rating cap and matchmaker logic in particular can be problematic), let alone of course that the game is perfect and could not be changed to "weigh" the different skills differently in its actual gameplay and potentially thereby become a more desirable experience in and of itself, but it does mean the system functionally fits the game as it actually is and produces desirable results (as long as and insofar the player pools and matchmaker algorithms allow for it).

    I think the system could be improved a little by looking at group survival rather than individual survival (so the win conditions instead are 0-1 kills/3-4 escapes versus 3-4 kills/0-1 escapes), and by calculating rating adjustments based on the average rating of the group rather than that of individual survivors, as well as factoring hook stages into the equation a bit (i. e. a 9-stage 3k would increase killer rating/decrease survivor rating by less than a 10/11-stage 3k, and a 3-stage 1k would decrease killer rating/increase survivor rating by more than a 4-9-stage 1k). Ultimately I do not think this is necessary and the per-survivor kill and survival rate calculations are doing their job well-enough for the purposes of matchmaking, but I suspect it could account for some performance aspects a little more accurately in the sense that it would take less trials to adjust ratings appropriately, as well as make rating adjustments on the killer side inflate less from facing very mismatchedly-rated groups. I think the devs also mentioned that they want to move to looking at group survival. Do you agree that these would be improvements?

  • ActualPainedFrog
    ActualPainedFrog Applicant Posts: 279
    Options

    I could already tell from skimming through the post initially that you have a lot of experience with these systems, mainly due to the utilitarian way you explained it. I completely agree that this is probably the best system this game can have, but I just don't think it belongs in the game, this is an excellent post though and I hope more people read and understand it.

  • Deferlo
    Deferlo Member Posts: 131
    Options

    For me, the only point that i would like to change is survivor's MMR. Since you play as a team to escape, i would prefer that every adjustment to the rating is made by taking the overall team result instead of an individual one.

    To take the 5 gen run as an example, chances are that every other survivor escaped thanks to that play, making this game a 75% escape rate for the team.

    Making such change could help reflect "skillfull" action into the result, since every play ultimatly lead to an end result.


    Also about skill, i feel like most player think of it's meaning in symmetrical pvp game where, if you "outplay" the opponing team, you will most likely win, but that logic doesn't apply here since both side aren't aiming for the same thing.

  • Lochnload_exe
    Lochnload_exe Member Posts: 1,360
    Options

    Thank you for writing this and it does make much more sense put in this way. I can see it differently now and I can understand why it was done this way, thank you lots.

  • jesterkind
    jesterkind Member Posts: 7,253
    Options
    1. Yes it does, that'd be the part where it explains that the system matches you over large batches of games- if you're cocky enough to go for the kinds of plays that can get you killed first after a five-gen chase, you're not going to fail that every time, and you're going to escape more often than you don't.
    2. That'd be the matchmaking half, not the MMR half itself, which is the focal point of this post. I don't know if OP would acknowledge the matchmaking portion being broken, but they don't have to- the devs did, in the QnA stream.
  • jesterkind
    jesterkind Member Posts: 7,253
    Options

    The post still addresses it, which is my point.

    No MMR system is going to work on a game by game basis, they're all supposed to place you over time. That's just how matchmaking works- progress systems often go game by game, but matchmaking takes much wider pools of data.

  • fblurbg
    fblurbg Member Posts: 78
    edited January 2022
    Options

    Nothing about what they said shows that the MMR system's standards is actually a good indicator of skill—which it isn't. In a game like DBD where there is more RNG-based variance than there is player input-based variance, not only is a skill-based matchmaking system a bad idea in the first place, but when you throw out all that variance and the infinitely huge list of factors that came into the situation and instead boil it down to a small number, it's not going to be accurate. And it isn't. MMR does not do a good job of indicating your skill or if you are even remotely close to as good or as bad as the other players in your lobby. Especially when you put a hard cap on your MMR number that can be reached with laughable ease, meaning there are probably more players at the highest MMR bracket than any other bracket in the game. And even if it WERE a good skill indicator, it doesn't matter because the system is broken and doesn't match you with players of similar MMR ratings anyways.

    So what we have is a skill-based matchmaking system that does a terrible job of rating your skill and also does a terrible job of matching you with people of similar ratings. It is, objectively, a trash, broken system and should be revised from the ground up.

  • SilentShepherd
    SilentShepherd Member Posts: 527
    Options

    Exactly, well put. And reading the Lead Game Designer's hockey analogy makes my head hurt.

  • Advorsus
    Advorsus Member Posts: 1,033
    edited January 2022
    Options

    Very well said. I also can't understand people's complaining about the matchmaking. The only problem I had was the backfill. Because it was taking average to bad killers and putting them against really good survivors and vice versa, just so they don't have to wait in queue longer, and that's dumb and makes it just like the old system. But they said they were looking to fix that next patch so I can't complain.

    I actually like the 2 survive and 2 die scenario. My most unfun matches are where I 4k on my main killer, and then at the end game screen there's 2 survivors with 10 hours combined. It feels unrewarding and feels like a hollow win because of their inexperience. Same thing if I'm playing a killer the first time, and I'm going against the sweatiest swf possible, it feels like you're completely helpless simply due to poor matchmaking.

    But having 1-2 survivors actually survive feels legit. Those are the matches I have the most fun in. Plus, it feels like an actual horror movie. You have the hero/heroes that always manages to survive at the end. That's how all the movies go.

  • jesterkind
    jesterkind Member Posts: 7,253
    Options

    As in any other MMR system (assuming that the number is actually used, which tbf it currently isn't, the matchmaking compromises on it too quickly as the devs have admitted to), the person who hides all game and waits for the team to die so they can open the exit gates (as hatch doesn't count for MMR) will climb up to the point where that no longer works. That's how MMR systems should work, if there's a cheap tactic it should let you climb so you get to the point where it stops working.

    I have no reason to disbelieve the devs when they say that it does. It makes sense to me- if you're the kind of person who excels at looping and runs the killer for five gens sometimes, that means you're going to escape more often than you're not, or you're going to slide down to the point where you will start escaping. That's how it should work, that's what a hypothetical perfect matchmaking system would look like.

    That, or you're hiding to escape every game, which I addressed above.

    As for skill-based matchmaking being a bad idea, that's why I generally agree with the assertion that they shouldn't have ever used that term, that was a massive mistake. It's not really a skill based system, it's a system designed to produce matches where the default assumption a 50% chance for you specifically to achieve your win condition- and from what I'm hearing, that part works, it's the sorting and matching half that compromises too quickly.

    (All of which is in this post, btw.)

  • SilentShepherd
    SilentShepherd Member Posts: 527
    edited January 2022
    Options

    And? That person who hides to escape, STILL got to a rating they do not deserve. bad system.

    And the person who actually tries to play the game and is doing everything for the team, this player puts themselves at risk more, and WILL die more, but they are an overall NET POSITIVE for the team in every game they play, as they get things done and help the team accomplish many things. This should be taken into account.

    Since this is a team game, and the Lead Ded used a hockey analogy, I will use a basketball analogy.


    Plus/Minus +/-

    A player who does 3 generators, 3 safe unhooks, puts up 3 boons, and escapes 4 chases is an overall NET POSITIVE. And we could assign numerical values to all of those efforts and find a +13, for example. This person should climb MMR and not be held back because they died.


    And another POSITIVE RESULT of this new system, would be that Survivors would not feel as bad if they die. Players would be more team-oriented and the game would be more fun as it would not be all about escaping only.

  • jesterkind
    jesterkind Member Posts: 7,253
    Options

    Yeah, I think this is where most people's problems come from. This system does not reward or punish anyone, it does not "hold back" or "elevate" anyone, there's no "deserve". It's not a progression system, there's no reason to assert that players who play very aggressive and cocky "deserve" to be elevated higher. If you're not escaping, you should be going down until you can, that's how matchmaking works.

    It's not actually a strict, game-to-game evaluation of your personal skill. That's what the emblem system is for, not the matchmaking system.

  • jesterkind
    jesterkind Member Posts: 7,253
    Options

    I don't think even OP is trying to say that this system is actually working? Even in the message you're responding to I point out that this is all hypothetical since the actual matching half isn't working properly at the moment.

    What we're saying is that it's not because it's based on kills and escapes, it's because the number you're boiled down to - while accurate - isn't being used properly when you're actually in a queue.

    And yes, BHVR shouldn't be using the word skill, it has a clashing definition here. My understanding is their use of "skill" has more to do with aggregate games and general brackets of players, not the minute-to-minute skilful plays that people generally use it for. They aren't lying or broadcasting a lack of understanding, but it is poor communication at this point.

  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    I think this is a misunderstood or misinterpreted point.

    Every player has an MMR level or number... whatever you want to call it (well, more accurately, one number per killer and one to represent survivors, but I'm going to hope you guys understand the point without me having to mention all these specifics).

    So, the system is trying to match you with people who are near enough your level. They may be slightly higher or lower, but this is all relative based on the range of possible values out there who are currently looking to match-make at the same time as you.

    For argument's sake, lets put some real numbers in. Let's say your level is 2000, and the system determines everyone should be within +/-100 of this... so the system will try to find a killer and survivors who each have a level of 1900-2100.

    But what if it can't find any? Maybe there's not enough killers to satisfy the survivor queue, etc... in this event the system "widens" its search criteria to anyone +/-200. And it keeps widening until it's found a match.

    Again, imagine you're a killer who's achieved 1000 4ks in a row, you'll have a high MMR value. And you want to play at stupid o'clock in the morning when most people are asleep. It's possible there's no-one at your level, so again the system will widen the search criteria which will invariably mean you match with people well below your rank (and those poor shmucks will end up with a killer who'll destroy them!)

    It's not that your level has been capped... there is no cap. They specifically stated there's no high or low value, and that would make complete sense in a self-calibrating system like MMR. As more matches are played, and skill levels diversify, the range of values will increase (which is why your actual number is irrelevant, what matters is where you are in relation to everyone else).

    Rather, the system just becomes more liberal with finding matches, because it also doesn't want you to wait an absurdly long time for a game.

    So the basic mechanics are:

    1) If there are players near enough your skill level, you match with them

    2) If there is no-one available, and it's been a while, you'll just get matched with the closest available

    There are also edge cases where people are pulled in to fill in gaps in lobbies already created when one person leaves, etc, but that's generally how it works. There's no cap. There's just fluctuating tolerance the system applies to try to ensure you get into a game soonish.

  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    I hear what you're saying, and this boils down to a phenomenon known as "The Curse of Knowledge", in that people who are very familiar with a subject often forget the nuances they're implicitly aware of when explaining to people outside of the knowledge circle.

    In other words, Patrick is a developer, and understands the background mechanics of what's going on. Because of the way the questions were posed, he only explained bits and pieces of the system at a time, rather than the whole, which made it harder for people without the background knowledge to understand what he meant. I'm a developer, and I've worked with this kind of stuff before, and I understood completely what he was saying. It all makes absolute sense. But unfortunately the general community didn't.

    I also do believe there are some bad-faith actors who've deliberately stirred things up for sensationalism. His hockey analogy was absolutely correct... there's no need to count/measure shots on goal when you can just measure wins overall as an indicator of a person's skill. If they're winning, they're obviously scoring more, and if they're scoring more, they're obviously taking more shots. It's all inferred, so you can cut out the middle dynamics (he said this almost verbatim).

    The strimmer I happened to see comment on this literally went on a rant of "hockey is a game of equal sides, people in different roles, blah blah blah" which has absolutely nothing to do with the analogy he was making. This is a straw-man!

    What's a straw-man? It's where you oversimplify the point to such a degree that it bears no resemblance to the original, and you defeat the oversimplification. You might be handy with a rifle, and you can shoot straw-men in the field all day... but when you're up against real soldiers who are shooting back, that's a completely different point.

    Reducing and dismissing the point of "hockey is a different" game is exactly the same... his point wasn't about hockey! His point was about not needing to focus on the individual inner-game mechanics when the outwardly result (i.e. win/loss) is a good enough indicator as to what happened. It's the same analogy regardless of sport... but too many people who didn't understand the analogy got fixated on the wrong point and made their rants based on that. And that's embarrassing.

  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    You're definitely not understanding... and this is the key part of your message which highlights that:

    "A player who does 3 generators, 3 safe unhooks, puts up 3 boons, and escapes 4 chases is an overall NET POSITIVE. And we could assign numerical values to all of those efforts and find a +13, for example. This person should climb MMR and not be held back because they died."

    You're looking at the MMR system like it's a reward/punishment system. It's not. There's no inherent advantage to climbing, nor are you being "held back" if your value doesn't change. What the system is doing is placing you with other players where it believes you have a 50/50 chance of escaping... that is it!

    A player who does the gens, and escapes chases, etc, will surely also escape the trials... and therefore overall their rank will climb. A few matches here and there won't disrupt the overall balance because the system only makes small adjustments to your level anyway (think of it like a moving average).

    But a player who's constantly dying all the time... regardless of howsoever that happens, will go down.

    You can't tell me there's a player out there who always does 3 gens, 3 unhooks and escapes 4 chases in every game... like every game they're matched with a team doing nothing else? They'll either be doing better than their team mates and will escape, and will rise in rank. Or will die and reach a point where they can escape more consistently. What you're describing is a specific incident, whereas the system is balanced around overall behaviour.

  • dugman
    dugman Member Posts: 9,714
    Options

    Good read. A few quick comments

    • MMR does assign a person a number, but of course how that number is assigned is where the devil in the details lies. One debate there is whether a "kill" should be considered a win versus something like "scoring more bloodpoints than all the survivors" or "getting X number of emblem points" for example. Rational people can disagree on what actually constitutes a win versus a loss in DbD and whether or not a kill versus an escape is the best such gauge. (Personally I think kills and escapes are perfectly fine, I'm just saying that I can understand why other people might think the "winner" of a match is actually the person who got the most BP or whatever.)
    • Another detail that matters is how much the MMR shifts up or down after each match. Typically these kinds of systems shift it more or less based on relative ratings of the opposing players and other factors, such as weighting the first kill more than the second kill in terms of up/down movement. There are some people who incorrectly seem to think the system just does a flat up or down each kill regardless, from what I recall other factors do play into the size of the shift.
    • There's also a consideration of, even though each player has an individual rating, how do you match four people with four different ratings against one killer with another rating? The simplistic method is to just take the average of the survivor ratings, but it seems pretty likely that a more accurate system would be doing something like training a neural net to take in four survivor ratings and spit out a closely matched killer rating it should aim for. It's quite possible for example that small to moderate differences in survivor ratings make a big difference in escapes, but larger rating gaps have diminishing effects on escapes so that it doesn't matter as much if someone has three times your rating versus four times your rating, for example. If the existing system has an inaccuracy in its estimations of results based on MMRs I suspect this is one area where they could possibly improve the algorithm.
  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    OK, let me address this part:

    "And the proof is in the pudding. We can see that their measurements used for calculating a 50% win rate game in the MMR system are bad because that isn't happening. That's why people are still complaining, because the system isn't doing what it's designed to do. Matches still have wildly differing results across all the brackets, and when they are consistent, they're not consistent in the way BHVR wants, usually resulting in strings of 3-4 kills or strings of 3-4 escapes, instead of the "ideal" 50% kill/escape ratio."

    This is an example of confusing the theory with reality (which in fairness is an easy mistake to make), so let me be clear.

    The system is NOT trying to create matches where 2 survivors escape, and 2 survivors die! It can't, because of the asymmetric nature of the game, and it's something the devs (implicitely) acknowledged by ascribing different values to survivors based on the order in which they die. I'm sure we all agree that once two survivors are dead, the other two are going to find it almost impossible to escape... to the point where the devs won't count the hatch as a "win" even though it's an escape.

    What the system is trying to do is CREATE A MATCH WHERE EACH SURVIVOR HAS 50/50 CHANCE OF ESCAPING, i.e. given the level of all players involved at the outset, they have an equal chance of escaping or dying. That - in theory - should be a balanced match.

    Mathematics dictate that with 4 survivors, the standard outcome would be 2-escapes and 2-deaths.

    But that's just the theory. In practice we know it doesn't work like that, and the system isn't trying to do that.

    Everyone - devs included - acknowledge that once a single survivor goes down, the odds of the others escaping decreases also, but that's outside of what the MMR was trying to predict and cater for. To do that level of prediction is stupidly hard, and unrewarding because there's no way to prove it works or not. You can get the same players to repeat the same game multiple times and have wildly differing outcomes every single time just because of a small thing that happened. I've had games where I've got from a 0k to a 4k, or vice-versa just because of some small thing.

  • Smuk
    Smuk Member Posts: 735
    edited January 2022
    Options

    Correction to 2) - BHVR QnA.


    If there is nobody or somebody leaves = completely ignores MMR.


    So if a full lobby has 2000 +/- 100

    Killer leaves the lobby, the filler will be satisfied with killer who have just finished tutorial and wants to enter the world of dbd. SBMM does not sort killers by MMR desc and selects first available one


    You as a gamer and developer. Your objective opinion, is the game in such state (from boons to DH and grab validations, size of the maps, speed of the gens etc…) is in fact suitable candidate for SBMM.

    SWF has 15% more escape rate then soloQ. Comms do help a lot in the game. Even just by luring killer north while gens pop south .


    In the aspect of the game, playing a killer is not something you can do to ease your mind and enjoy the game. As soon as you try to relax a bit, you will start to loose.

    Queues are long as there is less killer population. So it makes sense in symmetrical game, where population is one, but here is logically divided. And as QnA dodged the question regarding ratio, in the average, i would not be suprised if its atleast 10:1

  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    I invite you to take up the challenge... draw up an Excel spreadsheet and label all the columns with the required data that you can perform calculations on in order to arrive at a consistent and fair value.

    It's all well and good coming up with your theories and suppositions, it's another thing to turn it into code, and another again to make it workable and viable.

    But hey... I don't know what I'm talking about with over 2 decades of experience, nor do the devs with their collective experience of actually making the game. I'm sure you know best, so please... show us all the design of how this would work exactly.

  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    Did they say (explicitely) that it completely ignores MMR if someone leaves, or just that it widens the scope really wide very quickly to try and find someone suitable ASAP (i.e. queue-jumping)?

    If it's the former, that'd be really bad... but I wouldn't put it past them.

  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    Yeah, good point re: the 2nd. I was going to mention this but decided against it because the post was becoming very long as is.

    The MS Trueskill system used to assign two numbers actually... one was your rank, and the other was "confidence", i.e. how confident it was that your rank is "true". So obviously you'd start off with some arbitrary value and low confidence, but the more you play, the more accurate the rank would get, and depending on how close the match result was to what the system predicted, it'd adjust the confidence value accordingly too.

    Once nice feature of the system was that, if you went up against really skilled players and won, it'd raise your rank a lot faster because it was much more aware of the discrepancy in perceived skill level... and if you lost to a player who was much higher than yourself, it wouldn't adjust your rank so much because it expected for you to lose anyway.

    There's a number of ways to make such systems, but they boil down at the end of the day to reducing everything to a number. And I think that's the part that throws everyone off... because they want that number to represent how many unhooks they did, how many gens they did, how many chases they escaped, etc.

  • Smuk
    Smuk Member Posts: 735
    Options

    Aprox 55 minutes.

    I am sorry as Scott is bafling through whole explanation. But after he ends his commentary monologue, Patricks finalize with statement (speed over equality)

  • Marc_123
    Marc_123 Member Posts: 3,369
    Options

    A new killer should never get a SWF and a pro killer should never get newbies.

    The MMR system may work perfect (in theory).

    But the fact that backfilling or long wait times completely ruin it is in the end nothing more than a system which is not working.

  • fblurbg
    fblurbg Member Posts: 78
    edited January 2022
    Options

    And this is where your argument fails. You claim over and over that the system isn't trying to create a 50% kill/escape rate for each match... but it is. The devs have explicitly stated, over and over again, on a multitude of occasions, that this is the end goal. That is what they want. That is their ideal balance. It's not a 50% chance of escaping per Survivor, it's a match where 50% of Survivors escape and 50% are killed.

  • dugman
    dugman Member Posts: 9,714
    Options

    Right. If they wanted a single number that actually directly includes factors like how long you were in a chase or on hooks then you would probably just do something like use the endgame score screen and try to have the MMR make it so killers and survivors have about 50/50 chances of having the most points on that screen. But I can't remember the last time I saw anybody on the forums actually say "yeah, the winner of the match is the one who got the most points".

  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    So he addresses it at 1:09:10 in the original video

    He doesn't say they ignore MMR to backfill. He says they massively prioritise speed, which I would interpret to mean the system literally pulls in whoever's in the queue with the closest value, regardless of their position in the queue. The survivor queue may be 10m, but if someone joins the queue at the right time, they may get pulled into a lobby to backfill it immediately if they're near enough the level for which the lobby was created.

    That's a different thing to ignoring the MMR completely and pulling in anyone.

    That said, there is a point where the two can and do overlap a lot, so depending on the actual scenario there may be little difference. But in fairness, the way he explains it (to my ears) sounds like it's still "respecting" the MMR levels, but it just becomes massively lax in the range of values it'll accept for a match for the case of a backfill (as opposed to having no criteria whatsoever).

  • TheSubstitute
    TheSubstitute Member Posts: 2,261
    Options

    I don't think the reaction is embarrassing but rather misplaced. As it is Patrick didn't explain things as well as it could have been, which I understand, since he's a game developer and not a soft skills professional. He was attempting to be relatable but it backfired since the differences between the two are so great and it confused the issue. The point isn't goals, the point is DbD is defining a win condition.

    For a video game like Dbd, that's quite a bit different as other people have different win conditions but the majority would think kills on one side and escapes on the other means they achieved what they want for the majority.

    For a video game, calling it skill based though carries a lot of baggage that could be avoided by saying only match making. To those who suffer from RNG it sounds like an insult and to many in the gaming community saying you're bad at a game is a cutting insult. Whether someone thinks video game skill should be a point of pride or not many do feel that way and calling something as inaccurate as measuring all the variables in a match as the MMR system we have skill based would create pushback.

    It just boils down to better communication skills with the caveat that, no matter how perfectly phrased or made something is, at least a few people will still complain. And the main question for the devs to ask themselves, in my opinion,is 'is it more fun than what it replaced?'. People sometimes get too wrapped up in details but the goal is to improve how fun the game is and items such as MMR are just tools to try to achieve that end.

  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    I know they've said a balanced match in their view is one where there are 2-escapes and 2-deaths... that's obviously a result of the maths I've already explained. Like seriously, what else do you want them to say: "A balanced match is 4 deaths... or 4 escapes!"

    But anyway... please do provide a link/evidence to support what you're saying, since you're so confident of it. Let's have a look at what they said in context, rather than your opinion of what they said. Then we'll see whose argument fails.

  • RainehDaze
    RainehDaze Member Posts: 2,573
    Options

    Though if they did say that, we have to admit there's something of an issue with the goal they're setting up: two escapes would mean the Survivors must always complete their objective (do gens and open exit gate), but the Killer must only half complete their primary objective (Kill Survivors).

  • PlaysByShady
    PlaysByShady Member Posts: 590
    Options

    Let me address this point: "As for win conditions... it's a 1v4 game. There's an entire gameplay option for putting together proper teams. Teamwork is necessary to win. Treating matchmaking as a series of 1v1 encounters is just weird, especially given how easily one person can be snagged in endgame. It's just a bizarre choice all round--and it means that in what we're supposed to think is an even draw scenario, two people definitely gained MMR, two people definitely lost MMR, and the Killer's position probably changed based on who those are as well. In a draw. Er... just weird."

    Yes, I agree, it's weird to us as humans. But as far as the maths go, it's sound. In the game, each of the 4 survivors had a 50/50 chance of escaping or dying, and they achieved that with perfect statistical accuracy (50% escaped, 50% died).

    The ones who died, need to face less skilled killers to improve their odds of escaping. The ones who escaped should face better killers.

    And apply this to a whole range of matches.

    When you look at individual matches on a case by case basis, and analyse it as a human does, it doesn't make sense. But that's because the human isn't thinking how to rank the players against each other to match make. The maths is doing that and from a maths perspective it works.

    I literally don't know how else to explain it any more clearly.

  • Smuk
    Smuk Member Posts: 735
    Options

    You cannot proof it. It can also be selectized words by him. But you cannot deny facts as creating massive distance between mmr values and facts that your opponents has 16 hours ingame compared to somebody that has 2000. Which can be visible on profiles.

    Which to my eyes it seems as it takes (almost) whole rooster or the algorithm cannot do its job due to lack on player base on one side.


    And ps: I have not been working on mmr cases itself, but i was in development of AI / machine learning for almost a decade, so I do underastand the problem from developer and consumer side

  • RainehDaze
    RainehDaze Member Posts: 2,573
    edited January 2022
    Options

    We, as humans, designed the maths. We know how the game works, we know what the game requires, and we're defining what a win is.

    My point is that they decided to model MMR over a game as a series of 1v1 competitions and then adjust MMR based on the individual results there: i.e., modelling it as if the outcome per-Survivor is (mostly) a matter of the Survivor's actions on their lonesome, with the factor built in if someone else has died first.

    The maths would not care and would still do the same abstraction of skill into a single number if BHVR had modelled it instead as a 1v4, with the Killer gaining MMR for 3 or 4 kills and losing MMR for 0 or 1, and the Survivors gaining an inverse amount on all four of them regardless of their individual outcomes.

    It's a baffling design choice to model MMR in such a way when your game is not built around the idea of a single Survivor's skills being enough to secure their win condition, and it only necessarily needs one bit of bad luck for a single person to lose despite their overall team victory. It's why the sports analogy was flawed, because you don't suddenly demote goalkeepers in, say, professional football for letting in one goal right at the end if the team still won everything.

    Furthermore, it means that if the MMR achieved an even result from people with even starting MMR, you're still changing everyone's MMR. If your system can get a perfectly accurate result and it will then make adjustments anyway, I'm questioning your update logic. Looking at it from the perspective of spending way too much time with neural net loss functions, 100% accuracy should mean a loss of 0 and thus no parameters change.

  • TheSubstitute
    TheSubstitute Member Posts: 2,261
    Options

    The math does work. The problem lies in what you just alluded to; two games, one of which is a 4K and the other is a 4E, achieves the goal of a 2K 2E average. Neither of these games are fun for both parties.

    It really highlights some of the game imbalances though and what I consider to be a problem. Someone who is just a locker gnome will get harder killers until they end up escaping half of the time. A facecamper will get harder opponents until they regularly get stomped half the time. Neither playstyles though are fun and it would be super frustrating for both the locker gnome and face camper to relearn how to play the game. Most likely result in my opinion? They stop playing since frustrating a player usually doesn't result in the player trying harder but rather quitting. It's recreation time which should be fun.

    This negatively affects the game since a lower player base gives longer queues and worse matches. Long term effects? The game becomes worse.

    I think a lot of the issues arise because there seems to be a focus on balancing matches for MMR on BHVR's side instead of just asking if MMR makes DbD more fun or not.

  • RainehDaze
    RainehDaze Member Posts: 2,573
    Options

    Oh, no, I meant if they were outright aiming for 2E as a most common outcome, you're trying to create a balance where one side must 100% complete its objective in all games and the other side needs to only half do its one. That's just troublesome.

    Things being stompy one way or the other and averaging out is the more likely outcome, but if that's not what they're aiming for then it's weird.