World War Matchmaking - Design Spotlight 2019

TinSoldier

New member
Joined
Jun 4, 2017
Messages
1,568
Hello DomiNations fans! Joe “Muet” Grubb here with Big Huge Games. This spotlight covers a long awaited change to DomiNations: World War Matchmaking.

Why Change Matchmaking? What are the Problems We’re Trying to Solve?

The high level goal is twofold: Improve the competitive integrity of World War and improve the match quality.

Bear with me. I’m trying something new with this Spotlight; we’re going to go deep. We’re going to talk about how the old system worked and how the new system is attempting to solve problems. If that’s not appealing, there’s a tl;dr “what are the changes?” if you skip to the end. It’ll be clearly marked and I won’t fault you for skipping forward, after all you’ve got a Nation to run and World Wars to fight.

Didn’t skip forward? You sure? Cool. Let’s take a look at the problems in the current system and break down each of them:
  1. Method for comparing the Offensive and Defensive Ratings of a base
  2. Sandbagging/Imbalance in Alliance compositions
  3. The Leaderboard was not part of the evaluation
  4. Queue functionality
  5. Is this match fair to my Alliance?
  6. Sparse queues
Method for Comparing the Offensive and Defensive Ratings of a Base

Did you know that the matchmaking system creates a Rating of your base’s offensive and defensive capabilities? It does. Everything on your base contributes to these Ratings. However, the method that the old system (the current matching system on live) used to compare the final Ratings with other bases wasn’t a great predictor of win rate. There was some directional value in its ability to predict win rate. But, we knew this was an area where we could improve match quality.

That old system compared the absolute value of your Alliance’s Offensive/Defensive (O/D) Ratings with other potential Alliances. We ran some tests internally and discovered that when we match based on a percentage delta between those same Alliance’s O/D Ratings, we actually get a much stronger predictor for winning. So, we’re updating the system to compare Ratings by the percentage delta rather than absolute values.

Fun fact: If your Alliance has a ~10% higher Defensive Rating in the new system, your Alliance’s chances of winning are increased to 60% (Yes, defense matters. Start upgrading your neglected Catapults!).
Additional fun fact: The new system would not consider a 10% delta a “good” match. So don’t worry.

Sandbagging/Imbalance in Alliance Compositions

As with other team-based games, there is an ability to “carry” in DomiNations. If we only checked the percentage delta between two Alliance’s O/D Ratings, we may end up making the following match:
Alliance A: 5 Cold War, 5 Medieval
Alliance B: 1 Cold War with a mix of Enlightenment+.
Seem familiar? I’m sure all of us have been in Alliance B’s position before. The O/D Ratings may look fair, but this is not a fair fight; Alliance B is going to lose this War.

We don’t want to limit Alliances to bringing specific compositions to War. Sometimes you just don’t have enough players at close Ages. But we do want to address the sandbagging issue. So, we’re adding a second comparison to the Ratings formula. The new system now also compares the delta within an Alliance and tries to find other Alliances with similar deltas. The result? The system now knows that Alliance A and B are wildly inappropriate to match. It’ll find Alliance A another Alliance with a large Ratings delta in its players while protecting B from an unfair fight.

Ready for another fun fact?: If your Alliance is sandbagging in the new system, it may result in increased queue times as the system waits to find another sandbagging Alliance for you.

The Leaderboard was not Part of the Evaluation

The old system did not use Glory as part of its calculation. In order to preserve the competitive integrity of our Leaderboards, we felt that it was important to start using Glory. In order to do that, Glory gained/lost needed to be updated.

We had several discussions about potential directions to take Glory and what systems to use.
A system like Elo (initially popularized as a chess ranking system) provides a single rating that can be used to predict the winner between opponents. Since Glory is serving as our Leaderboards, using a system that would suggest the higher ranking Alliances are more likely to win made sense. Afterall, if you’re Rank 1 on the Leaderboards, you should have earned it. Elo also does some other great things for Glory like awarding you more Glory if you happen to beat an Alliance that is higher ranked than you. That was a tough War and you deserve it.

Elo is somewhat of a barebones rating system, however. Even chess has been modifying parts of the formula based on player ratings. So, we also considered more robust systems like Glicko-2, a more complicated evolution on the Elo formula that introduces a confidence interval. This confidence interval is a way for the system to determine how accurate it thinks an Alliance’s rating is. The interval can then modify points gained/lost in the system based on this confidence. For example, if your Alliance is 20-0 in War and the system says you have a rating of 18,000 Glory, what are the odds that your Alliance is probably better than 18,000? Probably high. Glicko-2 would take this into account and likely accelerate your subsequent Glory gained.

While more complex systems may have offered some extra bells and whistles, we ultimately decided that we could get most of what was important to DomiNations by using a modified version of Elo we’re calling DOM MMR (DomiNations Matchmaking Rating). It’s pretty close to standard Elo with some targeted modifications to K-factor and streaking.

It’s a bit of an oversimplification, but you can think of K-factor as a constant in the Elo formula that sets a maximum on the number of points that can be won or lost from a given match. There’s a handful of problems with this value being constant. Even chess has been modifying the K-factor of players based on their rating.

So what modifications are we making? DOM MMR will dynamically change an Alliance’s K-factor based on the following:
  1. New Alliance - New combatants in any ranking system are hard to properly evaluate. It is near impossible to properly evaluate a new Alliance with only 2 Wars compared to an Alliance that has 300 under their belt. Therefore, new Alliances will have a higher K-factor for their first several Wars. The result should help get sharks out of the kiddie pool and help new Alliances find a win faster.
  2. Streaking - Winning or losing several Wars in a row will also cause the system to modify your K-factor. Sometimes you’re just not as good as you used to be, you know? Or maybe your Alliance recruited some top-tier players and it’s time to step into the big leagues.
  3. High Glory Alliances - K-factor modifications sound great but K-factor can break down at the extreme ends of a ranking system. Therefore, Alliances with particularly high Glory will have their K-factor treated differently. The main difference is that streaking won’t apply here.
Now that we’re modifying how Glory works, we’re going to need to crunch the Leaderboards back toward 12,000 Glory. When the new matchmaking system goes live, all Alliances will be pulled toward 12,000 Glory (the strength of the pull being relative to your Alliance’s current Glory). This shouldn’t affect standings much. But, it will bring everyone closer together in Glory for a fresh start.

Think of it as a season reset.

Queue Functionality

The old queue would search the list of available Alliances at the moment your Alliance entered queue. If you were not matched, you were then put in the bucket for other Alliances to potentially pair with when they entered queue.

In theory this system sounds fine. And it worked okay for the most part. But, this created a rigid queue. Your potential pool of matches never expanded except when other Alliances entered the queue, regardless of how long you waited.

This is changing in the new system to better incorporate industry best practices of time vs. match quality. The result is a system that is constantly evaluating the entire pool of Alliances against each other. The longer you wait, the less restrictive the system will be.

Is this Match Fair to My Alliance?

Some of you that are Alliance leaders may have noticed that your Alliance would sometimes instantly get matched with what appeared to be a poor match. What gives? Why? To the forums! The matchmaking is awful! Well, not quite. This was because the old system didn’t have a concept of handshaking. It compared Ratings at the time your Alliance entered queue and if they were within an acceptable range of another Alliance, bam. Matched.

Moving forward, the new system will require the match to be acceptable from the perspective of both Alliances. This isn’t an in-game option or agreement that is made. Instead, you can think of it as the system viewing the potential match from both Alliance’s perspective to see if it thinks it is fair.

What is fair to my Alliance after being in queue for only 10 seconds may be different from what another Alliance is looking for after 5 minutes.

Sparse Queues

I know that this will be one of the least popular changes. I apologize in advance for killing your Alliance’s favorite queue. The reality is that DomiNations can’t—and has never truly been able to—support seven different queues. Think of it this way: Each queue added to DomiNations splits the player base. Each of those splits greatly reduces the number of potential matches your Alliance can make and reduces the quality of those matches. Theoretically, the most optimal system would be to force everyone into a single XvX queue. We can’t really do that and aren’t entertaining the idea. But four is better than seven.

We’re reducing the queues to the following:
10v10
20v20
30v30
50v50

We feel that this division of queues will flow more Alliances into more pools and increase the odds of finding your Alliance a better match. It should be noted that 50v50 is not and has never been a popular queue. But, we felt it was important to preserve 50v50 for community events like the Continental World Cup (https://continentalwar.wordpress.com/). If you find your Alliance jumping into 50v50 expect long wait times and extremely relaxed matching requirements.

Conclusion (“What are the Changes?” tl;dr is here!)

In short, this is a complete overhaul to World War matchmaking. Standard PvP battle matching is not being touched. The changes are as follows:
  1. The matching logic has changed how it compares base power between Alliances. The new logic is proving to be a strong predictor of win rate and should allow us to make more fair Wars.
  2. The matching logic now understands imbalances within an Alliance’s composition. It understands “sandbagging” and will try to find Alliances opponents with similar compositions.
  3. How Glory is calculated has been redesigned.
  4. All Alliances are being pulled back toward 12,000 Glory when the update goes live. The pull is relative based on your Alliance’s Glory and it shouldn’t affect the standings much. Think of it as a season reset.
  5. Reduced the number of queues from seven to four:
    1. 10v10, 20v20, 30v30 and 50v50
  6. The behind-the-scenes functionality of the queue itself is changing to constantly evaluate the entire pool of Alliances in queue.
  7. The longer you wait in the queue, the more relaxed your matching requirements become.
Finally, the entire system has been built with iteration in mind. Every piece of the system I explained has a knob that can be turned. We don’t expect the system to be perfect when it first launches but we’ll be actively studying it and tweaking behind the scenes to improve the quality of your Wars. We built these changes based on a lot of data from our internal systems and tools, but a major factor comes through player comments and discussions from all of our community pages. We'd love to get more feedback on this blog and the upcoming changes, so let us know what you think!

Thanks for reading!
- Muet
 

Berend_War

Approved user
Joined
Jul 16, 2018
Messages
179
Bin reading the notes and looks like a good start. Do think the new system needs some time to settle how ever..
 

LordStark263AC

Approved user
Joined
Oct 22, 2017
Messages
870
Now if you could just give a preview of what are your plans towards getting rid of hackers with unlimited citizens, that would be great.
 

BHG_Muet

Design Lead
Joined
Nov 12, 2018
Messages
72
Oh, absolutely.

While the initial launch of the system is tuned to historical data, I fully expect it to create some sub optimal matches. In preparing for the launch of this system, it was very important to make the matching logic easy to tune. We'll be actively collecting data and tuning as Alliances wage war and we observe the quality of those wars.
 

Omegaman

Approved user
Joined
Apr 5, 2016
Messages
69
BHG_Muet
We keep our fingers crossed that this will introduce fairer matching. But in all honesty WW is completely broken until you address the cheating.
Would you not agree having 5 teams out of the top 10 are Cheat Alliances is rather embarrassing ?
Restoring the integrity of the Glory system can only be achieved by culling these teams that have been endlessly reported by the community. #1 has been using hacks for over a year.
Pink Unicorn alliances popping up with every player level 332 at every turn. What is the point of a Glory System if you can lose two months gains overnight against these scum ?
I believe in a few weeks time the Top 10 will be completely filled with Garuda Sakti, Top Killers,Romania Stars,Swift/Fatality,Ardor TW and the other tossers.

So please if WW matchmaking work is concluded direct your attention to sorting out the cheats once and for all.

One further question..will the new system allow CS to modify an alliances glory if it has been matched with a cheating team ?

PS, But first fix the AR, it must be killing your revenue stream currently,as nobody in their right mind spending on the game while it is so hopelessly bodged
 
Last edited:

BeerMan

Approved user
Joined
May 10, 2017
Messages
415
Will the glory payout calculation be fixed/tweaked in the new system? In the current system, you could face a reboot of a top 10 alliance and lose 1000 glory simply because you have more glory than they do. That, along with cheating alliances not being punished, are the main issues with the current system IMO.
 

Prodigal Clint

Approved user
Joined
Apr 17, 2016
Messages
129
So I’ll ask the obvious question.... how do you define a sandbag in the new matchmaking system?
 

BHG_Muet

Design Lead
Joined
Nov 12, 2018
Messages
72
Part of the reason this was happening before is that Glory wasn't part of the matching logic. It is now. While these matches might still occur if the system can't find the two Alliances a better match, it'll try to find you a match close to your Glory first.

So, you shouldn't really see too many matches created where there is a major delta in the Alliance's Glory.
 

oddin

Approved user
Joined
May 17, 2018
Messages
1,586
So, we’re adding a second comparison to the Ratings formula. The new system now also compares the delta within an Alliance and tries to find other Alliances with similar deltas. The result? The system now knows that Alliance A and B are wildly inappropriate to match. It’ll find Alliance A another Alliance with a large Ratings delta in its players while protecting B from an unfair fight.
 

helix900

Approved user
Joined
Dec 23, 2015
Messages
327
Thanks for the information and thanks for trying to improve the game. Please test this update before releasing that’s all Im asking. Looking forward to see fair wars.
 

turmaffe

Approved user
Joined
Jul 17, 2015
Messages
31
Sounds really good and thought out! Let's see how these changes perform. One thing though:

7. The longer you wait in the queue, the more relaxed your matching requirements become.

This was in my opinion a problem in the old system. A Leader searching for a good match had to constantly quit and restart the search. This leads to less alliances in the pool, reducing match quality. How will this practice work in the new system?
Wouldn't it be a good idea to reduce preparation day to 22h and adding 1h minimum search time to accumulate a bigger pool for the system to choose from? Maybe even match only every full hour?
 

QuébecGlory

Approved user
Joined
Jul 22, 2016
Messages
149
If pourcentage delta is the the same as standard deviation, it will finally be the end of what is called sandbagging. I calculated standard deviation basically from the start of world wars and at 99% accuracy, the highest number always win. At least it matches same level players so that skill rise to the top.

Standard deviation is basically the difference between your tops, your bottoms, compare to your overall average. To understand, if you have 10 max SA player and 10 iron age, you would have the highest std dev, and you could have been matched with 20 Global players, which would be the worst std dev (all players same level). Standard deviation is the greatest indicator of win/loss. Matching std dev will result in close exciting matches for all.

Elo, or glory points will not be a good indicator on the other hand. An alliance with 10 crazy good Atomic players who never lose, can't be matched by rank with 10 SA players. The leaderboard will be a race to who can recruit more max SA players. It should be about skill compared to your level. If you want to separate leaderboards into a Spage age leaderboard, an Atomic leaderboard, etc... where your top player is of that level, that's fine, but you can't put SA and AA in competition just because they're both high on the leaderboard.
 

BHG_Muet

Design Lead
Joined
Nov 12, 2018
Messages
72
It looks at all members the Alliance is bringing to the War and does a calculation to determine the delta in their Offensive and Defensive Ratings to create a Variance Rating. The matching system then uses that Variance Rating as an additional window to compare against other Alliances and tries to find one with a similar rating.
 

BHG_Muet

Design Lead
Joined
Nov 12, 2018
Messages
72
It depends on what you value as a leader. Most every matching system is a battle between finding a quality match and letting players play. I will say that the new system is tuned in a way that understands the impact of pairing for a War that eats multiple days of your time. Typically PvP games are trying to match you within seconds. Since the War queue in DomiNations is passive and the Wars are long, we have *much* larger windows before match quality starts to degrade.

So, don't feel pressure to queue and cancel queue if you don't find a match in the first few minutes. Your search is the same in that window.
 

sulbin

Approved user
Joined
Jan 25, 2017
Messages
18
I read your long article well and look forward to the change of matching system in the future.

I wrote comments in your last article, but I don't know if you remember.
Let me introduce myself again. I am a user who is currently using ID SuLBiN, the leader of 3rd place ARES.

1. The point about your matching system ELO is very interesting.
I have been actively suggesting to the management and development team the introduction of the ELO system.

Your example tells of a alliance that is defeated helplessly by too wide a gap.
But you know what?
It is also very boring and annoying for the upper alliance.
We want to have a better World War.
But in the existing system, it's not easy to do that quality World War. (I've been waiting for more than eight hours now. LoL)
So I like your debate very much.

But there's one thing you're overlooking.
Did you know that there are two cheater guilds in the top 10 in honor?
How many are there in the top 100?
Users ask countless report, but they're not being processed.
All the top alliance were damaged.
But you don't listen to that question at all.
The first time I reported them was last October and it didn't work.
It's meaningless to change the match system in this situation.

For a fair game, such bugs, cheater alliance, need active sanctions against alliances beyond individuals.
Can you do that? This is a very important matter.
You need to understand the helplessness of normal alliances.
Such fairness is essential to the introduction of the ELO systems.


2. Do you know why the top-ranked alliance put in the sandbag?
In the current system, matching is determined by level, so without a sandbag, matching is not formed.
So I have to play a game with a sandbag.
But you don't have a definite answer to that in your words.
You're just saying it's taking a long time.
I want you to be clear.

Does the new system allow you to match without sandbags?
Or are you still going to need a sandbag for the level?

If it's the same as it is now, the new system you're talking about is useless.


3. I heard that you will reset your glory score to 12000. So, are you ready to compensate for the glory score you have earned?
We simply lose what we've built up by changing your matching system.
Are you preparing to make amends for it?

I want to introduce a league system if this matching system changes.
I hope there's a reward for that when the league is over.
The current World War does not have a goal.
The reward is too salty.
Users need something to focus on.

I'd like to say more, but I'd like to make sure I'm asking these three questions.
Don't selectively reply to either one, but think and answer clearly as a whole.
 
Last edited:

BHG_Muet

Design Lead
Joined
Nov 12, 2018
Messages
72
So, there's been a few questions crop up regarding cheating. I don't have a lot to share right now (sorry). But know that we are looking into the issue and are working through how we want to address it. As with anything, I don't like band-aid solutions to these types of problems and I want to make sure we're taking the correct steps. Hopefully the deep dive on the matchmaking overhaul is a good demonstration of that. We'll certainly message it loud and clear when we have more to share.
 

Ch0s3nByG0dz

Approved user
Joined
Mar 19, 2019
Messages
58
Before the new matchmaking system is released:

- Everyone is excited and cant wait.

After it is released:

- You broke the game, bring back the old matchmaking system

#SameThingsAllOverAgain. :D
 

BHG_Muet

Design Lead
Joined
Nov 12, 2018
Messages
72
Hi sulban. See my post below regarding the topic of cheating. For your other questions:

One of the pitfalls of the old system is that it didn’t take time in queue into account. Your Alliance could have been waiting for six hours and it wouldn’t have mattered for how it was attempting to match you. The new system is a dance between time in queue and match quality. As your Alliance waits longer in queue, the range of acceptable opponents widens. While the result may be a lower quality match, one will be made if you wait long enough. So, to answer you directly, yes. The new system will eventually match you without adding sandbags (even if it can’t find another perfectly paired Alliance).

The Glory reset isn’t a reset as much as it is a pull toward 12,000. The formula will pull harder the further away from 12,000 an Alliance is, but position to other Alliances shouldn’t really change much, if at all. If an Alliance had less Glory than you before, they’ll have less than you after. I did an exercise when designing the new system and the current top 100 leaderboard should end up somewhere between 15,000 and mid 17,000s.
 
Top