Community Question: Measuring Player Skill

Christophicus · 2012-07-23 07:41:24 UTC

My initial instinct is to vote for “Number of Objectives Completed per Match”, however you do have to consider those players who are supplying/healing the team in order to maintain a sustained assault. In that regard, one would hope that the scoring system would suitably reward those types of players, and so I would suggest that the score per minute option would be best, but only when designed incredibly well. People will of course find some way to abuse it in one way/shape/form.

tokamak · 2012-07-23 12:00:21 UTC

If you’re rewarding a player with doing one thing and praising him to do another then you’re admitting that you have no idea what you really want the player to do. If they’re not the same then at least one of them will be a perverse incentive.

Why?

XP is the only thing that allows us to quantify a combination of valuable things. Right now it feels like I’m arguing the merits of currency to a tribe that is still into bartering. XP allows us for the optimal level of detail in which contribution can be measured without losing track of the bigger picture. Like ELO modifies the W/L rating and makes it a more useful indicator, so does XP allow us to quantify all the different actions and express it one number that then allows us to enhance it further by modifying it.

This is a great position to create to backpedal to later. If I come up with any good/better indicators then you can grab them and say they should support your thing all along.

You’re not going to come up with a better indicator I’m afraid. But besides that, you miss apprehend me. I’m saying that any other indicator ALREADY ought to contribute to a higher xp/min under this system. Accuracy, boydcyount, k/d, battlesense, laps around the Valley lake, they’re all useless if they don’t contribute to the match, and the only way to asses whether or not they actually contribute is when a player uses these competencies to raise his xp. If the xp doesn’t get raised then we’re looking at situations where a sniper guy is constantly hitting an enemy deployable with his sniper rifle while an engineer is repairing it. Incredible for boosting his hit ratio, but all in all a completely futile enterprise which of course leads to a lower xp/min, doubly so under my model where he’s lowering his chances of getting the bonus through winning as well as ignoring the hot engineer repairing that thing.

Yeah, I get this which is why I mentioned some in long form in an earlier post. However, your claim that XP is the sweet spot and that its actions are tangible are unsupported. There have been pages of examples of how these actions are not remotely able to be measured well because the players intentions are unknowable.

If you find that I failed to address any of those examples then by all means point them out because I believe I went through them all. But I can address it again:

Intention itself is a completely worthless factor. Something either contributes to winning or it doesn’t. Whatever the player intended in the first place doesn’t matter. It could be that a player unintentionally made a huge difference while statpadding and it could be that a player with the best intentions was completely worthless or even detrimental to his team (I think we’ve all seen THAT airstrike over the objective at one point or another in our gaming careers).

Intention ultimately doesn’t matter. What matters is the outcome.

The (ETQW or Brink with its bias adjusted) system is already complete enough in all the interactions it rewards that you can say that if something doesn’t lead to more xp, then it’s likely not going to contribute to the match. I say likely because of course it’s still possible. You mention that later on and I’ll get back to that. What makes the whole thing tons more accurate is that at the end of the match your way of playing gets retro-actively adjusted by that outcome modifier. It means that whatever you did during that match gets reviewed again on how effective it actually was and you get the modifier applied according to whether you won or lost based on the team’s skill difference and the side of the map you were playing for.

Hindsight is such a splendid luxury in this case and it’s a rare opportunity we get to use it so easily. In many other fields this is not possible, but in game development, we can.

I’d also like to see how you think an XP system can be made to address the point made by an SD member earlier that any system which doesn’t evaluate the quality of players on the server probably isn’t very good.

This question is problematic as it leads to the schism I mentioned earlier. How you define skill and how you go about quantifying it are two different discussions. DAUK thinks that skill is being able to consistently make headshots and making beautiful trickjumps into hard to reach places and a whole range of impressive but necessarily outcome-related feats. Like I said, some appear to have the value as the laps around the Valley lake, impressive but useless.

I define skill as doing whatever is the best thing you can do in order to win. This means that no matter how impressive and skilful an action is, if it doesn’t bring your team closer to a win, it should be discounted or even punished.

When does an action bring you closer to a team? Whenever it leads to more favourable player/player and player/objective interaction. Why these interactions? Because these interactions are often the most valuable things you can do in order to win a match. What about the few cases they aren’t favourable to win the match? Those cases are discouraged by the outcome modifier. If a player finds a juicy opportunity to rack up his xp in the game but by doing so would jeopardise his chances of winning and thus see his overall xp drop by 20% (10% from not winning and -10% for losing) then he’ll leave that opportunity to statpad and go for the win as that’s ultimately more beneficial for his score.

I’m sure you’re trying to articulate a very lovely belief here in your perfect magical system but it just isn’t coming across.

If you could point out which part relies on ‘magic’ then I’d be grateful.

I don’t get this. Why wouldn’t it just take longer for a players elo to stabilize at its correct value? (and I thought elo was near entirely based on W/L) You are the only consistent feature in all your games (assuming a typical ranked server system enforcing standards maps/modes) and with enough games played the uncertainty on the elo will decrease.

Even ELO is an approximate of the player’s worth because a player never gets to play against all players in the pool. In team-games this becomes even less accurate because the amount of team combinations shoots up exponentially. It doesn’t just take 8 times longer for Brink and 12 times longer for ETQW (which are already HUGE numbers) to get to the same significance as a 1v1 game. It takes A LOT more than that: It’s a bit more nuanced than I’m going to put it now (it’s mutual exclusive rather than mutual inclusive probability) but for the sake of not letting the discussion devolve in too much maths it’s suffice to say that an 8v8 takes to the 8th power longer and a 12v12 takes to the 12th power longer to get to the same significance as a 1v1.

This already accounts for that fact that in SC2, a couple of my bronze/silver league friends play in the 3v3 platinum and 5v5 diamond league. The significance of the ELO simply takes a nosedive when you go beyond 1v1. It would eventually stabilise after an enormous amounts of matches but I fear that would take several lifetimes.

We can’t wait for that so we need to asses a player’s worth per match. Xp allows us to do that, and the ELO serves us well enough to make the outcome modifier more accurate which in turn raises the value of the player’s xp/min. But it’s just not significant enough for it to be used for anything else or even to be displayed at all. It simply takes far too many (lifetimes of) matches before it becomes useful for that.

XP systems have other uses beyond measuring skill. They’re a reward system, a tutorial system, a progression/unlock system. Most players will get more value/information out of a scrolling list of XP rewards at the bottom of their screen than an incomprehensible system that will (to them) seem to arbitrarily hand out high and low scores. And can you imagine the forum posts?

Players don’t need to know all the algorithms and on the surface the presentation of these things can be deeply simplified and instinctive. And I think the game can help a hand by giving visual cues. Think your cross-hair colouring warm or cold depending on how ‘hot’ the player or object you’re about to interact with.

But that’s not even necessary. Just a tooltip saying ‘you receive more xp from helping active players or killing active opponents’ is already sufficient enough because even without visual cues a player can already know that the ‘hot’ interactions (okay this is starting to sound raunchy now) are in the thick of the fray.

The average team ratings are hidden, the game only says ‘favoured, slightly favoured, equal’ and the outcome modifier is a single percentage bonus at the end of the match. Really nothing too complex.

Better to keep XP simple and easy to understand and banish the skill rating to a small corner of the online stats site since it doesn’t actually affect the game in any way, its just a measurement.

Oh agreed, the online stats site can go into more detail. During the gaming sessions all you need is the xp/min of that match, the team favour and the outcome modifier (and a separate tab with individual rewards because people simply love that).

I will address the rest of the comments later on. There’s a few things I haven’t addressed yet.

DarkangelUK · 2012-07-23 12:26:29 UTC

That whole post shows a severe lack of understanding that it’s no wonder you can’t grasp that it won’t work. If you think that that’s my thoughts on the subject then you’re way off base. If you can’t even grasp the basics, then you’ll never grasp the complexities.

edxot · 2012-07-23 13:04:49 UTC

you will never get it right

stop this thread and make 12 new maps for etqw

after, sell the game again, like a new one.

dont even reply to this, time is being wasted.

Chris2 · 2012-07-23 13:22:20 UTC

Thought we were measuring skill here though?

In one of the last matches I played in ET:QW, I placed a plasma charge opposite to where the GDF place an HE charge on the second stage of the map. I timed it so there was some chance it would take out some people in the event of us getting overrun and losing the room, I did this based on spawn times and estimate time it takes for them to reach the objective. I killed 2 soliders before they planted and a supporting medic. If someone just did the same in a match but was just dicking around hoping for some teamkills your system would score us equally.

Chris2 · 2012-07-23 13:28:48 UTC

In one of the last matches I played in ET:QW, I placed a plasma charge opposite to where the GDF place an HE charge on the second stage of the map. I timed it so there was some chance it would take out some people in the chance of us getting overrun and losing the room, I did this based on spawn times and estimate time it takes for them to reach the objective. I killed 2 solider and a supporting medic. If someone just did the same in a match but was just dicking around hoping for some teamkills your system would score us equally.

tokamak · 2012-07-23 13:30:06 UTC

Now ask yourself why exactly we want players to use their class abilities.

I didn’t understand this.

Brink’s xp distribution had a strong bias to teamwork and buffing. The reasoning was that combat happened anyway and therefore didn’t need to be rewarded as much. It did result in people using their team abilities but it went so far that people just kept buffing everyone indiscriminately regardless of whether it was the best thing to do at that moment. In turn this lead to a highly unreliable scoreboard.

Sports and economics are filled with cases like these.
http://www2.lse.ac.uk/newsAndMedia/videoAndAudio/channels/bigIdeas/player.aspx?id=403

This is my field of study (at least the part I actually like). We constantly have to deal with policy and the way it conducts society. What makes me so passionate about game design is that the mechanics you implement are of extremely high fidelity. Whereas real life policies are always merely abstractions that already get stuck at implementation, a game mechanic always absolutely one-on-one. No bureaucracy or transaction costs just exact algorithms always doing what they’re supposed to do without noise and without corruption.

If only public institutes and academics would embrace gaming more and test their ideas through simulations like these first then there would be far less mess-ups.

Pretty sure BF3 already does something similar to this for W/L. However, once again you have neglected to tell us how this XP system accounts for different games and servers will have different qualities of player in them.

It doesn’t account for different game modes. Thanks to the /min and the map adjustment the objective and stopwatch become very similar. But as for the rest I have no idea what SD has up their sleeve right now so I’m restricted to apply this theoretical framework to their previous games. The game does account for different quality of players by comparing the average team-ratings. Come on please try to make myself repeat myself a bit less. I have ample patience but it only causes more words for people to skip over and make more assertions about what they think they just read.

I see a problem here in that XP/min is used to measure skill, will be used as a multiplier to further reward and feedback on skill, but that XP earned in a match is uncapped and not zero-sum. Some games will naturally have higher XP totals and some lower but this will have absolutely no bearing on actual skill displayed in the match, yet XP/Min is treated as this huge incredible thing that both says how good a player is and what he deserves.

If I get this right then your concern is that players that play longer in a match are favoured by having more upgrades than players who recently join in right?

It’s a valid concern, but I don’t see how this is different for any alternative. XP/min actually goes a long way in offsetting the bias while many alternatives do not.

Let’s start with the advantage itself. This is an actual gameplay problem that is inherent to the way players acquire upgrades though a campaign. In another thread (and I mentioned it here as well) I proposed the idea of having three layers character development rather than one. I’m not going into this further now because that would be a different discussion entirely but it does seek out to lower that advantage without taking too much away from the path-dependency that adds so much depth to the game.

Now XP/min offsets the disadvantage through the following: The later on a player enters in on a match, the bigger his disadvantage but also the shorter the time left to take part in the match. In other words, the bigger the disadvantage the less weight it has in your overall score because the match is about to end sooner.

But yeah, I don’t see how any other skill indicator doesn’t suffer from this given equally or even worse.

[QUOTE=.Chris.;403518]Thought we were measuring skill here though?

In one of the last matches I played in ET:QW, I placed a plasma charge opposite to where the GDF place an HE charge on the second stage of the map. I timed it so there was some chance it would take out some people in the chance of us getting overrun and losing the room, I did this based on spawn times and estimate time it takes for them to reach the objective. I killed 2 soliders before they planted and a supporting medic. If someone just did the same in a match but was just dicking around hoping for some teamkills your system would score us equally.[/QUOTE]

Yes it would, because the outcome is the same. I don’t even see how such a thing like intention could even be measured. However, once you start repeating this situation then the difference between your insight and his lucky malice will grow drastically. He’ll score more team-kills ruining his xp/min and you will score more kill bonuses on top of your objective completion.

What your example includes however, is on top of mere prestige there’s also a tangible benefit of scoring kills along your destroyed objective. That’s something the xp/min system rewards.

What I think a system should not reward is prestige without any merit in the outcome of the match. And that’s something a few players here object to.

It’s a shame I failed to convey why exclusive W/L is utterly worthless but I hope my reply to Senethro clarifies it more.

What were you referring to then?

Actually it isnt necessary. The direct experience of it is enough, with or without a reaction from the players that got owned.

Right now you’re arguing for no indication whatsoever. Maybe that’s enough for you. But people like me like recognition and want no ambiguity of what they contributed to their team, they want it expressed in an illustrative number for everyone to see. Feats of strength become so much sweeter when they have an audience.

Again this is my point. One gets excited over the best medic award, another is aroused over the most objectives completed category, another moans at battlesense ,etc

Sure, but the goal of the game is not to be the best medic or to complete the most objectives. The goal of the game is to win. And the only way to reconcile the value of all these different ways a player can contribute is xp/min. Then the individual awards are still adorable enough to keep in a separate tab. Besides, it’s perfectly possible for an exceptional medic to score best medic, highest xp, most objectives completed and most kills if he was really up to speed. In the end it’s the xp/min that defines all the rest, and that still leaves us room for individual awards, just with less emphasis on them.

Creating a single score which encompasses every “legitimate” aspect of the game, is not going to prompt the k/d whore or the KPM player to concede that the player xyz with the highest score is the best player. The game might, but the players wont.

I have the following goal in my head:

The moment a player pursuing the skill indicator leads him to contribute to a victory in the best way he can, and the moment a player contributing the best way he can leads to being recognised by the skill indicator, that’s when this mission is accomplished.

And sure there’s nothing stopping from people to keep on pursuing their k/d ratio or the most laps around the Valley lake. But as long as those feats aren’t publicly displayed but rather kept in their personal private file then it becomes stale pretty fast as there’s nothing for them to brag about.

The player thats highest on that leaderboard matters to the degree players buy into the concept of XP and the formula that was used. The moment anything in that formula warrants scrutiny, they discard it completely and view it as “just a number”. The solution is to not actually advertise is as a “skill score”

I think that merely being the best option available is enough to warrant it as ‘the’ skill indicator. That leaves room for scrutiny but as long as no better option is presented then that scrutiny doesn’t really hold much ground.

Again I have no intention of misconstruing what you say. From your post I gather that the prestige of certain ways of play counts more than the way they contribute towards winning. If that’s not the case then chances are we’re actually in agreement.

tokamak · 2012-07-23 13:32:58 UTC