Community Question: Measuring Player Skill


(MoonOnAStick) #221

[QUOTE=DarkangelUK;403406]Either the XP farmer is considered skilful and raised up the ranks, or the awesome tactician is considered a farmer and held down the ranks. Which one is it? Lemme guess, more modifiers?[/QUOTE]I think the all-seeing XP system knows which deployables are the ‘right’ ones.

What tokamak has been describing doesn’t seem impossible, just a little… complicated. As INF3RN0 points out, the only outcome you can really measure is W/L ratio. It’s the only thing that everyone (well >90%) on a server is striving for. Then try to model the probability of a win against all the independent variables you think might be important (linear regression maybe?)

Skipping over the issue of controlling for: team size, team experience, team class make-up, map choice, …
and looking at the W/L ratios for all Engineers: ‘Is your team more likely to win if you repair deployables than if you don’t?’ We could perhaps look at time spent repairing against W/L ratio. In tokamak’s model I think you break it down further: ‘Is your team more likely to win if you repair APTs in this designated hot zone than if you don’t?’ You would quickly end up with a large number of variables which I imagine could be whittled down after checking for correlation with W/L percentage.

Having collected the data and fitted a model you could assess a player’s performance against those variables which were strongly correlated with W/L ratio. I’m sure the weighting for kills-per-minute/damage-dealt-in-hot-zones/whatever would be high in any fitted model. Maybe the most effective strategies change over time (or at different skill levels) and the model has to be updated.

Or you could just sketch out an XP system on the back of a cigarette packet. If the actions prioritized help contribute to a fun game that’s cool too. :wink:


(Humate) #222

If a game truly disintegrates at high levels to such a point that everything but combat becomes irrelevant then there’s something seriously wrong with the mechanics.

Everything isnt about combat, everything is about area control - combat happens to have the biggest impact in area control. When we are talking specifically about skill though, thats when everything is about combat.

As for there being something seriously wrong with the game, we saw the one eyed approach which was taken in Brink. ET and ETQW at least are inclusive of all levels of play.


(INF3RN0) #223

I think a very straightforward system can consistently have better distribution of XP via what I might call a domino effect- that multiplies reward based on the repercussions of an action rather than just the action itself. Not to mention it should encourage players to think their actions through, rather than just perform them mindlessly, as well as furthering reward based on measurable consequences. Here are some random scenarios I can think of which can be applied across the board under the same concept;

-Medic revives player, player outputs damage, medic receives XP multiplier based on damage output of player in X time frame.
-Medic revives player, player completes/disarms objective, medic receives XP multiplier based on % output on objective in X time frame.
-Engineer repairs friendly turret, turret outputs damage, player receives XP multiplier based on output in X time frame.
-XP multiplier for kills based on distance from objective and enemy control of map zones (map control needs to be taken into account).
-XP multiplier for kills that are gibbed.
-XP multiplier for extended defensive/fast offensive times per objective (under 5min completion=bonus XP, 5min intervals of defense per obj=bonus XP).
-XP multiplier for killing objective classes.
-XP multiplier for kills based on opponent’s cumulative XP.
-XP bonus for overall win.

The basic rules to the system would be that every action gives XP regardless, but the combination of variables present during the action as well as the measurable results of the action would grant much more XP in return. For example; a player can get X number of kills and get some base XP, but once the downed enemies are gibbed, the result is an XP bonus per kill. Furthermore, if the kills occur at the objective or the closest contested zone, the XP will be further multiplied. If the enemies killed are obj class, then the kill XP will be further multiplied. If the objective is completed faster than the nominal time or if the objective is being defended over the nominal time, the XP of every player will receive an additional multiplier.

This type of system would promote an ultimate goal of objective focus while enforcing players to think their actions through, as actions that actually produce positive results will increase in XP reward. The results of these actions being directly tied into the overall progression of the game and instigating a ripple effect of reward when teamwork is present.


(Humate) #224

Sniper lines up defuser, gets counter-sniped before shot is fired.
FieldOP randomly throws airstrike - kills entire team earns awesome xp
FieldOP selects his airstrike but gets tripled before he gets chance to throw it. Airstrike would have killed planter, possibly entire team :wink:
Soldier scratches himself, while planting a charge in a hallway. Databrain player stops at threat of it blowing up. Gets backraged and drops the objective.
Hyper player and technican become best friends on the last objective of sewer. Hyper spams the sewer entrance doing zero damage while tech feeds stroyent . GDF get too scared to move in, and lose the game.
Tokamak throws smoke grenade


(Senethro) #225

I wouldn’t play any game that didn’t reward me with precisely the XP I deserve in each of those circumstances. I now require you to make a table of the XP awards and the formulae used to derive them.


(Senethro) #226

Its an FPS, I’m not sure that shooting being the most important thing is anything less than a feature. There might be room for a dedicated supporter in a class based RPG but in most FPS every player is a rifleman first and their class second. Theres also the issue that being good at your class is very easy but being good at combat is very hard.


(DarkangelUK) #227

I don’t think you quite got my example, we’re not discussing the deployables, we’re discussing the users ‘skill level’ or intention that decided he should repair those deployables, and using XP to determine the skill level involved in that decision is flawed. Two different players doing the same thing, 1 because he realises the strategic value of those, another because he spots an XP hotspot and camps them to rack up some points. At the end of the day, it was as simple as walking up to the turret and holding X

I’m also curious as to just how varied we’re meant to be determining skill as well. Take movement for example, in games like W:ET there were those that honed their movement skills to a point that could completely sway a match. On Fueldump as Allies once I managed to spawn at the start as engineer, the sewer gate was blown open. I snuck along there, trickjumped into the fuel depot, planted the dyno and won the round… without ever firing my gun. As XP dictated, that was an absolutely terrible performance and I was considered the worst on the server.


(INF3RN0) #228

Up to that point you were the worst player :p.

And… LOL @ Humate, sh1ts da truth!


(DarkangelUK) #229

Had I given a damn about XP, or there was a completely unnecessary and overly convoluted system in place that forced me to care about it then I wouldn’t have bothered attempting it :tongue:


(INF3RN0) #230

That’s why winning needs to give the most XP! I personally don’t need any XP or any of that crap to motivate how I play, but I do know that 99% of the 12yr old kids that power the gaming market do unfortunately :/.


(DarkangelUK) #231

Yeah indeed, but the other point I was making was that it was skilful movement that won that match. Probably a small percentage of players have the ability or skill to jump into the fueldump with repeated succession, but an XP system doesn’t take that into account when calculating a players skill level.


(.Chris.) #232

I don’t think you can really measure skill, only observe it.


(tokamak) #233

Cheers, that was the type of feedback I was hoping for!

The main concern is that, because a tactical shooter is good as infinitely complex, any attempt at codifying the actions within the game will always be an abstraction, a simplification. That in and of itself isn’t an issue for most folks here. But what someone like DAUK is concerned about is that should this model be used to asses a player’s worth, all kinds of loopholes in the system can cause players to gain either undue credit, or lack credit that was due.

I think the ‘Engineer maintaining a battery of deployables’ is an excellent example as it cuts right to the core of the issue. It’s an activity that would generate a lot of xp but can range between completely pointless to utmost important in the entire match. Under the current system, there’s no way to discern between a statpadder not only gaining undue xp but also being a burden to his team by doing something that doesn’t contribute substantially to the team’s progress, and a skilled player doing something really important in order to secure a win.

Two points are then raised:

Basically you’re putting value on menial, non-skilful tasks and the system has no way of interpreting intention, luck, abuse or tactics.

Senethro is saying the same thing

Its an FPS, I’m not sure that shooting being the most important thing is anything less than a feature. There might be room for a dedicated supporter in a class based RPG but in most FPS every player is a rifleman first and their class second. Theres also the issue that being good at your class is very easy but being good at combat is very hard.

I think it’s important to get this argument out of the way because it’s polluting the rest of the discussion. What’s consistently going on here is that you both keep driving things on the absolute. In essence the argument here is that cognitive, hand-eye coordination skill is the only skill that should be recognised and rewarded. The rest, the support, the objectives and employing all kinds of toys, that’s all things that doesn’t require right timing and accuracy and is therefore trivial. It may be important in the game but it’s not something that requires skill.

I just can’t agree with this. I think it’s regrettable that people can even view it in such a way. Yes, buffing someone, constructing something and employing something doesn’t require precise action. But that doesn’t mean the skill necessary to use these tools effectively is negligible. It’s the cerebral part of the game. It’s the part of tactical insight, macro-oriented decision making and prioritisation. If these things truly didn’t require skill then it wouldn’t matter when and how they were handled. You know that’s not true and you know that any decision in these games is rarely cut and dry. To shove all that aside merely because they happen at the push of a button rather than at a well aimed click of the mouse is selling the game short.

On top of that, many of these actions are highly reliant on the combat competency of a player. A medic that can’t fight his way to his victims is limited in his options. Players that know how to fight will fight themselves more often in an area full of dead enemies which allows them to employ their class tools and be a huge asset to their team. I personally like to see the game reward the players that are good at fighting AND at prioritising and wouldn’t mind to see people who are only good at either one of them struggling to maintain a high score as in my view, that makes you a less valuable player.

But hey, this may all be very subjective. I agree that if you think that indiscriminate killing is the only thing that should be recognised as ‘skill’ then I agree that an xp/min system isn’t suitable and you’re best off with just a k/d ratio.

The second point raised is far more interesting for this debate:

Either the XP farmer is considered skilful and raised up the ranks, or the awesome tactician is considered a farmer and held down the ranks. Which one is it? Lemme guess, more modifiers?

Yes under the current system there’s no way to discern between these two scenarios. Repairing a deployable always results in the same xp/min which makes the xp/min not only inaccurate for skill but also a perverse incentive for players to do the wrong thing.

Under my proposal there’s two things that prevent this from happening and it doesn’t need this fractal like system of branching out xp bonuses Inferno suggested.

- The result modifer at the end of the match already goes a long way in stopping this.

Currently, it doesn’t matter if you’re winning or losing. You always get the same amount of xp from your tasks plus a little flat reward for win. It means that in both scenarios both players still get the same base xp/min value from their task, but in the scenario where it’s a worthless task the probability of losing is higher than in the scenario where it’s an important task. This means that being a burden to the team by chasing these type of ‘xp leaks’ will increase the rate in which you lose matches which means that you end up having the loser’s penalty applied more often to you than to a player that contributes to a team.

Statpadders will find themselves on the losing side more often (after all, the side they’re fighting on is at least down one useful player, which does make a major difference in these games) and therefore will have the negative percentage penalty for losing more often.

Yes, this means that sometimes they strike the jackpot of statpadding and winning. But because they lose more often than not this approach won’t favour their average xp/min and that’s by itself already enough to stop it dead in its tracks.
[B][I]

  • The 'hot modifier goes further in increasing the chances that xp distribution is accurate.
    [/I][/B]
    I don’t see why the ‘hot’ modifier, as explained earlier, shouldn’t be applied to deployable as well. If a deployable has been active recently (stopped a missile, damaged a vehicle, or even locked on to a player) it means it’s a relevant actor in the situation. Therefore it makes sense to hand out more xp for someone repairing, hacking, damaging or destroying it while it’s ‘hot’. Again, it doesn’t rule out exceptions, it’s only saying that the probability of the player doing a relevant action is higher. The few situations that this might not be the case is something I’m willing to take for granted because it averages out towards a more accurate system.

The hot modifier is doing the same thing as Inferno’s domino’s effect but only in a far more simplified manner. This doesn’t mean that we can still include chain-bonuses (games like War Inc. and even BF3 occasionally reward these bonuses). The hot modifier is simply a highly flexible way of punishing statpadding and encouraging a proactive approach to the game.

Here’s one more point that hasn’t been raised. I hoped other people would raise it but it’s worth discussing nonetheless:

-Next to the modifier being connected to the difference in the average team-rating, the modifier should also be connected to the average win-rate for that side on the map.

This is really simple. If statistically, defenders win 60% of the time on that map, then their modifier is cut by 20% to make up for the map-advantage they enjoyed. Including an externality like geometry advantage is something I’ve been thinking about a lot. You don’t want the masses of players to completely dictate the outcome, but a slight modifier like this helps in making the xp/min score more fair and it prevents players from bailing on being assigned sides that they experience as disadvantageousness.

Without xp/min such a modifier would be a lot more crude, but including it in the outcome modifier is quite elegant I think.

I hope I addressed most concerns. I’m sure I didn’t convince or satisfy everyone so by all means rebuke this again.


(Humate) #234

But hey, this may all be very subjective. I agree that if you think that indiscriminate killing is the only thing that should be recognised as ‘skill’ then I agree that an xp/min system isn’t suitable and you’re best off with just a k/d ratio.

Theres huge potential for SD’s playerbase to turn their nose up at their score system, irrespective of whether its an accurate score. Its safer to actually provide the stats that they want to look at. This also includes all the class based stuff players are interested in.

The game doesnt need to asses a players worth.


(SockDog) #235

It does if you’re then strapping on RPG elements, upgrades and other trinkets based on that worth. The system exists to feed other systems.


(tokamak) #236

[QUOTE=Humate;403472]Theres huge potential for SD’s playerbase to turn their nose up at their score system, irrespective of whether its an accurate score. Its safer to actually provide the stats that they want to look at. This also includes all the class based stuff players are interested in.

The game doesnt need to asses a players worth.[/QUOTE]

I understand that risk and I’m certainly not suggesting that these things should be omitted from the player. But the point remains that a player’s contribution to the team and a player’s ‘skill’ ought to be defined in the same number. A player may believe otherwise but I see no issue in stating that that player is wrong. There’s enough Starcraft players who believe they’re not in the league they ought to be in, or who believe their actions per minute need to be factored into it. Tough, a developer needs to dare to stand for what he believes is the right value to be recognised and appreciated.

In the end the point of these games is whether or not that final objective is completed. It’s the entire sum of things a player can do that lead up to that outcome that matters and not one singular aspect that may or may not be somewhat useful.

I do think the game needs to asses a player’s worth. Especially in a type of game where so many different ways of play can be valid (and even more ways of plays can be invalid). It’s like losing a chess game but still believe you’ve somewhat won because you captured more pieces, or because your queen is still alive. And chess is easy like that because it’s duels between two players. Shooters are fought in teams and not every player is as useful as the other. That needs to be recognised and valued in order for the game to be satisfying and rewarding to pay.


(Humate) #237

My point is to simply supply the information players value, instead of devising a score and advertising as the be and end all.


(Humate) #238

[QUOTE=tokamak;403475]

I do think the game needs to asses a player’s worth. Especially in a type of game where so many different ways of play can be valid (and even more ways of plays can be invalid). It’s like losing a chess game but still believe you’ve somewhat won because you captured more pieces, or because your queen is still alive. And chess is easy like that because it’s duels between two players. Shooters are fought in teams and not every player is as useful as the other. That needs to be recognised and valued in order for the game to be satisfying and rewarding to pay.[/QUOTE]

Sure there are a many methods of playing, but as long as you value your playstyle, theres no reason it needs to be validated by the game right?
If you know unequivocally whatever creative out there method contributed heavily to the win, why does it need to be reflected in a score?
I would have thought the direct experience of seeing the impact of the play happen, would be satisfying. Or do you need other players to recognise that creative out there playstyle indirectly via a score, to make it truly satisfying?

This is why I state to list all the stats, classxp and all, because ultimately each individual player has their own score system, their own leaderboard.


(tokamak) #239

You would be completely right if we were talking about a sandbox game, but that’s not the case here. There’s clearly right and wrong ways to play. Trying to get as many laps with your platypus around the Valley lake may be satisfying, it may require skill and may even result in an impressive feat if you break someone else’s record, but it’s not the point of the game and it doesn’t contribute towards it in any shape or form, in fact you’re being a nuisance by putting the team behind one man.

And now I can already hear the objection ‘that’s an absurd example’, yes it is absurd. But the moment you admit that there’s a clearly wrong way to approach this game you must start to recognise the value of a system that discerns between worthless and valuable players.

If you know unequivocally whatever creative out there method contributed heavily to the win, why does it need to be reflected in a score?

Wouldn’t it be totally frigging awesome to see the game recognise what you did and reflect it in a score? That’s the entire point of this discussion, to come up with something that is capable to do so. The challenge is to make it as elegant and simple as possible without losing accuracy. Additions like ‘hot’ tags and outcome modifiers have those qualities.

I would have thought the direct experience of seeing the impact of the play happen, would be satisfying. Or do you need other players to recognise that creative out there playstyle indirectly via a score, to make it truly satisfying?

Of course your own feats may be satisfying but having no one around to see it puts a damper on the whole thing. And that’s what happening under the current system. You may have done something unique, something huge, but it’s the statpadder that gets the most xp, and then there’s the dude in the chat bragging about his k/d, both completely oblivious to that tour the force you just pulled off and wont them the game.

This is why I state to list all the stats, classxp and all, because ultimately each individual player has their own score system, their own leaderboard.

I’m not against each player having access to a detailed personal file with those stats if they want to (and according to your logic, it can stay private), but the score board should parade around the dude with the highest xp that match as he’s the one that mattered the most.

Plus, a few posts ago I already voiced the idea of ’ special feat’ where the system picks an extraordinary achievement during a match that day, like someone reviving 12 people in a minute, or someone making having 8 mine kills in a row, or someone disguising as every member of the enemy team in one match. Just funny stories the game picks upon and displays it next to the leaderboard with the xp/min ladder.


(DarkangelUK) #240

[QUOTE=tokamak;403482]
I’m not against each player having access to a detailed personal file with those stats if they want to (and according to your logic, it can stay private), but the score board should parade around the dude with the highest xp that match as he’s the one that mattered the most[/QUOTE]

And once again you’ve just stated exactly why it’s completely the wrong way to go. Inherently broken system is something that can never be fixed… that’s what ‘inherent’ means. There are too many examples of how this won’t work that should be obvious to someone that’s played all the games for a decent amount of time at varying skill levels… you don’t have enough experience with them which is clearly showing now.