Community Question: Measuring Player Skill

[tokamak]

Come on, I know you can do better than just filling an entire post with assertions. Give me an example which breaks my system. That’s something I’m breaking my head over right now but due to my lack of experience I just can’t come up with anything.

[Breo]

There’s is no value of the poll that will show skill. But you can use the first 7 answers to categorise the type of players. The attempts to make a skill counter are based on these basic ratios that you can find in almost every shooter.

Also I don’t think it’s possible to make a accurate counter to measure skill, especially when the skill levels are mixed up in a game.

[SockDog]

That one individual is considered more valuable (Top XP) and in turn is likely more rewarded than other members of the team.

[tokamak]

Yeah that’s the point. Come on, any objections of actual substance? This is just people stamping their feet rather than engaging with the idea.

[DarkangelUK]

You’ve provided no actual substance yourself, all you say is “the system fixes that” when clearly it doesn’t because it can’t be fixed. Classing a non-skilful action as skilled and placing fake value on it does nothing for a skill assessment, nothing at all. Until you’ve actually grasped the concept then the ball is always in your court, so you can say you’re waiting for actual objections all you want but that changes nothing, you have no idea what you’re looking for. XP is easily obtainable, end of, anything easy is not a skill. Screwing in a lightbulb is not a skill, rewiring a system that runs it is… yet you’re wanting to give reward for putting the bulb in place because that provides the actual light when really the most skill was used to get it to that point in the 1st place.

[tokamak]

Classing a non-skilful action as skilled and placing fake value on it does nothing for a skill assessment

This discussion can be split up in two different subjects. The accuracy of the system itself and the semantic discussion on how ‘skill’ should be defined. I addressed both and I already provided why I think the discussion on the definition of skill isn’t really worth having. You say only the cognitive aspect should be recognised while I think the cerebral aspect needs to be included as it’s just as important if not more so. If we can’t agree on that part then you’re not going to agree on anything no matter what I say.

[DarkangelUK]

So in other words, you have no idea how you’re going to achieve this mystical classification but think magic will make it work when brought down to actual numbers using a system that can never and will never determine intention. Human driven multplayer gameplay is built on chaos theory, your proposed system just guesses that everyone is doing as they should, when they should and at the maximum efficiency they can when we all know that’s complete bull****. Perfect world scenarios are the worst type of scenario to base any system off of yet you think a crude numbers system can be used to determine a players mood, be arsedness, intention or even long term plan in their actions when it can’t. It will always miss-label the action across multiple players and therefore give an inaccurate skill level.

You have general skill and specific skill. General skill in any shooter comes under movement, shooting, accuracy, self preservation, time management and indeed knowing when to press X when a big button on the screen tells you to, we’ll call these cognitive triggers. You need to be looking at all of those aspects before being able to come close to accurately ranking skill for that player. The problem is, not all of those functions are methods of gaining XP, for example movement.

First we need to clarify if you even class movement as a general skill. Player 1 works the corridors and paths to make it to plant the dyno in 1min 30secs, Player 2 knows the movement system intricately, ramps some walls and CJ’s a large gap and makes it there in half the time… since movement isn’t factored then both are considered equally skilful in your system, that is wrong. Hell player 2 may have had to fight more enemies on his way there so racked up more XP where as Player 2 avoided the firefight and slipped in unnoticed, caught the enemy off guard and now we have self preservation and time management not being rewarded.

So that’s general skill ranking broken already before we’ve even begun, and that was 1 scenario (sorry dude but I’m going to stop humouring you soon). Then there’s the specific skills relating to role. Since I’m lazy i’ll point back to the deployables scenario, which you never fixed in your method btw because it was purely based on intention and your system doesn’t take that into account. Both got the same XP, both won the round, but one did it purely to farm XP where the other did it to win the match for his team. Both are classed as equally skilled which is wrong.

So general skill and specific skill can’t be calculated properly with your system, and this is something you want to use? I hope you never work on any real world projects that could put people at risk… oh dear lord. You keep asking for examples of how it can be broken because you think its full proof, any one with real investment in their idea would be thinking of them himself instead of sticking his fingers in his ears and going LALALALALALA!

[tokamak]

I wouldn’t say it’s magic. Just brilliant. If you had been paying attention then you’d know this isn’t based on perfect world scenarios at all, it’s based on probability. See, the intention becomes irrelevant when you can retroactively boost or penalise the xp for actions based on their outcome. Your deployable battery example was perfect for that. It’s based on the premise that if you’re after cheap xp that doesn’t contribute to your team, you’re by definition wasting time, being a burden to your team and increasing the chance of a loss. This then fires back on you because winning or losing suddenly matters a lot more under this system. Consistently being a burden to your team means an increased loss rate which automatically translates in a drastically lower xp/hour than someone who’s consistently a great asset to his team.

The moment the outcome of the match is calculated into the score, even the crudest, simplest xp-system becomes a whole lot more accurate. It means you no longer need to monitor every single minute action of a player. A player’s skilful movement or accuracy doesn’t matter at all because it already makes for a higher acquisition of xp. A player that navigates and fights well is more often at the right place to grab those relevant (hot) xp-rewards than a player who isn’t.

Or let me put it in another way. Even if you device some utterly genius way of tracking the skill of a player’s movement: If a player is doing some INCREDIBLE acrobatics on the ocean wall and getting inside the large bunker by some ridiculously beautiful trick-jumps, then that would still be a completely futile enterprise if it was given that everyone else was fighting inside over the EMP generator.

That’s the issue you’re ignoring here. By going to deep into detail with valuing your ‘specific skill’ you end up getting too much false positives. Accuracy is completely meaningless even detrimental if it means that a player becomes completely miss-adverse and stops taking worthwhile shots because the chance of missing becomes too likely to mess up his hit percentage.

Cognitive skills are utterly, utterly worthless if they don’t result in the right outcomes. And luckily the outcomes are a lot more tangible than all these skills and combination of skills you’re trying to work out. You can slap an xp-value on player-player interaction and on player-objective interaction. By simply focusing on the tangible results, all the complex, chaotic handling that preceded it become superfluous and therefore not worth tracking.

First we need to clarify if you even class movement as a general skill. Player 1 works the corridors and paths to make it to plant the dyno in 1min 30secs, Player 2 knows the movement system intricately, ramps some walls and CJ’s a large gap and makes it there in half the time… since movement isn’t factored then both are considered equally skilful in your system, that is wrong. Hell player 2 may have had to fight more enemies on his way there so racked up more XP where as Player 2 avoided the firefight and slipped in unnoticed, caught the enemy off guard and now we have self preservation and time management not being rewarded.

Player 2 has a higher probability of completing a key-objective each time he plays while player 1 has not. Player 2 spends his time completing the actual objective which does not only result in a heft flat-score of winning that round but also receives a multiplier due to the outcome modifier more often than player 1.

And how can you even say that time-management is not being rewarded? We’re talking xp PER MINUTE here. Even without the outcome modifier the xp/min of player 2 quickly completing a key-objective is way higher than a dude having to spend his time on detours and fighting opponents only to complete the same objective. Now, with the win-modifier this difference between both player becomes much bigger because not only is the time-factored into it, also the chance that a tactic is effective is factored into it. Employing tactics with a high success rates grants you the win-modifier more often than tactics with a low success rate. Statpadding is a tactic that has an extremely low success rate which means you’re suffering that xp-percentage-penalty so often that really messes up your overall score.

So in short. My proposition is that ‘specific’ skill is worthless if it isn’t displayed in ‘general skill’ which is a lot easier to track (especially now with a hot modifier). A ‘specific’ skill not resulting in a higher display of general skill (and thus higher xp) should just be called ‘frivolous activities’. Yeah, shooting an anansi with an obliterator while diving away from your icarus is very impressive. Very. But it’s only worth as much as one dead anansi.

It doesn’t really matter how you do it, it only matters that you do it.

[ailmanki]

So basically as a XP-Whore I have to make sure to be in winning team…

Then there is a problem with your system, sometimes teams are completely unfair, and mixing the teams newly would be required, but we only know after the end of the round, how good each one is.

[tokamak]

Yes, for effective xp-whoring it’s tantamount to win. Even if the modifier bonus ranged around 10% then that would still be enormous as it both stretches positively and negatively for winning and losing. And on top of that, DAUK made me realise that because of the flat team-wide rewards added to the team for either completing or defending an objective, it actually pays for attackers to win as fast as possible and it pays for defenders to stretch each round as long as possible, because now the time is calculated into it.

And yes, although randomly shuffling 24 people over 2 teams ought to give dependable averages, in reality people rarely stick to the team they’re assigned to and rarely pick the side randomly. People have preferences depending maps and team-mates that are playing. To solve this, the win-modifier needs to be adjusted by the win statistics of the map, and the average team-rating. The average team-rating is the average ELO of all the players in the team. The player ELO itself isn’t accurate enough to be displayed prominently so I think it should be hidden. It only serves to adjust the win modifier and make for more fair xp-score and to discourage players from stacking in the team.

Players shouldn’t be punished for wanting to play with a group of less competent friends and the game shouldn’t encourage highly skilled players ganging up on lowly random teams.

[Senethro]

Wait, why are you so deadset on making XP and Skill be the same thing? Why can’t XP just be a bunch of fake numbers for whatever progression system is in place and skill tied to good indicators such as W/L ratio?

[tokamak]

What you’re rewarded for and what you’re lauded for should be the same thing. Because XP is the only way skill can be accurately quantified. It’s not just the best indicator, it’s also better than any combination of indicators you can come up with, no matter how detailed. This is because all these smaller indicators ought to contribute towards a higher xp/min, if they’re still high and don’t contribute then they’re false positives.

We’re dealing with scaling here. You need to draw a line between too crude an indicator (W/L) and too precise an indicator (accuracy or any combination of similar indicators). Too crude means no significant distinction, too precise means too much chance at false positives. XP is exactly the sweet spot because it allows you to connect the smallest actions that are tangible (Player/player and player/objective interactions) with the overall outcome and even the overall outcome in the context of all the other games played.

It’s just incredibly powerful, and all of that expressed in one single, illustrative number. That’s just beautiful.

These tactical shooters are team-games played by a large number (16-24) of people of varying skill-levels, that makes W/L a completely meaningless indicator. Even a person’s personal ELO, which already goes much further in displaying a player’s worth than just W/L is still highly inaccurate because of the low stake a player holds into the outcome (compared to 1v1 games like Chess or Starcraft).

[SockDog]

And now we’re back to the contradictory values thing that we saw in Brink with the class changes and class building mechanics.

A Win is important enough to ensure people stick by the system but wait! It’s not so important as to encourage other bad behaviour like stacking teams or quitting poor teams. Yet another side effect of an unnecessary system.

I look forward to the catty retort.

[Senethro]

What you’re rewarded for and what you’re lauded for should be the same thing.

Why?

Because XP is the only way skill can be accurately quantified.

Why?

It’s not just the best indicator, it’s also better than any combination of indicators you can come up with, no matter how detailed. This is because all these smaller indicators ought to contribute towards a higher xp/min, if they’re still high and don’t contribute then they’re false positives.

This is a great position to create to backpedal to later. If I come up with any good/better indicators then you can grab them and say they should support your thing all along.

We’re dealing with scaling here. You need to draw a line between too crude an indicator (W/L) and too precise an indicator (accuracy or any combination of similar indicators). Too crude means no significant distinction, too precise means too much chance at false positives. XP is exactly the sweet spot because it allows you to connect the smallest actions that are tangible (Player/player and player/objective interactions) with the overall outcome and even the overall outcome in the context of all the other games played.

Yeah, I get this which is why I mentioned some in long form in an earlier post. However, your claim that XP is the sweet spot and that its actions are tangible are unsupported. There have been pages of examples of how these actions are not remotely able to be measured well because the players intentions are unknowable.

I’d also like to see how you think an XP system can be made to address the point made by an SD member earlier that any system which doesn’t evaluate the quality of players on the server probably isn’t very good.

It’s just incredibly powerful, and all of that expressed in one single, illustrative number. That’s just beautiful.

I’m sure you’re trying to articulate a very lovely belief here in your perfect magical system but it just isn’t coming across.

These tactical shooters are team-games played by a large number (16-24) of people of varying skill-levels, that makes W/L a completely meaningless indicator. Even a person’s personal ELO, which already goes much further in displaying a player’s worth than just W/L is still highly inaccurate because of the low stake a player holds into the outcome (compared to 1v1 games like Chess or Starcraft).

I don’t get this. Why wouldn’t it just take longer for a players elo to stabilize at its correct value? (and I thought elo was near entirely based on W/L) You are the only consistent feature in all your games (assuming a typical ranked server system enforcing standards maps/modes) and with enough games played the uncertainty on the elo will decrease.

XP systems have other uses beyond measuring skill. They’re a reward system, a tutorial system, a progression/unlock system. Most players will get more value/information out of a scrolling list of XP rewards at the bottom of their screen than an incomprehensible system that will (to them) seem to arbitrarily hand out high and low scores. And can you imagine the forum posts?

Better to keep XP simple and easy to understand and banish the skill rating to a small corner of the online stats site since it doesn’t actually affect the game in any way, its just a measurement.

[tokamak]

I’m not really sure what you’re objecting to here so I may be misconstruing your post. I highly disapprove of the way Brink favoured and encouraged the frivolous use of team-buffs. It not only was a perverse incentive, it also made xp a worthless indicator. Just because some behaviour (combat) happens automatically doesn’t mean players can still distinct themselves in it and it certainly doesn’t mean it’s not worth rewarding as it completely messes up the actual challenge of prioritising your moves.

The ‘hot’ tags improve the accuracy of the flat xp/min rate and the outcome modifier is there to encourage players to think for themselves and prevent them from valuing certain xp leaks over doing what needs to be done.

A Win is important enough to ensure people stick by the system but wait! It’s not so important as to encourage other bad behaviour like stacking teams or quitting poor teams. Yet another side effect of an unnecessary system.

And quitting because of poor map balance. Don’t forget that.

Stacking teams and quiting poor teams have been accounted for. First there’s the time dimension in xp/min: It’s how long you played in the match that matters as the modifier will just always apply regardless of whether you’re still connected. On top of that, the average team-rating difference as well as the statistical map advantage adjusting the outcome-modifier no longer means external factors pollute your xp/min score.

[Senethro]

I’m not really sure what you’re objecting to here so I may be misconstruing your post. I highly disapprove of the way Brink favoured and encouraged the frivolous use of team-buffs. It not only was a perverse incentive, it also made xp a worthless indicator.

Its always going to be a worthless indicator but as I said, XP systems have other uses. It succeeded at making players use their class abilities and to try stuff.

Just because some behaviour (combat) happens automatically doesn’t mean players can still distinct themselves in it and it certainly doesn’t mean it’s not worth rewarding as it completely messes up the actual challenge of prioritising your moves.

I didn’t understand this.

The ‘hot’ tags improve the accuracy of the flat xp/min rate and the outcome modifier is there to encourage players to think for themselves and prevent them from valuing certain xp leaks over doing what needs to be done.

Its going to be very hard to communicate to players that the XP they earn is dependent on so many factors. (imagine the forum posts!)

Stacking teams and quiting poor teams have been accounted for. First there’s the time dimension in xp/min: It’s how long you played in the match that matters as the modifier will just always apply regardless of whether you’re still connected. On top of that, the average team-rating difference as well as the statistical map advantage adjusting the outcome-modifier no longer means external factors pollute your xp/min score.

Pretty sure BF3 already does something similar to this for W/L. However, once again you have neglected to tell us how this XP system accounts for different games and servers will have different qualities of player in them.

I see a problem here in that XP/min is used to measure skill, will be used as a multiplier to further reward and feedback on skill, but that XP earned in a match is uncapped and not zero-sum. Some games will naturally have higher XP totals and some lower but this will have absolutely no bearing on actual skill displayed in the match, yet XP/Min is treated as this huge incredible thing that both says how good a player is and what he deserves.

[DarkangelUK]

jibba-jabba

So in other words, your system is broken and doesn’t in any way, shape or form actually calculate skill, be it general or specfic at all, it just rewards those for doing what they’re told and actual general skill isn’t counted. Glad we got that over with. Btw if that was you trying to argue the opposite, you did a poor job and just made my point more valid than ever. As has been said, as long as you win you’re ok… you just said it yourself, cognitive skill is worthless if they don’t result in the right outcome. Wait, isn’t that the same for absolutely everything? Basically you’ve created a system unnecessarily shrouded in rhetorics to settle down to one thing… W/L. Lol well done. I love that you think it’s brilliant when just about no one is sold on it because it’s broken.

[Senethro]

Yeah, seriously, and heres another thing I thought on while in the shower. XP is only awarded for actions but sometimes inaction is the skilful, game winning response. You can’t award XP for being retreating from a bad situation with a long reinforcement timer hanging over you, or for not performing the stupid unsafe revive or doing anything that makes your character shout and give your position away.

[TruGamer97]

I think W/L ratio is the most important do I have an argument supporting that? Nope not at all.

[Humate]

And now I can already hear the objection ‘that’s an absurd example’, yes it is absurd. But the moment you admit that there’s a clearly wrong way to approach this game you must start to recognise the value of a system that discerns between worthless and valuable players.

Thats not what Im referring to, and yes silly example.

Wouldn’t it be totally frigging awesome to see the game recognise what you did and reflect it in a score? That’s the entire point of this discussion, to come up with something that is capable to do so.

Actually it isnt necessary. The direct experience of it is enough, with or without a reaction from the players that got owned.

Of course your own feats may be satisfying but having no one around to see it puts a damper on the whole thing. And that’s what happening under the current system. You may have done something unique, something huge, but it’s the statpadder that gets the most xp, and then there’s the dude in the chat bragging about his k/d, both completely oblivious to that tour the force you just pulled off and wont them the game.

Again this is my point. One gets excited over the best medic award, another is aroused over the most objectives completed category, another moans at battlesense ,etc

Creating a single score which encompasses every “legitimate” aspect of the game, is not going to prompt the k/d whore or the KPM player to concede that the player xyz with the highest score is the best player. The game might, but the players wont.

I’m not against each player having access to a detailed personal file with those stats if they want to (and according to your logic, it can stay private), but the score board should parade around the dude with the highest xp that match as he’s the one that mattered the most.

The player thats highest on that leaderboard matters to the degree players buy into the concept of XP and the formula that was used. The moment anything in that formula warrants scrutiny, they discard it completely and view it as “just a number”. The solution is to not actually advertise is as a “skill score”