Community Question: Measuring Player Skill

(DarkangelUK) #81

[QUOTE=tokamak;402909].
Like I said, this is short-sighted and unimaginative. You can put all kinds of diminishing returns, context sensitivity and positive and negative feedbacks within the distribution to ensure of that.[/quote]
“all kinds of”, and you’re preaching unimaginatives to me. You’ve put no thought what so ever into the process, and your lack of forward thinking can’t even comprehend the basics flaws. Are you saying an XP system can’t be easily exploited therefore rendering a system based on it redundant?

Now you’re going in circles.

Nope, bang on the straight on narrow.

The value of the ‘skill’ based actions, IE stuff like accuracy, K/D and battlesense is highly situational and frequently detrimental to the team. Someone who refuses to fire from a distance because it messes up his accuracy, someone who camps on a hill with a sniper rifle or refuses to jump in for an important team-member because it messes up his K/D or battlesense is being a burden.

Now you’re going round in circles.

(light_sh4v0r) #82

I knew this would end up a discussion between DAUK and Toka from just the topic title :tongue:

(tokamak) #83

Yes I’m saying that xp-distribution can be done in a more elaborate fashion than it is now.

Nope, bang on the straight on narrow.

Your argument once again rested on the premise that the xp-distribution was insufficient. Circle.

Now you’re going round in circles.

That was not a response to your post.

[QUOTE=Humate;402910]The manipulation of stats, doesnt mitigate what skill actually is,
The stats are merely indicators, nothing more nothing less.[/QUOTE]

Flat stats can be manipulated. An intelligent dynamic score can not.

(DarkangelUK) #84

No you’re not, you’re guessing it can with no substance or reasoning as of how. Still waiting on an example of something that can’t be easily exploited, until then, all you’re doing is blowing hot air… smoke and mirrors.

Your argument once again rested on the premise that the xp-distribution was insufficient. Circle.

Still no reason, evidence, argument or validation of how it can be… until then i’m sticking to my point.

(Dormamu) #85

A combination thereof (please specify in reply) :
off topic/on topic
From my humble experience i will exemplify a combination of a player skill list i would like to see:

  • Kills or K/D ratio - only the player will now about this, not to be used in calculating the player skill or to have a very low effect. Mostly used to give some on the screen gratification and to cater to those who care about K/D ratio, or to help you improve your game…yeah right.
  • Assists - Mostly team-play actions: Obj. defend assists, obj. attack assists, kill assists, obj. completion assists, team actions near an obj or actions like, heal, give health pack, disarm, give ammo, etc. You will need a circle with a radius of (variable) to give the award (coins, fluff, xp). So you will gain 3 coins/min. if you are 2 feet from the obj/team-mate when you do the assist, gain 1 coin/min. when you are 6 feet from the obj./ team-mate.
  • Discipline - This will be your Mass Effect Paragon and Renegade bars :D. If you want to enforce the team-play we will give the Renegade bars an negative effect to be extracted at the end of a map from you coins earned during the match. In this category will enter Rambo actions(Achievement for full Renegade bars at the end of a map :D), actions made by a player to far from team/obj, Sniping/Camping as long as they are not Assists Actions, etc.
  • Shots - More like a spreadsheet with shots fired /match/total, shots fired while walking/running/crouching/strafing/jumping/up/down/etc. Time it takes you to fire your gun as soon as you take damage/and more. Team bled, tk/etc.
  • Hits - another spreadsheet who will show the hits on a human body and with a highlighted number of head-shot/body shots/ hits wile walking/running/crouching/strafing/jumping/up/down/etc.
    (Accuracy - another spreadsheet with a human body same as “Hits” but in % :D)
  • Target scanning - preferred class to kill, Nemesis class, preferred first class targeted near an obj. or in a band(+2) of enemies, preferred weapon to kill a specific class, etc.
  • DPR (distance per round -Breo) wile walking, running/ jumping/etc.
  • Experience - the 0 to 400 game mark for a player, you will need to achieve in a match a minimum of a 5 (from a scale of 1 to 10) or 3 stars (from 1 to 5 stars) to increase your counter by 1 and after much work to win the 400 game mark (achievement included :D) an to be considered a seasoned player :D. Stars/numbers awarded by Accuracy/Discipline/Assists Actions/etc.

PS. I reckon most of the above was already stated here or in another thread/forum. I only did it to raise my posts number :smiley: . Also, a good read about ranking Competitive Ranking Systems , Microsoft TrueSkill™

(Humate) #86

[QUOTE=tokamak;402917]

Flat stats can be manipulated. An intelligent dynamic score can not.[/QUOTE]

Its irrelevant.

It.does.not.change.what.skill.based.actions.are.
Therefore if one were to create a formula for skill it would consider ONLY those actions.

(Smooth) #87

I personally think that the main flaw with rating players using the currently available stats is that none of them really take into account the quality of the other players on the server.

XP/Min is the ideal measurement since it encapsulates everything else (the system does need to be robust enough to avoid exploits/cheap play - which is no small feat) including the important ones like WLR, KPM and to a lesser extent KDR.

If this value (or a combination of values) could be modified based on the 'skill’of your team-mates and opponents then I think you would eventually get a much more accurate rating for players.

(Paul) #88

[QUOTE=Smooth;402922]I personally think that the main flaw with rating players using the currently available stats is that none of them really take into account the quality of the other players on the server.

XP/Min is the ideal measurement since it encapsulates everything else (the system does need to be robust enough to avoid exploits/cheap play - which is no small feat) including the important ones like WLR, KPM and to a lesser extent KDR.

If this value (or a combination of values) could be modified based on the 'skill’of your team-mates and opponents then I think you would eventually get a much more accurate rating for players.[/QUOTE]

Which is why I voted for “A combination thereof (please specify in reply)”

As some of you might know Trackbase is experimenting lately with a new rating method, still based on the ELO system but to get the points we use a combination of factors, which are the following at the moment:

total points of the round = ((((totalHits + totalHeadshots*2) / 8.0) + maxKills) 2.5) + (total heavy weapon kills * 3) + (revives4) + engineering + battle sense xp.

This divided by the match time is the points per minute, if you use the modified ELO system on this one, you get TSP-like ranking which is a far better indication then just using XP/PM or just battle sense etc :slight_smile:

(tokamak) #89

I can’t name a concrete example because it would be too crude, the entire point of an encompassing xp-distribution is that it’s intricate and detailed. The point is that the system puts the actions into the context of the mission. This means that there’s a base xp given for the handling and on top of that all kinds of modifiers, positive and negative.

Brink was already well under way when it comes to context sensitivity. The problem with Brink was that the proportions weren’t right. Nonsensical buffing was worth way too much xp and the context-modifiers (killing an objective class near an objective) were too low.

All in all this is a system you can keep on tweaking. You say you can name an exploit for each system but I can name a solution for each exploit just as easily.

That’s something I can see happening actually. A rating could be a multiplier on top of the amount of xp you earned during the match. Most ELO ratings are adjusted with 0-20 up or down depending on the rating of the opponent. In a shooter that can translate to adjusting your gained xp from 80% to 120%.

A rating alone is too inaccurate for it to be the sole defining factor but as a multiplier it has just the right amount of weight for players to accept slightly skewed matches (so they don’t defer to the other team as easily).

(DarkangelUK) #90

[QUOTE=tokamak;402929]I can’t name a concrete example because it would be too crude, the entire point of an encompassing xp-distribution is that it’s intricate and detailed. The point is that the system puts the actions into the context of the mission. This means that there’s a base xp given for the handling and on top of that all kinds of modifiers, positive and negative.

Brink was already well under way when it comes to context sensitivity. The problem with Brink was that the proportions weren’t right. Nonsensical buffing was worth way too much xp and the context-modifiers (killing an objective class near an objective) were too low.

All in all this is a system you can keep on tweaking. You say you can name an exploit for each system but I can name a solution for each exploit just as easily.[/quote]
XP distribution in itself is flawed, do X get experience, do X,Y and Z get lots of experience OR repeat X over and over get lots of experience. The end result is the 2 players have reached the same number in different ways yet both are categorised as the same skill level. The system would need to determine if the user should be doing X, Y and Z because it’s part of the end game or if the player is just hopping from task to task to gather XP and not lending to the team. On the flipside it also needs to know if the player should be doing X over and over or not for the same reasons… that type of system does not exist and cannot exist unless you plan on creating some awesome AI to look after it all.

Brinks context was indeed broken, but I still believe there is no way to fix it. Players were given XP for defending the gate on Container City and being in its radius, that was a correct thing to do for defence on that stage yet those that done it from a distance or by covering alternate routes weren’t given the XP bonus of those that stuck by the gate, allowing the enemy to get close. Was he wrong for sticking close to the gate? Was he right for letting the enemy get so close then killing? Was he there to gather the XP flurries that came from being within the radius or was here there as a last defence for those that slipped by his team mates. Every conceivable intention regards all of them as of equal skill, and that’s simply untrue.

(Humate) #91

[QUOTE=Smooth;402922]I personally think that the main flaw with rating players using the currently available stats is that none of them really take into account the quality of the other players on the server.

XP/Min is the ideal measurement since it encapsulates everything else (the system does need to be robust enough to avoid exploits/cheap play - which is no small feat) including the important ones like WLR, KPM and to a lesser extent KDR.

If this value (or a combination of values) could be modified based on the 'skill’of your team-mates and opponents then I think you would eventually get a much more accurate rating for players.[/QUOTE]

The notion of a number representing skill is flawed anyway, even if it does take into account opponents and team-mates. Nonetheless, one aspect of it that stands out, is that its based on the belief that players are actually trying their hardest all the time in a pub game.Immediately that destroys the credibility of any skill rating.

Edit:
In SC2 a common thing that occurs- players either buying new accounts, sharing accounts or deliberately demoting themselves from their leagues to play troll matches against bronze league players. Blizzard have almost solved this, by introducing an unranked matchmaking system so that players can still troll match away from legit leagues.

(tokamak) #92

Yeah how does SD plan to counter this dreadful practice of buying extra accounts to play the system? Oh wait…

In a way he was. Not every kill is equal. Killing a player closer to the objective means the player wasted more time getting to that point and should carry a bigger reward. Killing player near their spawnpoint may be an effective strategy, perhaps the geometry is just right (coughcontainercitycough) and even a part of the lower xp-reward is offset by getting to kill the player more frequently.

Because the system can’t look in the future all we can go by is probability. The amounts of different things that can happen as a player gets closer to an objective diminishes. And it’s safe to say that, as a player gets closer to the objective (and even start doing the objective), the probability of finishing that objective increases way beyond his starting probability.

It’s the same with football. Perhaps that one pass by an attacker way at the opposite side of the field was the butterfly flap that prevented a disastrous match but we can’t be sure. What we can be sure about is that if that goalkeeper didn’t jump for that ball, that the match would be lost.

The danger is that you can let this particular context weigh too heavy. If a player is out to get the highest amount of xp (as he should be) then he needs to weigh the risk of killing an opponent at the very last moment (IE granting him the least effective way of spending his life) versus the xp reward versus the advantage the map offers. Certain bottlenecks simply grant a higher probability of stopping a player. If the reward for killing a player near an objective is too high then players may accept too high a risk for it to be tactical justifiable.

We don’t have to go as far as letting the xp-system reward people for killing players at the best bottlenecks though. Using the map to its tactical advantages already rewards you with more kills in and of itself.

(Humate) #93

Yeah how does SD plan to counter this dreadful practice of buying extra accounts to play the system? Oh wait…

Given I had over 55 aliases in etqw, not something I think they would consider. :wink:

(tangoliber) #94

W/L Elo is cool… but only in a random matchmaking mode where players cannot choose their teams, and cannot group up with friends. Otherwise, it just encourages teamstacking. A good thing about Brink was how freely players would switch to a weaker team to try and balance matches… a lot of games punish you for that.

(If the community is big enough, then you could offer both custom servers, and Ranked Elo matchmaking… but if you don’t want to split up the community too much, then custom servers are more important.)

I think the Brink XP system was quite good…though I suppose that the XP rate for medics should be scaled down. In terms of what actions gain XP, I don’t think you can do much better.

(DarkangelUK) #95

[QUOTE=tokamak;402938]
In a way he was. Not every kill is equal. Killing a player closer to the objective means the player wasted more time getting to that point and should carry a bigger reward. Killing player near their spawnpoint may be an effective strategy, perhaps the geometry is just right (coughcontainercitycough) and even a part of the lower xp-reward is offset by getting to kill the player more frequently.[/quote]
I was discussing the extra XP flurry given for simply being within the radius of the gate. I remember someone posting that they would start a private match then just stand at the gate and simply let the XP rack up. Letting an enemy get that close in the 1st place is more dangerous if you don’t have the aiming skill to kill them, it’s all subjective.

It kinda veers off into subjectivity, you’re again catering towards the highest XP prize and not taking into account the impact that player had. Lets take your hatred of K/D. Someone situated away from the gate has been keeping the enemy at bay the entire game, they’ve been mowing them down where the guy placed at the gate has been racking up XP for simply being there. Since kills seemingly aren’t valued, the 1st guy gets very little XP but a lot of kills or 1 or 2 deaths. 2nd guy has been racking up XP, he’s killed 1 or 2 that have slipped through and died a few times. He gets more XP and is considered the more skilled player. In a shooter, he’s stood still, killed less and died more times, yet is considered the more skilled player due to his XP… that’s very, very wrong.

(tangoliber) #96

[QUOTE=DarkangelUK;402950]I was discussing the extra XP flurry given for simply being within the radius of the gate. I remember someone posting that they would start a private match then just stand at the gate and simply let the XP rack up. Letting an enemy get that close in the 1st place is more dangerous if you don’t have the aiming skill to kill them, it’s all subjective.

It kinda veers off into subjectivity, you’re again catering towards the highest XP prize and not taking into account the impact that player had. Lets take your hatred of K/D. Someone situated away from the gate has been keeping the enemy at bay the entire game, they’ve been mowing them down where the guy placed at the gate has been racking up XP for simply being there. Since kills seemingly aren’t valued, the 1st guy gets very little XP but a lot of kills or 1 or 2 deaths. 2nd guy has been racking up XP, he’s killed 1 or 2 that have slipped through and died a few times. He gets more XP and is considered the more skilled player. In a shooter, he’s stood still, killed less and died more times, yet is considered the more skilled player due to his XP… that’s very, very wrong.[/QUOTE]

You are right, the entire territory from the objective zone to the other teams spawn should have the same XP rate…Just getting kills around the objective should not be worth less than cutting the enemy off while traveling.

But KDR is useless in almost any game where players have to complete objectives. In a evenly-matched game in Brink, you can’t usually clear out the defense… the objective often has to be performed in chaos, like the crane repair on shipyard. Engineers probably die 20 times by suicide repairing the crane, but it works and is necessary. Or the retrieval objective on CC, where the operative repeatedly throws an EMP into the room with the data, grabs the data, runs out, dies, repeat 10 times until he finally makes it out. KDR doesn’t always matter for defense either, as sometimes you gotta suicide defuse or something like that.
In Tribes: Ascend, the capper sometimes gets no kills, and always gets a lot of deaths deaths… the capper just suicides over and over so that he can occasionally have a successful run. In Killzone 2 clan competition, some of the best players have very low KDRs due to rushing plants in chaos, or preferring to take out turrets before players.

XP for kills and for objectives and teamwork (giving ammo, healing) is probably the best way to score. Its just an indicator of how active you were during the match…not really how skillful you are.

(DarkangelUK) #97

Which is exactly my original point, using a single variable to calculate skill is the wrong way to go, be it KDR or XP/min.

(Smooth) #98

I would say that’t an issue with the XP system that can be solved given time.

If the system gets the balance right between awards for kills along with everything else, then it should be the best measurement of value-to-the-team (aka skill) that is available. It’s a lot of work and would require many iterations and a ton of real-world statistical data to get right, but I think it’s possible.

Edit: My point is that XP (if done right) should be able to accurately reflect things like high KPM, KDR and WLR.

(DarkangelUK) #99

Just seems to me that it’s a system that can’t be tweaked as XP will always be open to abuse. Seems akin to piracy, as soon as something’s cooked up to prevent it, new tactics are employed to get round it and abuse it. At the end of the day it turns into a fight between the developers and the abusers, and the legit players are the ones that end up suffering because of it.

(tokamak) #100

Yeah you keep saying that but apart from the obvious double supply crate there’s actually very little to exploit in ETQW and even less in Brink. That supply crate trick didn’t even give a whole lot of xp, it was just to count towards a statistic milestone for the medic.

Now I absolutely agree that such milestone achievements are silly and reward players for doing the wrong things. But that’s not what we’re talking about here is it? In a game of ETQW, the players with the highest XP are simply always the players that have been the most useful. The only polluting factor is the time spent on the server which is solved with xp/min.

[QUOTE=DarkangelUK;402950]I was discussing the extra XP flurry given for simply being within the radius of the gate. I remember someone posting that they would start a private match then just stand at the gate and simply let the XP rack up. Letting an enemy get that close in the 1st place is more dangerous if you don’t have the aiming skill to kill them, it’s all subjective.
[/QUOTE]

Right that’s a clear example of a perverse incentive. It shouldn’t matter where the player is defending from. Still the location where the enemy dies does matter and is not just subjective. Another perverse incentive in ETQW is how a disabled deployable still gives the same amount of xp per damage you deal to it while it obviously should be much less. This caused players to waste a lot of time on redundant deployables rather than getting back into the fight.

These are all flukes rather than the norm though. A rough system is still puts the best players on top. I just think that it can be more accurate and rewarding by putting more nuance in it.