Scoring

I'm still not quite happy with the scoring.
Right now it looks like the more sites you play the higher you rank. Which says more about the amount of free time you have than your skill.

Perhaps apply an exponential scale?
So that scoring 50% on 1 site gives more points than scoring 10% on 5 sites?

Let us know what *you* think.

Totally agree. I have been thinking that going on percentage complete is not that fair as I could just do the first 25% on each site and get quite high. Getting 75% plus on any site is not going to be easy. But getting a high percentage of challs on any site takes time and its not that fair to penalise those who have not done any/many challenges on 1 site but have done almost every challenge on another site. Just my opinion

Also forgot to say that the number of challenges on a site should mater too - tbs over 300 challs compared with WeChall of 4. Percentages would be a little unfair.

Final point - need to wrap the text in forum posts - but the site is coming along very well

Yeah, we currently do not yet autmatically count the nr of challenges on a site, but that could well be a factor we're going to incorporate into the scoring.

We have an adjustable base_score for each site so that we can tweak the score a bit to represent harder or easier sites.
The problem is that the difficulty if a sites challenegs is subjective.

Maybe masters of sites should get more points

Just an idea:

final_score_at_a_site = actual_score *(1+(1/rank)).

So if you have 5000 points now, and are ranked 1., then your score will be muliplied with 2 (thats 10000 point), and if you are ranked 200. with 5000 point, your score will be multiplied with 1.005 with final 5025 point

Edit:
After discussing it with Gizmore we have found a new formula:

final_score_at_a_site = actual_score *(1+(1/(rank+a)))^b

with a and b you can tune this formula, maybe a=5 and b=2 is a good start.

And for the ranking part wechall can calculate a local "wechall rank" for every site, so if someone who is ranked 2. on xyz site but is not registered on wechall then the 3. gets rank 2. on wechall.

I really agree about making difference for the "50% on 1 site and 10% on 5 sites" cases. This will be more important as more and more sites will connect.

Perhaps there should be another page where some of these suggestions can be tested? I still think spoj scoring is a problem as well, half an hour to an hours work to solve a problem usually gives 0 points, whereas on other sites 2 minutes work can give a lot of points.

What about just linking SPOJ on WeChall but having the site removed from the ranking?

Citazione da Caesum

06 Mar 2011 - 11:59:46

Perhaps there should be another page where some of these suggestions can be tested? I still think spoj scoring is a problem as well, half an hour to an hours work to solve a problem usually gives 0 points, whereas on other sites 2 minutes work can give a lot of points.

Yes, SPOJ scoring is a serious problem. I've looked through some of the challenges at SPOJ and from what I can tell the challenges are easily equivalent to what you find at Rosecode or CSTutoring as far as difficulty, but at either of those sites you get around 20 points per challenge at the low end when you are just starting to work the site. Much more per challenge later on. Caesum, assuming this is the same Caesum we hear about in hushed whispers in the dark , has essentially solved close to the whole of CSTutoring, or solved Rosecode three times over, for 1788 (4.67%). That is seriously broken.

My thoughts towards a solution:

I've been thinking about WeChall's scoring mechanism, mostly because of this thread and this one (starting on page 3), and I think the scoring issue is twofold. First, WeChall's scoring tries to 'normalize' points such that 100% is about the same number of points no matter how many challenges are on the site. That shows up here:

Citazione da Gizmore

site_score = site_basescore + (avg*site_basescore) + (challcount*basescore_per_chall)
...
userscore = site_score * pow(percent_solved, 1+100/site_challcount);

From http://www.wechall.net/forum/show/thread/465/New_Scoring/page-1#post2675

Of course, several other factors change the final site score but 'normalizing to challenge count' is a significant factor. Until very recently, I thought that this was a good idea, now I believe otherwise. It badly breaks scoring on sites like SPOJ, and generally it results in equivalently difficult challenges scoring differently purely because the site challenge count is different. If you pasted a challenge from CSTutoring into Rosecode, the challenge would score differently because total challenge counts are different on the two sites. That seems wrong to me. This, I think, is the primary problem. I've come to think that total challenge count should not be a factor in WeChall scoring. I know that is a radical break but I've tried to walk through the reasoning.

Second, I am beginning to wonder if weighting scores by percentage solved is a good idea. This, also, is a radical break so I'll try to make a case. Again, this breaks badly on sites like SPOJ where you'd need to solve more than a thousand challenges to get 50% but even that number may be badly off since Caesum's 240 solved at SPOJ is about 11.5 % there but translates to 4.67% here. The idea behind weighting challenges by percentage solved seems to be to prevent people from solving a bunch of easy challenges on multiple sites and thereby racking up a lot of points for easy work. That makes sense. Until very recently I also thought that this was a good idea. Now, I think that it only makes sense for some sites, namely sites with 1) challenges of varying difficulty that 2) internally score challenges evenly.

This is a critical consideration so I'll elaborate. Imagine a site where all of the challenges are roughly equivalent in difficulty. I'm not sure such a site actually exists though Ma's or Electrica might qualify. I haven't played through enough of either of those so I can't say. CSTutoring's programming section mostly fits the description though. I stress, "mostly". It isn't as good of an example as I'd like so pretend. CSTutoring internally scores all challenges evenly, as far as I can tell. There are no challenges worth ten points vs. challenges worth 30 or 100. They are all scored the same. Assuming the somewhat fictional idea that all of the challenges are equally difficult it makes no sense to penalize a player for solving only one or two challenges. All challenges are equally difficult so it makes no sense to penalize a player for solving only the 'easy' ones.

Now, not many sites score challenges evenly. I've been playing Rosecode so I'll use that as an example. Rosecode has an internal difficulty rating. Challenges score 2 to 25 points or so depending upon how hard the challenge is. That internal difficulty rating is reflected in WeChall's scoring. That is, a site's internal difficulty calculations are already taken into account when the score is sent to WeChall. When WeChall further weights by percentage solved it is compensating, or punishing, a second time for difficulty, which doesn't make sense to me.

The only time it does make sense to weight by percentage solved is when, as I said, a site internally scores at a flat rate for challenges of varying difficulty, but I'm not sure how many of those sites there are. They are certainly pretty rare.

Something like this makes sense to me at this point:

site_score = (site_difficulty*site_basescore);
userscore = (site_score + (usercount/rank) + score) * (score/maxscore)

site_difficulty is pulled from the 'Dif' column on the 'Sites' table. This value is generated by WeChall user votes. The rest comes from values returned from linked sites-- username:rank:score:maxscore:challssolved:challcount:usercount. I have no idea how this would effect scoring. It may be terrible, and I'm sure it far from ideal. It does keep raw challenge count from the equation, which I think is the biggest issue, and keeps scoring relative to the user rank and the number of users who have solved a challenge both of which should be some kind of indication of difficulty whereas raw challenge count is not. At any rate, my primary point was not to suggest a new algorithm but to point out what seem to me to be errors in the logic behind the existing algorithm. There may be much better ways to correct those errors than mine.

I second Caesum's suggestion that we have 'test' page of some sort-- maybe something like the 'Rankings' table but with a number of algorithms side by side.

Anyway... flame away

I also never liked having challenge count entering the equations, but for a different reason, I feel it rewards people for doing the same repetitive things over and over. Which in my opinion is what happens on sites with a high challenge count.
WeChall's scoring aims to reflect the amount or level of skill someone has. Not how much time they are willing to spend doing very similar things.
"points per challenge" is never going to work with that. And the difficulty of individual challenges can never be taken into account here. So yes, solving a challenge of difficulty X on a site with a lot of challenges is going to be worth less than solving a challenge of difficulty X on a site with less challenges. This is fair when you think about it for a bit and factor in the choice of sites to pay on and the challenges to choose from on each site.

Try to shift your thinking from being rewarded for solving a challenge to being rewarded for solving a site.

I recomend against using the user-voted "dif" column. It is extremely unreliable. For some people it represents how difficult it is, for others how easy it is and for yet others how balanced the difficulty is. Also, most sites have no more than a handful of votes.
The "average" column is a much better indicator of the difficulty of completing a site.

Whether or not a site internally also uses difficulty for scoring (like rosecode or hackquest) makes little difference. 0% complete is still 0 points and 100% complete is still max points. It's just the curve that's a it different.

My solution:
I propose we ignore all challenges over 200 per site for the score. That should take care of the problems you described.