Judges

I don’t think anyone was suggesting otherwise. The impression I got was that some people wanted to be more Web 2.0-ish rather than to replace the judges.

The solution to the issue of users not rating games is fairly simple: scale each user’s rating by a function of the number of games they’ve played, and normalise them. The function should probably be of games on the site as a whole rather than games in a given year. I suggest something along the lines of

f(x) = x > 20 ? 1 : x < 4 ? 0 : Math.pow(0.9, 20 - x)

For normalisation, don’t display any scores (show a placeholder “Insufficient ratings”) until you have N people with more than 20 ratings. Then take the mean of their mean ratings as the mean of the normalised rating, and the mean of the variance of their ratings as the variance of the normalised rating. Possibly re-evaluate these once a month.

Do a Youtube to the extent that if someone has rated a game then you show them their rating rather than the overall one. That way they won’t notice that their first rating had weight 0. If a game has fewer than 10 ratings show the placeholder.

Note: I’m not a statistician - I have an A/S in statistics and nothing more. Someone who really knows what they’re talking about can probably suggest improvements to this scheme. In particular, it needs a workaround to the fact that each person’s distribution will probably be non-normal - if ratings are 1 to 5, then I expect 2 to be less used than 1 or 3.

The judges list was actually the thing that pushed me over the edge in deciding to enter this competition, so I’d like to see that stay the same.

However, I wouldn’t mind seeing a user judging system if it was kept separate. Perhaps ratings could be weighted by the number of games the rater rated? You could just do a weighted average with %reviewed as the weight. For example, if there were 10 games, and your game got a 5/5 from somebody who rated all of them, a 4/5 from somebody who rated half of them, and a 1/5 from somebody who just rated yours, you’d get a user rating score of (51+40.5+1*0.1)/(1+0.5+0.1)=4.4375.

Since the raters would all know this is how it works, it’d also be motivation to rate more games, as it makes your vote count more.

That might motivate people to just dole out totally random reviews.

I don’t think it’d motivate people to dole out random ones. Your motivation in sending out extra reviews would be that you want your voice on one game to be heard more, so applying random reviews is counter to your goal: it muddies the whole scoring system, so your review matters less.

If somebody came in and gave one game a rational, fair-minded view, and wanted to increase what they have to say, the natural course of action would be to go do that to other games.

But if they hated your game, they’d probably just go say they love the rest of the games to broaden that impact. Or vice-versa. So that’s definitely a weakness in the system.

I don’t see why the details of the system have to be published so that people can game them.