I’m working on this, along with a lot of other statistical analysis. It’s interesting to note that appel has the largest variance in scores in all categories.
I admit this is a problem. The reviews got finished so late that there was little time for afterthoughts.
I am now pondering with the idea of skipping these zero scores, and only averaging the score that is above zero. This WILL affect the order of the games, even in the top 10 and even in the top 3, although the #1st place is firmly held by Left 4k dead.
I can easily correct this, and in fact have the code ready for it, but I need “go ahead” from you guys.
Just do it ™.
However you should probably also put up the following variants:
- Remove all the highest scores for each entry
- Remove all the lowest scores for each entry
- Remove all both the highest and the lowest for each entry
- Remove a particular judge’s results for each entry
- Make all the scores for every game but the one selected zero (ok, maybe not this one ;)).
It’d hardly be fair to judge some entries on 4 judge’s results and some on 5 given the disparity in the judges ranges of scores and approach to judging.
This could go on and on and on…
One could also argue that it’s a coding contest and failing with an NPE is a failure to code to some API or framework given how many other games managed to run on the same setup.
Kev
Without any hesitation, go ahead, appel ! Skipping these zero scores would be fairer for everyone.
And too bad for me if my own game looses thirty places ! :o ( humor of course… )
Morre helped me realize this problem, and I’ve taken the steps to correct it.
The score chart has now been corrected!
This is entirely my fault, so anyone who now sees his game drop a few spots can only blame me. But in the end, I think this is more fair for a whole lot people.
I would certainly appreciate it, appel. Let’s see what the rest of the community thinks. Perhaps a poll?
EDIT:
I see you just changed it. Thanks a bunch!
Please, just keep in mind that I want the community to have a say in this! If they preferred the old version, I will accept it (although I don’t have to like it!). I don’t want to be the one ruining the contest for somebody. 
No, this was clearly an error, which has now been corrected.
An error doesn’t become an mistake until you refuse to correct it, and I don’t want to make this into a mistake.
Oh, I forgot:
Congratulations, Markus!
Left 4k Dead was a very worthy winner, and one of the best 4k games to date. Well done!
EDIT:
Also, I would again like to thank all the judges and the organizers for amazing work! I’ve had a lot of fun this year (as always!)
I almost forgot also:
Congratulations to the WINNERS and all the game developers! Hopefully you’ll participate again next year 
Omg, yay! =D
I’m SO spending all day tomorrow bragging.
I love the reviews, they’re very detailed and shows that the judges actually care about the games they’re judging. Thanks for putting in the effort, and a BIG thanks to appel.
So what am I supposed to do until Java4k 2010? :-\
The way to handle the disparity in ranges of scores is to normalise. Doing that and discounting 0s gives:
Sorting category Overall by Normalised means ignoring zeros
1. Left 4k Dead 94.5
2. 4bidden Fruit 90.1
3. Bridge4k 88.8
4. Pixeloids4k 88.2
5. Hunt 4k Bread 86.4
6. Keggle 85.1
7. Robo rampage 84.1
8. jm00 - a boomshine clone in 4k v1.2 84.0
= Red Baron 4K 84.0
10. Fortress4K 83.5
11. Splosh4k 82.9
12. Putty Shuffle 82.3
13. World Rally Driver 4k 82.0
14. Tekicars4k 81.6
15. Just Get Bigger 4K 81.4
16. BlockBuster4k 80.3
17. NiGHTS 4k 79.8
18. MEG4kMAN 79.1
19. Q*Bert4K 78.7
20. Conquest of Planets 78.5
21. Invaders 78.2
22. RevolvoMan-4K 77.8
23. 4x4k 77.5
24. Jetp4k 76.4
25. 4bsolution 75.9
26. Space Paranoids 2D4K 75.6
27. Critters4k 75.5
= Ice Fighters 75.5
29. F-Zero 4K 75.4
30. Kart 74.6
31. J-Type 74.5
32. Asteroid Alert! 74.4
33. Run Over Zombies 74.0
34. Super Marble World 73.2
35. Sea Spin 72.8
36. Word Twister 72.3
37. Sea through 70.5
38. RhythmSphere 69.8
39. Treasure4k 69.5
40. submarin 68.1
41. Bio 4K 67.3
42. HexoDama 66.6
43. Honey4kKeeper 66.1
44. Doodles 66.0
45. Grasshopper4k 65.6
46. Crimsonland4k version 1.001 65.5
47. Alienz 4K 64.5
= Star 4x 64.5
49. Crack Tower Defense 63.2
50. Gravitational Fourks 61.2
51. Ultimate Tic-Tac-Toe 60.9
52. Anyone 4 Tennis? 59.8
= PacPitfall4k 59.8
54. Coffee Shop Puzzle 59.0
55. Infinite Platformer 4k 57.0
56. Frequent Flier 56.4
57. Deathchase 55.5
58. BlisterBall 53.0
59. Scr4mble 52.5
60. Dragon 4k Boxing 52.2
61. Arena4k 51.9
62. Beez Helper 51.6
63. i4kopter 50.6
64. QuadSquad4k 50.1
65. Virtual On 4k 44.3
66. 4K Maze 37.5
67. Desert Bus 24.7
Code to follow, because it doesn’t fit in one post.
[quote]One could also argue that it’s a coding contest and failing with an NPE is a failure to code to some API or framework given how many other games managed to run on the same setup.
[/quote]
The fact that you don’t know which API or framework is a hint that it’s not necessarily the coder’s fault. My game broke with a version of Java which was released after I finished writing it. I don’t know which API or framework broke, nor in what way, because Webstart is rather sparse with its error messages. The only error message it shows seems to be an internal one.
Had to split it because of message length restrictions. See the TODO.
import java.util.*;
public class Analyse4k{
private static String[] areaNames={"Overall","Technical","Presentation"};
public static void main(String[] args){
// Note: values extracted automatically from the HTML page.
List<Game> games=new ArrayList<Game>();
// TODO Copy lines from next post here.
// Analysis by category. Ignore zeros.
double[][]areaJudgeMeans=new double[3][5];
double[][]areaJudgeVars=new double[3][5];
for (int area=0;area<3;area++){
for (int judge=0;judge<5;judge++){
int t=0,t2=0,n=0;
for (Game game:games){
int val=game.scores[area][judge];
if (val!=0){t+=val;t2+=val*val;n++;}
}
double mu=t/(double)n;
double var=(t2/(double)n)-mu*mu;
areaJudgeMeans[area][judge]=mu;
areaJudgeVars[area][judge]=var;
}
}
// Original values:
//printResults(games,new SimpleMean(),0);
// Current values:
//printResults(games,new MeanIgnoringZeros(),0);
// Fairest algorithm, assuming there should be no penalty for scoring 0s?
printResults(games,new NormalisedMeansIgnoringZeros(areaJudgeMeans[0],areaJudgeVars[0]),0);
}
private static void printResults(List<Game> games, ScoreAlgorithm alg, int area){
Collections.sort(games,new GameComparator(alg,area));
int longestName=0;
for (Game game:games) if (game.name.length()>longestName) longestName=game.name.length();
System.out.println("Sorting category "+areaNames[area]+" by "+alg);
int prev=Integer.MAX_VALUE,i=0;
for (Game game:games){
i++;
int score=alg.score(game,area);
String pos=score==prev?"= ":(i+".");
System.out.print(pad(3-pos.length())+pos+" "+game.name+pad(longestName+4-game.name.length()));
String sc=(score/10)+"."+(score%10);
System.out.println(pad(5-sc.length())+sc);
prev=score;
}
}
private static String pad(int n) {
char[] foo=new char[n];
Arrays.fill(foo,' ');
return new String(foo);
}
private static class Game {
public String name;
public int[][] scores=new int[3][5];
public Game(String name,int[][] scores){
this.name=name;
this.scores=scores;
}
}
public static interface ScoreAlgorithm {
public abstract int score(Game game, int area);
}
public static class GameComparator implements Comparator<Game> {
private final int area;
private final ScoreAlgorithm alg;
public GameComparator(ScoreAlgorithm alg,int area) {
this.alg=alg;
this.area=area;
}
public int compare(Game g1,Game g2) {return alg.score(g2,area)-alg.score(g1,area);}
}
public static class SimpleMean implements ScoreAlgorithm {
public int score(Game game,int area) {
int total=0;
for (int i=0;i<5;i++) total+=game.scores[area][i];
return (total*10+2)/5;
}
public String toString() {return "Simple mean";}
}
public static class MeanIgnoringZeros implements ScoreAlgorithm {
public int score(Game game,int area){
int total=0,n=0;
for (int i=0;i<5;i++) if (game.scores[area][i]!=0) {total+=game.scores[area][i];n++;}
return (total*10+n/2)/n;
}
public String toString() {return "Mean ignoring zero values";}
}
// Normalise to mean 70, variance 250.
public static class NormalisedMeansIgnoringZeros implements ScoreAlgorithm {
private final double[] means;
private final double[] vars;
public NormalisedMeansIgnoringZeros(double[] means,double[] vars) {
this.means=means;
this.vars=vars;
}
public int score(Game game,int area) {
double total=0,n=0;
for (int i=0;i<5;i++) {
if (game.scores[area][i]!=0) {total+=70+(game.scores[area][i]-means[i])*250/vars[i];n++;}
}
return (int)(10*total/n+.5);
}
public String toString() {return "Normalised means ignoring zeros";}
}
}
That may be a hint to you, but the end user doesn’t give a rats ass do they. They click the link and it didn’t work. The fact its an NPE is a hint it is coder error. All of that’s beside the point now really isn’t it, the contest is over, the results are finalized.
Kev
games.add(new Game("4bidden Fruit", new int[][]{{96,76,90,100,90},{97,70,90,86,90},{96,84,85,96,90}}));
games.add(new Game("4bsolution", new int[][]{{91,65,70,80,90},{98,60,75,80,85},{98,80,70,99,80}}));
games.add(new Game("4K Maze", new int[][]{{30,50,20,72,70},{15,55,20,74,70},{10,50,20,1,60}}));
games.add(new Game("4x4k", new int[][]{{67,80,70,82,92},{83,85,75,90,85},{70,85,70,90,80}}));
games.add(new Game("Alienz 4K", new int[][]{{53,82,55,92,70},{63,75,55,75,65},{60,85,55,75,70}}));
games.add(new Game("Anyone 4 Tennis?", new int[][]{{60,65,70,62,70},{62,82,75,84,80},{75,80,65,79,65}}));
games.add(new Game("Arena4k", new int[][]{{69,62,60,75,50},{62,73,60,60,55},{55,70,60,40,50}}));
games.add(new Game("Asteroid Alert!", new int[][]{{59,82,80,87,75},{78,92,80,94,65},{60,87,75,94,75}}));
games.add(new Game("Beez Helper", new int[][]{{45,60,40,86,70},{50,73,45,75,65},{55,64,55,89,70}}));
games.add(new Game("Bio 4K", new int[][]{{54,93,70,68,70},{50,97,60,72,65},{60,90,60,81,65}}));
games.add(new Game("BlisterBall", new int[][]{{62,40,80,62,60},{50,50,75,45,65},{40,40,75,15,50}}));
games.add(new Game("BlockBuster4k", new int[][]{{48,92,80,92,85},{46,85,70,86,85},{55,82,80,93,85}}));
games.add(new Game("Bridge4k", new int[][]{{92,90,0,92,90},{92,92,0,100,95},{70,80,0,89,85}}));
games.add(new Game("Coffee Shop Puzzle", new int[][]{{78,45,75,72,65},{68,75,70,86,65},{84,83,80,89,75}}));
games.add(new Game("Conquest of Planets", new int[][]{{92,82,75,82,80},{80,88,80,71,80},{80,78,70,68,75}}));
games.add(new Game("Crack Tower Defense", new int[][]{{65,69,75,62,70},{40,79,75,78,75},{30,74,70,45,65}}));
games.add(new Game("Crimsonland4k version 1.001", new int[][]{{67,58,70,86,0},{55,50,80,82,0},{48,50,65,82,0}}));
games.add(new Game("Critters4k", new int[][]{{84,72,85,89,70},{95,78,75,90,75},{60,68,75,55,70}}));
games.add(new Game("Deathchase", new int[][]{{60,57,70,70,60},{65,70,75,65,60},{70,80,65,70,60}}));
games.add(new Game("Desert Bus", new int[][]{{65,65,20,5,50},{55,75,10,30,50},{55,72,40,50,50}}));
games.add(new Game("Doodles", new int[][]{{72,85,70,78,60},{75,94,80,89,75},{62,83,65,80,55}}));
games.add(new Game("Dragon 4k Boxing", new int[][]{{18,73,60,70,65},{25,70,70,80,65},{38,82,60,90,65}}));
games.add(new Game("F-Zero 4K", new int[][]{{66,90,75,66,85},{67,90,80,84,75},{78,85,75,87,90}}));
games.add(new Game("Fortress4K", new int[][]{{94,92,70,81,90},{88,95,70,86,85},{82,92,75,89,80}}));
games.add(new Game("Frequent Flier", new int[][]{{25,78,0,68,0},{45,88,0,91,0},{25,75,0,42,0}}));
games.add(new Game("Grasshopper4k", new int[][]{{68,82,60,60,80},{70,89,60,78,80},{75,87,65,88,85}}));
games.add(new Game("Gravitational Fourks", new int[][]{{70,66,0,65,0},{73,50,0,68,0},{65,70,0,61,0}}));
games.add(new Game("HexoDama", new int[][]{{72,86,80,45,70},{83,85,75,76,75},{64,80,75,74,65}}));
games.add(new Game("Honey4kKeeper", new int[][]{{42,83,65,81,75},{40,86,65,73,70},{45,90,65,89,75}}));
games.add(new Game("Hunt 4k Bread", new int[][]{{82,82,80,91,95},{90,91,70,88,95},{68,87,75,90,90}}));
games.add(new Game("i4kopter", new int[][]{{56,63,65,40,65},{58,68,65,70,70},{42,72,60,55,60}}));
games.add(new Game("Ice Fighters", new int[][]{{92,86,70,80,75},{98,90,75,89,75},{92,93,75,82,75}}));
games.add(new Game("Infinite Platformer 4k", new int[][]{{48,0,70,74,60},{30,0,75,72,55},{38,0,75,70,60}}));
games.add(new Game("Invaders", new int[][]{{94,85,75,85,75},{85,96,75,82,75},{85,92,70,79,70}}));
games.add(new Game("J-Type", new int[][]{{86,65,75,88,80},{78,60,70,87,75},{72,60,70,86,70}}));
games.add(new Game("Jetp4k", new int[][]{{75,85,75,80,80},{68,90,75,75,70},{76,88,75,85,75}}));
games.add(new Game("jm00 - a boomshine clone in 4k v1.2", new int[][]{{86,82,80,82,92},{74,90,75,92,85},{72,75,75,88,100}}));
games.add(new Game("Just Get Bigger 4K", new int[][]{{85,87,80,68,90},{78,95,75,80,85},{90,90,80,90,90}}));
games.add(new Game("Kart", new int[][]{{80,75,80,82,75},{82,86,80,80,80},{82,82,80,80,75}}));
games.add(new Game("Keggle", new int[][]{{91,90,80,78,90},{90,95,80,86,90},{82,85,75,89,90}}));
games.add(new Game("Left 4k Dead", new int[][]{{100,97,80,99,95},{100,95,75,99,95},{97,93,75,100,90}}));
games.add(new Game("MEG4kMAN", new int[][]{{95,95,75,86,70},{97,98,80,92,70},{95,95,80,92,65}}));
games.add(new Game("NiGHTS 4k", new int[][]{{90,84,0,85,80},{90,88,0,87,75},{95,87,0,100,80}}));
games.add(new Game("PacPitfall4k", new int[][]{{65,0,65,81,60},{80,0,60,92,60},{70,0,65,86,55}}));
games.add(new Game("Pixeloids4k", new int[][]{{92,89,0,91,90},{92,88,0,100,95},{88,82,0,94,75}}));
games.add(new Game("Putty Shuffle", new int[][]{{93,92,75,97,75},{85,90,75,86,80},{85,94,70,85,70}}));
games.add(new Game("Q*Bert4K", new int[][]{{80,0,80,82,80},{75,0,75,86,75},{70,0,75,94,85}}));
games.add(new Game("QuadSquad4k", new int[][]{{55,50,0,65,65},{40,60,0,65,70},{45,45,0,65,65}}));
games.add(new Game("Red Baron 4K", new int[][]{{85,96,70,98,0},{87,95,75,81,0},{92,100,75,96,0}}));
games.add(new Game("RevolvoMan-4K", new int[][]{{94,88,80,81,70},{92,92,75,84,70},{92,92,80,74,70}}));
games.add(new Game("RhythmSphere", new int[][]{{60,73,75,83,75},{75,76,75,89,80},{65,69,70,62,75}}));
games.add(new Game("Robo rampage", new int[][]{{82,93,75,93,85},{80,92,75,84,75},{82,90,74,96,90}}));
games.add(new Game("Run Over Zombies", new int[][]{{70,83,70,92,75},{85,90,75,99,75},{85,87,65,95,75}}));
games.add(new Game("Scr4mble", new int[][]{{55,60,60,52,70},{70,60,65,73,70},{50,60,75,72,70}}));
games.add(new Game("Sea Spin", new int[][]{{78,76,75,81,75},{78,86,75,77,70},{80,84,70,88,85}}));
games.add(new Game("Sea through", new int[][]{{45,73,75,78,85},{60,87,75,96,95},{40,40,70,84,80}}));
games.add(new Game("Space Paranoids 2D4K", new int[][]{{82,86,75,70,80},{90,90,70,87,75},{78,94,75,87,80}}));
games.add(new Game("Splosh4k", new int[][]{{90,73,75,87,95},{85,70,75,90,90},{82,65,70,96,95}}));
games.add(new Game("Star 4x", new int[][]{{60,88,0,61,70},{84,93,0,91,75},{60,80,0,85,60}}));
games.add(new Game("submarin", new int[][]{{40,84,75,66,80},{65,89,70,89,75},{25,95,75,76,85}}));
games.add(new Game("Super Marble World", new int[][]{{76,70,75,92,75},{85,75,75,90,70},{68,65,70,92,80}}));
games.add(new Game("Tekicars4k", new int[][]{{79,90,80,79,85},{82,92,75,74,85},{72,88,75,72,80}}));
games.add(new Game("Treasure4k", new int[][]{{78,70,75,82,70},{72,82,75,72,70},{92,78,70,87,65}}));
games.add(new Game("Ultimate Tic-Tac-Toe", new int[][]{{75,60,60,50,85},{75,60,70,65,90},{84,60,60,85,85}}));
games.add(new Game("Virtual On 4k", new int[][]{{50,41,60,68,55},{72,40,70,79,60},{45,35,60,52,50}}));
games.add(new Game("Word Twister", new int[][]{{56,83,70,84,80},{75,60,60,68,70},{68,74,75,82,80}}));
games.add(new Game("World Rally Driver 4k", new int[][]{{82,90,60,91,95},{82,93,60,89,95},{84,90,50,86,95}}));
??? Did you get an NPE with Gravitational Fourks or are you confusing me with someone else?
I’m not confusing anything, I wrote and you quoted:
If your game didn’t get a zero for failing with an NPE the comment above doesn’t really apply to it does it?
Kev
My game got two zeros. Versions of Java not specified, but with 1.6.0_12 I get
java.lang.InternalError:
****************************************************************
ERROR: the javaplugin.version system property wasn't picked up
by the com.sun.deploy.Environment class. This probably happened
because of a change to the initialization order in PluginMain
where the deployment classes are being initialized too early.
This will break jar cache versioning, and possibly other things.
Please undo your recent changes and rethink them.
****************************************************************
at sun.plugin2.applet.Applet2Environment.initialize(Applet2Environment.java:113)
at sun.plugin2.applet.viewer.JNLP2Viewer.run(JNLP2Viewer.java:195)
at sun.plugin2.applet.viewer.JNLP2Viewer.main(JNLP2Viewer.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.javaws.Launcher.executeApplet(Launcher.java:1388)
at com.sun.javaws.Launcher.executeMainClass(Launcher.java:1246)
at com.sun.javaws.Launcher.doLaunchApp(Launcher.java:1066)
at com.sun.javaws.Launcher.run(Launcher.java:116)
at java.lang.Thread.run(Thread.java:619)
No mention of any NPEs.
So then my comment wouldn’t apply to your game then. Why quote it? Since the results have now been updated to ignore zero anyway, whats to worry about?
Kev
Wow! Excellent Judging! Very helpful reviews. Big thanks to Appel, Chris, and Demonpants for the great comments! Darkfrog, errr, thanks for playing it!
Appel, the Java4K Result page looks great! Excellent job!
Congrats to Markus for a well deserved victory. Gloat away!
Congrats Markus! But you really should have left it till the last week to submit, you probably put off a number of potential entrants.
Anyone else think percentage scores are unnecessarily precise given that the scores from each judge differed by over 50% in some cases. I think out of ten scores would probably be better for future contests.