Fewer games due to...

[pjt33]

Bear in mind that the judges aren’t paid to do this, have limited free time, and may not be in the same timezone.

[Demonpants]

@zeroone

I think the judges should certainly collaborate. When they worked alone, they came out with independent scoring systems that were incompatible with each other. Taking the averages of those numbers to determine rank never made any sense. In fact, the numbers themselves never really made any sense. Judges seemed to be able sort the list, but the game evaluation scores made little or no sense. The score categories made even less sense and none of the judges seemed to pay attention to them anyway. For example, there were many games that had very little game play value, but were technically very difficult to implement. I never saw that reflected in the judge’s scoring. I never saw the judge write in his comments something to the effect of, well this game is no fun, but the 3D graphics are unbelievably amazing. Hence game-play = 10, technical = 95.

I think the most efficient way for the judges to collaborate would be to hold elimination rounds. The judges could separately create ranked lists. Then they could collaborate and agree together to discard half the entries. Then they could go back to the remainder and re-rank. This could be repeated until we have a top 5. The judges could argue together until they agree on ranking those top 5. The results of each phase could be posted and it would be interesting to see these partial results over time. In fact, a chat log between the judges arguing over which entries stay and which goes would be awesome to read.

I don’t know if you’re paying attention. From what I have seen of the system, doing this elimination would make very little difference. It’s just a weighted average. If I give every single game a score between 90 and 100 (giving out nothing below 90), then effectively 90 is a 0 for me and 100 is still a 100 - I’m just reducing my effective range so I only have 10 options. Or, if I only give out a 0 through 10, then all 10s are basically 100s. You might see a 10 on my entry and a 100 on Appel’s, but in terms of finding the winner nothing changes.

More importantly, I don’t know if you actually read all the reviews we painstakingly wrote (for free).

Here is a small sampling of reviews I did:
4bidden Fruit - Overall 96% Technical 86%

This is a truly wonderful little 4k gem and I had a great time collecting all the crescents. One of the greatest strengths of 4bidden Fruit is its very calming atmosphere and simple gameplay. Despite the 4k file size, I really felt like I was playing in a fully fleshed out garden that hadn’t been hampered by any such limitations. The crescents were just difficult enough to find that I was not bored or frustrated - instead I enjoyed climbing higher and higher in my attempts to find more. Some sort of radar would have aided the game, however, as towards the end it could get frustrating finding the last few crescents.

Technically the game is also quite impressive, mostly because of the multiple fractal generating algorithms you were able to pack into 4k. The control and gravity are fairly standard fare, however.

F-Zero 4K - Overall: 66% Technical 84%

I think you did a great job capturing all the different elements of F-Zero into this little 4k game (except boost power), which is definitely an achievement. In addition, the dynamic 3D flat-down Mario Kart style level looks great, and the cars are very detailed for 4k.

I didn’t feel that you really captured the sprit of F-Zero, nor was the gameplay particularly exciting. I think F-Zero should inspire a sense of extreme speed and craziness, but your game felt more like I should be driving a kart.

Frequent Flier - Overall: 68% Technical 91%

I’m absolutely amazed you pulled it off in a 4k game. I (honestly) wouldn’t even know how exactly to do this in any Java game.

Maybe I was singing out of its allotted range, but I couldn’t help but to think that even a real singer would have trouble with this.

Ultimate Tic-Tac-Toe - Overall: 50% Presentation: 85%

Well, there’s not much to say about this. It’s very slightly larger tic-tac-toe. The enemy AI is very strong (not bad for a solved game), to the point where I either tied or I lost. The balls reflecting each other looks really great… really, what else can I say about this?

4bsolution - Overall: 80% Presentation: 99%

I found this game fairly entertaining but it tired me out relatively quickly. The idea is sound enough, and certainly has potential, but it all just takes too long and is too easy.

The presentation in 4bsolution is easily close to the absolute best this year. I really felt like a cool little critter swimming underwater gathering sentient little dots.

Are those examples perhaps indicative of the fact that “There were many games that had very little game play value, but were technically very difficult to implement. I never saw that reflected in the judge’s scoring.” is probably the result of you not actually reading all the reviews? From what I saw of every judge, if there was a game they thought overall wasn’t very good but was very technically amazing, they gave out low overall and high technical. Also, when games looked great, they got high presentation. So… what’s your point, other than that you’re going on about something that’s not lacking?

One thing I will agree with you about is that a chat log or something like that would be very cool. If I were you I’d be very interesting in seeing the thoughts of my judges laid out on paper. Unfortunately, this is not feasible for the reasons pjt33 already mentioned. We’re all over the world and we’ve all got limited time. Usually I play a few games when I’m on the train or something like that, every day to and from work. There is no internet when I’m on the train, so chats are not an option. Plus, last year there were so many games that Appel had to extend the deadline so many of us (including me) could finish up our reviews. Pushing it even further so that we could then discuss our favorites and all that seems like it would be in no way worth it. Had we discussed last year’s entrants, we would have all overwhelmingly said that Left 4k Dead was the clear winner. And guess what won? We may have had a bit of argument about the next 4 places (my tie for first ended up in 4th place), but really the changes are so insignificant that it’s not going to make a big difference. And, once again, having to get all the judges together at the same time would be borderline impossible.

PS - Last year’s reviews are here, if you want to look over them again. http://www.java4k.com/index.php?action=games&method=reviews&cid=5

[zeroone]

I greatly appreciate the long written reviews that the judges made and I actually did read the majority of them. I never understood the categories. Just off the top of my head, some things I look for in these games: was it fun to play, does it have replay value, how much depth does the game have (level complexity and variations between levels), graphics quality, music/sound quality, how faithfully it captures the spirit of a game it is imitating, technical complexity of the algorithms used, game difficulty, and how well the controls work. I do not think that Overall, Presentation and Technical capture enough and I noticed that the percentage of all 3 seemed closely related. I would also like to know how far the judges explored a game. In some games, the depth of the game is not revealed until you get to higher levels. If the judges give up early, they never get to see significant details of the game. That is part of the reason I feel the collaboration between the judges would help. Some will venture further and share their findings.

[Demonpants]

@zeroone

I greatly appreciate the long written reviews that the judges made and I actually did read the majority of them. I never understood the categories. Just off the top of my head, some things I look for in these games: was it fun to play, does it have replay value, how much depth does the game have (level complexity and variations between levels), graphics quality, music/sound quality, how faithfully it captures the spirit of a game it is imitating, technical complexity of the algorithms used, game difficulty, and how well the controls work. I do not think that Overall, Presentation and Technical capture enough and I noticed that the percentage of all 3 seemed closely related. I would also like to know how far the judges explored a game. In some games, the depth of the game is not revealed until you get to higher levels. If the judges give up early, they never get to see significant details of the game. That is part of the reason I feel the collaboration between the judges would help. Some will venture further and share their findings.

Okay, that makes some sense. Yeah, we’re going to eliminate the 3 categories this year which should help things. Without the categories, I would give a crappy game with cool technical achievements a higher score than I would otherwise, etc. I typically try to play through the game as much as it appears it has - that is, until I win or I get so fed up I want to quit. For games that don’t have a way of winning I usually play until I feel like I’ve gotten the gist. Obviously I’m just using my judgement here - I can certainly be wrong and miss something. But if someone’s game doesn’t have a way to hook me until I get to that point, then does it really deserve the points?

[pjt33]

@zeroone

how faithfully it captures the spirit of a game it is imitating

That requires you to know it. I did feel that there was a bias in favour of clones of games the judges knew last year - games which I thought were better got worse marks than games which got comments about nostalgia.

[Demonpants]

@pjt33

That requires you to know it. I did feel that there was a bias in favour of clones of games the judges knew last year - games which I thought were better got worse marks than games which got comments about nostalgia.

That’s very true. Being the youngest judge (I think) but a decade or more, I noticed a lot of judges say “awesome remake, reminds me of being a kid!” whereas I had said “feels retro, but it’s kinda boring,” etc. Their scores would often be mountains higher than mine for these entries. Nothing we can about this, though, and I think it’s fine actually. Think of the Olympic judges for figure skating - if someone does a piece that reminds them of their childhood they will probably give an uncharacteristically high score. It’s just the fact of it.

[TheAnalogKid]

Which is still biased.

[Alan_W]

Hum… maybe I should remake pong then ;D
My nostalgic favourites: Asteroids, Pacman, Defender, Missile Command, Zaxxon.

[steveyO]

hehe yeah… My 2nd entry this year will be based on a 1987 Acorn Electron game… Would be suprised if anyone on JGO has heard of it, never mind played it… So no unfair advantage for me on this one

[Demonpants]

@TheAnalogKid

Which is still biased.

Well yeah, but that’s the nature of asking one person’s opinion of something else. You can never had objective judging.

[nonnus29]

Sorry, but this all sounds like whining to me. When I judged in 2006(?) I specifically scored on a few criteria and gave bonus points for things like technical difficulty and innovative game play. There are clearly better ways to judge, you can never make everyone happy (ie the ‘losers’). Seems to me that best games have always won or been at the top anyway so the point is moot (imho).

Edit; I even documented my process in a thread: http://www.java-gaming.org/index.php/topic,12824.0.html