Judging

Contests 4K Game Competition - 2007
#1

I was just wondering, have you decided on which judging system to use? I myself would still like to see one or another type of voting, possibly approval voting (http://en.wikipedia.org/wiki/Approval_voting) as suggested by one of the members last year. In either case, I think this is a perfect time to discuss this, as the contest has not been launched and will not be for a while.

As opposed to the way it’s been some of the earlier years, let’s make sure the rules for judging are clear, reasonable, and fair before the competition starts :slight_smile:

#2

the past 2 contests (the ones I ran) were resonable and fair, and from what I could tell, clear.

the judging system is already in place. there will be a judges’ choice award, and a player’s choice award. more details on both as the time approaches.

#3

Sounds good, I think a peoples choice award is looonnggg overdue.

#4

Oh, woogley, you misunderstand me :slight_smile:

I never meant to accuse you of anything, I think you did a great job running the contests. However, circumstances and late rules prevented things from being entirely fair. Vague rules on judging criterion prevented the excellent Xero from making the top ten (top three?), instead sending it way down - only because one of the judges couldn’t start it. Had rules said that judge should merely be excluded from that game, the results would have been very different. Clear rules and criterion could have helped!

Besides, I see no point in withholding the judging and/or scoring system from us. The big surprise lies in the games, not in the rules. Should you print them out for us now, we might help you find any flaws and settle any disputes in time before the contest starts. I’m not saying they won’t be good, or even great; I’m saying that no man is perfect. There might be things you haven’t thought of. Give us a change to look at the rules, and things might be better for all parts. Less confusion for us, less final judging headache for you :smiley:

#5

… more details as the time approaches :wink:

most of the judging is based on past experience of the last 4 java 4K contests. this being the 5th contest, I’m making it a point to not involve the community into the mechanics of the contest. for reasons why, take a look at threads such as “who needs judges?” etc.

don’t misinterpret that, I’m not going to be the almighty 4K contest tyrant - I will always have my ears open… but there will be no more “so how do you think we should do this?” kind of threads :wink:

edit: on another note, you can’t blame just the judging system on Xero’s failure (there are many more variables than that). you can also blame the judge who probably only gave it one shot. as much of a jbanes fan that I am… you can also blame him! why? obviously not enough testing… it is well known that java fullscreen on linux is a gamble… heh, even on windows! providing a backup “windowed” version could have maybe saved him the bad surprise. it sucks that the only game that really felt like a genuine arcade game (probably cuz of the fullscreen) scored so low.

also, removing 1 judge because of 1 game is not a flaw in any system - it simply isn’t fair. for example… for various reasons, a while ago I generated a result list without the influence of nonnus29 (mostly because he was a particuarly low voter) to see what the results would be. the difference is HUGE and affects ALL games! check out out:

http://javaunlimited.net/results/ - normal results
http://javaunlimited.net/results2/ - results if nonnus29 had not judged

#6

Approval voting needs many people who vote… especially lots of those who played all games. While its certainly a good voting system it doesnt really work all that well in this case. With only a handfull of entries it would be a different thing tho.

€: trimmed voting is good btw. Discarding the highest and lowing scores and using the average of the rest or median where the vote in the middle is taken (if its an odd number of judges the two in the middle get averaged).

#7

When I judged, I tried to 1) use a systematic process and 2) went for a bell curve distribution. It never occurred to me that the other judges wouldn’t do the same thing. Some judges just awarded 100 points for every game they liked. I, on the other hand felt there should only be a few ‘A’ games.

But like I said before, judging in this contest is pretty meaningless as well. There really needs to game categories, because there are just too many types of games entered. People who like shooters vote high for them, people who like driving games vote high for them, people who value technical pinache vote for that etc…

So how about ‘Best of’ categories? Best shmup, best platformer, best gfx, best etc…

#8

we read your “how I judged” post last year :wink:

we need more diverse judges, yes. but not categories… the major flaw last year with the judging system is that not all the judges used the same system, although I did say give each game your 1-100 vote which I thought was clear. but apparently you saw it differently -shrug-

that’s one of the reasons this year is not going to have a “wanna be a judge? write your name down on this roster” post. an organized judges board will yield better results since they will all use the same system…

#9

Of course I see why you would want to avoid the threads of the earlier years, with something close to flaming happened (I’m blaming nobody). However, I think that was caused primarily because the rules hadn’t been set in time. People hadn’t gotten the time to think about the effects of them, and when they came the unclear parts could not be fixed without somebody getting an unfair advantage or disadvantage. That’s just my point.

I’m not saying we should decide on the rules using another type of voting - I’m saying, reveal the rules quickly. That way, everyone can see how things will be done, and any minor issues can be fixed.

That’s what I meant. Nobody’s perfect. The rules last year were good enough, but the criterion for judging was unclear, despite a good effort to make them clear. If a few more members, perhaps even judges only, had been given the chance to agree on criterion, I think we wouldn’t have the case were one judge voted in a different way.

Oh, and categories can be good - don’t say no just like that, think about it for a while. Personally, I’d say having just a few categories could be a good thing - much like what you’re already doing (a players’ choice and a judges’ choice). These are categories too, you know. Elaboration on why you don’t want more categories would be nice, and I expect too agree with you on that matter. :slight_smile:

#10

I’m not just saying “no” quickly, we have indeed tried categories before. :stuck_out_tongue: for one thing, the categories you’re talking about would require more than 50 games IMO, to really make a serious difference.

I will reveal the rules when its time to reveal them. just trust me, I’m getting good at this stuff :wink:

concentrate more on what entry you’re going to make that’s going to win the contest, and I’ll worry about the things behind the curtain. I am indeed talking to others behind the scenes, particuarly I’m personally pulling together the judges board myself… so don’t worry about it. like I said earlier in this thread, I’m not doing the “wanna be a judge? sign here” post. that was one of the major issues last year [of why the criterion was skewed]

#11

I think you might be right about that, and categorizing games can also be hard. On the other hand I think it would be nice to have some awards like best GFX, Sound, innovative idea, and technically advanced. Doom buggy or that real-time ray-traced game 2005(?) finished somewhere in the middle which is OK since gameplay was nowhere near Miners for example. But I still think they should be encouraged (for being mad enough to implement such an advanced ideas into 4K) and should have won an technical achievement award. That would increase the load on the judges a bit, but I think it might be worth it.

Just my 5c

#12

Java 4K isn’t really big enough for categories. Out the outside it might support 2:

  • Best Game
  • Best Technical Achievement

but then I think that has the equal downside that it encourages people to just write tech demos rather than games, I thought 4K was about how much fun/enjoyment can be packed into 4K, not whether it’s possible to write an academic engine with close to zero gamepay in 4K.

I guess it’d be nice to know what the actual overall aim is meant to be - in the past it’s been kinda fluid.

Kev

PS. Personally I hate the idea of any sort of player/community vote on games. The result would be entirely to dilute, unlikely to get good feedback, hard to police and runs the risk faddy/populatrity winning the day - rather than some considered judging by people explicitly taking the time to do it. (Admitedly some of the comments last year were a bit week - “ok”)

#13

  1. There’s no guarantee that the judge would not have had a crash. (Unless he was running Linux.) While fullscreen tends to crash more often, Java occasionally crashes any time the BufferStrategy pipeline is initialized.

  2. From the comments, it doesn’t sound like the judge ran the game more than once. Why would he have tried the Windowed version if he automatically assumed that the program was broken?

I don’t mind taking my lumps (and I said so last year), but a judge’s evaluation of being unable to run a program should not be accepted until he’s tried some simple diagnostics, like rebooting. In fact, simply trying a second time is usually enough to clear whatever problem occurred. Only after the judge is certain that he can’t run it AND that it’s the fault of the coder should his score be accepted. While I appreciate that you wished you could have changed the score, Woogley (something that really wasn’t feasible), sending it back to him to have him try it again would have far more of a difference.

Similarly, it would have been prudent to reject the JSquares score until it was determined why the judge needed extra components. If the game was outside of the supported versions of Java and components (it wasn’t), then it should have been disqualified. If it was a supported version that was accidently triggering Java to take action, then it should have been worked out. Especially since none of the other judges reported issues in either instance. As it so happens, all the judge needed to do was wait for the 1.4 version of Java to download. He could have easily come back to the game later, and waited for the download to complete in the background.

Basically, 100% of the issues with last year’s judging could have been prevented by rejecting the judge’s submission until the issues were resolved to a satisfactory end rather than accepting the technical issues at face value. Remember, these are game consoles where everything works right the first time. Even commercial games regularly run into technical issues on computers.

takes deep breath

Hi everyone! Ready for another fun and exciting contest? Hope I haven’t missed too much while I’ve been away. ;D

#14

Hey JB :slight_smile:

I didn’t realise the games only had to be playable on Windows - interesting - I feel some nice new hacks coming on :slight_smile:

Kev

#15

Historically, judging was required to happen on Windows machines as they were the common denominator. (I think it was year 2 we discussed this? I’ll see if I can look it up.) Macs were excluded because not many people had them (not to mention 1.4 wasn’t available at the time), and Linux had too many oddities to make for a suitable judging platform.

For example, Abuses 4K Shooter in year 2 was only designed to run on Windows. (Feel free to pop in here Anon666/Abuse. ;)) Because of this decision, Fullscreen was deemed “okay”, and many games went on to use it. (Although you do take a HUGE hit for going fullscreen.) I don’t mind if this changes, but the rules should be codified on this up front. :slight_smile:

#16

Yeah, seems that’s where this cafuffle has come from. Need to get a nice clear target platform specification this year. Personal opinion here would be that it’s better to require support for all platforms in the now Java glory days where it’s actually possible :slight_smile:

Kev

PS. I love the fact you can refer back to “year 2” (and year 1 presumably). 4K it’s just a contest, it’s a legacy man! :slight_smile:

#17

[quote=“kevglass,post:16,topic:28235”]
+1 to that. I do all my work on linux, in fact i only have a litle 8G “Wintendo” partition ready for lanparties in wich i only test the games before posting them.

Even if windows is the main ( actually the only ) gaming plataform, we should encourage the use of java for games taking advantage of one of it’s most usefull features: it’s (almost) plataform independent.

#18

Fascinating, isn’t it? I wonder, has anyone besides me been in every contest since it began? There has to be someone. ???

#19

I kinda feel thats the benefit of these forums being the hub for all the activity. There are sooooo many people here willing to test stuff out on various platform. I don’t have a Mac or a Linux installation but with the help of folks here (and a few other places) I generally get some pretty rigourous testing done.

I was around for Year 2, but didn’t manage to get my entry done in time. :frowning: Year 3, was the big one for me :slight_smile:

Kev

#20

Present.
If you want any testing on Linux I’d be your man.