[slightly ot]pack200 and gzip

Networking & Multiplayer
#1

I was thinking again (oh oh).

Y’know all those image formats, loading time, awt depedancies and stuff.

With 1.5 they introduced that new compression stuff for webstart… see here and details here.

I just toyed around with it… and yea it seems like you get a slightly better compression. It’s actually sometimes even better than PNGs (yea PNGOUTed PNGs can be bigger than gziped TGAs).

“GZIP Compression
[…]
One can get the highest degree of compression using gzip to compress an uncompressed jar file vs. compressing a compressed jar file, the downside is that the file may be stored uncompressed on the target systems.”

Downside huh? Well, you use jars with zero compression (store) and put that into a highly compressed gzip archive. If the archive (the zero compressed jar) gets stored on the client side in an uncompressed state (without beeing gzipped) loading times should be pretty short (yea you wast some mb on the hdd, but who cares?).

In theory this should allow the shortest loading times, because the physical distance of the files is as short as possible (because they aren’t cluttered around the hdd - y’know fragmentation et all).

Well, we won’t get around ogging. Heh. But it looks like a good trade off. It seems to be reasonable for images you need quickly (splash/loading) or for those bad to jpg/high contrast images like fonts or hand drawn cartoon images. They are quite often even a bit smaller than jpg :slight_smile:

So far I can’t tell if they get stored uncompressed (de-gzipped) or not. For testing that I would need to write quite a lot php.

Oh and lwjgl.jar, lwjgl.dll and lwjglaudio.dll are 255kb (zip [store]->gzip [max]) instead of 310kb (zip [max]). And you can even strip it down to 202kb gasp if you do something like this:

pack200 --modification-time=latest lwjgl-win-new.jar.pack.gz lwjgl-win32-0.92_s.jar

And if you use 7zip for gzipping it gets down to… 195kb :o

(Oh and sorry for the lack of structure… I wrote this and tested that stuff at the same time ;))

#2

Just for the fun of it I tried it with super elvis (I wanted some real world game related data ;))

[btw/ot² the images.jar is ref-ed twice, Cas :P]


[compressed]

       347.077 alloy.jar
       120.654 gamecommerce.jar
       126.056 hallucinogenesis-code.jar
        74.513 hallucinogenesis-fonts.jar
        35.935 hallucinogenesis-game.jar
       772.686 hallucinogenesis-images.jar
     2.048.148 hallucinogenesis-sounds.jar
     1.605.498 hallucinogenesis-sprites.jar
         8.101 jogg-0.0.5.jar
        63.437 jorbis-0.0.12.jar
       267.898 lwjgl.jar
        85.310 lwjgl-windows.jar
        63.512 openal-windows.jar
       208.687 spgl.jar
---------------------------------------------------
     5.827.512 bytes


[stored jars->p200->gz]

       122.633 alloy.jar.pack.gz
        34.113 gamecommerce.jar.pack.gz
        43.176 hallucinogenesis-code.jar.pack.gz
        65.431 hallucinogenesis-fonts.jar.pack.gz
        33.563 hallucinogenesis-game.jar.pack.gz
       743.804 hallucinogenesis-images.jar.pack.gz
     2.022.725 hallucinogenesis-sounds.jar.pack.gz
     1.579.566 hallucinogenesis-sprites.jar.pack.gz
         4.501 jogg-0.0.5.jar.pack.gz
        28.369 jorbis-0.0.12.jar.pack.gz
        74.758 lwjgl.jar.pack.gz
        80.971 lwjgl-windows.jar.pack.gz
        61.171 openal-windows.jar.pack.gz
        55.256 spgl.jar.pack.gz
---------------------------------------------------
     4.950.037 bytes


[stored jars (client sided size)]

       626.028 alloy.jar
       239.690 gamecommerce.jar
       235.318 hallucinogenesis-code.jar
     1.028.851 hallucinogenesis-fonts.jar
       229.290 hallucinogenesis-game.jar
     2.113.141 hallucinogenesis-images.jar
     2.110.245 hallucinogenesis-sounds.jar
     9.878.503 hallucinogenesis-sprites.jar
        13.713 jogg-0.0.5.jar
       112.330 jorbis-0.0.12.jar
       641.848 lwjgl.jar
       214.795 lwjgl-windows.jar
       145.181 openal-windows.jar
       401.869 spgl.jar
---------------------------------------------------
    17.990.802 bytes

It’s about 15% smaller (traveling through the net)… client sided quite the bloat (308%), but it should load faster this way.

Well, I think it’s cool. Every 7th (or so) download is for free or to put it in other numbers… 202 downloads for one gb traffic instead of 171 downloads.

Oh and I think the thread should be moved over to OnLine Game Development :stuck_out_tongue:

#3

Thanks for moving :wink:

So far I got a minimalistic example up and running (apache mod_rewrite plus some lines of php). Well, what’s left is checking of the accepted encoding (it’s a blind guess right now) and some logic.

I hope this stuff will work on my host, too :slight_smile:

#4

Dear diary… :stuck_out_tongue:

It’s done proper now, it works and my host supports it, too. Hooray! ;D

Will do some testing tomorrow… I really want to check it on all major platforms+vm combinations. I’ve also written a small test application (for testing the server) and a really simple thingy which just tells you which archive has been picked (just a frame with a label… “I was a .jar”, “I was a .jar.gz” and “I was a .jar.pack.gz”)… So basically everything is there for doing some quick tests to check if your host supports it and if you’ve setup everything correctly :slight_smile:

#5

http://www.kaioa.com/src/tmp/BounceDemo.jnlp

That’s a small demo, which tells you which archive has been send to you. I forgot to use target/source 1.4 switches while compiling, which kinda voided the purpose of the demo. Well, that’s fixed now :wink:

So far it has been tested on a handfull of linux and win machines running either 1.4.x or 1.5.0. And it really works as intended. I also found out that webstart saves the “E-Tag”, which I generated… so yea it won’t download the archives again if it hasn’t been updated (the etag itself is actually just the timestamp of the last modification plus the filesize in bytes).

#6

Hi oNyx,

I’m facing the same problem now… no Servlet support but I’d like to provide pack200 files… would you mind sharing your php code here ?

many thanks

Lilian

#7

problem resolved with http://joust.kano.net/weblog/archive/2004/10/16/pack200-on-apache-web-server

But I’m still interested by your solution !

Lilian

#8

I actually have 3 solutions.

The one I like most (the one I currently use) uses a redirection header.

.htaccess

AddType application/x-java-jnlp-file .jnlp
AddType application/x-java-archive .jar
AddType application/x-java-archive .jar_gz
AddType application/x-java-archive .jar_pack_gz
AddEncoding pack200-gzip .jar_pack_gz
AddEncoding gzip .jar_gz
RewriteEngine On
RewriteBase   /jws
RewriteRule jar/(.*) jarbounce2.php?file=$1 [NE]

jarbounce2.php

<?php
if(!isset($_GET['file']))
{
	not_found();
}

$gzip_accepted=FALSE;
$pack_accepted=FALSE;

if(strpos($_SERVER['HTTP_ACCEPT_ENCODING'], 'pack200-gzip')!==FALSE)
	$pack_accepted=TRUE;
if(strpos($_SERVER['HTTP_ACCEPT_ENCODING'], 'gzip')!==FALSE)
	$gzip_accepted=TRUE;

if(file_exists('jars/'.$_GET['file'].'_pack_gz') AND $pack_accepted)
{
	bounce_off($_GET['file'].'_pack_gz');
}
elseif(file_exists('jars/'.$_GET['file'].'_gz') AND $gzip_accepted)
{
	bounce_off($_GET['file'].'_gz');
}
elseif(file_exists('jars/'.$_GET['file']))
{
	bounce_off($_GET['file']);
}
else
{
	not_found();
}

function bounce_off($filename)
{
	header('HTTP/1.1 301 Moved Permanently');
	header('Location: http://kaioa.com/jws/jars/'.$filename);
	die();
}

function not_found()
{
	//you could do something else here, like logging and/or redirecting
	//it to your usual 404 page
	header('HTTP/1.0 404 Not Found');
	die();
}
?>

Structure is like this:
/jws/.htaccess
/jws/jarbounce2.php
/jws/jars/foobar.jar
/jws/jars/foobar.jar_gz
/jws/jars/foobar.jar_pack_gz

In the jnlp I just refer to “jar/foobar.jar” (Note that its “jar” and not “jars”).

Oh and dont forget to adjust that header location line.

#9

thanks for sharing !

Lilian