soszip2 - brute-force comression utility

Contests 4K Game Competition - 2011
#1

Hi!
I believe this is my first post, and I hope it’s a nice way to say hello.
Since some time, I’m developing an entry for this compo (but I’ll make a thread when it’s playable enough to show off) and I thought about making a custom zipping utility aimed at Java4k compo. Well, I guess I did it.

I made an LZMA (7zip algorithm) based compressor, which compresses a single file, storing only critical ZIP stuff inside. It’s a dumb tool and uses brute-force trial and error way to get the best possible options. It does around million iterations, but for a 4k game it’s usually enough to stop after a thousand. OH, And it saves the zip any time it finds it smaller than the last iteration, so you can break by pressing Escape at any time. Small help is included in a command line.

I hope it will come in handy, since making this just for myself would be a major waste of time.

Coolest screenshot ever:

Download (~30kb)

Cheers,
Sos

!!! UPDATE !!!
I added a small batch file ( sospack.bat ) that should strip some more redundant stuff using pack200 before the actual compression. You need to have JDK in your path for that. I’d love to hear some performance feedback on that one as well

PS. As a proof, I compressed the application with itself, so get yourself a 7-zip before downloading!
PS2. I don’t compress app with itself anymore since I added a second file
PS3. Files produced by this are not intended for further stripping (mostly header stripping), but should work alright as-is. It’s probably not as good as other tools around but it might prove better in some circumstances. Do try it.

#2

To be useful the resulting file must be able to be decompressed under standard ZIP compressions, since that is what the Java libraries use.

#3

I tried it out on Oh My Glob! but after an hour the resulting file didn’t get any smaller than 6234 bytes, whereas it’s compressing to 3662 bytes using KZip under Riven’s tool.

Don’t let this discourage you; I’m sure you’ll get there if you keep working on it.

#4

Well, I’m using LZMA SDK for compression, so I don’t really have control over how’s that part done. Yet, I build zip headers myself., and that saves a dozen of bytes compared to 7zip zip. I might tweak stuff but it might not get any better as well.

Well, yes it is! LZMA is a standard compresion method for ZIP, no worries! Java runtime handles it rather easily too.

#5

One thing to note about the LZMA compression is that it will only work for Jar files. There’s a method that some of us use for converting a zip compressed file into a gz file (for pack200 compression), and the gzip format (from what I’ve seen) doesn’t support LZMA.

#6

Zip = multiple entries of GZ

If a compression works in ZIP, it works in GZ.

In my compile-and-shrink service, I use 7z (which is LZMA) too for GZ files.

#7

Frankly speaking, I haven’t done any benchmarks and comparisons, but I’d love to see peoples results.
Also: Can this be used with your tool too, Riven?

#8

If your program has good results within about 5 seconds, then I might try it.

#9

Trying to integrate it now…

#10

oh wait, the jar format supports the lzma algorithm? since when?

#11

No, 7z is using an algorithm like LZMA, yet ZIP compatible.
It works, as my shrinker output works :slight_smile:

#12

It usually gets best result within first 1000 tries, I can tweak it to do it within 5 seconds.

Also, ZIP specs:
http://www.pkware.com/documents/casestudies/APPNOTE.TXT

It states support for LZMA (method 14) in version 6.3. Yet, I guess many zippers would fail to decompress.
I tried creating a JAR and it runs just fine.

#13

Hm… so Sos… seems you’re really using LZMA, as opposed to 7z (which is something else than 7zip).

progrd.java6.s2.pack.gz => 3472 bytes => [url=http://indiespot.net/app/java-four-kay-test?task=html&resource=download.49c5cf2a366148c8eb4d229fbeb0a81b.progrd.java6.s2&className=W]test[/url] (this one doesn't work) progrd.java6.bj.pack.gz => 3691 bytes => [url=http://indiespot.net/app/java-four-kay-test?task=html&resource=download.49c5cf2a366148c8eb4d229fbeb0a81b.progrd.java6.kp&className=W]test[/url] progrd.java6.pk.pack.gz => 3698 bytes => [url=http://indiespot.net/app/java-four-kay-test?task=html&resource=download.49c5cf2a366148c8eb4d229fbeb0a81b.progrd.java6.s2&className=W]test[/url] progrd.java6.7z.pack.gz => 3839 bytes => [url=http://indiespot.net/app/java-four-kay-test?task=html&resource=download.49c5cf2a366148c8eb4d229fbeb0a81b.progrd.java6.7z&className=W]test[/url] (existing 7z version that works)

The resulting files are smaller, but they are not zip compatible. :-\

#14

Wow! cool :wink:

I don’t really know how your service works, but I think the issue is that gz is not exactly zip and output packed with my tool should be at the end of the chain, directly decompressed by JVM. I’ll provide a timeout functionality today (think 2-3 hrs from now).

EDIT: Updated! Added timeout function. Specify number of seconds (integer) as a third parameter, no timeout if 0 or omitted. Original download link from the first post applies.

#15

Well… it is. If you don’t believe me, read the ZIP spec.
You can simply strip data off of a zip and end up with a GZ.
It’s how all other versions (kzip, bjwdeflate, 7z) are ‘converted’ to GZ in my service. So if yours doesn’t work, it’s either because your LZMA isn’t zip compatible, or I have some silly bug in my code :slight_smile:

I guess it’s the former, as I can’t extract your soszip2.zip with any other unzipper than 7z. Even WinRAR can’t extract it.

#16

WinRAR cannot extract LZMA zips at all, try to make one with 7-zip and see. I guess really not many archivers can do that (WinZip 12 claims to do so, I have to try it and see if it works).

I can make it output raw LZMA with any additional info if you can integrate that. Just tell me what you need.

EDIT: WinZIP 15.0 does fine extracting my zips.

#17

Anything that the JVM fails to load (non-GZ and non-ZIP) won’t be of any use for compile-and-shrink.

I’m afraid your hard work isn’t usable for Java4K

The Zip files your utility creates are (obviously) much larger than the stripped-to-gz zips that kzip/bjwdeflate/7z create, IIRC your generated zip was 4050 bytes, way bigger than the kzip GZ.

#18

Sos just curious, can your tool be used generically just to produce small .lzma files (non 4k and zip stuff) ?

also what language is your tool written in? be useful to be able use it on non windows systems.

#19

I run his exe on Linux through Wine.

#20

:frowning:
I’ll try to think something out. ZIP headers weigh 98 bytes + 2*filename length, so the overhead is actually relatively big.

It’s written in C, actually file extension say cpp, since MSVC 6.0 name mangling is necromancy. And i might have thrown in some bools by accident :stuck_out_tongue:
Oh! I use some WinAPI functions for getting file time/date and polling keyboard. But i guess it’d be minutes to port that. I can provide source, but it’s really crappy, and I really don’t have time to port it myself or to tidy this up but i might be of assistance if needed.

PS. Cool, it runs in wine :wink: So I guess I won’t have to expose my crappy coding techniques :stuck_out_tongue: