In-Memory Data Compression

Miscellaneous Topics
#1

So I have an image cache set up for what could very likely be pretty big images that I’d rather not re-render if I don’t have to. The problem is that this quickly eats up all of my memory (we’re talking on the order of 100 MB or more). And seeing as how I don’t need to have ALL of the images displayed at once (in fact, I really only need a few of them), I want to take the image data for the rest of them and compress them, and then uncompress them into full-out images when I need to draw them (and forget about the previous uncompressed image data, as I won’t be displaying it anymore).

I know that Java comes with some Zip utilities, but most of this looks like it wants to work with FILES rather than just data in memory. I would prefer to leave the filesystem out of this completely, as there’s no point in saving these images to disk. (It just takes a while to render each one, and I don’t want the user to have to wait forever each time they look at one they’ve already seen – I’m talking specifically about pages in a PDF document, if that helps clarify at all.)

I have also found an ObjectOutputStream, and an equivalent input stream, but these require Serializable objects, and I can’t find any image classes that are. And even if it there were, I can’t find a way to take an OutputStream and use its data (perhaps right away) as an InputStream without writing to disk and reading the file back.

In a nutshell, I need to:

  1. Compress image data in memory (zip would be fine)
  2. Access compressed image data in memory
  3. Decompress data in memory to an Image

Does anyone have any ideas on even where to get started with this?

#2

DXTn compression is what you are looking for here. File formats like DDS can allow for DXTn compression…

The reason for choosing DXTn compression over zip for instance is how fast you can decompress. Decompression is rather quick with DXTn, quick enough that GPUs can decode them on the fly with little to no cost at rendering speed.

Just wikipedia DXTn compression on the theory behind it. Spasi has a nice tool (JSquish) that can compress files to DXTn, its in the Shared Code section i believe…

DP :slight_smile:

#3

That’s not correct. Zip works with Streams not Files. You can store them in memory simply with.


ByteArrayOutputStream bout = new ByteArrayOutputStream();
ZipOutputStream zout = new ZipOutputStream(bout);
// all operations on the ZipOutputStream zout will be stored in bout.

To retrieve the data just put bout.toByteArray() into a ByteInputStream that feeds a ZipInputStream.

On the other hand, what darkprophet said sounds better, except that is a separate library.

#4

The problem with DXTn is that it’s a lossy format. That might or might not be acceptable.

With GZIP you get roughly the compression factor of PNG… (that might or might not be acceptable)


   public static final byte[] gz(byte[] input)
   {
      try
      {
         InputStream in = new ByteArrayInputStream(input);
         ByteArrayOutputStream baos = new ByteArrayOutputStream();
         OutputStream out = new GZIPOutputStream(baos);

         transfer(in, out, 8192);

         return baos.toByteArray();
      }
      catch (Exception exc)
      {
         exc.printStackTrace();

         return null;
      }
   }

   public static final byte[] ungz(byte[] input)
   {
      try
      {
         InputStream in = new GZIPInputStream(new ByteArrayInputStream(input));
         ByteArrayOutputStream out = new ByteArrayOutputStream(input.length * 2);

         transfer(in, out, 8192);

         return out.toByteArray();
      }
      catch (Exception exc)
      {
         exc.printStackTrace();

         return null;
      }
   }

   public static void transfer(InputStream in, OutputStream out, int bufSize) throws IOException
   {
      byte[] buf = new byte[bufSize];
      int r;
      try
      {
         while((r=in.read(buf)) != -1)
            out.write(buf, 0, r);
      }
      catch(EOFException exc)
      {
          // ignore
      }
      in.close();
      out.close();
   }
#5

Thanks so much for the help!

For now, I think that lossy is not what I’m looking for, so I’ll go with the zip code that you provided as a basis. I don’t know why I didn’t see the ByteArrayOutputStream / *InputStream classes before. :-\

#6

They aren’t really magic. Even if they weren’t there, you could have written them yourself once you needed them.