ClassLocater...only, it's bugged

Based on duncanIdaho’s ClassPath class, I modded this code. Aim is to “find all classes that are instances of a given interface or class” at runtime. I’d been using it OK for a while, and adding small fixes as problems came up (e.g. additional Throwable’s).

But now…I’m getting a massive memory leak, and no idea how to solve it. Mem usage goes from 64MB to over 160MB before process is killed - just from this one method.

Note that I ran this on the same PC 12 hours earlier wihtout problems, and it produce approx 4 classes as a result of the method (which is correct - there are only 4 test instances for the code I’m currently working with).

So…AFAICS there is no reason why this shouldn’t at the very least GC if it spirals out of control; I’ve checked that it (theoretically) isn’t hanging onto references.

EDIT: see the rest of thread for details, but basically the OOME is from too much method calling, hence sidesteps GC. Silly me can no longer recognise an infinite recusive loop in java runtimes (probably simply because it looks different in logging frameworks compared to in DOS prompt where I learnt it :))

I could try to limit by package name using the regex, but I’d like to be fully automatic and not rely on unreliable voluntary naming conventions! Meanwhile, any ideas on why this code OutOfMemoryError’s on 1.4.2_05 instead of GC’ing?

NB: ignore the poor package name; that was from before i got the JGF domain - it will be fixed before I post the JAR!


package com.grexengine.jgf;

import java.util.ArrayList;
import java.util.Iterator;

/**
 * An essential part of Java - locates any Class, anywhere.
 * </P>
 * This feature would have been nice as part of the JDK for the last 7 years, but Sun hasn't
 * added it, so we did it instead.
 * <P>
 * Browse the different static methods to see what kinds of search you can do...
 */
public class ClassLocater
{
      /**
      * Find all instances of the given <code>Class</code> or interface by
      * loading all classes on the class path.
      *  
      * @param targetType the superclass of all returned classes.
      * @return an array of all subclasses of <code>targetType</code>
      */
      public static Class[] getSubclassesOf( Class targetType )
      {
            return getSubclassesOf( targetType, ".*" );
      }
                              
      /**
      * Find all subclasses of the given <code>Class</code> or interface by
      * loading only those classes with names that match the given regular
      * expression.
      *  
      * @param targetType the superclass of all returned classes.
      * @param regex a regular expression that will match with every subclass
      * @return an array of all subclasses of <code>targetType</code>
      */
      public static Class[] getSubclassesOf( Class targetType, String regex )
      {
            ArrayList matches = new ArrayList();
            ClassPath cp = new ClassPath();
            Iterator i = cp.classNameIterator();
            while ( i.hasNext() )
            {
                  String className = (String)i.next();
                  if ( className.matches( regex )
                  && !className.equals( targetType.getName() ) )
                  {
                        Class clazz = null;
                        try
                        {
                              clazz = Class.forName( className );
                        }
                        catch (ClassNotFoundException cnfx )
                        {
                              continue;
                              
                        }
                        catch (NoClassDefFoundError cnfx )
                        {
                              continue;
                              
                        }
                        catch( UnsatisfiedLinkError cnfx )
                        {
                              continue;
                        }
                        finally
                        {
                              if ( clazz != null && targetType.isAssignableFrom( clazz ) )
                              {
                                    matches.add( clazz );
                              }
                        }  
                              
                  }
                        
            }
                  
            return (Class[])matches.toArray( new Class[0] );
            
      }
}

Are you under the impression that this exact same piece of code worked fine earlier, or have their been “minor mods” since then?

Can’t see anything wrong with the code you’ve got there but then it’d be nice to see “class ClassPath” aswell.

Kev

[quote]Are you under the impression that this exact same piece of code worked fine earlier, or have their been “minor mods” since then?

Can’t see anything wrong with the code you’ve got there but then it’d be nice to see “class ClassPath” aswell.

Kev
[/quote]
No mods for a long while. Interestingly, with a regex of “com.grexengine.jgf…*” it still OOME’s - though I would ahve thought that would avoid most of the allocations.


package com.grexengine.jgf;

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import java.util.StringTokenizer;
import java.util.jar.Attributes;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.jar.Manifest;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
 
 
/**
 * Created with Eclipse  
 * User: duncanIdaho for java-gaming.org
 *
 */
 
/**
 * ClassPath find and records the fully qualified name of every Class  
 * on the classpath via the system property "java.class.path".
 *  
 */
public class ClassPath {
 
 public static final int SILENT = 0;
 public static final int QUIET = 1;
 public static final int VERBOSE = 2;
 
 public static final String JAR_EXT = ".jar";
 public static final String ZIP_EXT = ".zip";
 public static final String CLASS_EXT = ".class";
 
 private ArrayList classNames;
 
 private int outputLevel;
 
 /**
  * create a new ClassPath instance and find all classes on the classpath
  *
  */
 public ClassPath() {
  this( SILENT );
   
 }
 
 
 public ClassPath(int type) {
  super();
  outputLevel = type;
  findAllClassNames();
 
 }
 
 /**
  * Answers an <code>Iterator</code> over fully qualified <code>Class</code>
  * names found on the classpath.
  * @return an <code>Iterator</code> over the elements in this list
  */
 public Iterator classNameIterator() {
  return classNames.iterator();
   
 }
 
 /**
  * Answers an <code>ArrayList</code> of all <code>Class</code> names found
  * on the classpath.
  * @return an <code>ArrayList</code> of class names.
  */
 public ArrayList getClassNames() {
  return classNames;
   
 }
 
 /**
  * Initialize the member variable <code>classNames</code> and  
  * look for classes.
  *
  */
 private void findAllClassNames() {
  String path = null;
  classNames = new ArrayList();
  try {
   path = System.getProperty( "java.class.path" );
  } catch ( Exception x ) {
   x.printStackTrace();
   
  }
  if ( outputLevel != SILENT )
   System.out.println( "scanning classpath: " + path );
  StringTokenizer toke = new StringTokenizer( path, File.pathSeparator );
   
  analyzeClasspathTokens( toke );
 
  if ( outputLevel != SILENT )
   System.out.println( "found " + classNames.size() + " classes." );
   
  if ( outputLevel == VERBOSE ) {
   Iterator i = classNames.iterator();
   while ( i.hasNext() ) {
    String name = (String)i.next();
    System.out.println( name );
     
   }
   
  }
   
 }
 
 /**
  * Adds a file explicitly mentioned on the classpath to the list  
  * of classes.
  * @param classFile a class file listed on the classpath itself.
  */
 private void addClass( File classFile ) {
  classNames.add( getClassNameFrom( classFile.getName() ) );
   
 }
 
 /**
  * Adds all class names found in the jar.
  * @param jarFile a jar file explicitly listed on the classpath.
  */
 private void addJarContents( File jarFile ) {
  JarFile jar = null;
  try {
   jar = new JarFile( jarFile );
 
  } catch ( IOException iox ) {
   // boom!
  }
  if ( jar != null ) {
   Manifest man = null;
   try {  
    man = jar.getManifest();
   } catch( IOException iox ) {
    System.err.println("error obtaining manifest from: " + jar.getName());
     
   } finally {
    if ( man != null ) {
     this.scanClasspath( man, jar, jarFile );
     
    }
     
   }
   Enumeration e = jar.entries();
   while (e.hasMoreElements()) {
    JarEntry entry = (JarEntry)e.nextElement();
    if ( !entry.isDirectory() && entry.getName().endsWith( CLASS_EXT ) ) {
     String className = getClassNameFrom( entry.getName() );
     classNames.add( className );
 
    }
   
   }  
 
  }
   
 }
 
 /**
  * Adds all class names found in the zip mentioned  
  * @param zipFile
  */
 private void addZipContents( File zipFile ) {
  ZipFile zip = null;
  try {
   zip = new JarFile( zipFile );
   
  } catch ( IOException iox ) {
   
  }
  if ( zip != null ) {
   Enumeration e = zip.entries();
   while (e.hasMoreElements()) {
    ZipEntry entry = (ZipEntry)e.nextElement();
    if ( !entry.isDirectory() && entry.getName().endsWith( CLASS_EXT ) ) {
     String className = getClassNameFrom( entry.getName() );
     classNames.add( className );
     
    }
   }  
  }  
 }
 
 /**
  * This method takes a top level classpath dir i.e. 'classes' or bin
  * @param dir
  */
 private void addDirectoryContents( File dir ) {
  // drill through contained dirs ... this is expected to be the  
  // 'classes' or 'bin' dir
  File files[] = dir.listFiles();
  for( int i = 0; i < files.length ; ++i ) {
   File f = files[i];
   if ( f.isDirectory() ) {
    addDirectoryContents( "", f );
     
   } else {
    if ( f.getName().endsWith( CLASS_EXT ) )
     addClass( f );
       
   }
   
  }
   
 }
 
 /**
  * This method does the real directory recursion, passing along the  
  * the corresponding package-path to this directory.
  *  
  * @param pathTo the preceding path to this directory
  * @param dir a directory to search for class files
  */
 private void addDirectoryContents( String pathTo, File dir ) {
  String pathToHere = pathTo + dir.getName() + File.separator;
  File files[] = dir.listFiles();
  for( int i = 0; i < files.length ; ++i ) {
   File f = files[i];
   if ( f.isDirectory() ) {
    addDirectoryContents( pathToHere, f );
     
   } else {
    if ( f.getName().endsWith( CLASS_EXT ) ) {
     String absFilePath = pathToHere + f.getName();
     classNames.add( getClassNameFrom( absFilePath ) );
     
    }
       
   }
   
  }
   
 }
 
 /**
  * While the StringTokenizer has classpath elements, attempt to  
  * add any contained classes to the list.
  *  
  * @param toke class path elements
  */
 private void analyzeClasspathTokens( StringTokenizer toke ) {
  while ( toke.hasMoreTokens()) {
   String pathElement = toke.nextToken();
   analyzeClasspathElement( pathElement );
 
  }
   
 }
 
 /**
  * Make a file out of the String, determine which kind of  
  * interesting classpath file it might be, and add it to the list.
  *  
  * @param pathElement  
  */
 private void analyzeClasspathElement( String pathElement ) {
  File elementFile = new File( pathElement );
  String elementName = elementFile.getAbsolutePath();
  if ( elementName.endsWith( JAR_EXT ) ) {
   addJarContents( elementFile );
   
  } else if ( elementName.endsWith( ZIP_EXT ) ) {
   addZipContents( elementFile );
   
  } else if ( elementName.endsWith( CLASS_EXT ) ) {
   addClass( elementFile );
   
  } else {
   addDirectoryContents( elementFile );
   
  }
   
 }
 
 /**
  * replace ANY slashes with dots and remove the .class at the  
  * end of the file name.
  * @param entryName a file name relative to the classpath.  A class  
  * of package org found in directory bin would be passed into this  
  * method as "org/MyClass.class"
  * @return a fully qualified Class name.
  */
 private String getClassNameFrom( String entryName ) {
  String foo = new String(entryName).replace( '/', '.' );
  foo = foo.replace( '\\', '.' );
  return foo.substring( 0, foo.lastIndexOf( '.' ) );
   
 }
 
 /**
  * Use the manifest associated with the jar to determine if there are
  * any Class-Path elements names in the jar that should also be scanned.
  *  
  * @param man the manifest of the given jar
  * @param jar the jar associated with the given manifest.
  */
 private void scanClasspath( Manifest man, JarFile jar, File jarFile ) {
  Map map = man.getEntries();
  if ( map != null ) {
   Attributes atts = man.getMainAttributes();
   if ( atts != null ) {
    Set keys = atts.keySet();
    Iterator i = keys.iterator();
    while ( i.hasNext() ) {
     Object key = (Object)i.next();
     String value = (String)atts.get( key );
     if ( outputLevel == VERBOSE )
      System.out.println( jar.getName() + "  " + key + ": " + value );
     if ( key.toString().equals( "Class-Path" )) {
      if ( outputLevel != SILENT )
       System.out.println( "scanning " + jar.getName() +"'s manifest classpath: " + value);
      StringTokenizer toke = new StringTokenizer( value );
      while (toke.hasMoreTokens() ) {
       String element = toke.nextToken();
       if ( jarFile.getParent() == null )
        analyzeClasspathElement( element );
       else {
        analyzeClasspathElement( jarFile.getParent() + File.separator +  element );
       }
       
      }
       
     }
     
    }
     
   }  
   
  }
   
 }
 
}

Teach granny to suck eggs here I suppose, but does it get past constructing the ClassPath object?

Kev

IIRC Class.forName() not only loads a class but also makes any static initialization to occur (execution of static blocks and static fields). Maybe a new class has been added to the system that creates some huge static objects?

[quote]Teach granny to suck eggs here I suppose, but does it get past constructing the ClassPath object?

Kev
[/quote]
I thought so, but … I’ll check anyway. Heck, it needs some logging statements in there anyway.

[quote]IIRC Class.forName() not only loads a class but also makes any static initialization to occur (execution of static blocks and static fields). Maybe a new class has been added to the system that creates some huge static objects?
[/quote]
Doh. Of course, that could explain it…however, I don’t believe I have added anything with static inits of non trivial size.

I’ll go do some more logging and come back with more data. I’d been hoping I was missing something obvious, but obviously not :(.

Oops. Seen the problem - didaho’s algo only works on toy examples (no offense, but just think what happens when you get mutually referential manifests GULP).

Rewriting it now to be iterative rather than ultra-deep (potentially infinite) recursive. I think it was collapsing with OOME because of too much recursion, 99.99% of which was unnecessary :(.

EDIT: just to be clear, DI never presented his code as perfect, so it’s fair enough that it’s not optimized or anything. I adopted it in order to polish it a bit. It’s just that it had worked perfectly for lots of small situations and I’d got used to it doing so, without bothering to go through the code with a fine tooth comb.

OK, I rewrote the ClassPath from scratch and now it doesn’t fail even in mutually (infinitely) recursive situations, and it’s noticeably faster in most cases. ;D.

If someone wants the code urgently, email me (ceo @ grexengine.com) and I’ll give you a fixed-up binary JAR. Otherwise, it can wait until JGFv3 is up, and will appear as one of the first 5 items in the “source code” section.

Ironically I was using these classes to auto-discover JGF-specific modules during the JGF server’s boot process when I discovered this problem ;D.