According to this (closed) bug report, java.library.path is meant to be a read-only property. However this workaround solves that problem:
public static void addDir(String s) throws IOException {
try {
// This enables the java.library.path to be modified at runtime
// From a Sun engineer at http://forums.sun.com/thread.jspa?threadID=707176
//
Field field = ClassLoader.class.getDeclaredField("usr_paths");
field.setAccessible(true);
String[] paths = (String[])field.get(null);
for (int i = 0; i < paths.length; i++) {
if (s.equals(paths[i])) {
return;
}
}
String[] tmp = new String[paths.length+1];
System.arraycopy(paths,0,tmp,0,paths.length);
tmp[paths.length] = s;
field.set(null,tmp);
System.setProperty("java.library.path", System.getProperty("java.library.path") + File.pathSeparator + s);
} catch (IllegalAccessException e) {
throw new IOException("Failed to get permissions to set library path");
} catch (NoSuchFieldException e) {
throw new IOException("Failed to get field handle to set library path");
}
}
For those of you who really need to run a LWJGL application by double clicking the jar yet you want the natives in their own subfolder, this is for you 
Obviously this might not be portable across JVMs.
Source: http://nicklothian.com/blog/2008/11/19/modify-javalibrarypath-at-runtime/