ok tk, even if it still seems near useless for me in term of security I found the principe/idea of overriding the JVM is something really interresting
EDIT:
let me explain a little more why it is near usless for security : if one unassemble the DLL and modify it, it could for example modify the method that decrypt the encrypted class file to write them back on the harddisk (it may not work in this case but that’s the idea)
he’s just trying to make it harder - not bulletproof 
but yeah, in the end, someone will just make a new jarprotect.dll that will spit out the actual decrypted bytes, and people will download that dll and replace yours.
The real problem is now you have to build shared libs for every arch that you want to support AND reverse engineer each version of each JVM that you want to support (and bail for any unrecognized hash code as there are way too many potential points of attack). That’s a lot of work for very little gain. And of course continuosly ongoing since, ya know, new VMs come out every once in a while. If you really want to prevent cheating…server.
the thing is even if they do spit out the ‘decrypted’ bytes, a typical decompiler will not be able to understand it, as it is in a different format to the official class file format, so people would also need to write a new decompiler specifically for this purpose. in future, I will add random instruction generation so each opcode that is decoded is random, making it even harder to write a decompiler that works for the format.
example:
LDC is usually encoded as 0x12, after random instruction generation, it could be 0x07, a typical Java decompiler only understands 0x12, therefore it would crash or error upon encountering an unexpected opcode. this will work because it is possible to hotpatch ClassFileParser::parseClassFile during runtime.
if you change input opcodes there is a certain possibility for modifying it in an invalid way. But that probably goes for all obfuscators.
it also sounds like you’re working a lot with the existing hotspot code - which is GPL, so you are required to follow the license.
lastly, if you really want to have some degree of security, you should share the code, instead of relying on obscurity.
using hotspot’s source as a guide as to what payload code to write doesnt require me to follow the license, as i havent actually used the source itself. im simply using it as a guide to learn what functions to patch. it’s similar to antiviruses hooking system routines (ZwOpenProcess, etc), although they use the winapi as a guide to learn what to hook, their software isn’t required to abide by microsoft’s license, as it doesnt actually use microsoft’s code.
later i may opensource the project. right now, it would make it much easier to crack it if I did opensource it however. this isnt meant to be bulletproof - its only meant to give Java apps aorund the same degree of security as C/C++ apps have.
Neat ideas.
Do the internal hooks change much from JVM version to version, or even JVM operating system to operating system?
It would be pretty annoying to keep track of the changes if it did.
fair enough. I was just assuming you were re-writing large parts of parseClassFile, if not, then it would just be a matter of patching jvm.dll to output the bytes after it has been verified.
The general thought is, if you have an obfuscated class then at some point you need to de-obfuscate it and pass it into the VM, at that point we intercept the input and write it out. In your case, you made it harder by creating some native code, so a different skillset is required.
That said, however, its still relatively easy to thwart, if you have a reason for doing so (crackers etc).
Your method could also be very brittle, in that may rely on features not present in all VMs, including non-Oracle VMs.
obfuscated classes need not be de-obfuscated to be executed, in fact it should not be possible to deobfuscate a class. the whole point of obfuscation is to change a class so it is difficult to understand and so that the obfuscated class still works the same way regardless what VM it is loaded in.
to crack this type of protection, knowledge of many software concepts is required, including
the general idea is that it hotpatches ClassFileParser::parseClassFile during runtime in order to be able to make classes out of my own class formats (not the same formats Sun/Oracle uses or has documented), so thereotically, as long as i am able to write a parser for a particular format, i could make class files in whatever format i wanted to. because it also doesnt use the JVM’s default classparser, i can also invent new instructions, etc, as long as the injected code could convert these new instructions back into jvm instructions, although i havent tried this yet. my next idea is to make the software generate random opcodes for each instruction, so even if you did get the bytecode you couldnt load it into a decompiler as the decompiler would not recognize the opcodes.
Doing a little reverse engineering of previous JVM versions and matching them to hashes during runtime should be all i need to be able to patch previous versions too; however i havent done this yet.
Actually, by including a disassembler in my dll and scanning for the right calls, i should be able to find the required addresses dynamically
No, unless I am misunderstanding what your stuff does, then whatever format you make up, has to end up as PROPER bytecode at some point, else the VM cannot “run” it.
At that point, you intercept - using your modified VM - and dump the bytes.
it does not have to be official bytecode. when the function parseClassFile is hooked, the new code is allowed to do anything, even parse a completely different format to the one Sun has.
but yes, at the moment it converts the new format bytecode back into Sun bytecode, you could do an intercept there, although it would be a pain in the ass to convert it into a decompilable format.
later on, i’m also going to write some code to hook the instruction translator, therefore even if you did intercept the instruction stream, you wouldnt be able to make sense of it unless you RE’d the whole thing, once I get to that stage it would be safe to opensource it.
It has to be in the proper format before it get converted into the internal IR. Really all of this extra convolution doesn’t really add much except for more work to create and maintain. None of these proposed tricks even address Riven’s comment. Even if you do somehow detect and reject a modified VM, the attacker can simply just use a debugger, an ICE or simply inspect the JVM processes memory as well as the already mentioned reverse engineering of the shared library.
What does parseClassFile return?
Kev
returns a global handle to a klassOop instance. take a look at the function in mercurial.
but what the difference with a custom classloader written in java ?!
using defineclass method you can do any class bytecode encryption you want in java too witout the compatibilty problem, no ?
Riven, Matzon, and others are correct. It only adds a small layer of security.
thats in Java. my method doesn’t use any java code to do security. If you use the java defineClass method, I could just intercept it by patching rt.jar
yes that’s in java but what difference ? let say I use my own class format decrypted by my own ClassDecrypterLoader written in Java, one can disasemble it and hack my game/soft. let say you write you own ClassDecrypterLoader or other written in C++ and compiled in a DLL, one can the same way decompile the DLL and hack your game/soft, no difference ( unless in java case It will produce less trouble of compatibiliy and will be easier to maintain)
I believe the point is that hacking Java is trivial and hacking native code is somewhat harder. It requires a different skill set and there are fewer people who can do it.
hacking native code is not a lot harder comparing to hack an obfuscated jar, and as soon as this “encrypter” will become used it will be hacked, I mean you cannot rely on this kind of solution to seriously securize anything, but you may use it if you want to demo something without delivering your source code but … it is already so boring to read obfuscated project source code with more than 3 classes that I think it is also unecessary in this case