JVM monitoring, server-status monitoring, crackers

Engines, Libraries and Tools Tools Discussion
#1

A java game-server I’m deploying got attacked over the weekend by a script-kiddie (you can tell because the logfiles show attempts to cause buffer-overflows AND INVOKE CMD.EXE! On a server that is openly unix-based (if they’d bothered to look ;)) and runs a Java server). Anyway, once I’d finished laughing at them…

The server didn’t crash, and still accepts incoming connections, but for some reason one of the Selector’s that processes requests has just stopped functioning entirely. It could be that the garbage the skiddie was throwing at it (all sorts of rubbish characters) has upset the 1.4.1 Selector; I have no idea. However, I now want to add some JVM-monitoring tools to the server (it already has an interactive admin console) to do things like check “Are all the Selector’s OK? Are any of the incoming-queue’s unnaturally full (suggesting “hung” connections etc)? What’s the memory usage like at the moment?”.

Some of these (especially the simple stuff, like current free memory) must have been very useful to many developers for many years, but I cannot seem to find ANY java tools on the net that can be integrated into a compiled app (e.g. as a library) that can spew useful status-info. The nearest I could come up with is profilers and debuggers - neither of which is appropriate, because this is a production server where I do NOT need this stuff running 24/7, I just need to do an occasional “give me a snapshot of the status now, please” - even if that means server slowdown for seconds/minutes/etc.

Some of it is trivial to implement (e.g. getting at memory info from the Runtime class), but other things (like the status of a given Selector) are not so trivial. If I have to, I’ll implement my own framework whereby every module supplies String[]'s of it’s own status on demand - but surely this is reinventing the wheel?

#2

Java Management Extensions (JMX) is designed for this type of stuff. I have yet to make time to learn it but I want to.

If you can use BEA’s JRrockit JVM (they have win32 and linux versions, maybe more) it has a built in management console which I have not used either.

#3

[quote]Java Management Extensions (JMX) is designed for this type of stuff. I have yet to make time to learn it but I want to.
[/quote]
Thanks, that’s what I was looking for. Unfortunately it’s pretty complex by the looks of things, so this is going to take a loooong time to get through…

#4

[quote]However, I now want to add some JVM-monitoring tools to the server (it already has an interactive admin console) to do things like check “Are all the Selector’s OK? Are any of the incoming-queue’s unnaturally full (suggesting “hung” connections etc)? What’s the memory usage like at the moment?”.

Some of these (especially the simple stuff, like current free memory) must have been very useful to many developers for many years, but I cannot seem to find ANY java tools on the net that can be integrated into a compiled app (e.g. as a library) that can spew useful status-info. The nearest I could come up with is profilers and debuggers - neither of which is appropriate, because this is a production server where I do NOT need this stuff running 24/7, I just need to do an occasional “give me a snapshot of the status now, please” - even if that means server slowdown for seconds/minutes/etc.

Some of it is trivial to implement (e.g. getting at memory info from the Runtime class), but other things (like the status of a given Selector) are not so trivial. If I have to, I’ll implement my own framework whereby every module supplies String[]'s of it’s own status on demand - but surely this is reinventing the wheel?
[/quote]
FYI to anyone following this thread: I’ve had to roll my own. JMX is mainly just a poorly-documented wrapper around SNMP. SNMP unfortunately lives up to it’s name (simple NMP) and only supports 3 commands:

[]get
[]getnext
[*]set

So it doesn’t handle sending command commands (e.g. “reset now”) to devices. It’s an extensible protocol, and you could make special variables that whenever a manager called “get” on them actually executed some code - but that kind of hacking is almost always a very bad idea in the long run (especially when new people join your team and start trying to use the standard protocol in the ways it was meant to work - and find all sorts of weird stuff happening!).

It’s also extremely complex, for a simple protocol :). Once you are familiar with it all, it’s not complex at all, but the time taken to get that familiar is time I could instead spend making something more suitable for my purposes from scratch.

#5

Actualy, my work project is a project on SNMP.
SNMP does what its suposed to do and nothing extra.

The protocol is simple to implement, but very powerful.
You can get variables you need and doing a server restart might be done by creating a variable that should be set to restart (and which is set back to normal after it is done)

SNMP v1 doesn’t do much in the ways of security however…

Back on topic, im kinda interested in this too, especialy when I hear of little evils like selectors deadlocking on close() calls.

Perhaps looking at the behamoth called JBoss may reveal some perls of wisdom?

#6

Well, in the same way that the Turing Calculus is all-powerful, yes. But evaluated as a management protocol, it looks pretty pants to me ;P.

I’m not sure what you could get from looking at e.g. JBoss. Either there’s a standard, or you have to roll your own. If JBoss does something, that won’t per se make a standard, so you’ll still have to roll your own…unless they have a standalone system similar to JMX (but easier to use and much more powerful), and which can be embedded royalty-free without a darn GPL or similar?

Looks like we’ll be supporting SNMP for data management, but our preferred mgmt sytems are proprietary - just because there’s nothing suitably simple, powerful, standard AND inexpensive out there :(.

#7

JMX is its own technology, and really has little to do with SNMP. At my last job I implemented a large JMX server/client combo on a database system. JMX actually is pretty simple, and if you use MX4J (http://mx4j.sf.net/), it’s fairly nice. The reference implementation of JMX stinks, and the commercial ones, partically Sun’s, are very expensive.

At its heart, JMX is a server and a bunch of Management Beans (MBeans). MBeans attach to something (usually an object) in your application and monitor it. The MBean needs to be specifically created for whatever it attaches to, generic MBeans are not common and generally not useful. Then the server can query all the MBeans that are installed in it and gather the current state of the system.

MBeans are not JavaBeans, nor are they a specific class or interface. Really they’re just a class that is written in such a way that the JMX server can introspect on them with reflection. However, there are several “common practices” for creating MBeans.

The server by itself offers no interface. You need to attach what JMX calls an adaptor to the server to see what is going on. The default one is an HTML adaptor, which enables you to monitor your system from a webpage. The HTML Adaptor that comes with the reference JMX implementation is terrible, and MX4J’s is not much better. It’s cryptic, and you really need to know JMX quite well to use it.

JMX also offers timers to schedule things, and so you can schedule your system to do various things at certain times, much like a Java cron. There are also gauges, which keep track of a certain attribute of an MBean and trigger an event if they go over a specified threshold. MX4J goes a bit further and offers things like email notification, an RMI adaptor, etc.

JMX really is simple, but yes it is poorly documented. My main server program with MX4J was about 20 lines of code (pretty much just getting the server up and running and installing some MBeans). Then from there it was all just writing MBeans. I also wrote a Swing interface using the RMI adaptor, because not only is the HTML adaptor cryptic, it’s also very limited in what it can do. The end result was actually pretty cool.

JMX’s achilles heal is no predefined interface that’s easy to use and desirable. Really, the HTML interface is just terrible. The commercial implementations probably offer something better, and of course there is JBoss. Writing that Swing interface was easily 90% of the effort for the system, a pretty big waste IMO.

Also, JMX pretty much forces you to bend your application to its needs. Our application had to change such that the JMX server came up first, then it brought up the main app. Bringing up a new JMS queue became install a new queue MBean that would then start the new queue, etc. In certain situations this can be unacceptable.

I wish I had access to the intranet, I wrote pages and pages on JMX :slight_smile:

#8

Is this the stuff you were looking at before going to JMX?

http://java.sun.com/products/jdk/1.2/docs/guide/jvmpi/jvmpi.html
note: “The JVMPI is not yet a standard profiling interface.”

http://java.sun.com/products/jpda/doc/architecture.html

Intersting side note: jvmstat works on 1.4.2 now … I’m not sure when it was upgraded.
http://developers.sun.com/dev/coolstuff/jvmstat/

#9

Thanks for the info. I can honestly say that you’ve written more real info in just a few paras about JMX than any of us could find in two or three days of searching (everything was either articles that were 99% waffle, or else so completely impenetrable it was impossible to understand what on earth they were talking about).

But I’m still confused.

But, if I understand correctly what you’ve said, JMX actually is…nothing. It doesn’t do anything. Like the JavaBean, it is nothing more exciting than a convention for class design? (it sounds like it isn’t even a concrete interface).

My reading of the official docs from Sun (or at least the marketing spiel, which seems to be the main “documentation” they provide :() made it sound like the primary purpose was to simplify SNMP implementation in Java, whilst trying to be a little bit more generic - although no concrete reason was given for being more generic.

In particular:

You see, all that (and all the stuff you mention later) is so easy to do it sounds like you could create it from scratch in less time than it took to find, read, and understand the docs for JMX, then implement what you needed in JMX? It sounds like you end up doing all the hard stuff yourself anyway, so what’s the poin of jumping through a JMX-shaped hoop?

Now, I can see a lot of value in an SNMP-supporting mgmt tool. Most enterprises and/or serious network admins have fancy remote SNMP management apps which nicely integrate all sorts of functionality in one place, and they appreciate being able to keep on managing things centrally, so to speak.

But I’m still left wondering, What’s the point of JMX?

#10

[quote]Is this the stuff you were looking at before going to JMX?
[/quote]
Personally, I’m not interested in using JVMPI at the moment: that’s really for low-level info gathering. I’m mainly doing high-level metrics, e.g. my original example of checking how full the “pending” queues are on a Selector. Some of those things do require JVMPI, but for the most part they don’t.

I was mostly looking for a framework (e.g. JMX - although AFAICS this is a very sparse framework ATM; but I’m still struggling to appreciate it so I don’t really know :)) that could do a lot of the work in instrumenting things for you. At the very least a set of non-generic interfaces that define a cohesive architecture would be nice.

[quote]Intersting side note: jvmstat works on 1.4.2 now … I’m not sure when it was upgraded.
http://developers.sun.com/dev/coolstuff/jvmstat/
[/quote]
That sounds like it’s quite cool.

“sounds like”. Both the doc-link and the download link for the 1.4.2-compatible version result in:

"Page Not Found
We are sorry, the page your requested was not found on our system. Based upon the url that you requested, we would like to recommend pages that match your requested url. If you prefer, you may navigate through our site or use our search for your page.

If you are certain that this URL is valid, please send us feedback about the broken link.
Your URL: http://developers.sun.com/dev/coolstuff/jvmstat/v2/policy.html"

Given that 1.4.1 + NIO is like playing russian roulette with your sanity, and we’re using NIO exculsively, this makes it kinda difficult to play with it :(.

Also need to try and find more info on the licensing. The generic “CoolStuff license” linked to from the page is so generic that it’s not clear how applicable it is. It would be nice to get some clarification. I’ll see if I can find some.

#11

Thanks for that link…I’d not seen JDPA before (at least, not since it began working with Hotspot), and it has quite a lot of things I’ve been wanting.

#12

[quote]Thanks for that link
[/quote]
You might want to be careful about invoking methods on objects in the debugee VM. My personal experience was that, on an arbitrary basis, the thread would just seem to hang. There doesn’t seem to be any good sources for information on JPDA so I couldn’t really see if anybody else was stumbling on that problem - though I talked to the author of JSWAT and he said that he always does the invokes in a separate thread, because they seemed to have arbitrary latency. YMMV

God bless,
-Toby Reyelts

#13

[quote]That sounds like it’s quite cool.

“sounds like”. Both the doc-link and the download link for the 1.4.2-compatible version result in:

"Page Not Found …
[/quote]
Must have been a temporary glitch - I got it without problems.

t is neat to see how often the eden generation is collected compared to the old generation… That technique sure seems to pay off.

[aside]
My new fun thing to watch is the output of the JVM when you use -XX:+PrintCompilation
I just wish I knew what the info meant before the method signatures
1 b java.lang.String::charAt (33 bytes)
2 s b blah.blah.balh:…
3 !b blah.blah.balh:…
4 * b blah.blah.balh:…
The ‘b’ is always there… but the ‘s’ ‘!’ or ‘’ seem to mean something special. I think '’ means that it backed out the compiled version… because I always see that with “(0 bytes)”… maybe ‘s’ is for synchronized methods…

jvmstat gives you a graph of time spent compiling but it doesn’t tell you what was compiled.

Another cool bit of trivia is that it compiles 16 methods before it ever loads your"main" class (tested it by giving a bogus class on the command line)

#14

It’s definitely a loose technology. The heart of JMX is really the server, which is provided for you in the JMX implementation. It manages communication (MBean to MBean, MBean to adaptor, event to MBean, etc), provides MBeans with a context in which to send in their management data, and manages things like remote RMI connections, sending emails out etc (although the last two are MX4J extensions). It’s not a simple program, and all you have to do is instantiate one and tell it to start.

JMX is also very modular. Kind of like SiteScope but more sophisticated. If you add a new component, you can easily whip up an mbean for it and have it managed within your system very quickly. For example JBoss attaches MBeans to everything. You can fire up JBoss and connect to the JMX HTML adaptor, and you’ll get a list 5 miles long of MBeans and their stats. It’s cryptic as can be, but if you know JMX you can know every detail of your JBoss system at a glance, not to mention tweak anything you want.

If you had an existing java application that you wanted to manage, attaching bean like objects to various parts is easier than reconfiguring the app to have management info built into classes (not to mention more modular). Since you’re going to be writing these bean like classes anyway, mine as well follow the MBean conventions and end up being able to plug them into a premade server.

That’s what JMX does too, it’s just younger. So that central management app is that HTML interface I was talking about, which needs a lot of work. I’ll bet commercial JMX implementations have a nice management center.

I dunno. I’m definitely not an advocate of it or anything, just was told to use it. I think in time it could prove to be nice. I think a lot of the Java community (like JBoss) likes it just because it is a Java standard, and so is a “good bet”.