JVM_Monitoring_with_SNMP

Introduction

This is a HOWTO article about configuring SNMP to monitor different aspects of your JVM. This is useful when you are running a Java app on a server and want to monitor when it's resources are stretched thin, or when you are simply looking to collect data about your app's use and stress.

Maintenance of this article

I have no personal plans on checking if newer versions of Gentoo/SNMP/Java/whatever break this article. With that said, the use of these components should not likely change soon since they are all standards-based.

Also, if anyone does discover errors or if a newer version of "foo" does break this solution, I don't have a problem doing that kind of upkeep. I simply won't be testing this whenever a new version of foo package comes out. Requests for clarification in specific areas of this article will also be noted, and if I think it would be useful, I will likely make such clarifications.

Finally, any and all formatting corrections are greatly appreciated.

Installing Necessary Packages

In order to monitor your JVM with SNMP, you'll undoubtedly need Java and an SNMP implementation installed. The Java component only needs to be the JRE, but if you are doing development on the same machine, the JDK includes the JRE. For the purposes of this article, I'll be using the JDK. Additionally, I'm suggesting you emerge the optional "lsof" package. The lsof utility is not at all necessary, but it proves useful when looking at whether your services are listening on the correct ports or not.

Emerge the JDK, net-snmp, and lsof (optional) packages:

emerge -av sun-jdk net-snmp lsof

Through the wonderful magic that is Gentoo's Portage, the ebuild fairies should properly check for blocks on these packages and necessary dependencies, build these components (as well as dependencies), and install them while so you don't have to.

Configuration

Onto configuration of your system. The first step is to configure Java's JRE to report SNMP monitoring.

Configuring Java for SNMP Monitoring

The first thing is to create an snmp.properties file to hold your configurations. This allows you to start these options without manually passing them as command-line parameters every time you start your JRE. This file should be located in the jre/lib folder of the current Java install that you're using (/etc/java-config-2/current-system-vm/jre/lib/snmp.properties).

Configuring snmp.properties file

The observant person reading this article may notice that I only have JVM allowing connections from localhost (127.0.0.1). They may even notice that I'm also using a non-standard port (1161 instead of SNMP's default 161). That's because we're setting this system up so that the JVM isn't the only thing that can use SNMP monitoring. If the JVM monopolizes UDP port 161, other services being monitored will require the use of different ports. This can quickly scale to an administrative nightmare if the machine is running many services which require SNMP monitoring. Instead, we'll run snmpd on UDP port 161, and have it act as an SNMP proxy to the JVM's SNMP server. This works for your other services as well and port 161 is used as a centralized port for monitoring many services (as it is intended to be).

The other thing to note about this simple configuration file is that I have two lines (one commented out) about an access control list (ACL) for SNMP. For the security unconscious (shame on you), you can uncomment the third line and comment out the fourth line. This means no ACL is imposed upon monitoring as long as the requests originate from the localhost. Granted that's not completely awful, but it will allow any user on the system to gain a plethora of information about your JVM, including many configuration options. Since there isn't any ACL at all, it is quite likely that any user on the system can also use the administrative capabilities of snmp and change some of your JVM's configs (possible stop your app?). I have not tested this, but without an ACL to tell the JVM "read-only," it seems that write access should be granted.

If you want the JVM's SNMP service to listen on an external interface, you can change 127.0.0.1 to the specific interface's IP. If you want it to listen on all interfaces, change 127.0.0.1 to 0.0.0.0 in the first line of the snmp.properties file.

Configuring snmp.acl file

For the rest of us who are a bit security-conscious, we'll be editing the snmp.acl file.

acl = {
  {
    communities = public
    access = read-only
    managers = localhost
   }    
  }
 trap = {
  {
    trap-community = public
    hosts = localhost
  }
 }

This ACL says that anyone who supplies the community string "public" (FYI, don't actually use public or private as your SNMP community strings) from localhost has read-only access. This also sets your SNMP trap for that community. You can also set an ACL for a community-string for read-write access, but I personally don't do that with my servers.

Configuring SNMP for monitoring

As previously stated, we'll be running snmpd on UDP port 161 which will serve as a proxy for the JVM's SNMP monitoring. The configuration file for snmpd is located at /etc/snmp/snmpd.conf

com2sec local     127.0.0.1/32    public
com2sec nyc     192.168.10.32/29    ThisIsThePublicCommunityString

group MyROGroup v1         local
group MyROGroup v2c        local
group MyROGroup usm        local

group MyROGroup v1         nyc
group MyROGroup v2c        nyc

view all    included  .1                               80

access MyROGroup ""      any       noauth    exact  all    none   none

syslocation Chicago
syscontact John Doe <john.doe@xyzcompany.com>

proxy -m /usr/share/snmp/mibs/JVM-MANAGEMENT-MIB.txt -v 2c -c public localhost:1161 .1.3.6.1.4.1.42.2.145

The first two lines setup two security groups with different community strings. The first security group (local) uses the community string public and can only originate from the localhost (and must be sent to localhost, not a public IP on your local host). The second specifies a remote monitoring group (nyc). The requests from this group must originate from the 192.168.10.32/29 subnet. Keep in mind you'll want to change these values for whatever fits your organization.

The next few lines actually add the IP/community string collections I called security groups into an actual group called MyROGroup, and read-only access is given to this MyROGroup. The system location and system contact are then specified.

The final line is the line that actually sets up the proxy between snmpd and JVM's snmp.

• The "-m" flag specifies the MIB definitions for the JVM (available from Sun at http://java.sun.com/j2se/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib). For uniformity with other MIB definitions, I placed this in the /usr/share/snmp/mibs directory, and renamed it with a .txt extension as all the other MIBs in that directory use. I suggest you do the same.
• The "-v" flag specifies which SNMP version to use ("2c" in this case).
• The "-c" flag specifies the community string to use ("public" in this case).
• The next argument (localhost:1161) specifies where to forward incoming requests (remember that we set JVM's snmp to listen to port 1161 and only from the localhost).
• The final argument (.1.3.6.1.4.1.42.2.145) specifies where to start the OID tree for rerouting to the JVM. Specifically, what this means is that any request for an SNMP OID that is under .1.3.6.1.4.1.42.145 will be forwarded to our JVM SNMP service, and that the JVM-MANAGEMENT-MIB.txt file we specified will be used to map OIDs to textual names.

Testing SNMP monitoring with snmpwalk

Congratulations! If you followed the directions correctly, you should now have a working solution to monitor your JVM's statistics with SNMP. Let's test this one out with a simple snmpwalk command (the snmpwalk utility comes with the net-snmp package we emerged in the beginning). From the machine snmpd is running on, type the following:

snmpwalk -v2c -c public localhost 1.3.6.1.4.1.42.2.145.3.163.1.1.3.1

This is the OID for JvmThreadCount (JVM OIDs can be explored in more depth at http://support.ipmonitor.com/mibs/JVM-MANAGEMENT-MIB/oids.aspx). If your setup is correct, you should get an output similar to the following:

SNMPv2-SMI::enterprises.42.2.145.3.163.1.1.3.1.0 = Gauge32: 11

Final Words

If this article helps you and we meet one day, feel free to buy me a beer if you felt it was worth it.

-TDS-