Part 2: Troubleshoot Connection Leak - Using JMX API

Overview:
In Part 1 of this series, we have seen how to troubleshoot Connection leak by enabling switch via weblogic Administration console and looking at thread stack in console logs.
In this post, we will see how can we monitor Connection leaks remotely using JMX API by querying MBeans configured inside Weblogic server.

Introduction:
Weblogic server have tones of MBeans configured almost for every component/module to gather runtime statistics for a given module. In our case, to get Connection leak details, we got to get hold on "JDBCDataSourceRuntimeMbean" by connection to server.

Below example connects to Weblogic Admin server running on localhost and tries to loop through all the servers (including Admin and managed server (even in cluster)) and fetch db pool configured in each server and tries to check if there is any connection leak.

Currently, below sample code just prints the connection leak count. It can be enahced to send alert if connection leak count is > 0.

We can schedule this code to run as a job (unix cron or other medium) to run periodically and monitor server and take proactive action if such issue found.

NOTE : Before you run this code . "Inactive Connection Timeout" value must be set to some positive integer. How to set that value , see Part 1.

import java.util.Hashtable;

import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.naming.Context;


public class JMXConnectionLeakDetector {

 public static void main(String [] args) {
  try{
   String protocol = "t3";
   int port = 7001;
   String hostname = "localhost";
   String jndiroot = "/jndi/weblogic.management.mbeanservers.domainruntime";
   JMXServiceURL serviceURL = new JMXServiceURL(protocol, hostname, port,jndiroot);
   ObjectName service = new ObjectName("com.bea:Name=DomainRuntimeService," +
     "Type=weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean");
   
   Hashtable h = new Hashtable<>();
   h.put(Context.SECURITY_PRINCIPAL, "appdeployer");
   h.put(Context.SECURITY_CREDENTIALS, "deployer123");
   h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES,"weblogic.management.remote");
   JMXConnector connector = JMXConnectorFactory.connect(serviceURL, h);
   MBeanServerConnection connection = connector.getMBeanServerConnection();
   
   
   ObjectName[] number_of_servers = (ObjectName[])connection.getAttribute(service, "ServerRuntimes");
   int length = (int) number_of_servers.length;
   
   for (int i = 0; i < length; i++){
    System.out.println("Server Instance="+number_of_servers[i]);
    String name = (String) connection.getAttribute(number_of_servers[i],"Name");
    ObjectName[] number_of_dbpools = 
     (ObjectName[]) connection.getAttribute(new ObjectName("com.bea:Name="+name+
         ",ServerRuntime="+name+",Location="+name+",Type=JDBCServiceRuntime"),"JDBCDataSourceRuntimeMBeans");
    int pool_length = (int) number_of_dbpools.length;
    for (int x = 0; x < pool_length; x++){
     System.out.println("********* PoolName=" + (String)connection.getAttribute(number_of_dbpools[x], "Name")+"  ******");
     int leak_connection_count=(Integer) connection.getAttribute(number_of_dbpools[x], "LeakedConnectionCount");
     System.out.println("LeakedConnectionCount         : " + leak_connection_count);
     if(leak_connection_count>0){
      //Send email alert
     }
    }
   }
      
  }
  catch(Exception e){
   e.printStackTrace();
  }
 }
}

Above approach looks good but still involves manual efforts to write java code, run it manually or schedule as a job. Situation may arise, where we are not allowed to configure any job or remote client to access server remotely. For that, Weblogic Diagnostic Framework (WLDF) comes to rescue, which can be configured within weblogic server and guide it to manage your resources and notify via emails etc..

We will cover WLDF in Part 3, so stay tuned.