Part 1: Troubleshoot Connection Leak - Using Weblogic Server

Overview:
In past, all of us might have been in situation where all of sudden an application runs out of database resources (connection) which bring the application to its knees. There could be any reason of an application running out of database connection like,

1. Increase in number of users or batch processing (threads) where database pool size not able to cope up with the load.
2. Bug in application where connections are being grabbed from database pool but not released back to the pool , which is also known as "Connection Leak".

First issue is easy to fix , just close monitoring , understanding the load and sample statistics can help to arrive at database connection pool size.

Troubleshooting 2nd issue can be a nightmare. Just imagine, if we are asked to find a connection leak by looking at each and every single java file in huge code base. I can't even think of it :-)

Fortunately, most of the application servers provides a mechanism to enable profiling at runtime to let you troubleshoot Connection Leak.

Weblogic server also provides different ways to find Connection leak and we will cover them in 3 parts of this series.

In this Part 1 of 3 series, we will see quick and simplest way by enabling some switches at runtime using "Weblogic Administration Console".
Lets assume that, below java Servlet code is deployed as part of application archive (EAR or WAR) in weblogic server. If you pay attention, when servlet's doGet() will be invoked, it grabs 5 connections from the pool but never returns them back to the pool.

package com.test.servlet.connection.leak;

import java.sql.Connection;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.sql.DataSource;

@WebServlet(name="ConnectionLeak",urlPatterns={"/ConnectionLeak"},loadOnStartup=1)
public class ConnectionLeaker extends HttpServlet{
 @Override
 public void doGet(HttpServletRequest request,HttpServletResponse response){
  System.out.println("ConnectionLeaker: doGet() called");
  Context ctx;
  try {
   ctx = new InitialContext();

   DataSource ds=(DataSource) ctx.lookup("TestPoolJndi");
   System.out.println("Got datasource="+ds);
   for(int index=0;index<5 data-blogger-escaped-con="ds.getConnection();" data-blogger-escaped-connection="+con);
   }
  } catch (Exception e) {
   e.printStackTrace();
  }
 }

}

Make sure, servlet is deployed properly

Lets trigger the servlet manually via browser as below.

You will see log statement as below in server console logs.

Now we have simulated that there is a connection leak in our application. Lets troubleshoot.

Login to weblogic administration console. Assuming you have weblogic server running on local host @ port 7001, then URL should look like as below:

http://localhost:7001/console

Go to Datasources ---> Click on your pool name -----> Select "Connection Pool" tab as below.

Click on "Advance" link which will expand the configuration on same page. Look for "Inactive Connection Timeout" which is by default set to "0" secs , change it to positive integer as below.

Based on the value you set (in our case 1 minute) , Weblogic will forcibly start taking connections back from the threads which had been holding the connection more than configured time.

While doing so, Weblogic will print thread stack trace of the connection holder on weblogic server logs as below. If you look closely, you will see the application code where that connection was grabbed from the pool but never returned back. See snapshot.

Now you know the place where to look and how to fix the code bug.

In this Post, we have seen easiest way to catch hold on issue. If we pay close attention to this scenario, we are attacking on this issue after we have seen the Connection Leak in our application. Also, this approach needs us to periodically monitor the logs to make sure no connection leak is happening.

What if we want to get alert when this kind of situation arises and then we go to logs to troubleshoot the issue.

In Part 2 of this series, We will see how we can get alert automatically to take proactive action.

Stay Tuned.