Part 3: Garbage Collection - Troubleshooting OutOfMemory Issue

Overview:
In Part 2 of this series, we have seen different types of garbage collectors , heap memory and how to configure them using JVM options. Also we have seen how to enable GC logs by setting different options to get more and clear information about the JVM.
In this last part of this series, we will end our "GC trip journey" by looking what to do when an OutOfMemory issue happens

1. How to troubleshoot and
2. Tools to use

Introduction:
Memory leak in application leads to OutOfMemory error in running JVM. When too much time is spent doing garbage collection, Garbage Collector throws OutOfMemory error.In other words, if 98% of the time is being spent in garbage collection and recovering less than 2% of heap , that leads to OutOfMemory error.

Memory Leak Sample:
Below program is intentionally developed to throw OutOfMemory error.

package com.test.gc;

import java.util.ArrayList;
import java.util.List;

public class OutOfMemory {

 private List list=new ArrayList();

 private void add(KeyValuePair str){
  list.add(str);
 }

 private void remove(int index){
  //this line causes a memory leak
  KeyValuePair st=list.get(index);
  st=null;
 }

 public static void main(String[] args) {
  System.out.println("Starting");
  Thread t=new Thread(new Runnable(){
   @Override
   public void run() {
    OutOfMemory oom=new OutOfMemory();
    for(int index=0;index<1000000000;index++){
     oom.add(oom.new KeyValuePair());
     oom.remove(index);
     /*try {
      Thread.sleep(3);
     } catch (InterruptedException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
     }*/
    }

   }
  });
  t.start();

  try {
   Thread.sleep(2*60*1000);
  } catch (InterruptedException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }

 }

 class KeyValuePair{
  String key;
  String value;
 }

}

First thing we need to know is PID (process ID) of the running process (which is causing OutOfMemory error). On a given host, there could be number of processes running and there are number of ways to find one. Fortunately java also provides a way to find PID and that is

jps command:

 jps command provides you list of all running java process ids.

Now we have got the process ID , so we can check heap statistics and see how does it look like. "jmap" is the way to go.

 jmap command with "-heap" option:

From jmap command , we have seen that both Young(New) Generation and Old generation heap size is 100% used. That indicates that there is a memory leak. But by looking at above memory statistics we can't figure out which culprit object is causing this issue. There are number of ways to figure the culprit object out. Lets start with simplest way:

 jmap command with "-histo" option:
jmap -histo:live PID

Above snapshot gives enough information about the culprit  which is sitting right there in the memory having 119905 LIVE instances on the heap. If you are a GUI fan and would like see such informations in GUI, then you got to follow below steps. First take memory dump of running process as below.

jmap command with "-dump" option: If you have not set
 "-XX:+HeapDumpOnOutOfMemoryError" option for the JVM then.

jmap -dump:file=FILE_NAME PID  will take the memory dump and save at the supplied location.

Now file dump has been taken and stored in "outofmem_heap.bin" file.Our job is to peak into dump file and find out what kind of instances are being created and how much space they are taking.

 How to read that dump file ?
Again , there are number of ways to peak into dump file , like

1. jhat utility bundled with JDK 5 and above.
2. Visualvm GUI base tool to read the heap dump (part of JDK 7 bundle).
3. Eclipse MemoryAnalyzer tool etc...

1st Option :
jhat command
jhat needs 10 times more memory to open given size heap dump. for e.g. to open 256MB heap dump, jhat needs atleast 2048MB memory. (it may vary but for me it required this much memory).  
jhat -J-Xmx2048m HEAP_DUMP_FILE_NAME.
Above command reads the heap dump file and runs a server at default port 7000.

Now open Internet explorer and type http://HOST_NAME:7000" (or localhost if running locally). You will see page as below.

Click the link as shown with red arrow above. It will show count of culprit objects taking lot of memory as below.

2nd Option: Visualvm GUI based tool

3rd Option: Eclipse Memory Analyzer:
Memory Ananlyzer tool (MAT) is more powerful as it opens up huge size dump file with no issue. you may have to set more memory (in case you run out of memory issue with MAT while opening huge size file).

Happy Troubleshooting !