09 February 2012

How to extract data from a Java heap dump

I will show how to extract data from the memory of a running Java application. Lets assume that this is the class with interesting data:
package se.lesc.blog.heap_extract;

import java.util.*;

/** Class that contain data to extract */
public class ArrayDataContainer {
    
    List<byte[]> arrays = new ArrayList<byte[]>();
    
    public String toString() {
        String result = "";
        for (int i = 0; i < arrays.size(); i++) {
            result += "Array " + i + ": " + new String(arrays.get(i)) + "n";
        }
        return result;
    }
}

An example main class that starts a Java process and populates the class with some data:
    public static void main(String[] args) throws Exception {
        ArrayDataContainer arrayDataContainer = new ArrayDataContainer();
        
        for (int i = 0; i < 20; i++) {
            String dataString = "This is my data " + i;
            arrayDataContainer.arrays.add(dataString.getBytes());
        }
        
        System.out.println(arrayDataContainer.toString());
        Thread.sleep(1000*300);
    }


Use your favorite tool to take a heap dump (for example jmap).

Open Java Visual VM (typically found in C:\Program Files\Java\jdk1.6.0_24\bin\jvisualvm.exe). It has an Object Query Language (OQL) feature that is very useful in this case. Go to File -> Load... Select Heap Dumps in the file open dialog and open the heap dump. Click on the "OQL Console" button.

Entry this query and press the execute button:
map(heap.objects("se.lesc.blog.heap_extract.ArrayDataContainer"), 
  function (it, index, array, result) {
    var res = '';
    for each (var element in it.arrays.elementData) {
      res += '<p/>';
      for each (var i in element) {
        res += i + ', ';
      }
    }
    return res;
  })

Copy the data from the Query Results and save it to a text file. Use this program to parse the file and recreate the List of byte[]:
package se.lesc.blog.heap_extract;

import java.io.*;
import java.util.*;

public class ArrayDataImporter {

    public static List<byte[]> importArrayList(InputStream in) throws Exception {
        List<byte[]> arrays = new ArrayList<byte[]>();
        
        BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(in));

        String line;
        while ((line = bufferedReader.readLine()) != null) {
            
            line = line.trim();
            if (line.isEmpty()) {
                continue;
            }

            List<Byte> array = new ArrayList<Byte>();
            
            String[] dataPoints = line.split(",");
            for (String dataPointString : dataPoints) {
                dataPointString = dataPointString.trim();
                
                int dataPoint = Integer.parseInt(dataPointString);
                array.add((byte) dataPoint);
            }
            
            byte[] byteArray = new byte[array.size()];
            for (int i = 0; i < array.size(); i++) {
                byteArray[i] = array.get(i);
            }
            
            arrays.add(byteArray);
        }
        
        return arrays;
    }
    

    public static void main(String[] args) throws Exception {
        ArrayDataContainer arrayDataContainer = new ArrayDataContainer();
        arrayDataContainer.arrays =
             ArrayDataImporter.importArrayList(
                 ArrayDataImporter.class.getResourceAsStream("query_results.txt"));
        System.out.println(arrayDataContainer);
    }
}

Things to note:
  1. The OQL help can be found at http://visualvm.java.net/oqlhelp.html.
  2. The OQL is mostly Javascript, so most of the "normal" Javascript functions will work.
  3. The heap.objects method targets every instance of a class (there was only 1 instance in this application).
  4. The it.arrays.elementData needs explaining. "it" is the iterator from the Javascript map method. "arrays" is the name of the attribute inside in my class ArrayDataContainer. "elementData" is the attribute inside ArrayList.
  5. The result of the query is rendered as a HTML page (thus explaining the <p/> in the middle of the query.

Mysql 5.1/5.5 and Unicode

I noticed that Mysql 5.1. and 5.5 has a difference with respect to the support for Unicode characters. Mysql 5.1 only supports the Basic Multilingual Plane (BMP), that is code points between 0 and 65535 (U+0000 to U+FFFF). Characters whose code points are greater than 65535 (U+FFFF) are called supplementary characters and support for these are included in Mysql 5.5. Read more at http://dev.mysql.com/doc/refman/5.5/en/charset-unicode.html.

Java String support both Basic Multilingual Plane and supplementary characters so this may cause some problems if a String is send directly into Mysql. Typically you will see this error message:
java.sql.SQLException: Incorrect string value: '\xC2\x9F' for column 'APN' at row 1


A little warning: even in Mysql 5.5 the "utf8" "charset" does NOT support supplementary characters. Use utf8mb4 instead.

07 February 2012

How to take a Java heap dump from command line

A heap dump is a great way to analyze a Java program in search of memory leaks. I mostly use Eclipse Memory analyzer for such an analysis. It is possible to acquire a heap dump from inside Eclipse Memory analyzer. But it may not always be possible to run a GUI in a foreign environment.

Use jps to find out the process id
C:\>jps
11448 MyLeakingApplication
12084 Jps

Use jmap to acquire the dump
C:\>jmap -dump:format=b,file=c:\Temp\mymemorydump.hprof 11448
Dumping heap to C:\Temp\mymemorydump.hprof ...
Heap dump file created

14 January 2012

Implemented choosable renderer for java quick search popup

I have now implemented so it is possible to choose (or implement one's own) renderer. I made a nice yellow renderer default:

I have made a web start version available. Try it out at http://code.google.com/p/java-quick-search-popup/.

08 January 2012

Created an open source project

I have for a while wanted to create an open source project, but has not had any good ideas. But yesterday I finally did it. I created http://code.google.com/p/java-quick-search-popup/. It is a small Java Swing component project.

24 November 2011

How to change file premissions in Git on Windows

Git manages file permissions for each file in the repository. It may be appropriate to have the executable bit set for shell/bash files to make them easier to execute in Linux systems.

Windows own file permissions does not map to the Git file permissions so it may be a bit hard to change file permissions when you are on Windows. This is how you do it:

Lets assume the file script.sh needs to have the executable bit set. Use the command git ls-tree to inspect the file permissions:
C:\views\myproject>git ls-tree HEAD
100644 blob 55c0287d4ef21f15b97eb1f107451b88b479bffe    script.sh


As you can see the file has 644 permission (ignoring the 100). We would like to change it to 755:
C:\views\myproject>git update-index --chmod=+x script.sh

C:\views\myproject>git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       modified:   script.sh
#


The file is now staged. Note that the file contents is not changed, only the meta data. We must commit the file so save the change:
C:\views\myproject>git commit -m "Changing file permissions"
[master 77b171e] Changing file permissions
 0 files changed, 0 insertions(+), 0 deletions(-)
 mode change 100644 => 100755 script.sh


Running git ls-tree again to see the change:
C:\views\myproject>git ls-tree HEAD
100755 blob 55c0287d4ef21f15b97eb1f107451b88b479bffe    script.sh

05 July 2011

Steam gives away Team Fortress 2

The popular multi player first person shooter game Team Fortress 2 is now free to play on Steam. I took the "bait" and installed Steam just to play this free game (and the game is really fun). So how can Steam money on this? Well there are two things that they can make money from:

1. In-game purchases
Many free 2 play games have an in-game store. In Team Fortress 2 there is a store where you can buy visual appearances (like hats) and also weapons. I was "suckered" in to buy some kind of instrument (called Wrangler) the engineer class in the game can use. For that I spent 0,39€:


2. Buy other games in Steam
The other thing that they can make money on is other games, that cost money. Since the user already has installed Steam the effort to buy something is lower. And since I could only transfer a minimum of 5 € to my virtual "Steam Wallet" I had some over. So I bought a game call Trine for 1,99€. Once I have bought one game (although cheep) there is a high probably that I will buy more games in the future.