Leveraging the Java Ecosystem in Matlab

Matlab is a statistical analysis software package widely used for computational finance. One very useful feature of Matlab is its integration with Java. A standard Matlab installation includes a customized version of Java and several 3rd party open source and commercial Java libraries When you start a Matlab session, an embedded JVM is also launched in the background by default which allows software developers to leverage Java code within Matlab.
However, at times, this integration comes at a cost when there is a conflict between the version of a 3rd party Java library that comes packaged with Matlab and the version that you want to use, it is very hard to resolve this conflict.

One instance where we ran into this issue at BlackRock is when we developed a Java library that is used to retrieve data from our servers. We wanted to leverage this library inside Matlab and we found that the version of Apache HTTP Client library that was embedded in Matlab was older and incompatible with the version that our software depended on.

Resolving conflicts in Matlab

An internet search revealed that this problem has been faced by several other users of Matlab and there are a few other suggested solutions to this problem that we read about:

1. Change our software to use the older version of Apache HTTP Client library
2. Update the JAR file containing the HTTPClient library that is part of the Matlab installation to the newer version that we required
3. Use the undocumented feature of Matlab wherein a tag is used in javaclasspath.txt to override the JAR files used by Matlab (see https://stackoverflow.com/questions/16366059/best-way-to-override-matlabs-default-static-javaclasspath)

We quickly discovered that none of these options were feasible. Reverting to an older version of the library meant we would lose some features that we needed. Second and third options would mean fiddling with the installation and potentially introducing incompatibility with Matlab and/or other libraries that Matlab depends on.

So we explored other options and finally arrived at a solution by building a custom classloader. We developed a class called BlackRockMatlabInterface that serves as the interface between BlackRock software and Matlab. The BlackRockMatlabInterface itself is loaded using the Matlab classloader, however BlackRockMatlabInterface then initializes a custom classloader. This custom classloader is java.net.URLClassLoader, which has been initialized with URLs to all JAR files that the BlackRock software needs. The parent classloader of this custom classloader is set to null, thereby forcing the resolution of classes using only the URLs that have been provided.

The BlackRockMatlabInterface exposes a method createClass that uses the custom classloader to create the instance of given class name. As the instance has been created using the custom classloader, all transitive dependencies are sourced using the custom classloader as well, this approach effectively isolates us from the JAR files that Matlab provides.

This approach successfully solved the problem around library conflict, as the inbuilt Matlab functions can continue to use the libraries that were packaged with Matlab and at the same time we can use the libraries we need from an isolated classloader. So you can use your desired version of Java libraries without compromising the integrity of the Matlab installation.

As a result of this solution we were able to use our custom Java library and the newer version of Apache HTTP Client library within Matlab with the following benefits:

1. Persistent connections – our Java library is used to retrieve data from our internal server and it makes several separate calls to get the data. Using persistent connections helped us avoid the overhead of handshaking involved with each new connection.
2. Binary data transfer – in order to efficiently transfer data, our internal server supports a binary data format. The inbuilt functions in Matlab (webread/urlread) do not support this binary format. Hence we resorted to a custom built Java library to retrieve data from our server.
3. Error reporting – The inbuilt functions in Matlab (webread/urlread) report only generic error messages, however our server sends verbose error messages that are very helpful in troubleshooting. Our custom library captures and shows these detailed error messages.

References
1. http://www.mathworks.com/help/matlab/matlab_external/bringing-java-classes-and-methods-into-matlab-workspace.html
2. http://undocumentedmatlab.com/blog/static-java-classpath-hacks
3. https://stackoverflow.com/questions/16366059/best-way-to-override-matlabs-default-static-javaclasspath

TECH-0035

Manvendra Gupta and Mehmet Yilmaz

Tags: ,