Install STIP Software With OpenCV v2

OVERVIEW


Action recognition is an exciting research area. A state-of-the-art software package called STIP is available for computing Spatio-Temporal Interest Points (STIPs) as well as the corresponding HOG/HOF features. This functionality lies at the core of many action recognition algorithms.  Many thanks to Ivan Laptev's research group for making this software available.  


In this post, I detail how to install this package and it's dependencies.  This is a little complicated, because STIP depends crucially on OpenCV for computing optical flow and visualizing results.  However, the install process for OpenCV is a bit delicate, and getting STIP's Linux binaries to find the shared libraries created by OpenCV is a bit of a hassle.  This is especially true for newer versions of OpenCV (current as of this post is 2.3), since STIP seems to have been built with OpenCV 2.0 or earlier. So hopefully other research students can learn from my experience and be more productive.

STIP TECHNICAL DETAILS


Everything in this post is specific to version 2.0, which has many important upgrades.  Most notably, it allows dense and sparse features and user-defined keypoint locations.  

Version 2.0 does not offer *any* source code, only compiled binaries specifically for 64-bit Linux.   So everything done here must be done on a 64-bit Linux machine.  I'm a little bummed that the original source code isn't available, but not much can be done about that.

STIP comes with a README file that is reasonable quality and should give all instructions necessary to run the code and compute features.

OPENCV TECHNICAL DETAILS

Here, I've based these instructions off of my use of OpenCV version 2.3 for Linux (which is the same source for both 32 and 64 bit).  I'm pretty confident these instructions will translate for later versions, as well as maybe 2.2 and 2.1.

When installing OpenCV, you have many options for compiling with various features enabled. These are all handled via the CMake utility, but the available options are difficult to understand at first glance.  To get a handle, I ran the command

cmake -i  <location of downloaded source>

which allows you to walk step-by-step through each option, and keep defaults unless you specifically say otherwise.

For the video applications of interest here, it is essential that we compile with FFMPEG support on.  Some other handy options I discovered were:
  • Defining my own custom installation directory
    • This allows you to run everything without root privileges, which is nice.
    • CMAKE_INSTALL_PREFIX  is the relevant option here
  • Python Support (so you can call OpenCV functions from Python)
    • BUILD_NEW_PYTHON_SUPPORT
Some default advanced features I decided I could live without were CUDA support (to use GPUs to speed up computation), Eigen (which seems to be some matrix math library), and OpenEXR (which is for high-dynamic range cameras.

All told, you can compile under my exact configuration using the following shell script.

cmake -D  CMAKE_BUILD_TYPE=RELEASE \
      -D  CMAKE_INSTALL_PREFIX=/home/mhughes/code/opencv64/ \
      -D  WITH_FFMPEG=ON \
      -D  WITH_CUDA=OFF \
      -D  WITH_EIGEN=OFF \
      -D  WITH_OPENEXR=OFF \
      -D  BUILD_NEW_PYTHON_SUPPORT=ON \
      -D  BUILD_EXAMPLES=ON \
      -D  INSTALL_PYTHON_EXAMPLES=ON \
      /home/mhughes/code/OpenCV-2.3.0/src/

After that runs, just typing make install should install the entire OpenCV library.  In my experience, the installation process takes about 10 minutes or so.

MAKING STIP AND OpenCV PLAY NICE TOGETHER


Because STIP is built with precompiled binaries, it is not able to find OpenCV's shared libraries out of the box, and will instead throw errors. I found that I could fix this in two steps.

1. Create symbolic links from the library files STIP expects to actual OpenCV shared libraries

STIP 2.0 expects the following files to be available.  Remember that ".so" is a shared libary file extension.  
  • libcxcore.so.2
  • libcv.so.2
  • libhighgui.so.2
  • libcvaux.so.2
  • libml.so.2
However, OpenCV seems to have changed the shared libary naming convention since STIP was compiled.  There is no longer a one-to-one correspondence between the above names and the current names, which include the following:
  • libopencv_core.so
  • libopencv_video.so
  • libopencv_highgui.so
  • libopencv_ml.so
  • libopencv_imgproc.so
  • ... (and a few more)
I'm not totally sure how these two library structures are related, but I found a workaround that makes everything work just fine on my setup.  Essentially, I figured that related name fragments (highgui, ml, core) were important indicators of correspondence, and that the video library would also be important (it seems to be how STIP computes optical flow and maybe some other features).

Within the OpenCV install directory's shared libary directory, I build the following links:
  • libcxcore.so.2   --->  libopencv_core.so
  • libcv.so.2         ---> libopencv_imgproc.so
  • libhighgui.so.2  ---> libopencv_highgui.so
  • libml.so.2        --->  libopencv_ml.so
  • libcvaux.so.2    --->  libopencv_video.so
via the commands:

cd </PATH/TO/OPENCV/lib>
ln -s  libopencv_core.so  libcxcore.so.2
ln -s  libopencv_imgproc.so  libcv.so.2
ln -s  libopencv_highgui.so libhighgui.so.2
ln -s  libopencv_ml.so libml.so.2
ln -s libopencv_video.so  libcvaux.so.2

Remember that these don't necessarily mean that each library on the right entirely captures the library on the left.  Instead, it means that somewhere in the collection of all libraries on the right, we can search and find each function that is supposed to come from the original shared library in STIP.    In fact, it seems that others have solved this problem by linking to "libopencv_contrib" instead of "libopencv_imgproc", so there may be many other valid configurations.  However, the "core", "ml", "high_gui", and "video" libraries seem to be essential.

Note: for compatibility, I link to the shared libraries whose names are "unversioned".  For example, if you look inside the lib subdirectory of your install, you'll find the following file structure for each library type (e.g. "ml", "video", "highgui", ... )
  • libopencv_video.so@       ( unversioned name ... just a symbolic link )
  • libopencv_video.so.2.3@  ( also a symbolic link )
  • libopencv_video.so.2.3.0  (actual library file both of the above links point to)
I chose to link to the *unversioned* names so this advice will work in future releases of OpenCV as long as (1) naming conventions are kept  and (2) OpenCV continues to install shared libraries along with symbolic links with "unversioned" names.

2. Adjust the terminal's library path to include the OpenCV install directory

cd </PATH/TO/STIP>
export LD_LIBRARY_PATH=<PATH/TO/OPENCV/lib>

where of course you insert the actual filepath on your machine.

That's it.  You should now be able to call any of the STIP binaries successfully.

A quick way to verify you've done this correctly is to just check the help file for any of the binaries in STIP.  For example:

./bin/stipdet   --help

If you actually get a help message and not some kind of error, you're home free.

RESOURCES


I found this discussion at Stack Overflow especially helpful: http://stackoverflow.com/questions/5212728/libcxcore-so-2-missing-in-opencv
Comments