Build Matlab MCC Application to Run on SGE

PURPOSE

Here I outline how to compile Matlab .m files into a stand-alone C application. 

The primary benefit of using a stand-alone is that it *does not* consume Matlab licenses at run-time, so you can run hundreds of parallel applications on a grid without upsetting the license manager at all.

To this end, I'll also show how to get that C app to run on the Sun Grid Engine (SGE).

WHY THIS TUTORIAL?

Unfortunately, other tutorials I found, while helpful, lack realism.  The sample applications built are usually the super simple "generate a magic square matrix" variety.  These don't help anybody deal with important concerns like: 
  • what if my application relies on many .m files in other directories?
  • what if my application uses some .mex files?
  • what if I want to make sure command line arguments can be passed to my compiled app?
  • what if I need to  read from disk or write to disk ?
This tutorial hopes to show how all this can be done with minimal headaches. None of this is too complicated, but it's worth not having to figure out yourself.

ACKNOWLEDGMENTS

Thanks to Soumya Ghosh for a good intro conversation about using mcc and discussion about dealing with addpath and related issues.

RELATED LINKS

For other tutorials on using mcc, check out:



DEMO APPLICATION OVERVIEW

I've chosen as a demo application a Gibbs Sampler for Latent Dirichlet Allocation, which I use for my research. 
The application reads in a bag-of-words representation about a document collection from a .mat file, runs a Gibbs Sampler to discover the latent topic structure, and saves the results of each run of the sampler to a .mat file for long-term safe keeping.

You can find a compressed Unix archive of all the code for this project at this link or as an attachment at the bottom of this tutorial.  Windows users will probably need to do some extra legwork to make it work on their machines.

Here's a view of the key files and folders involved in the project:


The main matlab script is driver_LDA.m. The bash script to build the MCC app is buildMCC.sh.

The bash scripts to run a batch of grid jobs are submit_LDA_jobs.sh and grid_driver_LDA.sh.

The data directory contains the bag-of-words data itself and the script to load it. 

The mex directory contains the C++ code for an LDA Gibbs Sampler, as well as compiled mex files for 32 and 64 bit Unix.  I compiled under Matlab R2010a and gcc C++ compiler version 4.2, so you may need to recompile if your setup is different.

The LDAutil directory contains utility scripts that initializing the sampler, calculate log likelihood, etc.

We'll store the output and error messages from the grid in the logs directory, and the results of the sampler in the results directory.

GATEWAY MATLAB SCRIPT

For the purposes of this tutorial, you don't need to know anything specific about the files buried in different subdirectories.  We concentrate on the main Matlab script driver_LDA.m.

The first section of the code looks like this:

function [] = driver_LDA( D, K )
% Main Function to Execute LDA Gibbs Sampler
%   on abstracts from Psych Review journal
% ...
% Author: Mike Hughes (mhughes@cs.brown.edu)
% Date: 3 March 2011

%  **** END USER MUST MODIFY THESE TWO LINES! ****
datafilepath = '/home/mhughes/mex/mcc/data/bagofwords_psychreview.mat';
resultsDir = '/home/mhughes/research/results/mccExp/';

if ( ~isdeployed && ~ismcc )
    addpath( genpath( '.' ) );
end

if strcmp( class(D), 'char' )
    Params.D = str2double(D);
elseif strcmp( class(D), 'double' )
    Params.D = D;
end

if strcmp( class(K), 'char' )
    Params.K = str2double(K);
elseif strcmp( class(K), 'double' )
    Params.K = K;
end
...

We notice a few important lessons that can generalize to any Matlab application we wish to deploy.
  1. We need to set Matlab's path via addpath, since dependent functions are in subdirectories
    1. Our standalone command will NOT use this command, so it's probably best to have MCC skip over it using the isdeployed or ismcc functions (these are default functions in the Matlab library). I have noticed the application runs fine without these though.
  2. We define an input file to read data from and an output directory to write to.
    1. These should be specified as absolute file paths, never as relative.
    2. Make sure if you run this example that you CHANGE these filepaths for your local system.
  3.  We want to pass numeric values (D,K) as arguments from the command line
    1. Need to be ready to cast these from strings to numeric via str2double
The rest of the script isn't too important for this tutorial, but hopefully is well-commented enough that the interested and diligent reader could follow along alone if necessary.

RUNNING THE LDA GIBBS SAMPLER IN MATLAB

It's probably best to make sure the code runs fine in Matlab first before we try to compile it to C.

As a demo, we call the function driver_LDA from the Matlab command line, passing arguments indicating to run the sampler on D=500 documents in the dataset and to discover K=10 topics.

>> driver_LDA( 500, 10)

Params =

    D: 500
    K: 10
    V: 9244

============= Run = 1/2 =================
Begin MEX LDA Sampler :10 iters
  completed iter 1/10 after 0.11 sec
  completed iter 2/10 after 0.23 sec
  completed iter 3/10 after 0.36 sec
  completed iter 4/10 after 0.49 sec
  completed iter 5/10 after 0.62 sec
  completed iter 6/10 after 0.75 sec
  completed iter 7/10 after 0.88 sec
  completed iter 8/10 after 1.00 sec
  completed iter 9/10 after 1.14 sec
  completed iter 10/10 after 1.26 sec
 ....................................   saved to file /home/mhughes/mex/mcc/results/K10/Samples_01.mat
============= Run = 2/2 =================
Begin MEX LDA Sampler :10 iters
  completed iter 1/10 after 0.11 sec
  completed iter 2/10 after 0.24 sec
  completed iter 3/10 after 0.37 sec
  completed iter 4/10 after 0.50 sec
  completed iter 5/10 after 0.62 sec
  completed iter 6/10 after 0.75 sec
  completed iter 7/10 after 0.88 sec
  completed iter 8/10 after 1.01 sec
  completed iter 9/10 after 1.13 sec
  completed iter 10/10 after 1.25 sec
 ....................................   saved to file /home/mhughes/mex/mcc/results/K10/Samples_02.mat

The results were saved to the results folder.

COMPILING VIA MCC

Our goal is to build -- in a separate directory -- a standalone C application that implements the functionality of driver_LDA.m.   We'll call this standalone directory mccLDA/. Make sure this directory exists and is empty.

We use the shell script buildMCC.sh to compile the application, since we need to give mcc lots of custom arguments.

This script looks as follows: (don't worry, all but one line are comments)

#!/bin/bash
# Script to build Matlab stand-alone application for LDA Gibbs Sampler
# NB: Make sure to run the following before trying to execute this script
#      >> chmod +x buildMCC.sh

mcc -m -v -I './data/' -I './LDAutil/' -I './mex/' -R '-nojvm, -nodesktop, -nosplash' -d 'mccLDA' driver_LDA.m

# Explanation of option flags for mcc
# -v  = verbose mode
# -d <outdir> = create all output files in specific directory <outdir>.  must exist beforehand.
# -m  = create stand-alone C application
# -I <mdir> = add the directly <mdir> to search path when looking for necessary .m files
# -R <matlabopt> = set the C application to use the matlab startup option <matlabopt> by default
#     in particular, the following are helpful options:
#         -nojvm =  disables java virtual machine (which shouldn't be necessary for grid jobs
#         -nodesktop = tells matlab to not be a desktop GUI application
#         -nosplash  = disables startup "splash" screen with matlab logo on startup  
# -? = displays mcc help

One key observation is that we need to
tell mcc explicitly where all the dependent .m files and .mex files are via the -I option flag.

This could get complicated for huge applications, but a good solution is just to use Matlab's built in genpath command at the root directory of the project.

>> genpath('.')

ans =

.:./LDAutil:./data:./mccLDA:./mex:

Feeding the resulting string to the -I flag should be just fine.


RUNNING THE C APPLICATION THROUGH THE TERMINAL

MCC generates a shell script called "run_driver_LDA.sh" that is all we need to run our app.

Its first argument is the location of the Matlab default libraries.

The remaining args are the arguments D,K for our original Matlab function driver_LDA.m.

Below is an example interactive terminal session running this app on a single 64-bit machine.

bench6 ~/mex/mcc $ cd mccLDA
bench6 ~/mex/mcc/mccLDA $ ls
driver_LDA*     driver_LDA_main.c                mccExcludedFiles.log  run_driver_LDA.sh*
driver_LDA.prj  driver_LDA_mcc_component_data.c  readme.txt            submit_LDA_jobs.sh
bench6 ~/mex/mcc/mccLDA $ ./run_driver_LDA.sh /local/projects/matlab/R2010a/ 500 10
------------------------------------------
Setting up environment variables
---
LD_LIBRARY_PATH is .:/local/projects/matlab/R2010a//runtime/glnxa64:/local/projects/matlab/R2010a//bin/glnxa64:/local/projects/matlab/R2010a//sys/os/glnxa64:/local/projects/matlab/R2010a//sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/local/projects/matlab/R2010a//sys/java/jre/glnxa64/jre/lib/amd64/server:/local/projects/matlab/R2010a//sys/java/jre/glnxa64/jre/lib/amd64/client:/local/projects/matlab/R2010a//sys/java/jre/glnxa64/jre/lib/amd64

Params =

    D: 500
    K: 10
    V: 9244
============= Run = 1/2 =================
Begin MEX LDA Sampler :10 iters
  completed iter 1/10 after 0.57 sec
  completed iter 2/10 after 1.19 sec
  completed iter 3/10 after 1.78 sec
  completed iter 4/10 after 2.37 sec
  completed iter 5/10 after 2.96 sec
  completed iter 6/10 after 3.54 sec
  completed iter 7/10 after 4.13 sec
  completed iter 8/10 after 4.71 sec
  completed iter 9/10 after 5.30 sec
  completed iter 10/10 after 5.88 sec
 ....................................   saved to file /home/mhughes/mex/mcc/results/K10/Samples_01.mat
============= Run = 2/2 =================
Begin MEX LDA Sampler :10 iters
  completed iter 1/10 after 0.56 sec
  completed iter 2/10 after 1.16 sec
  completed iter 3/10 after 1.72 sec
  completed iter 4/10 after 2.33 sec
  completed iter 5/10 after 2.89 sec
  completed iter 6/10 after 3.46 sec
  completed iter 7/10 after 4.05 sec
  completed iter 8/10 after 4.64 sec
  completed iter 9/10 after 5.24 sec
  completed iter 10/10 after 5.81 sec
 ....................................   saved to file /home/mhughes/mex/mcc/results/K10/Samples_02.mat



RUNNING THE C APPLICATION ON THE GRID

To get this C application to run on the Sun Grid Engine (SGE), we must have some helper scripts to (1) submit a bunch of jobs to the grid and (2) instruct the grid what commands to execute for each job.

1) Script that submits a bunch of jobs to the grid.

The script submit_LDA_jobs.sh contains the following code:

#! /bin/sh
# Bash script for sumbitting matlab jobs to an SGE cluster queue.
# This is the script you run from the command line to submit the jobs.
# Original script skeleton from David Black-Schaffer
#     http://cva.stanford.edu/people/davidbbs/cluster/matlab/

# Use the default settings configured for Brown University's grid
#  (non-Brown people definitely don't need this)
source /com/sge/default/common/settings.sh

# Submit 3 jobs to the grid
qsub -m bea -l arch=lx24-amd64 -t 1-3 grid_driver_LDA.sh 500

# Option explanations:
#  Mail Option (-m bea):  trigger email alerts at beginning, end, and aborting of jobs
#  Arch Option: (-l arch=...):  make sure 64 bit machines only
#  Batch option: (-t 1-3): submit array of jobs, each numbered 1,2, or 3

This script submits (via qsub command) 3 jobs labeled ("1","2","3") that each execute the job script grid_driver_LDA.sh with input argument D=500.

Here at Brown we run this script on the terminal after ssh-ing into the head grid node sge, but other users will need to consult with their grid system admin to figure out what to do.

2) Script that defines what each job should execute

The script grid_driver_LDA.sh sets up each grid job and launches our standalone application with the appropriate arguments. 

The script assigns a different number of topics to each of the 3 jobs initialized by the above code.  We denote K as the number of topics parameter.  It should be 10, 20, and the 30 for jobs 1, 2, and 3 respectively.

Note that it's better to assign input parameters based on SGE_TASK_ID in this wrapper shell script than in the Matlab code, since we don't want to recompile the entire MCC application just to change the range of input parameters.

The full code contents of grid_driver_LDA.sh are listed here:

#!/bin/bash
#$ -S /bin/sh
#$ -cwd
#$ -r y
#$ -q long.q -l arch=lx24-amd64
# put stdout and stderr files in the right place for your system.
#   NOTE that $TASK_ID is the correct var here, but not in rest of script (need SGE_TASK_ID)
#$ -o logs/$JOB_ID.$TASK_ID.out
#$ -e logs/$JOB_ID.$TASK_ID.err

echo "JOB ID: " $JOB_ID
echo "SGE TASK ID: "$SGE_TASK_ID

# Assign number of topics K based on task number assigned by SGE
#   remember that "=" as assignment operator CANNOT be surrounded by spaces
Ks=(10 20 30 40)
K=${Ks[$SGE_TASK_ID-1]}

echo "K: " $K

# Run the MCC main script
#  make sure the first arg is the path to the matlab libraries
#  remaining arguments are:
#          the D value passed to this script
#          the K value we computed based on the task number
mccLDA/run_driver_LDA.sh /local/projects/matlab/R2010a $1 $K

exit

The first section of this script sets some options for SGE.  Most of these aren't too interesting.

The most important thing is that it directs each job to send its standard output and standard error feeds to files with names of the form:

     path/to/
working/dir/logs/<JOB_ID>.<TASK_ID>.out
     path/to/working/dir/logs/<JOB_ID>.<TASK_ID>.err

I find this much easier to use than the default log files created by SGE, which get dumped directly in the current working directory en masse.

3) Actually running a batch of jobs on the grid

With the above scripts and a fully compiled MCC application, we can finally use the grid!

Here's a snapshot of my terminal session on the head grid node sge to launch the jobs.

sge ~/mex/mcc $ pwd
/home/mhughes/mex/mcc
sge ~/mex/mcc $ ls
LDAutil/      data/         logs/    mex/      submit_LDA_jobs.sh*
buildMCC.sh*  driver_LDA.m  grid_driver_LDA.sh*  mccLDA/  results/
sge ~/mex/mcc $ ./submit_LDA_jobs.sh
Your job-array 8385129.1-3:1 ("grid_driver_LDA.sh") has been submitted

That's all the feedback you get from the grid launching.  The log files contain much more information about the progress of each job.

We can check that our tasks were run with the correct parameters by examining the first three lines of each log file.  We should see that K is set to 10, 20, and 30 on tasks 1, 2, and 3.

sge ~/mex/mcc $ head -n3 ./logs/8385129.1.out
JOB ID:  8385129
SGE TASK ID: 1
K:  10
sge ~/mex/mcc $ head -n3 ./logs/8385129.2.out
JOB ID:  8385129
SGE TASK ID: 2
K:  20
sge ~/mex/mcc $ head -n3 ./logs/8385129.3.out
JOB ID:  8385129
SGE TASK ID: 3
K:  30


Finally, we can examine the complete standard output of a job to make sure its working properly.

 sge ~/mex/mcc $ cat ./logs/8385129.3.out
JOB ID:  8385129
SGE TASK ID: 3
K:  30
------------------------------------------
Setting up environment variables
---
LD_LIBRARY_PATH is .:/local/projects/matlab/R2010a/runtime/glnxa64:/local/projects/matlab/R2010a/bin/glnxa64:/local/projects/matlab/R2010a/sys/os/glnxa64:/local/projects/matlab/R2010a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/local/projects/matlab/R2010a/sys/java/jre/glnxa64/jre/lib/amd64/server:/local/projects/matlab/R2010a/sys/java/jre/glnxa64/jre/lib/amd64/client:/local/projects/matlab/R2010a/sys/java/jre/glnxa64/jre/lib/amd64
Warning: No display specified.  You will not be able to display graphics on the screen.
SGE_TASK_ID = 3

Params =

    D: 500
    K: 30
    V: 9244

============= Run = 1/2 =================
Begin MEX LDA Sampler :10 iters
  completed iter 1/10 after 0.70 sec
  completed iter 2/10 after 1.46 sec
  completed iter 3/10 after 2.19 sec
  completed iter 4/10 after 2.93 sec
  completed iter 5/10 after 3.65 sec
  completed iter 6/10 after 4.38 sec
  completed iter 7/10 after 5.12 sec
  completed iter 8/10 after 5.89 sec
  completed iter 9/10 after 6.62 sec
  completed iter 10/10 after 7.33 sec
 ....................................   saved to file
/home/mhughes/mex/mcc/results/K30/Samples_01.mat
============= Run = 2/2 =================
Begin MEX LDA Sampler :10 iters
  completed iter 1/10 after 0.67 sec
  completed iter 2/10 after 1.42 sec
  completed iter 3/10 after 2.12 sec
  completed iter 4/10 after 2.87 sec
  completed iter 5/10 after 3.61 sec
  completed iter 6/10 after 4.34 sec
  completed iter 7/10 after 5.08 sec
  completed iter 8/10 after 5.81 sec
  completed iter 9/10 after 6.57 sec
  completed iter 10/10 after 7.28 sec
 ....................................   saved to file /home/mhughes/mex/mcc/results/K30/Samples_02.mat

Alternatively, we can inspect the contents of all subdirectories of our results directory:

sge ~/mex/mcc $ ls -R results
results:
K10/  K20/  K30/

results/K10:
Samples_01.mat  Samples_02.mat

results/K20:
Samples_01.mat  Samples_02.mat

results/K30:
Samples_01.mat  Samples_02.mat

Looks like everything's there.  We have 2 sampler runs saved for each value of our parameter K.

That's it!  We've successfully deployed a standalone Matlab application on the grid.
ċ
mcc_demo_LDAsampler.tar.gz
(145k)
Mike Hughes,
Mar 4, 2011, 11:21 AM