OLD JULIA VERSION

The discussion about getting MKL to work on OSX is no longer applicable, but the discussion about chosing MKL versus OpenBLAS is still applicable.

Introduction

One of the great things about Julia for those in scientific computing is the ease of accessing highly optimized libraries. For matrix operations, Julia comes inbuilt with OpenBLAS, an open source implementation of BLAS, the Basic Linear Algebra Subprograms.

For the majority of people, that’s wonderful. OpenBLAS is quite fast and optimized.

BUT, when you want to diagonalize the large matrices that I do, there’s something better, Intel’s Math Kernel Library, MKL.

As Intel designed the chips and the hardware drivers for just about everyone, they can design their implementation of BLAS to take advantage of the specifics of the hardware and get a speed boost. More to my purposes, it also doesn’t start aborting on larger matrices, even though I had plenty of RAM left. The downside: they get this boost from trade secrets, and thus the software is propriety and behind closed doors. Moral objections for some, monetary objections for others.

If you want to get MKL for yourself, you have two possible routes:

  • A free community license through Intel Aviary . I got this for my workstation.

  • Convince your company/ university/ institute to get the fully supported and expensive version. For example, my institute’s cluster has all of Intel’s tools.

Do you need this?

Before you start trying to implement this on your system, take a second and decide whether or not it is worth your while. What kind of systems are you trying to diagonalize? Are you going to be diagonalizing systems at all? Or multiplying large matrices… that would count too…

I generated matrices through A=randn(n,n); and then diagonalized them through @time eigfact(A);.

All of these specs are for my Mac Pro, Late 2013 model, running OSX El Capitan. Processor: 3.7 GHz Quad-Core Intel Xeon E5. Memory: 64 GB 1866 MHz DDR3 ECC. I would be interested in seeing data for other processors.

Time Scaling

Time scaling for MKL and OpenBLAS. Performed for a matrix with randomly generated values according to a normal distribution of unit standard deviation.

Factor Scaling

The ratio between OpenBLAS and MKL. While comporable at small system sizes, at larger matrices, MKL shows a significant improvement.

Memory Scaling

Both MKL and OpenBLAS showed the same memory usage for a given calculation. The scaling appears quadratic, except for a deviation at small system sizes.

What you need to do

For Intel

So in my .zshrc, or .bashrc for those who haven’t discovered the wonders of zsh, I now have

export TBBROOT=fdsljkfds
source /opt/intel/mkl/bin/mklvars.sh intel64 ilp64

The value for TBBROOT is non-zero gobblety-gook. Once you have added that, either restart your terminal, or type

	source ~/.bashrc

to refresh your terminal.

Now, check that these environment variables are set up correctly:

  • MKLROOT
    • /opt/intel//compilers_and_libraries_2016.2.146/mac/mkl
  • DYLD_LIBRARY_PATH
    • /opt/intel//compilers_and_libraries_2016.2.146/mac/compiler/lib: /opt/intel//compilers_and_libraries_2016.2.146/mac/mkl/lib
  • LIBRARY_PATH
    • /opt/intel//compilers_and_libraries_2016.2.146/mac/compiler/lib: /opt/intel//compilers_and_libraries_2016.2.146/mac/mkl/lib
  • NLSPATH
    • /opt/intel//compilers_and_libraries_2016.2.146/mac/mkl/lib/locale/%l_%t/%N
  • MANPATH
    • /opt/intel//compilers_and_libraries_2016.2.146/mac/man/en_US
  • CPATH
    • /opt/intel//compilers_and_libraries_2016.2.146/mac/mkl/include

by typing

echo $NAME_OF_VARIABLE

Don’t just copy your variables against mine! Find your installation on the system, and checkout where the folders are.

For the Julia Installation

In the Julia file, edit Make.inc in this specific place

## Settings for various Intel tools
# Set to 1 to use MKL
USE_INTEL_MKL =1
# Set to 1 to use MKL FFT
USE_INTEL_MKL_FFT = 1
# Set to 1 to use Intel LIBM
USE_INTEL_LIBM ?= 0
# Set to 1 to enable profiling with Intel VTune Amplifier
USE_INTEL_JITEVENTS ?= 0
# Set to 1 to use Intel C, C++, and FORTRAN compilers
USEICC  ?= 0
USEIFC  ?= 0

Now in the Julia folder try

make
make install

How I eventually figured this out

I was getting complaints when makeing Julia, that

-L/opt/intel//compilers_and_libraries_2016.2.146/mac/tbb/lib

wasn’t found. There was good reason it wasn’t found. It didn’t exist. TBB stands for Threading Building Blocks, another one of Intel’s programs, but this one is meant for multicore C++ programs. Sounds fairly useful, but off topic to what I need right now.

So I wanted to figure out why it was trying to link to that directory. Looking in /opt/intel/mkl/bin/mklvars.sh, the program that sets environment variables for MKL, I discovered:

                if [ -z "${TBBROOT}" ]; then
                    mkl_ld_arch="${CPRO_PATH}/tbb/lib:${mkl_ld_arch}"
                fi

When the variable TBBROOT is zero, it adds this tbb folder to the path. Since I can’t change that file, proprietary stuff, my work around is making TBBROOT non-zero. Then DYLD_LIBRARY_PATH, which gets linked in the Julia make processes, only contains good locations.

Also, you can’t just run source ... to on the command line once and have the variables set for all eternity. When I restarted my terminal the next day, the variables had cleared. So I figured out the lines need to be put in the ~/.bashrc (or ~/.zshrc) instead of just run once.

Conculsions

I still got warnings when making Julia, but obviously none that broke the installation. Hopefully this work-around holds me over till this project is done, and hopefully it helps someone else too :)