Conservation Tracking dependencies and fixes for building with GCC on Linux and Mac

[chaubold]

This pull request contains various fixes to enable compilation of ilastik and its dependencies, including those needed for conservation tracking.

Successfully builds on:

• Ubuntu 14.04 using GCC 4.8
• Ubuntu 12.04 using GCC 4.7
• Mac OSX 10.9 using Homebrew's GCC 4.8
• a OSX 10.8 build is currently running

Most fixes are unrelated to Conservation Tracking and include:

• adding the pythonpath to BUILDEM_ENV_STRING
• fix openssl building on Ubuntu 14.04 - had Perl incompatibilities
• CPLEX path hints for Ubuntu
• boost, vtk and python patches for OSX

Important for Conservation Tracking:

• added mlpack and armadillo as dependencies
• armadillo currently needs OpenBLAS -> -DWITH_ATLAS=0
  • OpenBLAS builds fine on the aforementioned OSes
• add boost random
• update pgmlink.cmake

In kind collaboration with @martinsch

---

This change is 

[stuarteberg]

It looks like 132db22 refers to a patchfile called pgmlink-cmake.patch, but I think that file was accidentally left out of this PR...

[stuarteberg]

In this PR, commit 190ac3f reverts a commit from the master branch. Was it necessary to do this?

Backstory: Months ago, I added the CC and CXX environment variables to the BUILDEM_ENV_STRING because I thought it would help ensure that the same compiler is used for all BuildEM packages. But recently @DocSavage discovered that this change somehow breaks the go build. Since I couldn't think of a specific problem with simply removing the CC and CXX vars, we just took them out. In this PR, they've been added again. Presumably that breaks go.cmake again...

[martinsch]

I don't have write permissions on chaubold's branch atm, but the patch can be found here: martinsch/buildem@48d06c6

[chaubold]

Oh right, good catch, this is a relict. I'll remove the reference to the patch as we updated pgmlink's CMakeLists.txt.

[chaubold]

Regarding 190ac3f, you added this flag right around the time when I first tried to build on Mac using GCC, and so I relied on this way to pass the CXX and CC settings through to non-CMake-based ExternalProjects. Thus for the Mac-GCC build they are crucial.

I did never build go though, and thus did not experience errors with that.

Building ExternalProjects without these settings on OSX 10.9 results in a fallback to /usr/bin/gcc and g++, which are clang.

What do you suggest we should do there?

[DocSavage]

I think the issue is what CMAKE defaults to for CMAKE_C_COMPILER and CMAKE_CXX_COMPILER. We could change the CXX and CC settings to buildem-specific variables that are not initialized. So instead of this:
CC=${CMAKE_C_COMPILER} CXX=${CMAKE_CXX_COMPILER}
we use this:
CC=${BUILDEM_C_COMPILER} CXX=${BUILDEM_CXX_COMPILER}

This would let you set the compilers but if they aren't set, compilation falls through to the OS-specific default.

[stuarteberg]

I think it's strange that the go build script chooses a different default compiler than cmake chooses. I was going to propose a go-specific fix in go.cmake, and then we can use the standard CMAKE vars everywhere else:

# For some reason, the go build script can choke on Mac OS X if the CC environment variable is explicitly specified.
# Therefore, we remove (well, rename) that environment variable before we build.
# (See also: 190ac3ff0a449b0a8ac8640018297176d16c19be)
set(GO_BUILD_ENV_STRING ${BUILDEM_ENV_STRING})
string(REGEX REPLACE "(^| )CC=" " CC_IGNORED=" GO_BUILD_ENV_STRING ${GO_BUILD_ENV_STRING})
string(REGEX REPLACE "(^| )CXX=" " CXX_IGNORED=" GO_BUILD_ENV_STRING ${GO_BUILD_ENV_STRING})

ExternalProject_Add_Step(${golang_NAME} stupid_step
    COMMAND     ${GO_BUILD_ENV_STRING} GOBIN=${BUILDEM_BIN_DIR} ./all.bash
    DEPENDEES  patch
    WORKING_DIRECTORY   ${golang_SRC_DIR}/src)

(The above changes simply remove the CC and CXX variables from the environment before building go.)

[DocSavage]

That works for me :)

[JensNRAD]

Additionally to the PR of @chaubold I had to do the following changes to compile on a Ubuntu 14.04 machine:

diff --git a/opengm.cmake b/opengm.cmake
index 6741c35..90273ed 100644
--- a/opengm.cmake
+++ b/opengm.cmake
@@ -47,6 +47,7 @@ ExternalProject_Add(${opengm_NAME}
         -DBUILD_PYTHON_WRAPPER=ON
         -DBUILD_PYTHON_DOCS=OFF
         -DWITH_OPENMP=OFF # Mac doesn't support OpenMP
+   -DPYTHON_INCLUDE_DIR=${BUILDEM_DIR}/lib/python2.7/site-packages/numpy/core/include/
         ${CMAKE_CPLEX_ROOT_DIR}

     BUILD_COMMAND       ${BUILDEM_ENV_STRING} $(MAKE)
diff --git a/pgmlink.cmake b/pgmlink.cmake
index fce0945..1e8ce8e 100644
--- a/pgmlink.cmake
+++ b/pgmlink.cmake
@@ -60,8 +60,10 @@ else()
             -DWITH_TESTS=ON
             -DWITH_CHECKED_STL=OFF
             -DPYTHON_INCLUDE_DIR=${PYTHON_INCLUDE_PATH}
+            -DPYTHON_INCLUDE_DIR2=${PYTHON_INCLUDE_PATH}
             -DPYTHON_LIBRARY=${PYTHON_LIBRARY_FILE}
-            -DVigranumpy_DIR="${BUILDEM_DIR}/lib/vigranumpy"
+            -DVIGRANUMPY_DIR="${BUILDEM_DIR}/lib/vigranumpy"
+       -DVIGRA_INCLUDE_DIR="${BUILDEM_DIR}/include/vigra"
             ${CMAKE_CPLEX_ROOT_DIR}

         BUILD_COMMAND       ${BUILDEM_ENV_STRING} $(MAKE)

Furthermore, I had to
a)
make symbolic links in ${BUILDEM_DIR}/lib/python2.7/site-packages/:
futures -> futures-2.1.6-py2.7.egg/futures/
concurrent -> futures-2.1.6-py2.7.egg/concurrent/
b) ocasionally re-install nose

Everything then compiles flawlessly and the tests run fine. However, I do not see all the workflows I am supposed to see: Either tracking or object classification is missing, accompanied by that kind of message:

No handlers could be found for logger "ilastik.workflows"

In this case object classification is missing.

[chaubold]

@JensNRAD: for OpenGM the change is to set -DPYTHON_NUMPY_INCLUDE_DIR to the path mentioned above, not -DPYTHON_INCLUDE_DIR, right?

[JensNRAD]

@chaubold: Here are my ccmake settings (which compiled fine):

 PYTHON_EXECUTABLE                /home/share/code/ilastik/bin/python          
 PYTHON_INCLUDE_DIR               /home/share/code/ilastik/lib/python2.7/site-packages/numpy/core/include
 PYTHON_INCLUDE_DIR2              /home/share/code/ilastik/include/python2.7   
 PYTHON_LIBRARY                   /home/share/code/ilastik/lib/libpython2.7.so 
 PYTHON_LIBRARY_DEBUG             PYTHON_LIBRARY_DEBUG-NOTFOUND                
 PYTHON_NUMPY_INCLUDE_DIR         /usr/include

On the other hand, you are probably right ... But I did not test it.

[chaubold]

Your settings probably work as well because there is this second PYTHON_INCLUDE_DIR. I think this is the supposed configuration:

PYTHON_EXECUTABLE                /home/chaubold/ilastik-buildem/bin/python                                                                
 PYTHON_INCLUDE_DIR               /home/chaubold/ilastik-buildem/include/python2.7                                                         
 PYTHON_INCLUDE_DIR2              /home/chaubold/ilastik-buildem/include/python2.7                                                         
 PYTHON_LIBRARY                   /home/chaubold/ilastik-buildem/lib/libpython2.7.so                                                       
 PYTHON_LIBRARY_DEBUG             PYTHON_LIBRARY_DEBUG-NOTFOUND                                                                            
 PYTHON_NUMPY_INCLUDE_DIR         /home/chaubold/ilastik-buildem/lib/python2.7/site-packages/numpy/core/include   

Now the question is: what were the settings before we touched them? Was only the PYTHON_NUMPY_INCLUDE_DIR pointing to /usr/include or PYTHON_INCLUDE_DIR as well? I think the former was the case, but not 100% sure.

[JensNRAD]

As far as I recall the PYTHON_INCLUDE_DIR was set to the system one, that's why I changed it.

[JensNRAD]

We compiled (successfully) on OSX 10.9 merging this PR with @jakirkham's PR #31. Additionally, the following changes had to be made:

buildem(constracking)$ git diff
diff --git a/openblas.cmake b/openblas.cmake
index 0724059..ca8e34a 100644
--- a/openblas.cmake
+++ b/openblas.cmake
@@ -1,7 +1,6 @@
 #
 # Install OpenBLAS from source
 #
-
 if (NOT openblas_NAME)

 CMAKE_MINIMUM_REQUIRED(VERSION 2.8)
diff --git a/opengm.cmake b/opengm.cmake
index 6741c35..7a6e245 100644
--- a/opengm.cmake
+++ b/opengm.cmake
@@ -14,6 +14,7 @@ include (PatchSupport)
 include (boost)
 include (hdf5)
 include (python)
+include (numpy)

 external_git_repo (opengm
     576dc472324a5dce40b7e9bb4c270afbd9b3da37
@@ -47,6 +48,7 @@ ExternalProject_Add(${opengm_NAME}
         -DBUILD_PYTHON_WRAPPER=ON
         -DBUILD_PYTHON_DOCS=OFF
         -DWITH_OPENMP=OFF # Mac doesn't support OpenMP
+       -DPYTHON_NUMPY_INCLUDE_DIR=${BUILDEM_DIR}/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include
         ${CMAKE_CPLEX_ROOT_DIR}

     BUILD_COMMAND       ${BUILDEM_ENV_STRING} $(MAKE)
diff --git a/openssl.cmake b/openssl.cmake
index 76968f2..87cc124 100644
--- a/openssl.cmake
+++ b/openssl.cmake
@@ -13,10 +13,16 @@ include (BuildSupport)
 include (zlib)

 external_source (openssl
-    1.0.1c
-    openssl-1.0.1c.tar.gz
-    ae412727c8c15b67880aef7bd2999b2e
-    http://www.openssl.org/source)
+    1.0.1i             
+    openssl-1.0.1i.tar.gz
+    c8dc151a671b9b92ff3e4c118b174972
+    http://www.openssl.org/source
+    FORCE)
+    # 1.0.1c
+    # openssl-1.0.1c.tar.gz
+    # ae412727c8c15b67880aef7bd2999b2e
+    # http://www.openssl.org/source
+    

 if (${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
     # On Mac, 64-bit builds must be manually requested.
diff --git a/readline.cmake b/readline.cmake
index 4b14958..aaf1e18 100644
--- a/readline.cmake
+++ b/readline.cmake
@@ -29,10 +29,10 @@ ExternalProject_Add(${readline_NAME}
     UPDATE_COMMAND      ""
     PATCH_COMMAND       ""
     CONFIGURE_COMMAND   ""
-    BUILD_COMMAND       ${BUILDEM_ENV_STRING} ${PYTHON_EXE} setup.py build
+    BUILD_COMMAND       ${BUILDEM_ENV_STRING} CC=gcc CXX=g++ ${PYTHON_EXE} setup.py build
     BUILD_IN_SOURCE     1
     TEST_COMMAND        ""
-    INSTALL_COMMAND     ${BUILDEM_ENV_STRING} ${PYTHON_EXE} setup.py install
+    INSTALL_COMMAND     ${BUILDEM_ENV_STRING} CC=gcc CXX=g++ ${PYTHON_EXE} setup.py install
 )

 set_target_properties(${readline_NAME} PROPERTIES EXCLUDE_FROM_ALL ON)
diff --git a/vigra.cmake b/vigra.cmake
index 8d79dfb..856b7dd 100644
--- a/vigra.cmake
+++ b/vigra.cmake
@@ -120,7 +120,7 @@ ExternalProject_Add(${vigra_NAME}
         -DFFTW3_INCLUDE_DIR=${BUILDEM_DIR}/include
         -DFFTW3_LIBRARY=${BUILDEM_DIR}/lib/libfftw3.${BUILDEM_PLATFORM_DYLIB_EXTENSION}

-        -DCMAKE_CXX_FLAGS=-pthread
+        -DCMAKE_CXX_FLAGS=-pthread\ -std=c++11
         -DCMAKE_CXX_LINK_FLAGS=-pthread
         -DCMAKE_CXX_FLAGS_RELEASE=-O2\ -DNDEBUG # Some versions of gcc miscompile vigra at -O3
         -DCMAKE_CXX_FLAGS_DEBUG="${CMAKE_CXX_FLAGS_DEBUG}"

ilastik-build-Linux(master)$ git diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index cf6b4d2..836b323 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -51,6 +51,9 @@ else()
 endif()

 include (scikit-image)
+include (matplotlib)
+include (ipython)
+

[openblas-git]

in /opt/local/bin (mac ports dir)
sudo ln -s /usr/bin/ranlib gcc-ranlib
sudo ln -s /usr/bin/ar gcc-ar

[matplotlib]

${BUILDEM_DIR}/include
ln -s freetype2/freetype

[opengm-git]
pull master, the version included results in an import error

[JensNRAD]

Updated to newest stable matplotlib version

diff --git a/matplotlib.cmake b/matplotlib.cmake
index d08062a..956d3ef 100644
--- a/matplotlib.cmake
+++ b/matplotlib.cmake
@@ -28,10 +28,13 @@ include (freetype2)
 include (tornado)

 external_source (matplotlib
-    1.1.1
-    matplotlib-1.1.1.tar.gz
-    8cbeaae8ba9da703d926e74c3e7c8a57
-    http://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.1.1)
+    1.4.0
+    matplotlib-1.4.0.tar.gz    
+    0bf2e83b970806bd5a630992b80993c7       
+    https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.4.0/matplotlib-1.4.0
+    FORCE)
+

 message ("Installing ${matplotlib_NAME} into FlyEM build area: ${BUILDEM_DIR} ...")
 ExternalProject_Add(${matplotlib_NAME}

[chaubold]

When merging John's branch, a new version of h5py is installed which brings new modules.
To build an OSX binary with that, in the setup_mac.py file of ilastik we also need to add 'h5py.h5ac', 'h5py._objects' to the includes.

[jakirkham]

OpenSSL is updated in master ( 3990614 ). So, that doesn't need to be included in the PR, as well. Also, the download has been added ( janelia-flyem/janelia-flyem.github.com@5421664 ) so "FORCE" should not be needed.

I'm not clear on why CC and CXX are set to GNU compilers explicitly in readline.cmake. I built this using LLVM compilers without an issue. Was there a problem that you encountered? If not, I don't think we should be setting these.

This is a bit nitpicky, but do we need to delete the empty line in openblas.cmake?

As for vigra, does it now require C++11? Setting this flag would stop some people using older compilers from building this. Certainly, it doesn't affect me, but maybe other people care about this.

Finally, matplotlib, the freetype error is a bit interesting. Though I didn't run into this problem, it does seem that you are not alone with this issue. Does this problem go away with the new matplotlib version? If not, maybe we should add BUILDEM_DIR/include/freetype2/ to the include path for matplotlib.

[JensNRAD]

I'm not clear on why CC and CXX are set to GNU compilers explicitly in readline.cmake. I built this using LLVM compilers without an issue. Was there a problem that you encountered? If not, I don't think we should be setting these.

Using the gcc/g++ from mac ports I ran into assembler errors. Using these flags where the only way to built readline.

This is a bit nitpicky, but do we need to delete the empty line in openblas.cmake?

No, of course not. The whole purpose was to show what I needed to change on my Mac (with ports) to build ilastik. I did not pay attention to empty lines. I actually put a if (NOT APPLE) there to disable openblas.

As for vigra, does it now require C++11?

I don't recall the error (maybe @chaubold does?), but it was necessary for building vigra on both our Mac's

Does this problem go away with the new matplotlib version? If not, maybe we should add /include/freetype2/ to the include path for matplotlib.

Probably no, but I did not test it (I left the symlink in place). So I think it is a good idea to add it to the include path.

[jakirkham]

Using the gcc/g++ from mac ports I ran into assembler errors. Using these flags where the only way to built readline.

So, which gcc and g++ are they referring too then?

I actually put a if (NOT APPLE) there to disable openblas.

Also, you can run cmake using -DWITH_ATLAS=1 to get around OpenBLAS.

[chaubold]

So, which gcc and g++ are they referring too then?

We are building everyting with GCC 4.8 (installed via Homebrew or Mac Ports), as we need the CPLEX Optimizer, which does not work with the LLVM libc++ yet. Thus, for the whole buildem tree, we set CC/CXX to gcc-4.8/g++-4.8. But apparently building readline with GCC 4.8 did not work for @JensNRAD, so he specified to use the system compiler clang. We need to investigate more before before we can propose a fix for that, as that never occurred to me (Jens' GCC 4.8 comes from MacPorts, mine from Homebrew).

Also, you can run cmake using -DWITH_ATLAS=1 to get around OpenBLAS.

The best option here would be to build neither of them on OSX, as Apple ships the Accelerate framework that does have BLAS and LAPACK and is optimized for all CPUs used in Macs. Unfortunately scipy apparently does not fall back to Accelerate if neither OpenBLAS nor ATLAS is present.

Aside from that, Jens did get OpenBLAS to build by creating the symlinks to ar and ranlib. I was never able to build Atlas on my machine yet...

As for vigra, does it now require C++11?

It requires at least -std=c++0x, but that flag is not automatically set yet. Jens just noted down what he had to do to compile, I already talked to Uli what might be the best way to fix this. Unfortunately GCC 4.6 only supports the c++0x flag, whereas GCC 4.8 only supports c++11, so we need to add a check in our buildem cmake script that determines the GCC version number.

Finally, matplotlib, the freetype error is a bit interesting.

Not really, Qt has exactly the same behaviour. Creating the symlink once with cd $BUILDEM_DIR/include; ln -s freetype2/freetype fixes both. I think I even had to do that on some linux machines. We should add a script for that in both, matplotlib.cmake and qt4.cmake. @jakirkham it might be that you have a freetype installation in another path that matplotlib and Qt use when compiling on your machine.

[jakirkham]

Jens' GCC 4.8 comes from MacPorts, mine from Homebrew.

If it works with Homebrew, could he use that instead? Personally, I didn't trust either and installed a copy of the GNU compilers separately ( http://hpc.sourceforge.net ).

The best option here would be to build neither of them on OSX, as Apple ships the Accelerate framework that does have BLAS and LAPACK and is optimized for all CPUs used in Macs.

Using the Accelerate framework is a bit problematic. In particular, it has issues with forking ( ContinuumIO/anaconda-issues#133 ), ( numpy/numpy#4776 ) ( https://groups.google.com/forum/#!topic/comp.lang.python/b2Ayc8bQzMQ ) meaning the multiprocessing module can't be used. The last citation includes a response from Apple saying basically don't fork with Accelerate. This is not the only issue I have seen with Accelerate.

It requires at least -std=c++0x

Can't we use this flag with gcc 4.8 then as it is supported ( https://gcc.gnu.org/gcc-4.8/cxx0x_status.html )?

Just out of curiosity, which vigra commit are you using?

@stuarteberg is gcc 4.4 still required for some cases?

We should add a script for that in both, matplotlib.cmake and qt4.cmake.

Maybe just in freetype2.cmake, it would be best if there was a symlink command in CMake, but I don't know CMake well enough to say. Though, I am starting to have doubts as to whether this is really necessary. Looking into matplotlib, it appears that it will detect freetype2. Is it possible that there are two conflicting freetypes?

it might be that you have a freetype installation in another path that matplotlib and Qt use when compiling on your machine

I was wondering the same thing when I wrote it. Searching my computer, the only freetypes I find are in MacPorts, which I have not added to my path, and buried deep within some existing packages (i.e. Sage, Octave, etc.). None of those paths are exposed.

Further, using the Ubuntu VM ( https://github.com/ilastik/ilastik-test-vm ), which only has freetype2 installed, I had no problem going through the build of matplotlib or Qt, for that matter, without including any extra steps for freetype2. So, even if it linked against the non-BuildEM freetype2, it still would have had the same issue.

[JensNRAD]

I just wanted to document what problems occurred when compiling on a Mac 10.9 with Mac ports. That's why I didn't make a PR but instead posted the changes here. Some changes might be worth including in buildem, others can be made on the fly when people run into these problems.
Regarding freetype on Ubuntu 14.04: here I also had to set the symbolic links.

[jakirkham]

Understood. Perhaps, some of these changes do need to be integrated. I just want to make sure the set to be integrated is minimal so as not to break builds on other systems.

As for the Ubuntu case, I should have been more specific. Its version is 12.04 and I have added a link in my previous comment. This is a very barebones VM.

[jakirkham]

After looking into the freetype issue with matplotlib, it does appear to be a bug in matplotlib 1.3.1 ( matplotlib/matplotlib#3029 ). It's unclear when the bug emerged (at least after 1.1.1), but it is clear that a fix has been made and applied in 1.4.0 ( matplotlib/matplotlib@66ee42d ).

[chaubold]

@stuarteberg, we have look into this again next week, as buildem's master branch still does not build ilastik's conservation tracking dependencies...