Notes for how to setup a 64-bit Mac with important Python development tools

Created by Trond Kristiansen on 27.7.2009, and last updated 05.0.2.2012
Updated and modified after installation of Snow Leopard on 05/11/2009
Re-written for publication on www.trondkristiansen.com: 06.02.2010, updated after comment 10.4.2010
Updated completely and edited to work for Python v2.6.5 and Numpy v1.4 02.06.2010.
This page was updated again on 05.02.2012 after updating my installation of Intel Fortran to the XE composer (v12). A major step of the installation was to update the HDF5 and NetCDF4 , which caused several problems to arise.

This page contains my own notes for how to install a variety of packages that I use on a regular basis on Intel Mac computers. This particular setup described below is the latest version of these notes and was written when I wanted to re-install a number of packages as 64-bit on my new Mac with Snow Leopard. My particular interest is to have a solid python installation in 64-bit with a functional numpy, matplotlib, netcdf4, f2py, Intel fortran compiler, and hdf5. It took me a long time to get this to work correctly and I always seem to forget how I did this the previous time I had to install all these packages so I decided to write up everything I did. I hope other people will find these notes useful. Comments, hints, suggestion, and improvements on the 20 steps of installation procedures are very welcome.

A few hints/comments before start:

  • Some of the suggestions found on this page was originally found here
  • One critical thing to remember when going through these steps and you install using “python setup.py build …” is to always go into the build directory and delete everything that is there. If you don’t do that, you may involuntarily use old compiled files that was compiled with an old configuration setup.
  • Enable 64 bit kernel on your Snow Leopard following this tip can boost performance: 
    sudo systemsetup -setkernelbootarchitecture x86_64

    .

  • In some cases I transfer files from one computer to another using email and in some cases this has led to a “DOSified” version of the file. A very useful tool for making sure that the line end of critical files such as .bash_profile are correct is to use the dos2unix online tool found here. Or alternatively, use the command line option: tr ‘r’ ‘n’ < file-with-dos-newline.txt > file-with-mac-newline.txt

STEP 1: Important flags you have to set before you start compiling

export CFLAGS="-Os -no-cpp-precomp -mno-fused-madd -arch x86_64 -fno-common -fPIC -DHAVE_NETCDF"
export CXXFLAGS="-arch x86_64 -Os -pipe"
export FFLAGS="-m64"
export FCLAGS="-m64"
export MACOSX_DEPLOYMENT_TARGET=10.6
export LDFLAGS=""

STEP 2: Install/update your local installation of subversion
Download subversion for Mac (Universal build for 32 and 64 bit Mac) http://www.open.collab.net/downloads/apple/index.html. The download requires you to register at Collabnet before you can download. This package will install to

/opt/subversion/bin

so after installation add the following to your .bash_profile file

export PATH=/opt/subversion/bin:$PATH

STEP 3: Install LibPNG

curl ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-1.5.8.tar.gz -o libpng.tgz
./configure --prefix=/usr/local
make check
sudo make install

STEP 3 alternative: Install LibPNG
When I wanted to use version 1.4.2 I had to manually download the file from here, before I installed it with

./configure --prefix=/usr/local
make check
sudo make install

STEP 4: Install ZLIB
Zlib is used for compression/decompression in reading/writing netcdf4 files. The package can be downloaded here and installs by default to /usr/local.

./configure
make
make check
sudo make install

After installation you want to add this to your .profile (or .bash_profile) file:

if [ -z "${ZLIBDIR}" ]
then
ZLIBDIR="/usr/local"; export ZLIBDIR
else
ZLIBDIR="/usr/local:${ZLIBDIR}"; export ZLIBDIR
fi
export LIB=/usr/local/lib:$LIB
export INCLUDE=/usr/local/include:$INCLUDE

STEP 5: Install Python 64-bit

curl -O http://python.org/ftp/python/2.6.5/Python-2.6.5.tar.bz2
bzip2 -d Python-2.6.5.tar.bz2
tar xf Python-2.6.5.tar
cd Python-2.6.5
unset DYLD_LIBRARY_PATH
unset LD_LIBRARY_PATH
./configure --enable-framework=/Library/Frameworks \
MACOSX_DEPLOYMENT_TARGET=10.6 \
--with-universal-archs=intel \
--with-readline-dir=/usr/local/lib \
--with-zlib=/usr/local/include \
--enable-toolbox-glue \
--enable-universalsdk=/ MACOSX_DEPLOYMENT_TARGET=10.6
make
make test
sudo make install

Next, I made a copy/backup of the original Python installation (Python 2.6.1) by running:

file sudo mv /usr/bin/python /usr/bin/python-old

Then I relinked the new 64-bit Python 2.6.5:

 sudo ln -s /Library/Frameworks/Python.framework/Versions/2.6/bin/python-64 /usr/bin/python

Test to check that you get correct 64-bit versionof the Python binary

file /usr/bin/python

And you should see something like this:

/usr/bin/python: Mach-O 64-bit executable x86_64

And you should see this when you type in “python” in the terminal window:

Python 2.6.5 (r265:79063, Jun 2 2010, 14:46:12)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt;

STEP 6: Update your easy_install installation
Here we install easy_install for python 2.6.5 on a computer that previously had python version 2.5. The current easy_install is linked to python 2.5 and will install packages to site-packages/2.5 which is not what we want. Therefore we install easy_install-2.6. After the installation we will have both easy_install-2.5 and easy_install-2.6 available, so use the one that corresponds to the python version you use.

Also, see this link and this link for more details on these topics.
The installation of easy_install can be done in two ways 1) get the source codehere so you can build against the 64-bit python:

sudo python setup.py build
sudo python setup.py install

Or 2) get the egg from this link and install using

sudo sh setuptools-0.6c11-py2.6.egg

STEP 7: Install readline with support for X86_64

sudo easy_install-2.6 readline

STEP 8: install numpy for 64-bit
Since python was installed as a universal architecture (ppc,i386,x86_64), but I only wanted numpy as 64-bit, I needed to edit the python makefile. I removed all the references in python Makefile to other archs except x86_64. So I edited the file (see two options depending on Leopard or Snow Leopard installation)

/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/config/Makefile (Leopard)
/System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/config/Makefile (Snow Leopard)

and kept only the occurrence of “-arch x86_64″ and removed the other ones. Then I saved the file and build numpy (get numpy here) using the commands:

sudo python setup.py build
sudo python setup.py install

A note of warning: There is a problem using ifort and icc from the command-line under Snow Leopard. If you get an error message of the form
ld: in /opt/intel/fce/10.1.006/lib/libimf.a(libm_reduce_pio2.o), ObjectFileAddressSpace::mappedAddress(0xFFFFFFFFFFFFFFFC) not in any section at the end of the compilation process you’ve hit a known bug in the Intel suite. This problem is solved by adding the flag -shared-intel into the disutils file “/Library/Python/2.6/site-packages/numpy/distutils/fcompiler/intel.py” (relative path differs with installation of python 2.5 and 2.6).

Since I wanted to compile Fortran programs in 64-bit mode using the Intel Fortran Compiler (ifort), I edited the file “/Library/Python/2.6/site-packages/numpy/distutils/fcompiler/intel.py”. In the file, I found the section that contained specifications for my compiler which was “class IntelEM64TFCompiler(IntelFCompiler):”, and then I added “-shared-intel” in the “linker_so” options so that it looked like this:

class IntelEM64TFCompiler(IntelFCompiler):
compiler_type = 'intelem'
compiler_aliases = ()
description = 'Intel Fortran Compiler for EM64T-based apps'
version_match = intel_version_match('EM64T-based|Intel\\(R\\) 64')
possible_executables = ['ifort', 'efort', 'efc']
executables = {
'version_cmd' : None,
'compiler_f77' : [None, "-FI", "-w90", "-w95"],
'compiler_fix' : [None, "-FI"],
'compiler_f90' : [None],
'linker_so' : ['',"-O0","-shared-intel","-bundle",'-dynamiclib', '-Wl,-undefined,dynamic_lookup', '-Wl,-framework,Python'],
'archiver' : ["ar", "-cr"],
'ranlib' : ["ranlib"]
}
....

Update 21.04.2010: This problem disappears if you upgrade to the latest Intel Fortran compiler version 11.1 according to L. Pagani.

Update 05.02.2012: This problem is gone with the Intel XE composer.

After a successful installation of Numpy I added these lines to my .bash_profile script:

export PYTHONPATH=/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages:$PYTHONPATH
export INCLUDE=/Library/Frameworks/Python.framework/Versions/2.6/include:$INCLUDE
export LIB=/Library/Frameworks/Python.framework/Versions/2.6/lib:$LIB

STEP 9: Howto get F2PY to work for 64-bit python
To get f2py to compile together with 64-bit fortran Intel compiler, you have to edit the following file in disutils (notice two options, one for Leopard and one for Snow Leopard):

sudo chmod a+r+w+x /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/distutils/fcompiler/intel.py (Snow Leopard)

Edit the line that contains “linker_so” for both intel (32-bit) and intelem (64-bit Fortran compilers) so that it looks like this:

'linker_so' : ['',"-O0","-shared-intel","-bundle",'-dynamiclib', '-Wl,-undefined,dynamic_lookup', '-Wl,-framework,Python'],

Then you can for example compile Fortran modules using Intel Fortran compiler with the following option in f2py:

f2py --verbose --fcompiler=intelem -c -m barotropic barotropic.f90

The compiled module can be imported to python and called using import barotropic

Step 10: Install UDUNITS
Udunits is used by netcdf4 an hdf5 which will be installed later.
cd into the src directory:

./configure
make
make test
sudo make install

after installing UDUNITS add this to your .profile:

export INCLUDE=/Users/trond/GMT/udunits-1.12.7/include:$INCLUDE
export UDUNITS_INC=/Users/trond/GMT/udunits-1.12.7/include
export UDUNITS_LIB=/Users/trond/GMT/udunits-1.12.7/lib
export UDUNITS_PATH=/Users/trond/GMT/udunits-1.12.7

Step 11: Install IPython and the dependencies IPython relies on
Download source code from: http://ipython.scipy.org/dist/ or use curl below (see further down). IPython requires a number of other programs to function optimal:
1. pygments (syntax highlighter) which you can install using: sudo easy_install-2.6 Pygments
2. Next, to create documentation you should install sphinx: sudo easy_install-2.6 sphinx
3. Parallel computing with foolscap: sudo easy_install-2.6 Foolscap
4. Nose required for running test suite: sudo easy_install-2.6 nose
5. Pexpect required for running standalone doctest: sudo easy_install-2.6 pexpect

Next complete the installation by installing Ipython from scratch

curl -O http://ipython.scipy.org/dist/ipython-0.10.tar.gz
tar xf ipython-0.10.tar.gz
cd ipython-0.10
python setup.py build
sudo python setup.py install

An alternative to compiling the sources for IPython is to use easy_install:

sudo easy_install-2.6 ipython

STEP 12: Install Freetype2
Get the package here: http://mirror.its.uidaho.edu/pub/savannah/freetype/

./configure --prefix=/usr/local --prefix=/usr/local/ --target=x86_64-apple-darwin -build=x86_64-apple-darwin --host=x86_64-apple-darwin
make
sudo make install

STEP 13: Install the PIL (Imaging library)
To be able to for example to use the blue marble background image in matplotlib you have to install PIL (Imaging library). Download the package from: http://www.pythonware.com/products/pil/ and install by issuing the following commands:

sudo python setup.py build
sudo python setup.py install

STEP 14: Install PKG-CONFIG
Main website :http://pkg-config.freedesktop.org/wiki/
Download the package from here: http://pkgconfig.freedesktop.org/releases/

 ./configure --prefix=/usr/local
make
sudo make install

Step 15: Install Matplotlib
Matplotlib requires a number of libraries to be installed prior to installation. Howto install these libraries have been described in the previous steps, however, the paths to these ibraries (libpng, pkg-config, and freetype2) have to be set below in the make.osx file that you find in the root of the matplotlib source directory.

You can download Matplotlib from here: http://matplotlib.sourceforge.net/users/installing.html or grab the latest source from the subversion trunk:

svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/trunk/matplotlib matplotlib
cd matplotlib

The matplotlib package was installed using the suggestion in README.osx

sudo make -f make.osx PREFIX=/Users/trond/GMT/matplotlib PYVERSION=2.7 fetch deps mpl_install_std

STEP 16: Install Basemap for Matplotlib
Basemap is another very useful module for python created by Jeffrey Whitaker for making maps in matplotlib. Prior to the installation of Basemap you need to compile and install the GEOS package (I could only get version 2.3.3 to work and had no success with version 3.1). Download the GEOS package from here : http://trac.osgeo.org/geos/

Compile and install:

./configure --prefix=/usr/local
make
sudo make install

Add this line to .bash_profile

export GEOS_DIR="/usr/local"

Download the Basemap source code from here.

When your download has completed, cd to root level of basemap source directory and install Basemap using :

sudo python setup.py build
sudo python setup.py install

Finally, you can test the success of the installation by trying to import the Basemap module in python:

from mpl_toolkits.basemap import Basemap

STEP 17: Install HDF5
Download HDF5 here: http://www.hdfgroup.org/HDF5/release/obtain5.html

I compiled HDF5 (version HDF5 version 1.8.8) in static mode for use with Fortran (--enable-fortran) and netcdf4. If you want to compile HDF5 as a shared library you can not create Fortran libraries and have to use the option --disable-fortran.

./configure --with-zlib=/usr/local --enable-fortran --prefix=/usr/local/--target=x86_64-apple-darwin -build=x86_64-apple-darwin --host=x86_64-apple-darwin --enable-shared --disable-production
make
make check
sudo make install

Next add these lines to your .profile so that when we install netcdf4 HDF5 will be visible to the configurator:

export HDF5_LIB_DIR="/usr/local/lib"
If [ -z "${LD_LIBRARY_PATH}" ]
then
LD_LIBRARY_PATH="/usr/local/lib"; export LD_LIBRARY_PATH
else
LD_LIBRARY_PATH="/usr/local/lib:${LD_LIBRARY_PATH}"; export LD_LIBRARY_PATH
fi
if [ -z "${LIBDIR}" ]
then
LIBDIR="/usr/local/lib"; export LIBDIR
else
LIBDIR="/usr/local/lib:${LIBDIR}"; export LIBDIR
fi
export INCLUDE=/usr/local/lib:$Include
export PATH=/usr/local/bin:$PATH
export HDF5_LIB_DIR=/usr/local/lib
export HDF5DIR=/usr/local
export HDF5_DIR=/usr/local
export CPPFLAGS="-I$HDF5_DIR/include -I/usr/local/include"
export LDFLAGS="-L$HDF5_DIR/lib -L/usr/local/lib"

STEP 18: Install MacTeX
Download and install (DMG) the latest version of LaTeX from here. A working installation of LaTeX is required to create documemnts and man pages for NetCDF. When installation is complete, add this line to your .bash_profile script to make all the binaries available in PATH.

export PATH=/usr/texbin:$PATH

STEP 19: Install NETCDF
NetCDF source files can be downloaded from here. As of the writing of this, latest stable version is v4.1.3. This version was particularily difficult to get to compile in 64 bit mode (at least for me) and it took me several hours to make sure that both HDF 1.8.8 was compiled as 64 bit and that it could be linked to the comilation of the netCDF4. I had to set several environmental variables to get this to work, including setting the GNU GCC compileer to version 4.6 (as described here):

export CC=/usr/local/bin/gcc
export CXX=/usr/local/bin/g++
export FC=ifort
export F90=ifort
export F77=ifort
export FCFLAGS=-m64
export FFLAGS=-m64

Next, I still had problems with teh compilation until I found a tip at the Unidata website that suggested a solution to my problem. The trick to get NetCDF4+HDF5+Intel XE composer to work together was a simple statement:

 lt_cv_ld_force_load=no

. The following commands enabled me to compile and load NetCDF4 using Intel Fortran X:

FC=ifort lt_cv_ld_force_load=no CXX="" CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" ./configure --enable-f90 --prefix=/usr/local/ --target=x86_64-apple-darwin -build=x86_64-apple-darwin --host=x86_64-apple–darwin --enable-netcdf4 --enable-shared
make
make check
make install

After installation of netCDF4 add this to your .profile:

export NETCDFHOME=/usr/local
export NETCDF_PREFIX=/usr/local
export NETCDF_LIB=/usr/local/lib
export NETCDF4_ROOT=/usr/local
export NETCDF4_DIR=/usr/local
export NETCDF_INC=/usr/local/include
export PATH=/usr/local/bin:$PATH

STEP 20: Install NetCDF for Python
Jeffrey Whitaker has written an extremely useful module for interfacing python with netCDF4 called netcdf4-python. The module can be downloaded from here http://code.google.com/p/netcdf4-python/. After updating NetCDF to version 4.1.3 , HDF5 1.8.8, and netCDF4-Python to 0.9.9, I no longer had to manually edit the setup.py file to point to HDF5 and NC dirs.

I was able to install the package (version netCDF4-0.9.9) using these commands:

python setup.py build
python setup.py install
cd test; python run_all.py

STEP 21: Install H5PY (HDF5 Python support)
Se here for a link to h5py
Install cython

sudo easy_install cython

Then grab the latest source code of h5py :

svn checkout http://h5py.googlecode.com/svn/trunk/ h5py-read-only

Next, cd into h5py working directory (h5py-read-only) and run the following commands:

export HDF5_API=18
python setup.py cython
python setup.py configure --api=18 --hdf5=/usr/local/SharedButNoFortranHDF5
python setup.py build
sudo python setup.py install

If you update to a newer revision or have modified h5py, be sure to run the
following to prevent the use of stale files:

python setup.py clean
python setup.py cython --clean

STEP 22: Install/update MySQL
Download MySQL from MySQL downloads website. I also
installed a separate MySQL Startup Item package to enable the automatic startup of MySQL on system startup. The Startup Item starts the server by invoking mysql.server
and you can see this option as part of

/etc/hostconfig

. See this link for more details. I added the installation to my .bash_profile script:

export PATH="/usr/local/mysql/bin":$PATH

Next I started up MySQL running:

sudo /Library/StartupItems/MySQLCOM/MySQLCOM start

I then needed to set the root passwords for my MySQL installation (for details see here):

shell&gt;; mysql -u root
mysql&gt;; SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newpwd');
mysql&gt;; SET PASSWORD FOR 'root'@'127.0.0.1' = PASSWORD('newpwd');

Next I edited the rights for reading and writing to the data directory (/usr/local/mysql/data) by issuing these commands:

sudo chown -R mysql /usr/local/mysql/data
sudo chgrp -R mysql /usr/local/mysql/data

Finally, one of the reasons I use all of these packages is because of my interest in ecosystem modeling. I have a few projects related to ecosystem modeling where I combine Python and Fortran tools. These projects are available on my SVN website: http://www.assembla.com/profile/trond. My major project is my soda2roms toolbox that enables automatic generation of forcing files for the Regional Ocean Modeling System (ROMS). This toolbox is written using a combination of Python, and Fortran programming languages and compiles using f2py. This project could benefit tremendously if more people would be involved and help development. So please email me if you are interested in developing and improving the toolbox. One of the major goals is to expand it so that it can be used with a number of forcing files such as HYCOM or ROMS input data. Currently the toolbox supports SODA data as input (more info).