HOWTO_Setup_An_ICECREAM_Compile_Cluster

HOWTO Setup an Icecream Compile Cluster

This article shows how to set up an Icecream compile cluster with two or more machines. Icecream was created by SUSE and is based on ideas and code by distcc. Like distcc it takes compile jobs from your build and distributes it to remote machines allowing a parallel build on several machines you've got. But unlike distcc, Icecream uses a central server that schedules the compile jobs to the fastest free server and therefore it is dynamic. This advantage pays off mostly for shared computers, if you're the only user on x machines, you have full control over them anyway. Unlike distcc, Icecream does not require the same versions of gcc on all machines.

Note

After some research, this tutorial was found. http://www.mockies.de/main.php?seite=crosscompiling&lang=en It may help in the production of this tutorial at a later time because this is still a stub. Hope it helps.

Requirements

This setup requires two or more linux hosts, and they do not all need to be running Gentoo. If you use a different distro on the other machines, make sure to install the same version of Icecream on all machines, as the protocol may change between versions. The use of ccache is optional and Icecream plays nicely with ccache. If the compile job is not in the local cache already, it gets sent to the compile cluster.

Setting up

As of May 2007, Icecream is still masked in portage, unmask the latest version by adding the following line to /etc/portage/package.keywords

sys-devel/icecream   ~x86

and emerge icecream

[user@system ~] emerge sys-devel/icecream

Install the same version of Icecream on the other machines too.

Decide which computer will run the scheduler, this will be the machine that decides where to send the compile job. In general I run the scheduler on my local machine, that way even if the other nodes are not available (it's a laptop, and I don't bring my compile cluster around in my bag with me ;-), it still sees my local machine and sends the compile jobs there. The overhead added to compiling with only one node is minimal, and Icecream has been designed with this situation in mind.

Edit /etc/conf.d/icecream to suit your tastes, it has descriptions of the options available. All the machines in the cluster will run the Icecream daemon (which sits and waits for compile jobs), but only one should run the scheduler. If you want to run the scheduler set

ICECREAM_RUN_SCHEDULER="yes"

The ICECREAM_MAX_JOBS option should be set per-machine, depending on how many compile jobs that particular machine can process in parallel (think of 'make -j X', where X describes the number of jobs to compile at one time). Usually this is twice the number of (virtual) CPUS plus one.

ICECREAM_MAX_JOBS="3"

Next, add Icecream to the default runlevel

[user@system ~] rc-update add icecream default

and start the service:

[user@system ~] /etc/init.d/icecream start

Now start Icecream on all the other nodes. Your compile cluster is now set up.

To use icecream with normal make add it to the beginning of your PATH (e.g. in /etc/profile)

PATH=/usr/lib/icecc/bin:$PATH

To see if the compile cluster is working you can install the icecream-monitor, and see all the nodes on your network, and the number of compile jobs each one is capable of. The icecream-monitor will have been installed if you enable the "kde" USE keyword for the icecream build as it is a KDE program. The monitor has a number of different views, and will also show how many jobs have been processed on each node, and show which machines are currently compiling which jobs. Very nice eyecandy!! :-)

NOTE: The current stable release (0.7.14) of icecream no longer includes the monitor by default. There is an ebuild available here: http://bugs.gentoo.org/show_bug.cgi?id=139432

Icecream and portage

To use icecream with portage add the following line to /etc/make.conf

PREROOTPATH=/usr/lib/icecc/bin

Also, increase MAKEOPTS in /etc/make.conf to equal the total sum of compile jobs that your cluster can handle now, e.g.

MAKEOPTS="-j9"

Icecream and ccache

To use icecream with ccache, the ccache PATH should come first

PATH=/usr/lib/ccache/bin:/usr/lib/icecc/bin:$PATH

(Alternatively Andi Kleen provided the following code on LKML to achieve the same effect without worrying about the PATH -> ftp://ftp.firstfloor.org/pub/ak/smallsrc/icecache.c)

Icecream and firewalls

These are the ports you will need to open in your firewalls to allow Icecream to function correctly:

TCP/10245 on the daemon computers (required)
TCP/8765 for the scheduler computer (required)
TCP/8766 for the telnet interface to the scheduler (optional)
UDP/8765 for broadcast to find the scheduler (optional)

Icecream and cross-compiling

If you want to use hosts with different architectures together using icecream, you need to setup a cross-compiler. In Gentoo, this is easy. On the helper machines with different arches, run:

emerge crossdev
crossdev --target <arch>-<vendor>-<os>-<libc> --binutils <version> --gcc <version> --kernel <version> --libc <version>

where kernel refers to the version of linux-headers. An example for the target argument would be x86_64-pc-linux-gnu. To merge the latest version of a package, you can omit the corresponding parameter.

NOTE: If you get an error compiling cross-gcc, comment out the line: PREROOTPATH=/usr/lib/icecc/bin that you added to make.conf earlier in this howto.

Now icecream must explicitly call the full name of the compiler on the client machine running emerge, like x86_64-pc-linux-gnu-gcc, instead of just gcc, or else the cross compiler wouldn't become invoked on the icecream hosts. To achieve this, you just have to create a wrapper script for the short version of the compiler binaries:

cd /usr/lib/icecc/bin
vim x86_64-pc-linux-gnu-wrapper

#!/bin/bash
exec /usr/lib/icecc/bin/x86_64-pc-linux-gnu-g${0:$[-2]} "$@"

Now the short versions are simply linked to the wrapper script:

chmod +x x86_64-pc-linux-gnu-wrapper
for file in cc gcc c++ g++; do
  rm $file
  ln -s x86_64-pc-linux-gnu-wrapper $file
done

That's about it. This has to be applied to every machine that is running emerge by itself, not the helpers. If you emerge on every host in your network, you certainly have to setup all of them. This trick has been stolen from the Gentoo distcc cross-compiling howto.

Caveats

N.B. The following tip on genkernel is not needed from at least 3.4.10 onwards, as $PATH is added first in the genkernel script now.

genkernel

Icecream does not yet work with genkernel. To make it work, you need to edit /usr/bin/genkernel and add the Icecream path to the PATH. This is on the fourth line of /usr/bin/genkernel. You can also add ccache, as I have done below:

PATH="/usr/lib/ccache/bin:/usr/lib/icecc/bin:/bin:/usr/bin:/sbin:/usr/sbin"

Also, when running genkernel, you will need to specify the makeopts on the command line, e.g:

genkernel --oldconfig --install --makeopts=-j9 all

If using icecream to compile KDE, disable the kdeenablefinal USE flag. This flag converts all the source files in a given subdirectory into one big source file, and then tries to compile it. You lose a lot of the advantages of distributed compiling if there is only one big source file.

gcc / binutils / glibc

The icecream page warns not to build these with architecture-specific CFLAGS as it ruins multi-architecture icecream compile clusters. Set up the following dir structure/links to tell portage to compile gcc / binutils / glibc with a generic set of CFLAGS:

$ ls -lR /etc/portage/env/
/etc/portage/env/:
total 12
-rw-r--r-- 1 root root   52 2008-05-14 01:36 i686-cflags
drwxr-xr-x 2 root root 4096 2008-05-14 01:37 sys-devel
drwxr-xr-x 2 root root 4096 2008-05-14 01:37 sys-libs

/etc/portage/env/sys-devel:
total 0
lrwxrwxrwx 1 root root 14 2008-05-14 01:37 binutils -> ../i686-cflags
lrwxrwxrwx 1 root root 14 2008-05-14 01:37 gcc -> ../i686-cflags

/etc/portage/env/sys-libs:
total 0
lrwxrwxrwx 1 root root 14 2008-05-14 01:37 glibc -> ../i686-cflags

where the contents of i686-cflags is

CFLAGS="-march=i686 -O2 -pipe"
CXXFLAGS="${CFLAGS}"

Sources

Icecream homepage
Gentoo DistCC Cross-compiling Guide