HOWTO_build_a_LiveCD_with_Catalyst

It has been suggested that HOWTO build a LiveCD with Catalyst for newbies be merged into this article or section. (Discuss)

Forward to more recent article

This is poorly maintained now, try HOWTO build a LiveCD with Catalyst for newbies.

Introduction

This article will explain how you can build a Gentoo LiveCD with Catalyst. Catalyst is the tool used for building Gentoo releases. This includes stage tarballs, minimal CDs, package CDs, universal CDs, and the relatively new LiveCD with X and Gnome. This HOWTO will also give you hints, tips, and tricks to build your own version of a Gentoo Linux LiveCD. If this is your first time building a LiveCD using Catalyst, keep reading. Otherwise, advance to the end for customization help.

Keep in mind that if catalyst does not currently work the way you want it to, do not complain to the developers. Catalyst's primary purpose is Gentoo release building, and that's what it's designed to do. However, if you have ideas for enhancements, please file a bug on Gentoo's bugzilla with a patch.

What you need

1. A running Gentoo Linux system
2. The Catalyst utility
3. Knowledge
4. Time
5. This article

Some things to know about catalyst

1. The process of building a LiveCd using Catalyst needs certain steps that I'll show you in the next paragraph; to avoid encountering problems you have to do all the steps.
2. The steps I talked about in the previous point, are done by using a .spec file: these files tell catalyst what to do for that step. You'll use a certain .spec file for a certain step; this means that stage1.spec is different from livecd-stage1.spec, because the steps are different!
3. There are some bugs that change the behaviour you would expect from catalyst (read one example below), so keep your eyes open

Have a quick look at the syntax of each spec in the official documentation (may be sometimes outdated, but will give an idea)

http://www.gentoo.org/proj/en/releng/catalyst/index.xml

Overview of the process

In the table that will follow I'll show you each step of the process, and for each one, which sample .spec file to use as template and what we need to accomplish it. Remember that in any step, you'll need the output of the previous one ( of course not for the first one )

Install Catalyst

If you don't have Catalyst installed, install it:

# emerge catalyst

Creating your own .spec files

In the sample files that come along catalyst, you can find a full description for each part of the file; here are just some tips to avoid problems.

Portage snapshot

You can quickly and easily build a snapshot using the -s option to catalyst:

# catalyst -s nameofyoursnapshot

This will build a snapshot of your current /usr/portage at /var/tmp/catalyst/snapshots/portage-nameofyoursnapshot.tar.bz2. Alternatively, you can also fetch a snapshot from any mirror. Generating your own just allows you to easily make changes before the snapshot is created.

Stage 1, 2, and 3

no its not simple and straightforward this could be expanded for something so imperative for creating gentoo release iso's this deserves better attention to details

The spec files for stage 1, 2, and 3 are simple and straightforward. They cannot really be customized (adding/removing packages), since they would no longer fit the Gentoo definition of that particular stage. You will probably want to build your own set of stages to use as a seed for your LiveCD build. This will save you a lot of trouble down the line.

Livecd-stage1

You will run into problems if you add any ebuild which checks for a configured kernel to livecd/packages, since the kernel is not configured until livecd-stage2. Most ebuilds that fail for this reason don't really need to fail. The only packages which should fail if there is no configured kernel are ones that actually build against the kernel. If you encounter this, double-check that the package does not build against the kernel and then file a bug for the package in Gentoo's bugzilla.

Livecd-stage2

On this stage you will configure the kernel, emerge packages that require this kernel, build the iso, add an overlay, unmerge unnecessary packages etc. The documentation of this stage is quite self explanatory.

Second Step

What else do you need for building a LiveCD with Catalyst? You need some sample "spec" files for building e.g. the default Gentoo Linux LiveCDs. You will find them e.g. on the gentoo.org web server.

Getting the sample spec files

You can find the spec files used to create each release for each architecture in CVS.

Direct Links to the 2006.0 LiveCD specs:

1. livecd-stage1 Specfile Template
2. livecd-stage2 Specfile Template

LiveCD spec files are also available in the dev-util/livecd-specs package. This will install the spec files in /usr/share/livecd-specs-<version>.

You can also use the sample files that come with the catalyst install; You can find a sample for this .spec file in the folder:

/usr/share/doc/catalyst-2.0/examples/

Using our sample spec files

If you want to have a look directly in our used specfile templates for our example check this out:

## livecd-stage1 template for Gentoo Linux release bootable LiveCDs
## John Davis 
## Modified by: Stephan Hermann <sh@sourcecode.de>
## Modified by: Zac Slade <krakrjak@gmail.com>
## Modified to make compatible with newer portage:
##     Now using 2004.3 profile
## Added -opengl and changed gtk to -gtk
## Removed xorg firefox and xfce4 to prevent X from building
## Barrier for entry should now be lower to create a CD that works
subarch: x86
version_stamp: 2004.3
target: livecd-stage1
rel_type: default
profile: default-linux/x86/2004.3
snapshot: 2004.3
source_subpath: stage3-x86-2004.3
livecd/use:
        -X
        -gtk
        -opengl
        -svga
        ipv6
        socks5
        livecd
        fbcon
        ssl

livecd/packages:
        baselayout
        livecd-tools
        ucl
        kudzu-knoppix
        module-init-tools
        hotplug
        irssi
        aumix
        metalog
        pciutils
        parted
        mt-st
        links
        star
        strace
        raidtools
        nfs-utils
        jfsutils
        usbutils
        speedtouch
        xfsprogs
        xfsdump
        e2fsprogs
        reiserfsprogs
        hdparm
        nano
        less
        openssh
        dhcpcd
        mingetty
        pwgen
        popt
        dialog
        rp-pppoe
        gpm
        screen
        mirrorselect
        penggy
        iputils
        hwdata-knoppix
        hwsetup
        bootsplash
        device-mapper
        lvm2
        evms
        vim
        gpart
        pwgen
        pptpclient
        mdadm
        tcptraceroute
        netcat
        ethtool
        wireless-tools
        tcpdump
        nmap
        libpcap
        sniffit

This is the stage1 specfile template for our example. I provide you with the second stage later on. Save it e.g. as "livecd_gentoo_wiki_howto-stage1.spec"

Don't add "genkernel" to "livecd/packages". This will cause an error during stage1 catalyst run!

Take care about special entries in this specfile

In this example spec file you will see some entries like

snapshot: 2004.3

and

source_subpath: stage3-x86-2004.3

Those entries are pointing to special files. We will produce these files right now.

snapshot: 2004.3

To create this file, do the following steps:

root@somewhere:~/> emerge --sync
root@somewhere:~/> catalyst -s 2004.3

This will create the file /var/tmp/catalyst/snapshots/snapshot-2004.3.tar.bz2. This creates the portage snapshot to enable portage during stage1. You could instead download a snapshot here. This is REQUIRED!

source_subpath: stage3-x86-2004.3

This file we don't have to create. What you need to do is to fetch this file directly from one of your friendly Gentoo Linux Mirrors.

once you have this file, do this :

mkdir /var/tmp/catalyst/builds
root@somewhere:~/> mv stage3-x86-2004.3.tar.bz2 /var/tmp/catalyst/builds

This is also REQUIRED!

Third Step

Ok, now we have everything for a "livecd-stage1" build. It's time to actually attempt to build it.

Building livecd-stage1

Enter these commands:

catalyst -f livecd_gentoo_wiki_howto-stage1.spec

This process can take anywhere from 20-30 minutes plus. You might want to find alternate entertainment than watching your screen. You should however check it every 10-15 minutes to see if it encountered an error. If this step is successful you will have a directory /var/tmp/catalyst/tmp/default/livecd-stage1-2004.3 and you should see a success message from catalyst.

Some problems

1. If you set the portage configuration dir: package.use doesn't work how you think it should in catalyst. It's a known bug, but one that won't likely be fixed for some time. Basically, if you use $target/use, then it will ignore package.use settings. You should only use package.use when deviating from the profile defaults for a package, not when setting your own USE.}}

1. Likely, due to the previous bug, you will run into problems when emerging. It's hard to emerge so many packages without errors, so your options are either write a portage snapshot or use package.use as long as you don't set "use" in the spec, etc. This is what the developers do when building a release (at least the modification of ebuilds). Remember that if you build the packages and keep the snapshot unchanged you only need to compile a package once ;). There are some uglier hacks (more about that to add later).

Building livecd-stage2

Currently the livecd-stage2 specfile template has the most documentation available on catalysts' livecd-stage2 target building.

This is the stage where most of your customizations will happen.

So I will describe some of the settings available that you may want to consider which are otherwise not very well explained in the spec template.

• livecd/fstype:<looptype>
  • Type of loop image to put the filesystem in. (will need proper kernel settings for some)
  1. zisofs -- transparent compressed filesystem. Only transparent on linux.
  2. normal -- standard iso filesystem no compression.
  3. noloop
  4. squashfs -- creates the loopback file system as a squashfs. Modern livecds use this.
     • note: you must have squashfs kernel support for this to work. And you should have squashfs-tools merged on your system and in your stage1.
  5. gcloop
• livecd/archscript:<scriptname>
  • architecture dependent script to use
  1. x86-archscript.sh
  2. sparc64-archscript.sh
  3. sparc-archscript.sh
  4. ppc-archscript.sh
  5. hppa-archscript.sh
  6. alpha-archscript.sh
• livecd/cdtar:<name.tar.bz2>
  • cd boot scripts to use. The isolinux images are the only ones that work on x86. (located at /usr/lib/catalyst/livecd/cdtar/)
  1. aboot-0.9-r1-cdtar.tar.bz2
  2. isolinux-2.08-cdtar.tar.bz2
  3. isolinux-2.08-memtest86+-cdtar.tar.bz2
  4. isolinux-2.08-memtest86-cdtar.tar.bz2
  5. palo-1.2_pre20030630-cdtar.tar.bz2
  6. silo-1.3.2-sparc64-cdtar.tar.bz2
  7. silo-1.4.4-sparc32-cdtar.tar.bz2
  8. yaboot-1.3.11-cdtar.tar.bz2
• livecd/rcadd: service|runlevel
  • Services to add to specified runlevel. Warning: the syntax has changed in catalyst 1.1.10, it was service:runlevel.
• livecd/rcdel: service|runlevel
  • Services to remove from specified runlevel. Warning: the syntax has changed in catalyst 1.1.10, it was service:runlevel.
• livecd/gk_mainargs:<genkernel options>
  • Useful but not required options to pass to genkernel for all builds:
  1. --makeopts=-jX where X is 1+<number of CPUs> like -j3 on a dual processor machine
  2. --udev adds udev support for the initrd image
  3. --lvm2 adds lvm2 support to the initrd image (if you use lvm for disks)
     • Try man genkernel for a complete list of options
• livecd/type:<type>
  • Changes ##GREETING in /usr/lib/catalyst/livecd/files/generic-motd.txt
  1. gentoo-release-universal
     • Welcome to the Gentoo Linux Universal Installation LiveCD!
  2. gentoo-release-minimal
     • Welcome to the Gentoo Linux Minimal Installation LiveCD!
  3. gentoo-gamecd
     • Welcome to the Gentoo Linux <GAMENAME> GameCD!

The minimal required config options are:

• subarch
• version_stamp
• target
• rel_type
• profile
• snapshot
• source_subpath -- for using your stage1 you built set this to default/livecd-stage1-x86-2004.3
• boot/kernel
• boot/kernel/[boot/kernel:x]/sources
• boot/kernel/[boot/kernel:x]/config
• livecd/cdfstype
• livecd/runscript
• livecd/archscript

Once we have saved livecd_gentoo_wiki_howto-stage2.spec we can TRY the following:

catalyst -f livecd_gentoo_wiki_howto-stage2.spec

Download the kernel configs, specs, scripts and all the scripts used in the livecd's for reference. http://sources.gentoo.org/viewcvs.py/gentoo/src/releng/

To Be Continued

Possible Problem

Catalyst may fail because of a pam-login and shadow conflict. Removal instructions were given here: pam-login removal instructions

This worked for me by doing the following (Please change for your directory structure and filenames):

mkdir /root/livecd/stage3
/bin/tar xjpf /var/tmp/catalyst/builds/stage3-i686-2006.0.tar.bz2 -C /root/livecd/stage3
mkdir /root/livecd/stage3/usr/portage
mount -o bind /usr/portage /root/livecd/stage3/usr/portage
chroot /root/livecd/stage3 /bin/bash
# not sure if the following is needed
env-update && source /etc/profile
emerge --unmerge pam-login
exit
umount /root/livecd/stage3/usr/portage
cd /root/livecd/stage3
/bin/tar cjpf /var/tmp/catalyst/builds/stage3-i686-2006.0.tar.bz2 .

The following should not be done under any circumstances. For nearly the entire catalyst development, it has supported the "-v" or "verbose" option. This option enables "--tree" in the build, as well as a way to pause the build to allow for further investigation.

Thanks,

--wolf31o2

If such blockers occur, you might want to use the following hack to get more info:

Edit

/usr/lib/catalyst/targets/livecd-stage1/livecd-stage1-chroot.sh

and find your appropriate lst_emergeopts statement. You might want to change it to something like:

export clst_emergeopts="--usepkg --buildpkg --newuse --pretend --tree"

and rerun catalyst. You should get more info on blockers etc then.

Question and answers