HOWTO_diskless_in_a_heterogeneous_environment

Diskless in a Heterogeneous Environment

How does this differ from Regular Diskless

This HOWTO describes how to set up diskless in a heterogeneous environment. This means an environment with many different kinds of hardware, and different configurations. The Regular Diskless HOWTO is aimed at a single client configuration, and does not attempt hardware detection or dynamic configurations. (though this HOWTO is based off the Regular Diskless HOWTO, and as such will contain some very similar information).

To do this, we can set up an image with hardware detection, and dynamic configuration. The configuration is based on the hostname or IP address.

Notes

• This tutorial is based on a package of scripts that may have bugs, unimplemented and/or incomplete features or may not work in your setup at all.
• This HOWTO may have missing and/or incomplete information. If you notice something missing, feel free to contact the original author, or click "edit".
• You will need to prepare a chroot image to use as a base for creating the diskless image. Creating chroot images is out of the scope of this document.
• This HOWTO is based on a university environment, but may apply to other environments as well.
• This system is set up to be as flexible as possible, and the scripts are designed into Gentoo's init system, without needing any modification to existing scripts.
• This setup is considerably more complicated than the HOWTO network boot without write access on server configuration, and as such, it will require quite a bit more work to get working, and to maintain.
• This HOWTO uses 192.168.1.* network as an example IP range. Make changes to this as needed for your setup. It also assumes the server is at 192.168.1.1. You can replace this IP with the real IP of your server.

Server environment

• The examples here show putting the diskless images at /diskless on the server. You can put these directories anywhere you like.
• The tftproot is at /diskless/tftproot, this can also go anywhere you like.
• This setup assumes that you are going to have multiple diskless images (for example one for PPC, one for x86 and one for amd64). Because of this, the image and the TFTP files are stored in subdirectories.

Recommendations

In a diskless configuration, the percieved peformance of your client machines will depends entirely on your server configuration. You should select a server with a decent disk configuration and lots of RAM. Software or hard RAID is generally a good idea. You will also definitely want to have a gigabit network connection to the server.

My current server is a Dual Xeon 2.4Ghz with 2GB of RAM. It has 2 7200RPM IDE disks mirrored (Raid 1) using Linux software RAID (see HOWTO Gentoo Install on Software RAID for more information on setting up software RAID). This server serves around 200 client machines with very reasonable peformance, and a fairly low loadavg.

Installation

You will need the following packages installed on the server:

• net-misc/dhcp (or net-dns/dnsmasq for small networks behind a NAT server)
• net-ftp/tftp-hpa
• net-fs/nfs-utils

Configuration

Kernel Configuration

You will need to make sure to have NFS support in your kernel. Client support is not necessary, but is recommended to prevent some errors starting the NFS service at boot.

File systems  --->
 Network File Systems  --->
  <*> NFS file system support
  [*]   Provide NFSv3 client support
  ...
  <*> NFS server support
  [*]   Provide NFSv3 server support

TFTP

Set up the TFTP server

# /etc/init.d/in.tftpd

# Path to serve files from
# (this can be located anywhere, this is just an example)
INTFTPD_PATH="/diskless/tftproot"

# For more options, see tftpd(8)
INTFTPD_OPTS="-l -s ${INTFTPD_PATH}"
 

pxelinux

Next we need to set up pxelinux. This comes as part of sys-boot/syslinux, you can install this package on the server

1. emerge --pretend --verbose syslinux
   • or get the package from the syslinux homepage at "http://syslinux.zytor.com/".
2. cp /usr/lib/syslinux/pxelinux.0 /diskless/tftproot/gentoo/
   • Put the pxelinux.0 in the TFTP directory for your image, for this example that directory would be /diskless/tftproot/gentoo/.

Now we need to make a configuration file for pxelinux, the default configuration file for pxelinux is located at pxelinux.cfg/default relative to pxelinux.0, so we need to create the default configuration file:

DEFAULT vmlinuz
APPEND root=/dev/nfs nfsroot=192.168.1.1:/diskless/gentoo,ro,rsize=8192,posix
IPAPPEND 1
 

We will need to copy the kernel over to the TFTP root after the client image is completed.

yaboot

To set up yaboot, we first need to install sys-boot/yaboot on the PPC image/machine. Next copy /usr/lib/yaboot/yaboot binary to the TFTP server, something like scp /usr/lib/yaboot/yaboot root@192.168.1.1:/diskless/tftproot/ should copy the file to the appropiate place. Next you need to create a yaboot.conf file in the same directory as the yaboot binary. There is an example yaboot.conf that should work below.

timeout = 50
defaultos = linux

image = enet:0,maclinux
label = gentoo-mac
append = "root=/dev/nfs nfsroot=192.168.1.1:/diskless/mac,ro,rsize=8192,posix ip=dhcp"
 

NFS

Next you need to set up your exports, your /etc/exports should look something like this:

# /etc/exports: NFS file systems being exported.  See exports(5)
/diskless/gentoo 192.168.1.5(rw,async,no_root_squash) 192.168.1.0/24(ro,async,no_root_squash)
 

DHCP

Now we need to configure the DHCP server, this example configuration hasn't been tested.

default-lease-time 604800;
max-lease-time 604800;
ddns-update-style none;
authoritative;
get-lease-hostnames true;

shared-network mynetwork
{
  subnet 192.168.1.0 netmask 255.255.255.0
  {
    option broadcast-address      192.168.1.255;
    option routers                192.168.1.1;
    option domain-name-servers    192.168.1.1, 192.168.2.1;

    # uncomment this to give out addresses to any host
    # that requests one
    #range 192.168.1.150 192.168.1.200;

    group
    {
      # this is the address of the TFTP server
      server-identifier 192.168.1.1;

      # the name of the PXE file to load
      filename "gentoo/pxelinux.0";


      # these are individual host definitions
      host diskless-1
      {
        hardware ethernet 00:11:22:33:44:55;
        fixed-address 192.168.1.23;
      }

      host diskless-1
      {
        hardware ethernet 00:11:22:33:44:55;
        fixed-address 192.168.1.24;
      }

    }

    group
    {
      # this is the address of the TFTP server
      #server-identifier 192.168.1.1;

      # the name of the PXE file to load, you could load an alternat
      # image or kernel for this group of hosts
      #filename "gentoo2/pxelinux.0";
    }
  }
}
 

DNS Server

You will also need to set up a DNS server, if your network does not already have one. Every host that will be booting diskless needs to have an IP that can be reverse-DNS'ed so it can figure out what it's hostname is. You can look at this howto for more information on setting up a DNS server: HOWTO Setup a DNS Server with BIND

Image setup

This chapter assumes that you already have an image ready to be used as a diskless image.

Kernel Configuration

The absolute minimum kernel setup needs kernel-level network autoconfiguration and NFS booting:

Networking  --->
 Networking options  --->
  [*]   IP: kernel level autoconfiguration
  [*]     IP: DHCP support

You will need NFS and NFS root support. I also recommend NFS server so the nfs service starts cleanly on boot. NFS over TCP is also a good idea, so if a some brain-dead firewall on your network decides UDP is a bad idea, you will still be able to boot your diskless systems.

File systems  --->
 Network File Systems  --->
  <*> NFS file system support
  [*]   Provide NFSv3 client support
  ...
  <*> NFS server support
  [*]   Provide NFSv3 server support
  ...
  [*]   Provide NFS server over TCP support
  [*] Root file system on NFS

You will also need tmpfs support in the kernel for mounting directories that need to be read-write on the client machines. Procfs is also needed so the scripts can access various information on the running kernel.

File systems  --->
 Pseudo filesystems  --->
  [*] /proc file system support
  [*] Virtual memory file system support (former shm fs)

For the rest of the kernel configuration, you should make modules for every piece of hardware you might possibly have or ever have anywhere on your network. You can get the kernel configuration I use for gentoo-sources-2.6.19-r2 at "http://www.cs.mcgill.ca/~chutz/config", it should give you a fairly good starting point.

Preparation

Now that you have the server set up, it's time to get the image ready. This assumes that you have already made an image in a chroot, and installed all the software you will need on it, and set up all the servieces. You do not need to configure X or anything hardware specific on the image, that will be done automatically at boot.

Image Cleanup

First we have to a little cleanup on the image, we will get rid of any unnecessary files, and other junk.

1. Delete the contents of /tmp and /var/tmp/, a simple rm -rf /tmp/* /tmp/.X11-unix/* /tmp/.ICE-unix/* /var/tmp/* should get rid of most of the temporary files.
2. Clean out any logs that are on the image, cd to /var/log and delete any files except lastlog.

Copying Image to Server

Next we need to copy the image to the server, the easiest way to do this is to the NFS share set up earlier. Assuming you are on the machine that you specified as rw in /etc/exports on the server, do something like this:

# mkdir /mnt/diskless
# mount 192.168.1.1:/diskless/gentoo /mnt/diskless
# cp -a /path/to/image/* /mnt/diskless

This will probably take a long time as you are copying a huge amount of data, and creating a lot of small files (which is quite slow over NFS).

Copying Kernel to TFTP

Once you have the image copied over to the server, you can copy the kernel to the TFTP directory on the server. That will look something like this (depending on the particular version of the kernel you are copying.

# cp /diskless/gentoo/boot/vmlinuz-2.6.14-gentoo-r3 /diskless/tftproot/gentoo/
# ln -sfn vmlinuz-2.6.14-gentoo-r3 /diskless/tftproot/gentoo/vmlinuz

Having the actual kernel that the pxelinux configuration points to be a symlink allows you to keep several kernels around and easily switch between them simply by changing the file that the link points to.

Setting up the Diskless

Compile Machine

Now that the image is on the server, we probably want to set up a machine to do emerging, and maintenance on. We could do this directly from the server, but it's generally best to keep as much load off the server as we can, so I recommend that you set up a machine with the diskless image mounted read-write so you can chroot into the diskless.

Since you will be emerging on this chroot, you will need some bind mounts. I also suggest bind mounting directorys of non-critical cache files so you won't be storing all kinds of unnecessary data on the server.

#!/bin/bash
OPERATION="$1"
IMAGE="$2"

# this is the directory you want to store temporary files
# for the images in
DS=/usr/local/diskless

if [ "$IMAGE" == "" ]; then
        OPERATION=""
else
    if ! [ -d /diskless/$IMAGE ]; then
        echo "ERROR: directory \"/diskless/$IMAGE\" does not exist!"
        exit 255
    fi
fi

case "$OPERATION" in
    mount)
        mount -o bind /dev /diskless/$IMAGE/dev
        mount -o bind /proc /diskless/$IMAGE/proc
        mount -o bind /sys /diskless/$IMAGE/sys

        [ -d $DS/tmp/ ] || mkdir -p $DS/tmp/

        [ -d $DS/tmp/$IMAGE ] || mkdir $DS/tmp/$IMAGE
        mount -o bind $DS/tmp/$IMAGE /diskless/$IMAGE/tmp

        [ -d $DS/vtmp/$IMAGE ] || mkdir $DS/vtmp/$IMAGE
        mount -o bind $DS/vtmp/$IMAGE /diskless/$IMAGE/var/tmp

        [ -d $DS/distfiles/$IMAGE ] || mkdir $DS/distfiles/$IMAGE
        mount -o bind $DS/distfiles/$IMAGE /diskless/$IMAGE/usr/portage/distfiles
    ;;
    umount)
        umount /diskless/$IMAGE/dev
        umount /diskless/$IMAGE/proc
        umount /diskless/$IMAGE/sys
        umount /diskless/$IMAGE/tmp
        umount /diskless/$IMAGE/var/tmp
        umount /diskless/$IMAGE/usr/portage/distfiles
    ;;
    *)
        echo "Syntax: $0 <mount|umount> <image>"
    ;;
esac
 

This script takes two parameters, the action (either mount or umount) and the name of the image. This script assumes that your images are mounted at /diskless/<image name>. So to setup the bind mounts for an image names "gentoo" you would run the script with the parameters "mount gentoo".

Diskless Scripts

Installing Diskless Scripts

There is an overlay available for the diskless, to get it, install subversion on your image and do this (you can also do it with subversion on the server, just provide the path to the image before the /usr/local/diskless):

emerge --pretend --verbose subversion

Now edit /etc/make.conf and add /usr/local/diskless to the PORTDIR_OVERLAY on your image. Once you have that done, you can install the diskless scripts with emerge diskless.

"Layer" Introduction

These diskless scripts use a "layering" system to allow for separate configurations for different systems. Any files that need to change in different systesm, or change in the live system are replaced with symlinks to the "local" (the name of which is configurable) directory for the system directory they are in. For example /etc/fstab is actually a symlink to /etc/local/fstab, /etc/X11/xorg.conf is a symlink to /etc/local/X11/xorg.conf. The "local" directories are mounted read-write via either tmpfs or bind mounts to a locally mounted "scratch" drive (explained further down).

When the system boots, the initscripts consecutively copy each layer into the "local" directories on the system. By default the layers are stored in the /diskless directory on the image. This is configurable.

Configuration "Layers"

The order the layers are loaded are as follows:

1. The "base" layer, this layer contains every file that is considered a "local" file, every system copies this layer first. This layer is stored at /diskless/base.
2. The "network" layer, this layer is based on the machine's subnet (192.168.X.255), it often contains a resolv.conf and any other files that vary on a per-network basis. This layer is stored at /diskless/networks/*.
3. The "lab" layer. This layer is based on the hostname, it is determined by taking the part of the hostname before the first "-" character. For example, a machine named lab3-14 would be in the "lab3" layer. This layer is stored at /diskless/labs/*.
4. The "host" layer, this layer is simply the hostname of the machine, without the domain part. For example lab3-12.example.edu would be just lab3-12. This layer is stored at /diskless/hosts/*.
5. Finally, the IP layer, this layer is simply the full IP address of the machine. This layer is stored at /diskless/hosts/*.

Configuration

The diskless scripts need to be configured, there is a basic configuration file located at /etc/diskless/diskless.conf. This file should be fairly self-explanatory, the most important parts are the DIRS variable and the LOCALDIR variable. The DIRS variable is a space separated list of the directories that have a locally mounted subdirectory, and the LOCALDIR is the name of the local subdirectory.

There also should be a file in /etc/diskless/local.d/ for each directory listed in DIRS. These files contain the relative paths of all files in that directory that need to be stored locally. When the script sets up the diskless system, these files will be replaced with relative symlinks to the same path in the "local" directory. The "local" directory on the image will be populated with symlinks to the "base" layer. On booting, the client machines will mount a tmpfs filesystem over the "local" directory very early in boot, so they won't see these symlinks, only the files that are in the "local" directory after the layer copying finishes.

Finally, there is an informational file at /etc/diskless/about, this file contains the "name" for the current image and a short description. The "name" of the image has to be the entire first line, and cannot contain spaces or special characters. The name is an informational name, it is mainly used to name the prompt after chrooting into the diskless image. The contents of this file is displayed when the "about" command is typed from the image.

The "diskless" Script

Included with the diskless package is a script called diskless, by default this script is installed at /usr/sbin/diskless. This script provides several functions to help managing layers. The script has different options depending whether it is run on the master (with a rw root) or one of the client machines.

On the client machines it provides a "relayering" function. This function allows you to re-execute the "layering" procedure that was done at early boot, allowing you to apply a configuration change on the client machine. To relayer a client machine simply use the command diskless relayer from the client you want to relayer.

Another useful command is the diskless setup command. This command will perform all the initial steps to set up the layering on an image. It creates the /diskless/ tree and sets up all the symlinks for the local directories.

Initscripts

diskless

The "diskless" initscript is the first service that starts at boot. This script sets up the hostname and domainname. Then mounts the "local" directories with the tmpfs filesystem and copies the "layers" into them. Finally this script initializes the "mtab" with the currently mounted filesystems.

All the configuration options that this script accepts are in /etc/diskless/diskless.conf.

autoconfig

Autoconfig is the script that detects the hardware in the system and loads the modules for that hardware. This script uses sys-apps/hwsetup and sys-apps/hwdata-gentoo to detect the hardware in the system and determine the appropriate module to load. This is the same method that the Gentoo LiveCDs use to detect hardware. This script is a modified version of the script with the same name that the LiveCD's use.

The configuration for this script is located at /etc/conf.d/autoconfig. In the configuration file, you can turn off the sound detection, and set the mixer devices that the script automatically unmutes. It also has switches for SCSI, hotplug and GPM support.

This script generally runs very early in the bootup procedure, after the "diskless", "checkroot", "modules" and "hostname" services.

findlocal, uselocal

The "findlocal" and "uselocal" scripts work together to make use of the local disks in a system. "findlocal" finds local partitions that can be mounted and adds them to /etc/fstab. Between "findlocal" and "uselocal" the system runs the standard Gentoo initscript "localmount" that mounts local filesystems. Once the filesystems found by "findlocal" are mounted by "localmount", "uselocal" will make use of them.

The "findlocal" works by scanning all disks in the system looking for partitions formatted with filesystems listed in /etc/conf.d/findlocal. This configuration file is fairly well documented, and contains a switch as to whether to use any partitions found that are marked as Linux swap partitions. It also contains a list of possible filesystems to use as "scratch" drives. A "scratch" drive is a partition that is mounted at /mnt/scratch on the machine, and has subdirectories bind mounted to the "local" directories on the system, as well as /tmp and /local. The "scratch" drive's main purpose is to provide local disk storage for the local files so they do not need to be stored in tmpfs using up system memory. Anything found in the paths on the scratch drive where it bind mounts the "local" directories can be optionally wiped before being used.

"findlocal" uses sys-apps/parted to find partitions by formatted filesystem, rather than the marker on partition. This means that the marker is ignored, if a filesystem is marked "FAT32" and formatted ext2, it will be detected as an ext2 partition.

xorgconf

"xorgconf"configures the X server, it will generate /etc/X11/xorg.conf. This script generated an initial configuration file by using the X -configure, this executes X, telling it to output a new configuration file in /root/xorg.conf.new. The script then edits this configuration file, fixing the mouse setup to a working one (X -configure never actually makes a working mouse configuration). It also sets up the default resolution as specified in /etc/conf.d/xorgconf.

The configuration for xorgconf contains a setting to turn off X, turning this setting on will make the script modify the inittab so the desktop manager won't run when /etc/init.d/xdm start is executed, then the script will exit.

Various files

Now that we have configured the image how we want it, and initialized it as a diskless setup, there are a few finishing touches that we need to do.

First we should set up the fstab for the image, here is an example fstab that sould work for a diskless setup. The first line is critical, otherwise you will get odd errors from the initscripts as the system is booting.

# <fs>          <mountpoint>    <type>     <opts>                           <dump/pass>

/dev/root       /              nfs      ro,rsize=8192,posix,hard,nfsvers=2  0 0
/dev/cdrom      /media/dvd     iso9660  ro,user,noauto,owner                0 0
/dev/cdrom1     /media/cdrom   iso9660  ro,user,noauto,owner                0 0
/dev/floppy/0   /media/floppy  auto     rw,user,noauto,owner                0 0
/dev/sda1       /media/usb     auto     rw,user,noauto,owner                0 0

none            /dev/shm       tmpfs    defaults                            0 0
none            /tmp           tmpfs    defaults                            0 0
 

Image Maintenance

Updating

To update the image (update or install packages), simply chroot into the image and emerge as you would on a normal machine. If there are new files that need to be stored locally, use the diskless command to add them to the list of locally stored files. There is no need to do source /etc/profile once you chroot, as the /root/.bashrc installed by the diskless ebuild will do this for you, as well as change your prompt to match the first line in /etc/diskless/about.

From time to time, an update will silently overwrite or move files that the diskless scripts depends on for boot. The diskless script has a function to clean up the more commonly broken parts of a diskless setup. To run this cleanup, use the diskless fix command.

Client Machine Configuration

Hardware Requirements

There aren't a whole lot of requirements for the client machines. If you are going to be running something like GNOME or KDE, I would recommend at least 512MB of RAM.

If you want the machine to not seem absurdly slow, you probably want to have a 100 megabit network connection between all the clients and the server. Anything less (10 megabit, for example) results in the client machine generally being very slow.

You will also need to have a network card and BIOS capable of booting off the network, most modern systems can do this, but with older systems you may encounter some that can't. If you do encounter one that can't, you can purchase a PCI network card with a network boot ROM.

Getting Machines to Boot Diskless

PC BIOS Setup

If your motherboard has an onboard network card that supports PXE, you will have to enable PXE, then set the LAN as the first boot device in the system. It is a good idea to remove all other boot devices from the list. You may also want to set a setup password for the BIOS so no one decides to boot the machine off a CD or floppy.

If your network card doesn't support PXE, you will have to buy one that does, and install it in the machine. These cards often have an onboard BIOS that initializes the PXE and boots off the network. You may need to tell the BIOS to initialize the boot ROM.

Mac OpenFirmware Setup

First you need to get the OpenFirmware prompt, to do this boot up the machine holding down Apple-Option-O-F on the keyboard. At the OpenFirmware prompt, type these commands to change the default boot device to the network.

 > setenv boot-device enet:0,yaboot

 > password
 > setenv security-mode command

 > reset-all

Local Partition Setup (Swap and Scratch)

The diskless setup can taka advantage of the local disk to cache files and have a swap partition. by default, findlocal will use any reiserfs partitions it finds as scratch disks. This is configurable in /etc/conf.d/findlocal. findlocal will also use any swap partitions it finds on any disks, this is also configurable in the configuration file.

To prepare the local disk on a machine for running diskless, simply create a swap and Linux partiton and format the swap partition with mkswap and use mkreiserfs to format the Linux partition with reiserfs. The default findlocal configuration will ignore any other partitions it finds.

There are plans to implement an automatic partitioning system in findlocal, but as of yet it is not implemented.

Useful Extras

Automounting Home Directories

Distributed Logging

• syslog-ng
• tenshi

Useful Scripts

kill-procs

kill-procs is a script for managing usage of machines. It was primarily developed for usage on machines in a university computer lab. This script has two main functions, first it kills X sessions of users who have left the computer unattended for a certain period of time (the default is 3.5 hours). Second the script kills various background processes that desktop environments have been known to leave behind when a user logs off. This script will also report on other background processes that it finds running when the owner is not logged into the machine.

The script is currently available in the diskless overlay, you should be able to install it with emerge kill-procs. There are a few configuration settings at the top of the file.

bashrc

I have a custom bashrc that does a few things automatically. Such as running "diskless fix" and making sure that dispatch-conf is patched properly to handle symlinks in /etc (the default version doesn't handle this properly). This is a little dirty, but it does cut down on maintenance a bit. Here is the current version:

patch_dispatchconf() {
        local dry="--dry-run --quiet" patch=1
        while [[ "${patch}" ]] &&  patch -p0 -t ${dry} <<EOF
--- /usr/lib/portage/bin/dispatch-conf.orig    2006-09-28 11:10:33.000000000 -0400
+++ /usr/lib/portage/bin/dispatch-conf  2006-09-28 11:11:01.000000000 -0400
@@ -20,7 +20,7 @@
 import portage, dispatch_conf
 from portage_exec import find_binary
 
-FIND_EXTANT_CONFIGS  = "find '%s' %s -iname '._cfg????_%s'"
+FIND_EXTANT_CONFIGS  = "find -L '%s' %s -iname '._cfg????_%s' -exec readlink -q -f '{}' ';' | sort | uniq | sed -e 's://:/:g'"
 DIFF_CONTENTS        = 'diff -Nu %s %s'
 DIFF_CVS_INTERP      = 'diff -Nu %s %s | grep "^[+-][^+-]" | grep -v "# .Header:.*"'
 DIFF_WSCOMMENTS      = 'diff -Nu %s %s | grep "^[+-][^+-]" | grep -v "^[-+]#" | grep -v "^[-+][:space:]*$"'
EOF
        do
                if [[ "${dry}" ]]; then
                        unset dry
                else
                        unset patch
                fi
        done
}

post_pkg_postinst() {
        [[ "${PN}" == "portage" ]] && patch_dispatchconf
        /usr/sbin/diskless autofix
}

 

check-quota

Miscellaneous Extras

• Xsession customization

Trouble shooting