HOWTO_Setup_2_Node_Active_Passive_Cluster_With_DRBD_UCARP

The Setup

The primary setup configuration will be a 2 node Active/Passive High-Availability cluster. Data replication will be done using DRBD and IP Failover will be done using UCARP. Miscellaneous configuration file synchronization with Csync2.

Both nodes will be connected to a network through 1 network interface card. This network interface will be used for all network traffic including DRBD replication. There are many different setups you could use for this including separate paths, cross-over link and many more. The more redundant and complex you make your setup the more technologies you will need to use. Note that I do not cover STONITH (Shoot The Other Node In The Head) setup because you don't really need one in our case. So this is a very simple setup but it works.

Technologies Used

• DRBD for synchronus data replication
• UCARP for IP Failover
• Csync2 for synchronizing configuration files

Prerequisites

We assume that you have at least the following already setup and installed:

• 1: Gentoo Linux installed on 2 machines
• 2: The machines are identical in hardware setup (this makes it easier to trouble shoot problems in the long run)
• 3: You have 1 unformatted linux partition that is not being used (this will be your DRBD partition)
• 4: Both machines are connected to some sort of network where they can see each other

Name Your Hosts

It is very important that you assign your nodes hostnames and have a naming strategy. This makes it easier for DRBD and UCARP to know which hosts do what. It also makes it easier for you to keep track of nodes. In this document I will use a simple hostname naming strategy. My machines hostnames are 'gentoo1' and 'gentoo2', they are known on the network as 'gentoo1.company.net' and 'gentoo2.company.net' and I'd want the virtual IP address to be 'gentoo.company.net'

Edit your /etc/hosts file

 $ nano -w /etc/hosts

 
127.0.0.1       localhost
::1             localhost
192.168.1.1     gentoo1 node1 gentoo1.company.net
192.168.1.2     gentoo2 node2 gentoo2.company.net
192.168.1.3     gentoo.company.net
  

Emerge DRBD

Install DRBD

$ emerge drbd-kernel drbd

Create the DRBD config file. Note that this configuration puts the nodes in a Active/Passive configuration. On any network failure between the two nodes DRBD will continue in a stand alone state. Here we assume that '/dev/sda3' is your unformatted linux partition.

$ nano -w /etc/drbd.conf

 
#
# drbd.conf
#

global {
	usage-count no;
}

common {
	
	syncer {
		rate 10M;
		al-extents 257;
	}
}

resource "mirror" {

	protocol C;
	
	handlers {
		pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
		pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
		local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
	}

	startup {
		degr-wfc-timeout 120;
	}

	disk {
		on-io-error   detach;
		fencing dont-care;
	}

	net {
		ko-count 4;
		cram-hmac-alg "sha256";
		shared-secret "secretPassword555";
		after-sb-0pri discard-younger-primary;
		after-sb-1pri consensus;
		after-sb-2pri disconnect;
		rr-conflict disconnect;
	}

	on gentoo1 {
		device     /dev/drbd0;
		disk       /dev/sda3;
		address    192.168.1.1:7788;
		meta-disk  internal;
	}

	on gentoo2 {
		device    /dev/drbd0;
		disk      /dev/sda3;
		address   192.168.1.2:7788;
		meta-disk internal;
	}
}
  

Initialize the DRBD partition

$ drbdadm create-md mirror

Important Note

So far the above instructions must be done exactly the same on both machines. If anything DOES NOT need to be done on both nodes it will be mentioned.

Start DRBD on both Nodes

Both nodes will connect to each other at this point.

$ /etc/init.d/drbd start

Initialize your primary node. Do this on NODE 1.

$ drbdadm -- --overwrite-data-of-peer primary all

This will make drbd synchronize the block devices and make 'node1' or 'gentoo1' in our case as the primary node. This might take a while depending how big the partition is. It can be faster if you tweak the 'syncer' clause in the drbd config file.

Format The DRBD Partition

Once the drbd synch has finished we are ready to format our partition to something we can actually use. Format using your proffered file system, I will use reiserfs. Do this on NODE 1.

$ mkreiserfs /dev/drbd0

Check everything is normal by checking the status of DRBD.

$ /etc/init.d/drbd status

It should show 'primary/secondary' on Node1 and 'secondary/primary' on Node2

Add an entry into '/etc/fstab' to make mounting/unmounting your drbd partition easier. Here I created a directory called '/ha' which hosts everything I want to be "highly available" or rather synchronized to both machines via DRBD.

$ nano -w /etc/fstab

 
/dev/drbd0        /ha             reiserfs        noauto          0 0
  

Emerging UCARP

$ emerge ucarp

IMPORTANT! Please note that currently portage has UCARP v1.3 which is ok, if you want to use v1.4 you will need to patch it. There is a post of bugzilla with a v1.4 ebuild which includes the patch http://bugs.gentoo.org/show_bug.cgi?id=208156.

Configuring UCARP

This section I have borrowed the setup procedure from HOWTO_Setup_IP_failover_with_UCARP because it provides very useful scripts. I have modified the scripts a bit to add some extra options.

Create the UCARP scripts directory

$ mkdir /etc/ucarp

Create the Virtual IP Up script

$ nano -w /etc/ucarp/vip-up.sh

 
#!/bin/bash
#

# Read the configuration file
source /etc/conf.d/ucarp

# Set the current node as primary
# Then mount the high availability partition
drbdadm primary all
mount /ha

# Bring up the virtual interface
$IFCONFIG $INTERFACE $VIRTUAL_ADDRESS netmask $VIRTUAL_NETMASK broadcast $VIRTUAL_BROADCAST

# Start our services
/etc/init.d/postfix start
/etc/init.d/mysql start
/etc/init.d/apache2 start

# Do something extra while we're at it, like sending an email to a sysadmin
  

Don't forget to

$ chmod +x /etc/ucarp/vip-up.sh

Create the Virtual IP Down script

$ nano -w /etc/ucarp/vip-down.sh

 
#!/bin/bash
#

# Read the configuration file
source /etc/conf.d/ucarp

# If we want to send an email to the sysadmin notifying that the
# node will go down, here is the place to put it.

# Stop all services
/etc/init.d/apache2 stop
/etc/init.d/mysql stop
/etc/init.d/postfix stop

# Unmount the high availability partition
# Then set the node into secondary state
# Once the Slave node comes back online you need to run
# the command "drbdadm -- --discard-my-data connect all"
# from the Slave.
# Then run the command "drbdadm connect all" from the new master
# so DRBD can connect/sync and go back to replication.
umount /ha
drbdadm secondary all

$IFCONFIG $INTERFACE down
  

Don't forget to

$ chmod +x /etc/ucarp/vip-up.sh

Create the UCARP configuration file.

$ nano -w /etc/conf.d/ucarp

 
# The UCARP configuration file for NODE 1

# Location of UCARP executable
UCARP_EXEC=/usr/sbin/ucarp
IFCONFIG=/sbin/ifconfig
NODE_NAME=gentoo1

# Location of UCARP pid file
UCARP_PIDFILE=/var/run/ucarp.pid

# UCARP options
# -z run shutdown script on exit
# -P force preferred master
# -n don't run down script at start up when we are backup
# -B run in background mode
OPTIONS="-z -n"

# The ratio number to be considered before marking the node as dead
DEAD_RATIO=3

# The logging facility to use
FACILITY=daemon

# UCARP base, lower number will be preferred master
# set to same to have master stay as long as possible
UCARP_BASE=1
SKEW=0

# Interface for IP Address
INTERFACE=eth0:0

# Instance ID
# Any number from 1 to 255
# Master and Backup need to be the same
INSTANCE_ID=1

# Password so servers can trust who they are talking to
PASSWORD=someSecretPassword

# Read password from a file
# -o
#PASSWORD_FILE=/etc/ucarp/passfile

# Send extra parameter to down and up scripts
# -x
#XPARAM=<enter param here>

# The start and stop scripts
START_SCRIPT=/etc/ucarp/vip-up.sh
STOP_SCRIPT=/etc/ucarp/vip-down.sh

# The failover application address
VIRTUAL_ADDRESS=192.168.1.3
VIRTUAL_BROADCAST=192.168.1.255
VIRTUAL_NETMASK=255.255.255.0

# Maintanence address of the local machine
SOURCE_ADDRESS=192.168.1.1

# This line is never to be touched by anyone
UCARP_OPTS="$OPTIONS -b $UCARP_BASE -k $SKEW -i $INTERFACE -v $INSTANCE_ID -p $PASSWORD -u $START_SCRIPT -d $STOP_SCRIPT -a $VIRTUAL_ADDRESS -s $SOURCE_ADDRESS -f $FACILITY"
  

Note that when you copy/paste this file to NODE 2, make the appropriate changes to the SOURCE_ADDRESS and NODE_NAME. Also if you already have a UCARP cluster running on the same network then it is recommended to change the instance id to something that is not being already used.

Create the UCARP init script

$ nano -w /etc/init.d/ucarp

 
#!/sbin/runscript
# Copyright 2005 Mike Glenn & Homechicken Software
# Distributed under the terms of the GNU General Public License v2
 
depend() {
       use logger net
}
 
start() {
       ebegin "Starting UCARP"
       start-stop-daemon --start --background --make-pidfile --pidfile ${UCARP_PIDFILE} --quiet --startas ${UCARP_EXEC} -- ${UCARP_OPTS}
       eend $?
}
 
stop()  {
        ebegin "Stopping UCARP"
        start-stop-daemon --stop --pidfile ${UCARP_PIDFILE}
        eend $?
}
  

Installing Csync2

Emerge csync2

$ emerge csync2

Edit the configuration file. Here we sync various directories and files to keep both machines as identical as possible. If users are added on Node1 then we want it replicated to Node2, especially the passwords. /home can be on the drbd partition. The rest of the files here is what I found useful and currently use in my setup.

$ nano -w /etc/csync2/csync2.cfg

 
# Csync2 Configuration File
# ---------------------------------
#
# Please read the documentation:
# http://oss.linbit.com/csync2/paper.pdf

group gentooconfig
{
        host    gentoo1 gentoo2;
        key     /etc/csync2/csync2.key_gentoconfig;

        # Synchronized directories
        include /root/hints;
        include /etc/ha.d;
        include /etc/mysql;
        include /etc/xinetd.d;
        include /etc/vim;
        include /etc/postfix;
        include /etc/portage;
        include /etc/pam.d;
        include /etc/security;
        include /etc/mail;
        include /etc/init.d;
        include /etc/modules.d;
        include /etc/modules.autoload.d;
        include /etc/logrotate.d;
        include /etc/ssh;
        include /etc/ssl;
        include /etc/ssl/certs;
        include /etc/ssl/misc;
        include /etc/ssl/postfix;
        include /etc/ssl/private;
        include /etc/java-config-2;
        include /etc/java-config-2/build;
        include /etc/apache2;
        include /etc/apache2/modules.d;
        include /etc/apache2/vhosts.d;
        include /etc/apache2/ssl;
        include /usr/local/bin;

        # Synchronized files
        include /etc/drbd.conf;
        include /etc/fstab;
        include /etc/group;
        include /etc/group-;
        include /etc/gshadow;
        include /etc/gshadow-;
        include /etc/hosts;
        include /etc/lilo.conf;
        include /etc/lilo.conf.example;
        include /etc/make.conf;
        include /etc/make.conf.example;
        include /etc/make.globals;
        include /etc/resolv.conf;
        include /etc/locale.gen;
        include /etc/passwd;
        include /etc/passwd-;
        include /etc/shadow;
        include /etc/shadow-;
        include /etc/shadow.old;
        include /etc/services;
        include /etc/xinetd.conf;
        include /etc/updatedb.conf;
        include /etc/securetty;
        include /etc/modules.conf;
        include /etc/modprobe.conf;
        include /etc/logrotate.conf;
        include /etc/issue;
        include /etc/issue.logo;
        include /etc/gentoo-release;
        include /etc/etc-update.conf;
        include /etc/cron.deny;
        include /etc/crontab;
        include /etc/ca-certificates.conf;
        include /etc/conf.d/clock;
        include /etc/conf.d/mysql;
        include /etc/conf.d/sshd;
        include /etc/conf.d/apache2;

        # Do not include hidden or backup files
        exclude                 *~ .*;
        backup-directory        /var/csync2;
        backup-generations      3;
        auto none;   #left , right, bigger, smaller, younger, older
}
  

The resolution method is disabled. Since we will create a cron script to auto-run csync2 every hour. Csync2 doesn't know which node is master or slave. So if there are any errors it will be logged and the admin has to resolve it manually.

Create the csync2 key file

$ cd /etc/csync2
$ csync2 -k csync2.key_gentooconfig

Note if this process ends up being really slow then you may have to link /dev/random to /dev/urandom.

Enable csync2 on xinetd

$ nano -w /etc/xinet.d/csync2

 
# default: on
# description: csync2
service csync2
{
        flags           = REUSE
        socket_type     = stream
        wait            = no
        user            = root
        group           = root
        server          = /usr/sbin/csync2
        server_args     =  -i
        #log_on_failure  += USERID
        disable         = no
        only_from       = 192.168.1.1 192.168.1.2
}
  

Create the csync2 backup dir (or else csync2 won't work right)

$ mkdir /var/csync2

Add xinetd to default run level

$ rc-update add xinetd default

Start xinetd

$ /etc/init.d/xinetd start

Begin the initial csync

$ csync2 -xv

Solve any sync problems from here on by reading the csync2 docs (http://www.clifford.at/papers/2005/csync2/paper.pdf)

Create Csync2 Cron Script

$ nano -w /etc/cron.hourly/csync2

1. Runs the csync2 command to keep config files in sync on both nodes

 
echo " " >> /var/log/csync2_cron.log
/bin/date >> /var/log/csync2_cron.log
exec /usr/sbin/csync2 -xv 2>&1 2>>/var/log/csync2_cron.log
  

This can be put on both nodes, they will sync each other every hour.

Starting UCARP

Now that everything is setup it is time to put it all in action. First we want drbd and ucarp to start when our machines boot so...

$ rc-update add drbd default
$ rc-update add ucarp default

DRBD should be alredy started but not mounted yet. DRBD should be set to secondary/secondary state initially before UCARP starts. So go to your primary node and run

$ drbdadm secondary all

Now we start UCARP on NODE1

$ /etc/init.d/ucarp start

Look at the logs to see what's going on:

$ tail -f /var/log/daemon.log

You should see it putting NODE1 into BACKUP mode first.. then MASTER mode and advertize the virtual IP. You can make sure the services are running by checking their status scripts in '/etc/init.d' and look at the drbd partition to see if it has been automatically mounted '/ha'.

Now go to NODE2 and start ucarp over there. Look at the log file as well. You should see it set to BACKUP mode and stay in BACKUP mode.

The End

We now have a 2 Node HA cluster with block device replication and config file synchronization. If you unplug the network cable or power cable of NODE1; UCARP will automatically promote NODE2 as the master. DRBD will detach and run in primary mode on NODE2 and mount the partition and services will resume.

While NODE1 comes back online after an adim recovers it. UCARP will keep it in BACKUP mode and DRBD will stay in a disconnected state on NODE1 until an admin manually runs this command on NODE1

$ drbdadm -- --discard-my-data connect all

And then runs this command on NODE2 (new master)

$ drbdadm connect all

This will let NODE2 (the new master) update NODE1 (the new slave) with all the latest data.

DMA02 - 2008-MAR-29