Building your own supercomputer using Ubuntu and Kerrighed:
Introduction:
The following page describes how to build your very own supercomputer using household-type easily available parts and tying them all together using Kerrighed, a Single System Image operating system for clusters. This guide is mainly based on the install guide located at www.kerrighed.org, but includes many extra details!
Hardware Requirements:
There are few hardware requirements when it comes to choosing parts for building your own supercomputer, but there are a few options that will make the install significantly easier. The most important components of your new supercomputer are going to be the motherboards of each individual node and hence before you start you should choose these carefully.
1) Your motherboards must support PXE booting
2) Your motherboards should have VGA ports on the back. You will need to see what you’re doing in order to configure the BIOS, and additional graphics cards means extra costs as well as extra hassle!
3) Your motherboards should have onboard Gigabit Ethernet, if not you will have to buy extra network cards in order to connect them all together.
4) It’s also handy if your motherboards support multicore CPU’s so you can squeeze in more processing power into less space.
In this example, I chose to go for Asus P5B-V motherboards as these support PXE booting, they have onboard VGA and Gigabit Ethernet ports and also support Intel quad core CPU’s.
Once you have selected your motherboards you need to decide on your CPU’s and RAM. In our case we decided to go for 4GB of RAM per motherboard and to buy Intel QX6700 CPU’s. Only one of your computers (the head node) is going to need a hard drive so pick one large enough to hold all the data you’re going to generate. The head node will also need an additional Gigabit Ethernet card (I’ll explain why later) and a DVD drive to install your operating system.
So you’ve got all your parts now you need some standard PC cases to put them in. In this case we chose Gigabyte Triton cases as they are cheap and come pre-loaded with two 12cm fans to keep everything nice and cool on long runs! Last but not least you will need a gigabit Ethernet switch, with as many ports as you have computers in your cluster.
So put it all together and let’s get started!
Installing and setting up your head node:
So you need to choose a Linux distro to be the base operating system for your supercomputer. In this case I chose to use Ubuntu as it is a lightweight and very reliable Linux Distro
Install Ubuntu on your head node. Once installation is complete, set up a root account and log back in as root
Next we need to do some essential house keeping:
Remove the old sh directory and symlink bash to sh
root@server:~$ rm -f /bin/sh
root@server:~$ ln -s /bin/bash /bin/sh
Open the file “/etc/apt/sources.list” and append the following lines:
deb http://ftp.ubuntu.com/ubuntu/ gutsy main universe multiverse restricted
deb-src http://ftp.ubuntu.com/ubuntu/ gutsy main universe multiverse restricted
Open up a terminal and run
root@server:~$ apt-get update
root@server:~$ apt-get install kernel-package libncurses5-dev fakeroot wget bzip2 automake autoconf libtool rsync xmlto dhcp3-server tftpd-hpa initramfs-tools nfsbooted nfs-common nfs-kernel-server debootstrap dpkg autotools-dev syslinux ssh
Configuring the DHCP server:
Modify “/etc/dhcp3/dhcpd.conf” to contain the following lines:
#GRUB magic
option grub-menu code 150 = string;
option dhcp-max-message-size 2048;
use-host-decl-names on;
deny unknown-clients;
deny bootp;
option domain-name “mydomain”;
option domain-name-servers 192.168.0.0;
option ntp-servers ntp.network.net;
subnet 192.168.0.1 netmask 255.255.255.0 {
option routers 192.168.0.1;
option broadcast-address 192.168.0.255;
filename “/pxelinux.0”;
}
group {
option root-path “/NFSROOT”;
host node01 {fixed-address 192.168.0.101; hardware ethernet XX:XX:XX:XX:XX:XX; }
host node02 {fixed-address 192.168.0.102; hardware ethernet XX:XX:XX:XX:XX:XX; }
.
.
.
host nodeXX {fixed-address 192.168.0.1XX; hardware ethernet XX:XX:XX:XX:XX:XX; }
}
Modify “/etc/default/dhcp3-server” to contain reference to the ethernet device which is connected to the nodes of the cluster;
INTERFACES=”eth1″
Modify “/etc/network/interfaces” to contain:
auto lo
iface lo inet dhcpauto eth1
iface eth1 inet static
address 192.168.0.1
gateway 192.168.0.1
netmask 255.255.255.0
network 192.168.0.0
broadcast 198.162.0.255
Configuring the TFTPD-HPA server:
Modify “/etc/default/tftpd-hpa” to contain:
#Defaults for tftpd-hpa
RUN_DAEMON=”yes”
OPTIONS=”-l -s /srv/tftp”
Configuring the host list file:
Modify “/etc/hosts” to contain a list of your server and all your nodes:
127.0.0.1 localhost.localdomain localhost
192.168.0.1 server.mydomain server
192.168.0.101 node01.mydomain node01
192.168.0.102 node02.mydomain node02
.
.
.
192.168.0.1XX nodeXX.mydomain nodeXX
Configure the file which specifies which directories your server will be serving:
Modify “/etc/exports” to contain:
/NFSROOT *(rw,async,no_root_squash,no_subtree_check)
/NFSROOT/tmp *(rw,sync,no_root_squash,no_subtree_check)
/NFSROOT/var *(rw,sync,no_root_squash,no_subtree_check)
/NFSROOT/dev *(rw,sync,no_root_squash,no_subtree_check)
/NFSROOT/root *(rw,sync,no_root_squash,no_subtree_check)
Create GRUB directories:
Create the network GRUB directories and copy the pxelinux.0 file to the file-server root by running the following commands:
root@server:~$ mkdir “/srv/tftp/”
root@server:~$ mkdir “/srv/tftp/pxelinux.cfg”
root@server:~$ cp /usr/lib/syslinux/pxelinux.0 /srv/tftp/
Create a boot file for your nodes:
Use your favourite text editor to create a default boot config file for your nodes called “default” and place it in the “/srv/tftp/pxelinux.cfg/” directory you made earlier. The file should contain only the following lines:
LABEL linux
KERNEL vmlinuz-“uname -r”
APPEND root=dev/nfs initrd=initrd.img nfsroot=192.168.0.1:/NFSROOT/ ip=dhcp rw
Make a kernel and ramdisk image for your nodes:
Make a kernel and ramdisk image for your nodes by running the following commands in a terminal:
root@server:~$ mkinitramfs -o /srv/tftp/initrd.img
root@server:~$ cp /boot/vmlinuz* /srv/tftp/
Next we need to do some more essential house keeping:
Symlink a mount nfs command so that nfs is mounted regardless of the network state by typing the following in a terminal:
root@server:~$ ln -s /etc/network/if-up.d/mountnfs /etc/rcS.d/S35mountnfs
Next we need to create a “restart-server” script that we can run whenever we need to reset the server without rebooting. To do this you need to create a file called “restart-server” and put it in “/bin/” the script needs to contain only the following lines:
/etc/init.d/portmap restart
/etc/init.d/dhcp3-server restart
/etc/init.d/tftpd-hpa restart
/etc/init.d/nfs-common restart
/etc/init.d/nfs-kernel-server restart
exportfs -ra
Installing the node operating system:
Create the node operating system directory by typing the following in a terminal:
root@server:~$ mkdir /NFSROOT
Build the node operating system by typing the following in a terminal:
root@server:~$ debootstrap “version” /NFSROOT http://ftp.ubuntu.com/ubuntu/
Copy some essential housekeeping files to the node operating system from the server operating system:
root@server:~$ cp /etc/apt/sources.list /NFSROOT/etc/apt/
root@server:~$ cp /etc hosts /NFSROOT/etc/
root@server:~$ cp -r /usr/lib/locale/* /NFSROOT/usr/lib/locale/
root@server:~$ cp /etc/default/locale /NFSROOT/etc/default/locale
Change root to the node operating system and do some houskeeping
When logged into your server you can change root to the node operating system root at anytime by typing the following into the terminal:
root@server:~$ chroot /NFSROOT
Once you have changed root to the node filesystem you need to set the root password. You can do this by typing “passwd” into a terminal and following the instructions:
root@server:~$ passwd
Like on the server system we need to remove the old sh directory and symlink bash to s by typing the following into a terminal:
root@server:~$ rm -f /bin/sh
root@server:~$ ln -s /bin/bash /bin/sh
Now we need to mount the proc and dev file systems on the node operating system by typing:
root@server:~$ mount -t proc none /proc
root@server:~$ mount -t devpts none /dev/pts
Update and install the necessary node packages:
Next we need to install some essential packages to allow our nodes to boot and run from the server, we do this by typing:
root@server:~$ apt-get update
root@server:~$ apt-get install libncurses5-dev fakeroot wget bzip2 automake autoconf libtool lsb-release xmlto initramfs-tools nsfbooted nfs-common autotools-dev ssh pkg-config patch make dhcpd linux-image-“uname -r”
Configure the FSTAB files for the cluster nodes:
Next we need to set up the fstab file which tells the nodes when they are booting where to find all the important directories. We do this by creating a file called fstab and placing it in “/NFSROOT/etc/” replacing the existing file. The “fstab” file should contain only the following lines:
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0192.168.0.1:/NFSROOT / nfs rw,hard,nolock 0 0
192.168.0.1:/NFSROOT/dev /dev nfs rw,hard,nolock 0 0
192.168.0.1:/NFSROOT/var /var nfs rw,hard,nolock 0 0
192.168.0.1:/NFSROOT/tmp /tmp nfs rw,hard,nolock 0 0
192.168.0.1:/NFSROOT/root /root nfs rw,hard,nolock 0 0
# TMPFS /var/run tempfs defaults 0 0
Configuring node boot parameters:
Next we need to do some boot parameter configuration. Well start by modifying the boot statement in “/NFSROOT/etc/initramfs-tools/initramfs.conf” to read:
BOOT=nfs
Again we need to symlink a mount nfs command so that nfs is mounted regardless of the network state:
root@server:~$ ln -sf /NFSROOT/etc/network/if-up.d/mountnfs /NFSROOT/etc/rcS.d/S35mountnfs
Your done! congratulations!
Hopefully if youve reached this point then your redy to boot your nodes! First though we need to reboot the server/head node. Once its rebooted log in as root and run the “restart-server” command. Hopefully eveything will come up as OK! if not go back and double check the setting of the DHCP and TFTPD servers above.
root@server:~$ restart-server
Configure your nodes to enable PXE boot:
To do this you will need to consult the mamnual for your motherboard as all motherboards will have different procedures for enabling PXE boot.
Once PXE is enables hook up your ethernet cables and boot up your nodes. If your nodes boot into Linux then congratulate yourself on getting this far as your 99% there. Your now ready to install kerrighed so shut down your nodes again.
Get and install Kerrighed:
Kerrighed is constantly being updated so look here for the current gide for installing kerrighed on your supercomputer
And heres one I made earlier….
This is a cluster I built in 2008. Its running Ubuntu Server and kerrighed and it has only had one reboot since it was first switched on. The reboot was not because of a hardware or software failure but was so I could install a pci SSD drive to mount the client filesystem on to speed up disk access (highly reccommended!!!).
Since this cluster was built it has been used to find the origin-of-life. Understand how genome structure influences gene expression regulation in Trypanosoma brucei and other organisms. For the last year or so it has been mostly thinking about how to make better multiple sequence alignments.
Components:
Motherboards: SuperMicro X7DVL-E (X6)
Hard Disk: Western Digital Caviar (X3 1Tb in raid 1 on head node. No disks in compute nodes) mounted on “/”
RAM: 72Gb (6 x 12Gb. Each node has 6 Kingston 2Gb DDR2 modules)
CPU: intel quad core Xeon E5420 (X12, 48 Cores of low temperature CPU power!)
SSD: RevoDriveX2 (320Gb solid state PCI hard disk) mounted on “/NFSROOT”
Case: Gigabyte Triton (X 6, my favourite PC case, spacious, good airflow, no screws, what more could you ask for!)
Total cost in 2008: £5,100
