Zinc is the file and DNS server. It will eventually provide DHCP to netboot the shell servers.
Custom Supermicro 2U case, dual quad-core Xeon E5504 processors, 12 GB RAM, 4x 1.5 TB drives.
- File server
- Internal DNS server
- Netboot head node
Zinc is particular due to its mdadm and LVM setup - it has both RAID 5 and a special zinc-data LV that holds bulk-data folders (homedirs, ugcs, shellserver nfs root). This LV fills all of the remaining space on the array.
NFS is provided by the nfs-kernel-server package and configured via /etc/exports. Right now it's a pair of lines for each server, but once we have a netblock we can clean this up.
DNS is provided by dnsmasq and is configured in /etc/dnsmasq.conf. This file contains basic settings (including MX settings), but the dnsmasq server reads DNS information from /etc/hosts (for hostname definitions) and /etc/resolv.conf (for DNS forwarding). Note that dnsmasq also provides caching.
UGCS mirrors big stuff for convenience of the users to use. This includes OSS and caltech-only stuff. We just use apache and set the webroot over to the right folder in /mnt/media/ugcs/ whatever. The caltech-only stuff needs to be behind ldap.
Filesystem quotas are set per user.
We frist install the quotatool package. Do this before enabling quota in fstab or else the package won't install. Next, enable the quota on the filesystem.
sudo mount -o remount,usrquota /mnt/data sudo quotacheck -cu /mnt/data
Now we can set the quota per user with something like
sudo quotatool -u azhao -b -l 100M /mnt/data
This sets a hard block quota with size 100M
Users are allowed to SFTP directly into the server. They are chrooted to /home and have access to openssh's builtin sftp server. This is done using a match group directive.
TODO: some stuff online talked about needing "nosync" on the nfs. figure that out.
This server is the head node for netbooting the shell servers. This is done all on an internal network on the eth1 interface. This interface is configured with a static ip (172.31.255.254) on the 172.16/12 ip block.
dnsmasq is capable of providing both the dhcp and tftp required for netboot. We take advantage of this capability and have both of these set up.
The nfs server is configured to export the /mnt/data/shellroot folder to the shellservers in addition to the homedirs and ugcs dirs. These dirs are all exported on the internal network across the entire 172.16/12 block. We disable DHCP gateway assignment on the DHCP server, or else the machines will not be able to access the internet, as the server is not configured as a gateway.
Netboot is setup based approximately on the instructions in this guide https://help.ubuntu.com/community/DisklessUbuntuHowto
We will reproduce the relevant instructions here:
The OS is installed on one of the shellservers, then the files are copied onto the nfs share. It's a bit cleaner to debootstrap for the new installation, but we do this because it's easier.
warning, if you do this, then the installer will write the eth interface names into the udev config. You'll have to delete this shit so that it will work for machines that don't match the mac address of the original machine
initial ramdisk and kernel
We first make the initramfs. Start by editing /etc/initramfs-tools/initramfs.conf . We still want to have hdd scratch space, so we want to keep the "modules=most" the same instead of changing it to netboot. We also probably need to set "boot=netboot". There might be a "boot=local" already.
TODO: check if boot=netboot is really necessary
Then we create the initramfs file
mkinitramfs -o ~/initrd.img-`uname -r`
We then copy the initrd.img file and the kernel file (/boot/vmlinuz-...) into the tftp root
We are going to use the syslinux pxelinux bootloader. To get this, we will just download the debian pxe installer and take the files out. We need to put pxelinux.0 and ldlinux.c32 into the tftp root.
Now configure the bootloader
LABEL linux KERNEL vmlinuz-??? APPEND root=/dev/nfs initrd=initrd.img-??? nfsroot=172.31.255.154:/mnt/data/netboot ip=dhcp rw
TODO: multiple boot options
Next we copy the data to the netboot root
mount -t nfs -onolock 172.31.255.254:/data/netboot /mnt cp -ax /. /mnt/.
Now before we can make this work, we need to change the /etc/network/interfaces and /etc/fstab. Fstab needs to remount the root (and tmpfs shit) and interfaces needs to be changed so that it doesn't reconfigure the interface. (if it tries to reset the interface, we lose our root and the system explodes)
extra stuff on shell servers
git build-essential openssl
We also throw in shell in a box on the shell servers. This comes in the package so that's pretty easy.
TODO: We then need to change the port to 443 to make it easier and less jank to grab a shell