Sysadmin Documentation/zinc

< Sysadmin Documentation
Revision as of 23:20, 13 February 2017 by Azhao (talk | contribs) (Azhao moved page UGCS5 Server Setup/zinc to Sysadmin Documentation/zinc)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Return to Server Setup

Zinc is the file and DNS server. It will eventually provide DHCP to netboot the shell servers.


Custom Supermicro 2U case, dual quad-core Xeon E5504 processors, 12 GB RAM, 4x 1.5 TB drives.


  • File server
  • Internal DNS server
  • Netboot head node


Zinc is particular due to its mdadm and LVM setup - it has both RAID 5 and a special zinc-data LV that holds bulk-data folders (homedirs, ugcs, shellserver nfs root). This LV fills all of the remaining space on the array.

NFS is provided by the nfs-kernel-server package and configured via /etc/exports. Right now it's a pair of lines for each server, but once we have a netblock we can clean this up.

DNS is provided by dnsmasq and is configured in /etc/dnsmasq.conf. This file contains basic settings (including MX settings), but the dnsmasq server reads DNS information from /etc/hosts (for hostname definitions) and /etc/resolv.conf (for DNS forwarding). Note that dnsmasq also provides caching.

Direct Downloads

UGCS mirrors big stuff for convenience of the users to use. This includes OSS and caltech-only stuff. We just use apache and set the webroot over to the right folder in /mnt/media/ugcs/ whatever. The caltech-only stuff needs to be behind ldap.

Filesystem Quotas

Filesystem quotas are set per user.

We frist install the quotatool package. Do this before enabling quota in fstab or else the package won't install. Next, enable the quota on the filesystem.

sudo mount -o remount,usrquota /mnt/data
sudo quotacheck -cu /mnt/data

Now we can set the quota per user with something like

sudo quotatool -u azhao -b -l 100M /mnt/data

This sets a hard block quota with size 100M


Users are allowed to SFTP directly into the server. They are chrooted to /home and have access to openssh's builtin sftp server. This is done using a match group directive.


TODO: some stuff online talked about needing "nosync" on the nfs. figure that out.

This server is the head node for netbooting the shell servers. This is done all on an internal network on the eth1 interface. This interface is configured with a static ip ( on the 172.16/12 ip block.

dnsmasq is capable of providing both the dhcp and tftp required for netboot. We take advantage of this capability and have both of these set up.

The nfs server is configured to export the /mnt/data/shellroot folder to the shellservers in addition to the homedirs and ugcs dirs. These dirs are all exported on the internal network across the entire 172.16/12 block. We disable DHCP gateway assignment on the DHCP server, or else the machines will not be able to access the internet, as the server is not configured as a gateway.

Netboot is setup based approximately on the instructions in this guide

We will reproduce the relevant instructions here:


The OS is installed on one of the shellservers, then the files are copied onto the nfs share. It's a bit cleaner to debootstrap for the new installation, but we do this because it's easier.

warning, if you do this, then the installer will write the eth interface names into the udev config. You'll have to delete this shit so that it will work for machines that don't match the mac address of the original machine

initial ramdisk and kernel

We first make the initramfs. Start by editing /etc/initramfs-tools/initramfs.conf . We still want to have hdd scratch space, so we want to keep the "modules=most" the same instead of changing it to netboot. We also probably need to set "boot=netboot". There might be a "boot=local" already.

TODO: check if boot=netboot is really necessary

Then we create the initramfs file

mkinitramfs -o ~/initrd.img-`uname -r`

We then copy the initrd.img file and the kernel file (/boot/vmlinuz-...) into the tftp root


We are going to use the syslinux pxelinux bootloader. To get this, we will just download the debian pxe installer and take the files out. We need to put pxelinux.0 and ldlinux.c32 into the tftp root.

Now configure the bootloader

LABEL linux
KERNEL vmlinuz-???
APPEND root=/dev/nfs initrd=initrd.img-??? nfsroot= ip=dhcp rw

TODO: multiple boot options


root filesystem

Next we copy the data to the netboot root

mount -t nfs -onolock /mnt
cp -ax /. /mnt/.

Now before we can make this work, we need to change the /etc/network/interfaces and /etc/fstab. Fstab needs to remount the root (and tmpfs shit) and interfaces needs to be changed so that it doesn't reconfigure the interface. (if it tries to reset the interface, we lose our root and the system explodes)

extra stuff on shell servers

git build-essential openssl

We also throw in shell in a box on the shell servers. This comes in the package so that's pretty easy.

TODO: We then need to change the port to 443 to make it easier and less jank to grab a shell