This documents my attempt to create a diskless, PXE boot, NFS-supported Debian system. The device is an Intel NUC and it is served by OpenWRT routers. There are posts about PXE previously. But in summary, the workflow of such system is
- Client boot with PXE stack on network card, which requests for DHCP
- DHCP server respond with IP address and the location of the boot program
- Client, according to the DHCP reply, request for the network boot program (NBP) from TFTP
- Client pass the ownership to the NBP for the next stage of boot, which in case of pxelinux, will show boot menu and load kernel
My previous attempt using DD-WRT does not work well. OpenWRT however, has a better kernel to support NFS to make everything smooth.
DHCP set up
In OpenWRT, the DHCP server is dnsmasq,
which its configuration are located at /etc/config/dhcp
(OpenWRT specific)
and /etc/dnsmasq.conf
(dnsmasq default). When we run it as a daemon, OpenWRT
will create a new config at /var/etc
that loads the latter and apply the
attributes from the former. Hence it is better to modify at /etc/config/dhcp
.
Below should be a new section appended to the end:
config boot linux
option filename 'pxelinux.0'
option serveraddress '192.168.0.2'
option servername 'tftpserver'
The above will be converted into a line
dhcp-boot=pxelinux.0,tftpserver,192.168.0.2
in the dnsmasq.conf
, and
delivered as DHCP option 66 (TFTP server address) and option 67 (boot filename)
in the reply. In fact, we can make this more specific to one host, for example,
the below will be host-based configuration based on MAC address:
config host 'myhost'
option ip '192.168.0.123'
option mac '01:23:45:ab:cd:ef'
option tag 'NOPXE'
TFTP set up
TFTP is also supported in dnsmasq. What we needed to do is to add these two
lines into /etc/dnsmasq.conf
:
enable-tftp
tftp-root=/path/to/tftproot
or equivalently, add to /etc/config/dhcp
:
config dnsmasq
# ...
option enable_tftp '1'
option tftp_root '/path/to/tftproot'
config dhcp 'lan'
option interface 'lan'
option dhcp_range '192.168.0.1,proxy'
option start '100'
option leasetime '12h'
option limit '150'
option dynamicdhcp '0'
The above configuration is for using a OpenWRT device to serve TFTP other than
the one responding to DHCP. Otherwise, we do not need to modify the lan
section to set the dhcp_range
and dynamicdhcp
option. But we must ensure
that dnsmasq is listening to the interface that serves TFTP. It will not work
if we ignored the interface for DHCP. If we set ths up in web interface in LuCI:
- in Network → Interface → LAN → DHCP server → General setup, uncheck “Ignore interface”
- in Network → Interface → LAN → DHCP server → Advanced settings, uncheck “Dynamic DHCP” and uncheck “Force”
- in Network → DHCP and DNS → General settings, uncheck “Authoritative”
- in Network → DHCP and DNS → TFTP settings, check “Enable TFTP server”, enter the full path for “TFTP server root”
To test, run “tftp <address>” and then try to “get
NFS system
We used NFSv4 under OpenWRT. Kernel server is used, as it seems more robust
than the user server in the case of OpenWRT systems. The disks are mounted
externally as OpenWRT device usually would not have enough built-in storage. We
need to set up the mount in /etc/config/fstab
with the following:
config global automount
option from_fstab 1
option anon_mount 1
config global autoswap
option from_fstab 1
option anon_swap 0
config mount
option target /tmp/extstorage
option device /dev/sda1
option enabled 1
option enabled_fsck 0
option options 'noexec,noatime,nodiratime,nodev'
and for USB storage, we will also need to
opkg install usbutils kmod-usb-storage block-mount
For NFS then, we need to do
opkg install nfs-kernel-server
and then edit /etc/exports
with
/tmp *(ro,all_squash,insecure,no_subtree_check,sync,fsid=0)
/tmp/extstorage *(rw,all_squash,insecure,no_subtree_check,nohide,sync,fsid=1)
/tmp/extstorage/diskless *(rw,no_root_squash,insecure,no_subtree_check,nohide,sync,fsid=3)
Note here that the external drive are mounted under a mount point under /tmp
,
not the usual /mnt
. It is because in OpenWRT, the /mnt
is at an overlay
mount and mounting something under there will raise an error. In /tmp
,
however, is a tmpfs which does not have such problem.
Usually NFS will do root squash to make all files accessed through NFS to be owned by nobody (exact user ID depends on system). However, we do not want this for our diskless systems so that different users, root included, can still be distinguished.
PXElinux configuration
In the TFTP server, under the tftp root, we should make a copy of pxelinux.0
from syslinux and put the *.c32
file (e.g. 32-bit BIOS version) under
boot/isolinux/
. Then we can set up, for example, the below config at pxelinux.cfg/default
:
DEFAULT menu.c32
PROMPT 1
TIMEOUT 30
ONTIMEOUT Debian
LABEL reboot
MENU LABEL reboot computer
COM32 reboot.c32
LABEL local
MENU LABEL boot local drive
LOCALBOOT 0
LABEL Debian
MENU LABEL Debian Buster
KERNEL vmlinuz
APPEND vga=858 rw ip=dhcp initrd=initrd.img root=/dev/nfs nfsroot=192.168.0.2:/tmp/extstorage/diskless/NUC ipv6.disable=1
Then we also need to copy the kernel (vmlinuz
) and init ramdisk
(initrd.img
) to the tftp root. Which can be easily done if we have installed
a system in another machine.
Boot disk set up
The easiest way to create a boot disk is to get a separate computer, install Debian, then create a tarball out of the root mount. Or otherwise, we can also use
debootstrap --no-merge-usr bullseye /path/install http://ftp2.us.debian.org/debian
to create a barebone Debian install. After we copy over all files into the NFS
server under a dedicated directory (e.g., /tmp/extstorage/diskless/NUC
in my
case), we still need to make sure the kernel and the init ram disk are
accessible from TFTP. A symbolic link of the two files into the tftp root
should be suffice.
The next thing is to prepare the copied file to be usable in the diskless
environment. One dedicated copy for one machine would be the best. But at
least, we need to update /etc/hostname
and the /etc/network/interfaces
if
not using DHCP. Chroot to apt-get install openssh-server
would also be
necessary if the diskless device is also headless.
One key thing for the kernel: It will be delivered via TFTP and the root would
be mounted as NFS. Hence the kernel should support NFS access. If not, we need
to create a init ram disk with those modules. In debian, what we need is to add
BOOT=nfs
into /etc/initramfs-tools/initramfs.conf
and then run mkinitramfs
-d /etc/initramfs-tools -o path/to/initrd.img
.
First boot
After all these are done, the diskless device should be able to boot with the
root mounted via NFS. To test run (which fails in DD-WRT but works in OpenWRT)
is to sudo and run apt-get update; apt-get dist-upgrade
. This will test
whether NFS can handle the permission correctly. Also we should check /proc
and /dev
to make sure the procfs and device fs are mounted correctly (local,
not related to NFS)
Cavet: Diskless system over Ethernet is slow because Ethernet is way slower than SATA connection.