PXE booting of a FreeBSD disk image

Introduction

I had to set up a regression and network performance lab. This lab will be managed by a Jenkins, but the first step is to understand how to boot a FreeBSD disk by PXE. This article explains a simple way of doing it.
For information, all these steps were done using 2 PC Engines APU2 (upgraded with latest BIOS for iPXE support), so it's a headless (serial port only, this can be IPMI SoL with different hardware) .

The big picture

Before explaining all steps and command line, here is the full big picture of the final process (more readable SVG version of this file):

FreeBSD PXE boot steps

And the tasks we will do:

1. Creating image-miniroot and image.txz, with the help of poudriere
2. Setting up a DHCP (dnsmasq), TFTP (FreeBSD) and FTP (FreeBSD) server
3. Populating the TFTP and FTP server
4. Configuring the DHCP server
5. Test the result

Notice in my lab, the server is configured with IP 1.1.1.254 and the DHCP range will be between .1 and .10.

Instructions

Creating images

To create images we had to do:

1. Install poudriere
2. Configure it (I don't have ZFS on my small APU2, so disable it)
3. Create a poudriere jail of a FreeBSD 12.0-RELEASE
4. Configure custom configuration file we want on the image
5. Generate the poudriere images (main and miniroot)

These commands will do it:

pkg install -y poudriere-devel

echo "NO_ZFS=yes" >> /usr/local/etc/poudriere.conf

echo "FREEBSD_HOST=https://download.FreeBSD.org" >> /usr/local/etc/poudriere.conf

poudriere jail -c -j 120amd64 -v 12.0-RELEASE -K GENERIC

mkdir -p ~/miniroot-overlay/boot

echo 'console="comconsole"' >> ~/miniroot-overlay/boot/loader.conf

mkdir -p ~/miniroot-overlay/etc

cat >~/miniroot-overlay/etc/rc <<EOF

#!/bin/sh

PATH=/bin:/sbin:/usr/bin

# Reusing data from the pxeboot loader to configure network

ifconfig \$(kenv boot.netif.name) inet \$(kenv boot.netif.ip) netmask \$(kenv boot.netif.netmask) up

route add default \$(kenv boot.netif.gateway)

# Need to remount in read-write: Can't use uzip compressed image (read-only)

mount -uw /

mkdir /newroot

# An empty 12.0 base installation (no ports) consumme 1.2G

md=\$(mdconfig -s 2g)

newfs \$md

mount /dev/\$md /newroot

fetch -o - ftp://\$(kenv boot.tftproot.server)/image.txz | bsdtar -xpf - -C /newroot

umount /newroot

kenv vfs.root.mountfrom=ufs:/dev/\$md

# reboot -r needs tmpfs.ko loaded

reboot -r

EOF

mkdir -p ~/image-overlay/boot

echo 'console="comconsole"' >> ~/image-overlay/boot/loader.conf

mkdir -p ~/image-overlay/etc

cat >~/image-overlay/etc/rc.conf <<EOF

# IP configuration and routes will be preserved from the miniroot state

# But configure it as DHCP in case of an 'service netif restart'

ifconfig_igb0="DHCP"

# You need to install your SSH keys

sshd_enable="YES"

# Avoid "My unqualified host name (poudriere-image) unknown; sleeping for retry"

sendmail_enable="NONE"

# Hostname will be added by poudriere image here:

EOF

poudriere image -j 120amd64 -t tar -n image -m ~/miniroot-overlay -c ~/image-overlay/

The last 2 lines from poudriere should be:

Image `/usr/local/poudriere/data/images//image-miniroot' complete

Image available at: /usr/local/poudriere/data/images/image.txz

We will move these files later.

TFTP server

Now let's:

1. Enable TFTPD and inetd
2. Populate the directory with pxeboot, lua scripts, kernel, custom boot/loader.conf and unziped image-miniroot

These commands will do it:

sed -i "" -e 's/^#tftp/tftp/g' /etc/inetd.conf

sysrc inetd_enable="YES"

mkdir -p /tftpboot/boot

mkdir -p /tftpboot/kernel

cp /usr/local/poudriere/jails/120amd64/boot/pxeboot /tftpboot

cp -r /usr/local/poudriere/jails/120amd64/boot/lua /tftpboot/boot

cp -r /usr/local/poudriere/jails/120amd64/boot/defaults /tftpboot/boot

cp /usr/local/poudriere/jails/120amd64/kernel/kernel /tftpboot/kernel

cp /usr/local/poudriere/jails/120amd64/kernel/tmpfs.ko /tftpboot/kernel

cat > /tftpboot/boot/loader.conf <<EOF

# Disable menu

autoboot_delay="-1"

# Enable serial console only

console="comconsole"

comconsole_speed="115200"

# tmpfs is needed by reboot -r

tmpfs_load="YES"

# Download an md_image and use it as root fs

vfs.root.mountfrom="ufs:/dev/md0"

mfs_load="YES"

mfs_type="md_image"

mfs_name="/image-miniroot"

EOF

mv /usr/local/poudriere/data/images/image-miniroot.gz /tftpboot

cd /tftpboot

gunzip image-miniroot.gz

service inetd start

Check your TFTP server is correctly able to serve our files:

cd

tftp localhost
tftp> get pxeboot
Received 436224 bytes during 0.1 seconds in 853 blocks
tftp> quit

FTP server

Now let's:

1. Enable anonymous FTP server (by creating 'ftp' account)
2. Move image.txz into /home/ftp

These commands will do it:

sysrc ftpd_enable=YES

echo "ftp::::::FTP anonymous::/usr/sbin/nologin" | adduser -f -

mv /usr/local/poudriere/data/images/image.txz /home/ftp/

service ftpd start

Check your FTP server is correctly able to serve this file:

ftp ftp://anonymous:nobody@localhost
Trying ::1:21 ...
Connected to localhost.
220 apu2.cochard.me FTP server (Version 6.00LS) ready.
331 Guest login ok, send your email address as password.
230 Guest login ok, access restrictions apply.
Remote system type is UNIX.
Using binary mode to transfer files.
200 Type set to I.
ftp> get image.txz
local: image.txz remote: image.txz
229 Entering Extended Passive Mode (|||61982|)
150 Opening BINARY mode data connection for 'image.txz' (257213124 bytes).
100% |***********************************************************************************************| 245 MiB 26.63 MiB/s 00:00 ETA
226 Transfer complete.
257213124 bytes received in 00:09 (26.63 MiB/s)
ftp> quit
221 Goodbye.

DHCP server

The last configuration step:

1. Install dnsmasq
2. Configure (with the trick of generating a different answer if the request came from iPXE or from FreeBSD's pxeboot loader) and enable it

These commands will do it:

pkg install -y dnsmasq

cat >/usr/local/etc/dnsmasq.conf <<EOF

# Range of IP to distribute (mandatory to enable DHCP server)

dhcp-range=1.1.1.1,1.1.1.10,3h

# TFTP server name

dhcp-option=66,"1.1.1.254"

# Filename to download

dhcp-boot=pxeboot

# Magic trick to detect FreeBSD's pxeboot and avoid iPXE conflict

# Add tag 'fbsd' to clients using userclass 'FreeBSD':

dhcp-userclass=set:fbsd,FreeBSD

# Reply with root-path only to 'fbsd' tagged clients:

dhcp-option=tag:fbsd,option:root-path,tftp://1.1.1.254

EOF

sysrc dnsmasq_enable=YES

service dnsmasq start

Final test

Now time to power up a PXE client (still a PC Engine APU2):

Booting from ROM...
iPXE (PCI 00:00.0) starting execution...ok
iPXE initialising devices...ok

iPXE 1.0.0+ (f8e167) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT


---------------- iPXE boot menu ----------------

ipxe shell
autoboot

net0: 00:0d:b9:45:7a:d4 using i210-2 on PCI01:00.0 (open)
[Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net0 00:0d:b9:45:7a:d4)...... ok
net0: 1.1.1.1/255.255.255.0 gw 1.1.1.254
Next server: 1.1.1.254
Filename: pxeboot
tftp://1.1.1.254/pxeboot... ok
pxeboot : 436224 bytes [PXE-NBP]
PXE Loader 1.00

Building the boot loader arguments
Relocating the loader and the BTX

Starting the BTX loader
(...)

\Loading /boot/loader.conf.local
Loading kernel...
/boot/kernel/kernel text=0x1678aa8 data=0x1cd288+0x768b40 syms=[0x8+0x174cd8+0x8+0x19224a]
Loading configured modules...
/image-miniroot size=0xb00000
/boot/kernel/tmpfs.ko size 0x10c70 at 0x313d000
can't find '/boot/entropy'
---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
(...)
nfs_diskless: no server
Trying to mount root from ufs:/dev/md0 []...
2019-02-22T09:44arc4random: no preloaded entropy cache
:02.524970+00:00 init 26 - - login_getclass: unknown class 'daemon'
arc4random: no preloaded entropy cache
add net default: gateway 1.1.1.254
fstab: /etc/fstab:0: No such file or directory
uhub1: 4 ports with 4 removable, self powered
random: unblocking device.
/dev/md1: 2048.0MB (4194304 sectors) block size 32768, fragment size 4096
using 4 cylinder groups of 512.03MB, 16385 blks, 65664 inodes.
super-block backups (for fsck_ffs -b #) at:
192, 1048832, 2097472, 3146112
newfs: Cannot retrieve operator gid, using gid 0.
uhub0: 2 ports with 2 removable, self powered

ugen1.2: <vendor 0x0438 product 0x7900> at usbus1
igb0: link state changed to UP
- 245 MB 2074 kBps 02m01s
vfs.root.mountfrom="ufs:/dev/md1"
Trying to mount root from ufs:/dev/md1 []...
/etc/rc: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
Setting hostuuid: b1161b13-3686-11e9-acda-000db9457ad4.
Setting hostid: 0x123814de.

eval: cannot open /etc/fstab: No such file or directory

(...)

Fri Feb 22 09:46
FreeBSD/amd64 (poudriere-image) (ttyu0)

login: root
Feb 22 09:46:53 poudriere-image login[1023]: ROOT LOGIN (root) ON ttyu0
FreeBSD 12.0-RELEASE-p3 GENERIC

Welcome to FreeBSD!
(...)

Edit /etc/motd to change this login announcement.
root@poudriere-image:~ # df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md1 1.9G 1.2G 611M 67% /
devfs 1.0K 1.0K 0B 100% /dev
root@poudriere-image:~ #