vendredi, février 22, 2019

PXE booting of a FreeBSD disk image


I had to set up a regression and network performance lab. This lab will be managed by a Jenkins, but the first step is to understand how to boot a FreeBSD disk by PXE. This article explains a simple way of doing it.
For information, all these steps were done using 2 PC Engines APU2 (upgraded with latest BIOS for iPXE support), so it's a headless (serial port only, this can be IPMI SoL with different hardware) .

The big picture

Before explaining all steps and command line, here is the full big picture of the final process (more readable SVG version of this file):
FreeBSD PXE boot steps

And the tasks we will do:

  1. Creating image-miniroot and image.txz, with the help of poudriere
  2. Setting up a DHCP (dnsmasq), TFTP (FreeBSD) and FTP (FreeBSD) server
  3. Populating the TFTP and FTP server
  4. Configuring the DHCP server
  5. Test the result
Notice in my lab, the server is configured with IP and the DHCP range will be between .1 and .10.


Creating images

To create images we had to do:
  1. Install poudriere
  2. Configure it (I don't have ZFS on my small APU2, so disable it)
  3. Create a poudriere jail of a FreeBSD 12.0-RELEASE
  4. Configure custom configuration file we want on the image
  5. Generate the poudriere images (main and miniroot)
These commands will do it:

pkg install -y poudriere-devel
echo "NO_ZFS=yes" >> /usr/local/etc/poudriere.conf
echo "FREEBSD_HOST=" >> /usr/local/etc/poudriere.conf
poudriere jail -c -j 120amd64 -v 12.0-RELEASE -K GENERIC
mkdir -p ~/miniroot-overlay/boot
echo 'console="comconsole"' >> ~/miniroot-overlay/boot/loader.conf
mkdir -p ~/miniroot-overlay/etc
cat >~/miniroot-overlay/etc/rc <<EOF
# Reusing data from the pxeboot loader to configure network
ifconfig \$(kenv inet \$(kenv boot.netif.ip) netmask \$(kenv boot.netif.netmask) up
route add default \$(kenv boot.netif.gateway)
# Need to remount in read-write: Can't use uzip compressed image (read-only)
mount -uw /
mkdir /newroot
# An empty 12.0 base installation (no ports) consumme 1.2G
md=\$(mdconfig -s 2g)
newfs \$md
mount /dev/\$md /newroot
fetch -o - ftp://\$(kenv boot.tftproot.server)/image.txz | bsdtar -xpf - -C /newroot
umount /newroot
kenv vfs.root.mountfrom=ufs:/dev/\$md
# reboot -r needs tmpfs.ko loaded
reboot -r
mkdir -p ~/image-overlay/boot
echo 'console="comconsole"' >> ~/image-overlay/boot/loader.conf
mkdir -p ~/image-overlay/etc
cat >~/image-overlay/etc/rc.conf <<EOF
# IP configuration and routes will be preserved from the miniroot state
# But configure it as DHCP in case of an 'service netif restart'
# You need to install your SSH keys
# Avoid "My unqualified host name (poudriere-image) unknown; sleeping for retry"
# Hostname will be added by poudriere image here:
poudriere image -j 120amd64 -t tar -n image -m ~/miniroot-overlay -c ~/image-overlay/

The last 2 lines from poudriere should be:

Image `/usr/local/poudriere/data/images//image-miniroot' complete
Image available at: /usr/local/poudriere/data/images/image.txz

We will move these files later.

TFTP server

Now let's:
  1. Enable TFTPD and inetd
  2. Populate the directory with pxeboot, lua scripts, kernel, custom boot/loader.conf and unziped image-miniroot
These commands will do it:

sed -i "" -e 's/^#tftp/tftp/g' /etc/inetd.conf
sysrc inetd_enable="YES"
mkdir -p /tftpboot/boot
mkdir -p /tftpboot/kernel
cp /usr/local/poudriere/jails/120amd64/boot/pxeboot /tftpboot
cp -r /usr/local/poudriere/jails/120amd64/boot/lua /tftpboot/boot
cp -r /usr/local/poudriere/jails/120amd64/boot/defaults /tftpboot/boot
cp /usr/local/poudriere/jails/120amd64/kernel/kernel /tftpboot/kernel
cp /usr/local/poudriere/jails/120amd64/kernel/tmpfs.ko /tftpboot/kernel
cat > /tftpboot/boot/loader.conf <<EOF
# Disable menu
# Enable serial console only
# tmpfs is needed by reboot -r
# Download an md_image and use it as root fs
mv /usr/local/poudriere/data/images/image-miniroot.gz /tftpboot
cd /tftpboot
gunzip image-miniroot.gz
service inetd start

Check your TFTP server is correctly able to serve our files:

tftp localhost
tftp> get pxeboot
Received 436224 bytes during 0.1 seconds in 853 blocks
tftp> quit

FTP server

Now let's:
  1. Enable anonymous FTP server (by creating 'ftp' account)
  2. Move image.txz into /home/ftp
These commands will do it:

sysrc ftpd_enable=YES
echo "ftp::::::FTP anonymous::/usr/sbin/nologin" | adduser -f -
mv /usr/local/poudriere/data/images/image.txz /home/ftp/
service ftpd start

Check your FTP server is correctly able to serve this file:

ftp ftp://anonymous:nobody@localhost
Trying ::1:21 ...
Connected to localhost.
220 FTP server (Version 6.00LS) ready.
331 Guest login ok, send your email address as password.
230 Guest login ok, access restrictions apply.
Remote system type is UNIX.
Using binary mode to transfer files.
200 Type set to I.
ftp> get image.txz
local: image.txz remote: image.txz
229 Entering Extended Passive Mode (|||61982|)
150 Opening BINARY mode data connection for 'image.txz' (257213124 bytes).
100% |***********************************************************************************************| 245 MiB 26.63 MiB/s 00:00 ETA
226 Transfer complete.
257213124 bytes received in 00:09 (26.63 MiB/s)
ftp> quit
221 Goodbye.

DHCP server

The last configuration step:
  1. Install dnsmasq
  2. Configure (with the trick of generating a different answer if the request came from iPXE or from FreeBSD's pxeboot loader) and enable it
These commands will do it:

pkg install -y dnsmasq
cat >/usr/local/etc/dnsmasq.conf <<EOF
# Range of IP to distribute (mandatory to enable DHCP server)
# TFTP server name
# Filename to download
# Magic trick to detect FreeBSD's pxeboot and avoid iPXE conflict
# Add tag 'fbsd' to clients using userclass 'FreeBSD':
# Reply with root-path only to 'fbsd' tagged clients:
sysrc dnsmasq_enable=YES
service dnsmasq start

Final test

Now time to power up a PXE client (still a PC Engine APU2):

Booting from ROM...
iPXE (PCI 00:00.0) starting execution...ok
iPXE initialising devices...ok

iPXE 1.0.0+ (f8e167) -- Open Source Network Boot Firmware --

---------------- iPXE boot menu ----------------

ipxe shell

net0: 00:0d:b9:45:7a:d4 using i210-2 on PCI01:00.0 (open)
[Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net0 00:0d:b9:45:7a:d4)...... ok
net0: gw
Next server:
Filename: pxeboot
tftp:// ok

pxeboot : 436224 bytes [PXE-NBP]
PXE Loader 1.00

Building the boot loader arguments
Relocating the loader and the BTX

Starting the BTX loader

\Loading /boot/loader.conf.local
Loading kernel...
/boot/kernel/kernel text=0x1678aa8 data=0x1cd288+0x768b40 syms=[0x8+0x174cd8+0x8+0x19224a]
Loading configured modules...
/image-miniroot size=0xb00000
/boot/kernel/tmpfs.ko size 0x10c70 at 0x313d000

can't find '/boot/entropy'
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
nfs_diskless: no server
Trying to mount root from ufs:/dev/md0 []...
2019-02-22T09:44arc4random: no preloaded entropy cache
:02.524970+00:00 init 26 - - login_getclass: unknown class 'daemon'
arc4random: no preloaded entropy cache
add net default: gateway
fstab: /etc/fstab:0: No such file or directory
uhub1: 4 ports with 4 removable, self powered
random: unblocking device.
/dev/md1: 2048.0MB (4194304 sectors) block size 32768, fragment size 4096
using 4 cylinder groups of 512.03MB, 16385 blks, 65664 inodes.
super-block backups (for fsck_ffs -b #) at:
192, 1048832, 2097472, 3146112
newfs: Cannot retrieve operator gid, using gid 0.

uhub0: 2 ports with 2 removable, self powered

ugen1.2: <vendor 0x0438 product 0x7900> at usbus1
igb0: link state changed to UP
- 245 MB 2074 kBps 02m01s
Trying to mount root from ufs:/dev/md1 []...

/etc/rc: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
Setting hostuuid: b1161b13-3686-11e9-acda-000db9457ad4.
Setting hostid: 0x123814de.

eval: cannot open /etc/fstab: No such file or directory


Fri Feb 22 09:46
FreeBSD/amd64 (poudriere-image) (ttyu0)

login: root
Feb 22 09:46:53 poudriere-image login[1023]: ROOT LOGIN (root) ON ttyu0

Welcome to FreeBSD!

Edit /etc/motd to change this login announcement.
root@poudriere-image:~ # df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md1 1.9G 1.2G 611M 67% /
devfs 1.0K 1.0K 0B 100% /dev
root@poudriere-image:~ #