mercredi, septembre 16, 2015

Recipe for building a 10Mpps FreeBSD based router

First you need a server, with a minimum of 8 cores and a good NIC.
My setup is this one:

  • HP ProLiant DL360p Gen8: 8 cores Intel Xeon E5-2650 @ 2.60GHz
  • Quad port 10 Gigabit Chelsio TS540-CR

The first step is to entering the UEFI and disabling Hyper-Threading.
(note to myself: Need to generate benchmark comparing HT impact for a router use)

Once done, you can install a FreeBSD on it… but not a classical 10.2!

Default behavior of FreeBSD multi-queue NIC drivers is to create a number of queue equal to number of core (with a maximum number of 16 for Chelsio).
This mean for a 8 cores server, it will create 8 queues:
  • Each queue will obtain its own IRQ
  • The NIC will load-balance in/out frames between these queues
The NIC load-balance algorithm kepts same flow on the same queue by default: Then you need lot's of differents flow (different src/dst IP addresss or TCP/UDP ports) for a correct distribution among all theses queues: Don't bench your setup with only one FTP flow as example.

And FreeBSD meet a problem here because the number of queue/core didn't scale well after 4 cores:

=> On this 8 cores setup, you need to reduce your NIC queue number to 4 for the best performance.

But recently this problem was resolved by Alexander V. Chernikov (melifaro) on its experimental projects/routing branchs (lot's of cleaning regarding fine locking and testing new ideas):

Well… almost resolved: We see a big improvement and perfect linear scale up to 4 cores but still not linear to 8 cores. 
And surprisingly this non-linear problem isn't related to the improvement in forwarding code, but to the new random entropy harvester brings recently in head that is collecting first 2 bytes of each frame under single mutex.
Disabling INTERRUPT and NET_ETHER entropy sources (by adding harvest_mask="351" in /etc/rc.conf) solve the problem:

=> 9.5Mpps on this setup !

And for non-network people who didn't understand value in "paquet-per-second", here is a different graph regarding impact on forwarding performance with ipfw or pf enabled. With the equivalent IMIX on the right side:

Now how to reach 10Mpps? Just use a little more powerfull CPU ;-)

And if you want to test these new performance on your hardware you just need one USB flash disk and installing on it these BSD Router Project (nanobsd) EXPERIMENTAL images used for theses benchs. It's just a dd to the USB flash disk, and more installation instructions are on BSDRP web site.

mercredi, juin 24, 2015

Serial-PXE-TFTP install of FreeBSD(BSDRP,Xsense,NAS4Free)/OpenBSD/Centos


Remote  installation of multiples Operating systems using only:
  • FreeBSD server with a PXE and TFTP services
  • Serial console: IPMI Serial-over-LAN (sol)

I didn't found an easy way for PXE+TFTP (only!) serial remote installation for NetBSD or DragonFly.
FreeBSD was very complex too (need to recompile bootloader for TFTP and serial usage), but hopefully mfsBSD hides this problem.
OpenBSD and CentOS, by providing ramdisk natively and easy way of configuring their bootloader, were the most admin-friendly.


This step will install an all-in-once DHCP/TFTP server:
pkg install dnsmasq
Then, create a small configuration file (example with "bce1" as NIC and local subnet in
cat > /usr/local/etc/dnsmasq.conf <<EOF
pxe-service=x86PC, "pxelinux", pxelinux

And start it:
sysrc dnsmasq_enable=yes
service dnsmasq start


This step will install pxelinux binaries and configure PXE menu:
mkdir /tftpboot
cd /tftpboot
unzip -d syslinux
cp syslinux/bios/memdisk/memdisk /tftpboot
cp syslinux/bios/core/pxelinux.0 /tftpboot
cp syslinux/bios/com32/elflink/ldlinux/ldlinux.c32 /tftpboot
cp syslinux/bios/com32/menu/menu.c32 /tftpboot
cp syslinux/bios/com32/libutil/libutil.c32 /tftpboot
cp syslinux/bios/com32/modules/pxechn.c32 /tftpboot
cp syslinux/bios/com32/lib/libcom32.c32 /tftpboot

cp syslinux/bios/com32/chain/chain.c32 /tftpboot
cp syslinux/bios/com32/modules/reboot.c32 /tftpboot/
rm -rf syslinux
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default <<EOF

SERIAL 0 115200
UI menu.c32
LABEL freebsd
 MENU LABEL mfsbsd (FreeBSD, pfSense, BSDRP, NAS4Free, etc...)
 KERNEL memdisk
 APPEND initrd=/mfsbsd-10.1-RELEASE-amd64.img harddisk raw
LABEL openbsd
 KERNEL pxechn.c32
 APPEND ::/openbsd/pxeboot
LABEL netbsd
 KERNEL pxechn.c32
 APPEND ::/netbsd/pxeboot_ia32_com0.bin
LABEL centos
 MENU LABEL Centos 7
 kernel centos/vmlinuz
 append initrd=centos/initrd.img method= devfs=nomount ip=dhcp console=ttyS0,115200 earlyprint=serial,ttyS0,115200
LABEL local
 MENU LABEL local disk
 KERNEL chain.c32

LABEL reboot
 MENU LABEL reboot
 KERNEL reboot.c32


Download mfsBSD image and enable serial port:
fetch -o /tftpboot/mfsbsd-10.1-RELEASE-amd64.img
mdconfig -a -t vnode -f mfsbsd-10.1-RELEASE-amd64.img
mount /dev/md0a /mnt/
echo "-S115200 -h" > /mnt/boot.config

umount /mnt
mdconfig -d -u 0


Download OpenBSD's pxeboot and RamDisk image, then enable serial port:
mkdir /tftpboot/openbsd/
fetch -o /tftpboot/openbsd/pxeboot
fetch -o /tftpboot/openbsd/bsd.rd
mkdir /tftpboot/etc
cat > /tftpboot/etc/boot.conf <<EOF
stty com0 115200
set tty com0
boot tftp:/openbsd/bsd.rd


Download CentOS kernel and RamDisk:
mkdir /tftpboot/centos
fetch -o /tftpboot/centos/initrd.img
fetch -o /tftpboot/centos/vmlinuz

Installing BSDRP, pfSense, OPNsense, NAS4Free, or any nanoBSD

From mfsbsd, just dd their serial nanobsd/embedded image to the local hard drive.
For installing FreeBSD: just uses bsdinstall

Debugging PXE/TFTP process

From the server, start a tcpdump accepting only bootps and tftp packets:
tcpdump -ni bce1 -vv port bootps or port tftp