Recipe for building a 10Mpps FreeBSD based router

First you need a server, with a minimum of 8 cores and a good NIC.
My setup is this one:

• HP ProLiant DL360p Gen8: 8 cores Intel Xeon E5-2650 @ 2.60GHz
• Quad port 10 Gigabit Chelsio TS540-CR

The first step is to entering the UEFI and disabling Hyper-Threading.

(note to myself: Need to generate benchmark comparing HT impact for a router use)

Once done, you can install a FreeBSD on it… but not a classical 10.2!

Default behavior of FreeBSD multi-queue NIC drivers is to create a number of queue equal to number of core (with a maximum number of 16 for Chelsio).

This mean for a 8 cores server, it will create 8 queues:

• Each queue will obtain its own IRQ
• The NIC will load-balance in/out frames between these queues

The NIC load-balance algorithm kepts same flow on the same queue by default: Then you need lot's of differents flow (different src/dst IP addresss or TCP/UDP ports) for a correct distribution among all theses queues: Don't bench your setup with only one FTP flow as example.

And FreeBSD meet a problem here because the number of queue/core didn't scale well after 4 cores:

=> On this 8 cores setup, you need to reduce your NIC queue number to 4 for the best performance.

But recently this problem was resolved by Alexander V. Chernikov (melifaro) on its experimental projects/routing branchs (lot's of cleaning regarding fine locking and testing new ideas):

Well… almost resolved: We see a big improvement and perfect linear scale up to 4 cores but still not linear to 8 cores. 

And surprisingly this non-linear problem isn't related to the improvement in forwarding code, but to the new random entropy harvester brings recently in head that is collecting first 2 bytes of each frame under single mutex.

Disabling INTERRUPT and NET_ETHER entropy sources (by adding harvest_mask="351" in /etc/rc.conf) solve the problem:

=> 9.5Mpps on this setup !

And for non-network people who didn't understand value in "paquet-per-second", here is a different graph regarding impact on forwarding performance with ipfw or pf enabled. With the equivalent IMIX on the right side:

Now how to reach 10Mpps? Just use a little more powerful CPU ;-)

And if you want to test these new performance on your hardware you just need one USB flash disk and installing on it these BSD Router Project (nanobsd) EXPERIMENTAL images used for theses benchs. It's just a dd to the USB flash disk, and more installation instructions are on BSDRP web site.

Serial-PXE-TFTP install of FreeBSD(BSDRP,Xsense,NAS4Free)/OpenBSD/Centos

Objectives

Remote  installation of multiples Operating systems using only:

• FreeBSD server with a PXE and TFTP services
• Serial console: IPMI Serial-over-LAN (sol)

I didn't found an easy way for PXE+TFTP (only!) serial remote installation for NetBSD or DragonFly.
FreeBSD was very complex too (need to recompile bootloader for TFTP and serial usage), but hopefully mfsBSD hides this problem.
OpenBSD and CentOS, by providing ramdisk natively and easy way of configuring their bootloader, were the most admin-friendly.

dnsmasq

This step will install an all-in-once DHCP/TFTP server:
pkg install dnsmasq
Then, create a small configuration file (example with "bce1" as NIC and local subnet in 192.168.1.0/24)
cat > /usr/local/etc/dnsmasq.conf <<EOF
interface=bce1
dhcp-range=192.168.1.80,192.168.1.85
pxe-service=x86PC, "pxelinux", pxelinux
enable-tftp
tftp-root=/tftpboot
EOF

And start it:
sysrc dnsmasq_enable=yes
service dnsmasq start

pxelinux

This step will install pxelinux binaries and configure PXE menu:

mkdir /tftpboot
cd /tftpboot
fetch https://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.zip
unzip -d syslinux syslinux-6.03.zip
cp syslinux/bios/memdisk/memdisk /tftpboot
cp syslinux/bios/core/pxelinux.0 /tftpboot
cp syslinux/bios/com32/elflink/ldlinux/ldlinux.c32 /tftpboot
cp syslinux/bios/com32/menu/menu.c32 /tftpboot
cp syslinux/bios/com32/libutil/libutil.c32 /tftpboot
cp syslinux/bios/com32/modules/pxechn.c32 /tftpboot
cp syslinux/bios/com32/lib/libcom32.c32 /tftpboot
cp syslinux/bios/com32/chain/chain.c32 /tftpboot
cp syslinux/bios/com32/modules/reboot.c32 /tftpboot/
rm syslinux-6.03.zip
rm -rf syslinux
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default <<EOF
SERIAL 0 115200
CONSOLE 0
UI menu.c32
TIMEOUT 300
MENU TITLE PXE BOOT MENU
LABEL freebsd
 MENU DEFAULT
 MENU LABEL mfsbsd (FreeBSD, pfSense, BSDRP, NAS4Free, etc...)
 KERNEL memdisk
 APPEND initrd=/mfsbsd-10.1-RELEASE-amd64.img harddisk raw
LABEL openbsd
 MENU LABEL OpenBSD
 KERNEL pxechn.c32
 APPEND ::/openbsd/pxeboot
LABEL netbsd
 MENU LABEL NetBSD
 KERNEL pxechn.c32
 APPEND ::/netbsd/pxeboot_ia32_com0.bin
LABEL centos
 MENU LABEL Centos 7
 kernel centos/vmlinuz
 append initrd=centos/initrd.img method=http://mirror.centos.org/centos/7/os/x86_64/ devfs=nomount ip=dhcp console=ttyS0,115200 earlyprint=serial,ttyS0,115200
LABEL local
 MENU LABEL local disk
 KERNEL chain.c32
 APPEND hd0
LABEL reboot
 MENU LABEL reboot
 KERNEL reboot.c32
EOF

FreeBSD

Download mfsBSD image and enable serial port:

fetch -o /tftpboot/mfsbsd-10.1-RELEASE-amd64.img http://mfsbsd.vx.sk/files/images/10/amd64/mfsbsd-10.1-RELEASE-amd64.img
mdconfig -a -t vnode -f mfsbsd-10.1-RELEASE-amd64.img
mount /dev/md0a /mnt/
echo "-S115200 -h" > /mnt/boot.config
umount /mnt
mdconfig -d -u 0

OpenBSD

Download OpenBSD's pxeboot and RamDisk image, then enable serial port:
mkdir /tftpboot/openbsd/
fetch -o /tftpboot/openbsd/pxeboot http://ftp.openbsd.org/pub/OpenBSD/5.7/amd64/pxeboot
fetch -o /tftpboot/openbsd/bsd.rd http://ftp.openbsd.org/pub/OpenBSD/5.7/amd64/bsd.rd
mkdir /tftpboot/etc
cat > /tftpboot/etc/boot.conf <<EOF
stty com0 115200
set tty com0
boot tftp:/openbsd/bsd.rd
EOF

CentOS

Download CentOS kernel and RamDisk:
mkdir /tftpboot/centos
fetch -o /tftpboot/centos/initrd.img ftp://ftp.free.fr/mirrors/ftp.centos.org/7.1.1503/os/x86_64/images/pxeboot/initrd.img
fetch -o /tftpboot/centos/vmlinuz ftp://ftp.free.fr/mirrors/ftp.centos.org/7.1.1503/os/x86_64/images/pxeboot/vmlinuz

Installing BSDRP, pfSense, OPNsense, NAS4Free, or any nanoBSD

From mfsbsd, just dd their serial nanobsd/embedded image to the local hard drive.
For installing FreeBSD: just uses bsdinstall

Debugging PXE/TFTP process

From the server, start a tcpdump accepting only bootps and tftp packets:

tcpdump -ni bce1 -vv port bootps or port tftp

ipfw improvement on FreeBSD -current

Few days ago Alexander V. Chernikov posted on the FreeBSD -net mailing list an "HEADS UP: Merging projects/ipfw to HEAD" with lot's of promises:

• Tables are now identified by names, not numbers. There can be up to 65k tables with up to 63-byte long names.
• Tables are now set-aware (default off), so you can switch/move them atomically with rules.
• More functionality is supported (swap, lock, limits, user-level lookup, batched add/del) by generic table code.
• New table types are added (flow) so you can match multiple packet fields at once.
• Ability to add different type of lookup algorithms for particular table type has been added.
• New table algorithms are added (cidr:hash, iface:array, number:array and flow:hash) to make certain types of lookup more effective.
• Table value are now capable of holding multiple data fields for different tablearg users

I'm not an expert of ipfw(8), but I would check the impact of this improved-ipfw on forwarding performance. By "performance" I mean how this code impact the throughput (in term of packet-per-second) of my FreeBSD firewall (I didn't bench all the parameters requiered by RFC3511).
Once the code was committed as r272840 on -head, I've generated a new nanobsd(8) image on my 10gigabit bench lab… and here are the results:

More than 100K pps of differences! Now I dream of an ipfw_sync equivalent to pf_sync(4).
And here are the ministat output for statistician extremists.
Regarding ipfw in stateless mode:


x 272685.ipfw-stateless
+ 273009.ipfw-stateless
+----------------------------------------------------------------------+
|x      x     x    x                                  + + +      +    +|
|   |______A__M___|                                                    |
|                                                     |___M__A_____|   |
+----------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       1585928       1619817       1608891     1604564.2     12728.878
+   5       1683246       1712607       1690405     1695508.6      12250.89
Difference at 95.0% confidence
        90944.4 +/- 18219.1
        5.66786% +/- 1.13546%
        (Student's t, pooled s = 12492.2)

And regarding ipfw in statefull mode:


x 272685.ipfw-statefull
+ 273009.ipfw-statefull
+----------------------------------------------------------------------+
|xx    x   x    x                       ++   +    +                   +|
||_____A______|                                                        |
|                                    |_______M___A____________|        |
+----------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       1390415       1433678       1407058     1408663.4     18451.472
+   5       1502719       1589778       1517320     1529871.8     35404.181
Difference at 95.0% confidence
        121208 +/- 41172.4
        8.6045% +/- 2.9228%
        (Student's t, pooled s = 28230.4)

PuTTY and Solarized colors

I'm using the Solarized color palette on all my FreeBSD desktops, but at work I had to works from a MS Windows desktop :-(
Here are my PuTTY settings for a correct rendering of Solarized colors.

Softwares used

• PuTTY and it's authentication agent (There is KiTTY, an improved version of PuTTY too)
• MTPuTTY (Multi-Tabbed Putty) for adding tabbed feature to PuTTY/KiTTY
• Solarized PuTTY settings

Installation steps

Installing PuTTY or MTPuTTY didn't need specials instruction.
If you're using KiTTY, the Solarized PuTTY.reg files need to be adapted by opening them into a text editor and replacing the line:
[HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions\Solarized%20Dark]
By this one:
[HKEY_CURRENT_USER\Software\9bis.com\KiTTY\Sessions\Solarized%20Dark]
Then import the .reg files.

KiTTY/PuTTY configuration

Now start KiTTY or PuTTY, load the Dark or Light Solarized session and modify this session with:

• Window - Colours : Enable "Allow terminal to use xterm 256-colour mode"
• Connection - Data - Terminal details : Terminal-type string : "xterm-256color"

You can then add more customization, for example:

• Terminal - Bell : Visual Bell
• Window - Appearance - Font Settings - Font: "Consolas" - 12point
• Window - Appearance - Font Settings - Font quality: ClearType
• Window - Translation - Remote character set: UTF-8
• Window - Lines of scrollback: 10000
• Window - Selection - Control use of mouse: xterm (Right extends, Middle Past)
• Connection : Second between keepalives: 25
• Connection : Enable TCP keepalives
• Connection - SSH - X11: Enable X11 forwarding

Once all your customization done, save the session as "Default Settings".

Checking parameters

Start a KiTTY/PuTTY, check that your preferences are loaded by default and open a SSH session to an *nix machine.
Once logged, the command "echo $TERM" should answer "xterm-256color".
And, if you've solarized your VIM (you don't need to use let g:solarized_termcolors=256!), you should correctly see the column after entering a "set colorcolumn=80".

tmux

tmux need to be configured for advertise a 256color term by adding in ~/.tmux.conf the line:
set -g default-terminal "screen-256color"

Configuration IPv6 propre d'une Kimsufi sous FreeBSD

Pour configurer une passerelle par défaut IPv6 sur un Kimsufi, le guide officiel se résume à:

1. Paramétrer l'IPv6 de votre interface avec votre préfixe /64 (2001:41D0:1:46e::/64 par exemple)
2. Suivre la règle IP:v:6:FF:FF:FF:FF:FF pour déduire votre passerelle par défaut (dans notre exemple elle est donc 2001:41D0:1:4FF:FF:FF:FF:FF).

Sauf qu'avec cette règle l'IP de la passerelle est en dehors de votre réseau (/64)… donc injoignable !
«À ce qu'il paraît» cela ne pose pas de problème aux GNU/Linux…no comment.
Une autre section du guide propose de récupérer les RA pour trouver la route par défaut "link-local" annoncée par le routeur, mais cela ne fonctionne plus car ils ont été désactivés.
La solution la plus commune à ce problème est de paramétrer un préfixe /56 à la place du /64 sur votre interface: du coup la passerelle par défaut se trouve dans votre réseau et le problème est résolus.
Mais OVH m'a donné un /64, je ne vois pas pourquoi je lui déclarerai un /56!

Une jolie solution (soufflée par flo@) permettant de paramétrer un /64 tout en utilisant cette route par défaut est la suivante:

ifconfig_re0_ipv6="inet6 2001:41D0:1:46e::1 prefixlen 64"
ipv6_static_routes="mac"
ipv6_route_mac="-host 2001:41D0:1:4FF:FF:FF:FF:FF -iface re0"
ipv6_defaultrouter="2001:41D0:1:4FF:FF:FF:FF:FF"

Ethernet-Wifi failover on FreeBSD

I want a simple behaviour with my laptop:

1. If Ethernet cable connected use this connectivity, otherwise use the wireless;
2. I want to kept the same IP addresses, event if I'm using DHCP client.

The solution is quiet simple:

1. set-up an aggregate interface in failover mode with Ethernet as primary and wireless as backup;
2. Clone the Wireless MAC NIC to the Ethernet (opposite is not always possible with wireless chipset restriction).

Here are how to do it:

# ifconfig -l
iwn0 bge0 lo0
=> My Ethernet NIC is "bge0" and wireless is "iwn0" here

# set MAC=`ifconfig wlan0 | grep ether | cut -d ' ' -f 2`
# sysrc ifconfig_bge0="ether $MAC"
ifconfig_bge0:  -> ether 00:1c:23:25:ab:45
# sysrc wlans_iwn0=wlan0
wlans_iwn0:  -> wlan0
# sysrc ifconfig_wlan0="WPA up"
ifconfig_wlan0: WPA DHCP -> WPA up
# sysrc cloned_interfaces=lagg0
cloned_interfaces:  -> lagg0
# sysrc ifconfig_lagg0="laggproto failover laggport bge0 laggport wlan0 DHCP"
ifconfig_lagg0:  -> laggproto failover laggport bge0 laggport wlan0 DHCP
# sysrc ifconfig_lagg0_ipv6="inet6 accept_rtadv"
ifconfig_lagg0_ipv6:  -> inet6 accept_rtadv
# service netif restart

And now with Ethernet cable unplugged:

# ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 00:1c:23:25:ab:45
    inet6 fe80::41d:23ff:fe25:ab78%lagg0 prefixlen 64 scopeid 0x4
    inet6 2a01:e35:9b9d:a1a0:41d:23ff:fe25:ab45 prefixlen 64 autoconf
    inet 192.168.100.7 netmask 0xffffff00 broadcast 192.168.100.255
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: active
    laggproto failover lagghash l2,l3,l4
    laggport: wlan0 flags=4<ACTIVE>
    laggport: bge0 flags=1<MASTER>


Then If I plug the Ethernet cable:


# ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 00:1c:23:25:ab:45
    inet6 fe80::41d:23ff:fe25:ab45%lagg0 prefixlen 64 scopeid 0x4
    inet6 2a01:e35:9b9d:a1a0:41d:23ff:fe25:ab45 prefixlen 64 autoconf
    inet 192.168.100.7 netmask 0xffffff00 broadcast 192.168.100.255
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: active
    laggproto failover lagghash l2,l3,l4
    laggport: wlan0 flags=0<>
    laggport: bge0 flags=5<MASTER,ACTIVE>

Generating custom FreeBSD installation media

Objective

Generating a custom -current memstick image without all the debug feature enabled.

Prerequisite

Have the head source installed (I will use /usr/src as example).
If not, here is an example for synchronizing up-to-date head (-current) sources on /usr/src
svnlite co svn://svn.freebsd.org/base/head /usr/src

Customizing build compilation options

A little debug feature to disable on -current:
echo "MALLOC_PRODUCTION=yes" > /etc/src.conf

Building world and kernel

New we start the classical building of world and our customized kernel.
For information this step takes about 4 hours on my PC.
cd /usr/src
make buildworld; make buildkernel KERNCONF=GENERIC-NODEBUG

Generating install media image

Last step: Generating the install media.
Here is an example for generating memstick install media without port tree (long live to pkgng! neither doc):
cd /usr/src/release
make -DNOPORTS -DNODOC memstick

Replace "memstick" by "cdrom" (bootonly.iso and release.iso) or "ftp" for other media.
You can add a -DNOSRC option for avoiding to include sources too on the media.

Then copy the image to your usb key:
dd if=memstick of=/dev/da0 bs=64k

Cleaning your mess

Your system has lot's of file that you don't need anymore, here is how to clean it:
cd /usr/src/release
make clean
cd /usr/src
make clean