Following the cable misadventures I was in need of the ability to route at gigabit speeds on my home network. I had a venerable PCEngine APU2 that was able to do about 550-650 Mbit/s firewalled, but that’s just not enough. Exacerbating the issue is the fact that currently my main workstation has VMs that straddle multiple subnets, so bulk file-transfer traffic that should be switched (L2) is instead moving through the router (L3). In a perfect world I’d have an L2/L3 switch, but I haven’t gotten around to it yet.

Given that I was relatively happy with pfSense CE I had some idea that I wanted to build another BSD-based router on an x86_64 platform. I had been looking at potentially kitting-out an Optiplex with some half-height PCIe NICs, but frankly I didn’t know where I would put it. While it’s nominally a small form factor, it still takes up quite a lot of desk space, and in my experience they usually have relatively cheap (read: inefficient) PSUs on the order of about 400 watts. — If I was going to use such a beast it might as well be rack mountable, but that rapidly exits the realm of “quiet enough to live in my bedroom.”

While doing some research I stumbled across this awesome video from ServeTheHome where he talks about running Proxmox on these ultra-mini PCs. These things can be had for ~$200-300, cheaper if you buy them from the source (aliexpress) instead of drop-shipped from Amazon. I settled on a unit from TOPTON with the following specs:

  • Intel Celeron N5105 (4C4T @ 2GHz)
  • 8GB RAM
  • 4x Intel I225 NICs (2.5 GbE)
  • 128GiB NVMe storage (Some no-name SSD, don’t forget your backups!)

This thing also has USB 3.0 (3.1?), HDMI, room for a SIM/WWAN backup, and one of those goofy RJ45 console ports. (Thankfully I did not have to use the console port because I have no clue where all my serial adapter hardware got off to!) — This thing is awesome.

I had mine shipped with a pfSense CE image already on it, mostly out of curiosity, but I plan to replace it with Proxmox and OPNSense in the future. Something I was not expecting was that you can do PCIe passthrough on this device. I really did not expect “Celeron”-branded silicon to support that, but I guess because of the relatively unique mainboard layout the usual problems with Linux/ACS/IOMMU groups do not rear their ugly head. (Still the fact that this has virtualization extensions at all is a pleasant surprise to me. It should surprise no one that I am not Intel’s biggest fan.),

This thing booted up to a console no problem, I have just two gripes with this hardware:

  1. The power button has a bright blue LED
  2. It has a wall wart which also has a bright blue LED. (WHYYY?)

I restored to factory settings and inadvertendly locked myself out of the console - what I didn’t realize is that pfSense did not seem to output to the HDMI port by default, I guess that was part of the configuration they did at the factory. (That or it was using the “dual console” mode, which may not have functioned correctly on this unit.) Thankfully the pfSense guys thought of this and at the boot splash you can override this w/ a kernel command line flag.

After that I restored my old pfSense config which had some issues: it wanted to reassign my interfaces, which worked fine, but when it came to reassigning VLANs it would not remember that they were slaves to the new interface. In the end I had to drop my old VLANs, recreate my VLANs on the new interface and only then was I able to go through the reassignment wizard without issue.

It was at this time I realized my new router was not configuring for IPv6. Naturally I assumed it was something I did wrong, so I tried restoring the old DUID, generating a new DUID, etc. but nothing worked. In the end, per my blog post on Cable Misadventures, I realized it was in fact my new SB8200 modem that was at fault.

Apart from that it was fairly uneventful, some things I noted:

  • pfSense is running on ZFS now, which is great for me because my entire backup infrastructure is based around ZFS send/recv.

  • this box runs fairly hot, about 48-50c in a house that’s nominally 21c.

  • the heat rejection on this box is far better than the APU2, it is incredibly warm to the touch. the heatsink is definitely not just for show ;-)

  • I haven’t bothered to test at 2.5GbE speeds yet, but I’m easily able to saturate my gigabit uplink to my switch now.

Also you may remember I had to move my cable modem, and here at hime-enterprises we love cursed network setups, so for your enjoyment I present my current toplogy:

  • MB8611 cable modem
    • (VLAN 4) Unifi USW8 switch
      • WiFi AP (SSID on VLAN2)
      • WiFi AP (SSID on VLAN3)
      • Printer (VLAN2)
      • PCs (VLAN2)
      • (Trunk) Uplink to USW24 switch
        • (VLAN 4) Router WAN side
        • (VLAN 1-3) Router LAN side
        • PCs / Servers / etc on VLAN2

For context:

  • VLAN1: management net
  • VLAN2: resident net
  • VLAN3: guest/iot net
  • VLAN4: (new!) cable gateway net

So for a packet from my PC to reach the internet it goes:

PC -> USW24 -> Router[LAN] -> Router[WAN] -> USW24 -> USW8 -> Cable GW

… and of course the reverse is also true for the response packet(s).

Yeah… my latency is not great. I could move the router downstairs as well, but I’ve officially entered the “stubborn old fool” phase of my existence. Also I don’t wanna purchase a second battery backup for downstairs. >:(