_____                   _                  _____            _____       _ 
  |     |___ _____ ___ _ _| |_ ___ ___ ___   |  _  |___ ___   | __  |___ _| |
  |   --| . |     | . | | |  _| -_|  _|_ -|  |     |  _| -_|  | __ -| .'| . |
  |_____|___|_|_|_|  _|___|_| |___|_| |___|  |__|__|_| |___|  |_____|__,|___|
  a newsletter by |_| j. b. crawford               home archive subscribe rss

>>> 2023-02-13 my homelab (PDF)

I have always found the term "homelab" a little confusing. It's a bit like the residential version of "on-premises cloud," in that it seems to presuppose that a lab is the normal place that you find computer equipment. Of course I get that "homelab" is usually used by those who take pride in the careful workmanship of their home installation, and I am not one of those people.

Welcome to Computers Are Bad - in color.

Note: if you get this by email, the images may or may not work right. We're going to find out together! I don't plan to make a habit of including images and they don't look that good anyway, so I'm not too worried about it.

closet rack

They say that necessity is the mother of invention, but I think often mere desire will suffice, and I am sort of particular about how I want things to work. Perhaps the bigger problem is that I started my career in technology in a way that was both mundane and hands-on: in high school I found a poorly paying job as a sort of technical jack-of-all-trades for a local managed service provider (MSP). The term MSP is not even that familiar to many in the technology industry today. This was the kind of company that would set up and maintain Microsoft Active Directory for businesses that were big enough to have ten computers but not big enough to have an IT department. The owner, though, was a wheeler-dealer if I ever knew one, and generally jumped into whatever line of business he thought would make some money.

I was hired ostensibly as a computer technician, repairing laptops as a Lenovo contract warranty service center. Then I was repairing photocopiers, then I was selling them. Not long after I was running common-spaces WiFi for a fairly large office tower (the World Trade Center... of Portland, Oregon). Along with some video surveillance installation, I developed the kind of addiction that doesn't pay well enough to be a career unless you are smart enough to go to trade school instead of a university: cabling.

And I think that's how I became the person I am today: I want computer networks to operate in as straightforward and tangible a fashion as they did in 2009. And I want a lot of cabling.

I don't have a large house, and I do have a lot of stuff. Most equipment is crammed into a 14U wall-mount rack in the upper part of the office closet. Two sets of fan grilles, in a push-pull arrangement, ventilate the top of the closet and as a bonus circulate air from the office to the laundry room. Closet shelving stands in for things that are not amenable to rackmounting, such as my "breadbox" form factor AT&T Merlin model 206 KSU. This small-business telephone system dates back to around 1985 but still operates well after a repair to the power supply. It supports 6 extensions (conveniently connected by 8P8C cabling, ethernet-compatible) and 2 outside lines, which are provided by an ATA connected to the Asterisk server I run "in the cloud." It is one of two phone systems in the house, the other being all IP.

I installed the Merlin instead of the significantly more capable, late-'90s vintage Comdial PABX I have (with voicemail!) because it is incredibly fashionable and because I love the simple logic of key systems. I do also love the Comdial for how over-the-top complicated its hybrid PABX/key system design is, complete with text messaging, but it just doesn't have the charm of a system where phones were offered in a color called Cinnabar. Unfortunately I don't have any phones in Cinnabar; they've proven very hard to find on the second-hand market.

Also on the shelf, due to lack of motivation to mount it more neatly, is a PiStar/MMDVM hotspot. While it is configured for DMR (I sometimes monitor the Southwest and New Mexico Brandmeister groups, AE5JL) I use it mostly as a POCSAG pager transmitter. A simple daemon I wrote bridges messages from MQTT to the MMDVM remote control interface, notifying me of various events like violation of the IR optical fence across the end of the driveway via the finest communications technology of the '80s: a beeper. I have started acquiring hardware to replace it with a 35 watt transmitter which will properly introduce DAPNET amateur paging to Albuquerque, but I only have so much free time and money.

closet rack

I take great pride in my work, but no one pays me for this, so I try not to consider it work. About once a year I make a sincere effort to tidy the patch cables but it never lasts.

An Arris cable modem is where The Cloud arrives in my home. I am fortunate enough to have slightly faster than gigabit internet service, although I haven't bothered to set up link aggregation so it is de facto 1gbps. It's okay, the router doesn't really make 1gbps in some scenarios due to PPS performance limitations anyway. I am unfortunate enough to obtain that internet from Comcast, which means that it is expensive and the upstream only hits 45mbps on a good day. My favorite feature of this Arris modem is that no matter how many times I reset the password for the management interface I can never get back into it later. I'm pretty sure this is my fault, but cable modems are loathsome so I'll blame it on Arris anyway. The city recently completed a franchise agreement with an FTTH provider out of Texas and it is possible I will be able to get service from them inside of the next six months. Given the history of new ISPs in this area I am not holding my breath.

Because of my strident objection to Comcast's existence, for about the first six months after I bought this house I obtained my internet connection only via LTE, using a used Cradlepoint and roof-mounted diversity antennas. The performance was actually quite good at night, but it was very poor during the day. I live very close to downtown and so I assume this was determined mostly by the occupancy of the office towers. The bigger problem is that the tiny MVNO I used, on a grandfathered contract with AT&T that had exceptionally good terms, was also one person with a FedEx Office mailing addresss that was not very good at subscription management. Every couple of months the internet would stop working and I would have to call them to nag them to update the expiration date on my service plan in their provisioning system, which was of course not at all integrated with their billing system.

From the modem, bits flow downstream to a PC Engines APU4D4 SBC running Opnsense. This is one of two APU4D4s that sit side-by-side in a very tidy 1U enclosure I imported from France at a completely exorbitant price. Why I spent something like EUR 150 on getting this nicely silk-screened front panel for the APUs only to Tetris most of the rest of the equipment onto a rack shelf is a mystery to me as well.

I am mostly pretty happy with Opnsense except for all of the ways I hate it. It replaced a Unifi Security Gateway which replaced an old Sonicwall, so I figure I am at least moving upwards in usability. My favorite thing about Opnsense is that it brings me the warm comfort of using BSD. My least favorite thing about it is how many clicks it takes to get to the DHCP lease table, which I am constantly looking at because I do not keep the internal DNS records up to date at all.

The core switch is a TP-Link 24-port PoE switch. It's Omada-manageable, along with a couple of other TP-Link switches elsewhere in the house, and I figure I will eventually buy into Omada when I get tired of mapping VLANs by hand. This switch does have fans but is very quiet, an impressive feat in a PoE switch. I am only using around 50W of its 250W capacity, if I ever go for that PoE++ troffer lighting I like to window shop for it might end up a whole lot louder. Currently the PoE load is mostly the result of infrared illuminators in exterior surveillance cameras. The SFP cages will be much appreciated when I finally lose my mind and run fiber to the shed.

Next to the router, the second APU4D4 runs Pihole, Home Assistant, and Plex Media Server in Docker containers. I run Plex in a docker container because they only build it for ARM as a Debian package, and I'm a Red Hat person. Well, Red Hat in the streets, Fedora, erm, at home. It's also a Tailscale subnet router, although I haven't really bought into Tailscale that much yet and still have a lot of manually-configured Wireguard tunnels.

Home Assistant is perhaps the most complicated thing here. I am not as bought into Home Assistant as I maybe should be, and so I make extensive use of various homegrown services that speak MQTT. I have, at times, been tempted to improve performance and "simplify" (for select definitions of "simplify") by writing my own simple logic engine to implement automations, but I'd probably just end up creating a bad version of Home Assistant with fewer features. A chintzy USB Z-wave stick is a major bridge to the Real World, and I am particularly fond of the Zooz multi-relays as a practical way to handle various physical inputs and outputs. A Philips Hue hub tidily slapped on the side of the rack controls most lighting, though, besides a few Z-wave wall dimmers for integral LED fixtures.

My latest home automation achievement is something I call "Giant Voice" after the historic Altec outdoor address system once popular on military bases. It receives simple commands via MQTT and plays back audio clips and speech synthesis via Microsoft Azure Cognitive Services Speech (a Microsoft product name if I have ever seen one). So it's sort of like a doorbell, and basically functions as one, except it plays clips of Star Trek computer beeps and announces which part of my small lot a visitor has intruded on. It's not at all reliable because, for reasons of being built out of things I had on hand, it's running on a Pi Zero W connected to a cheap Bluetooth speaker. Trying to keep a reliable connection to a Bluetooth audio sink on Linux without X running may actually be impossible.

Pihole forms part of a split-horizon DNS arrangement on the top-level domain I use, which is such a nice name I made it available on FreeDNS where it is used by a dozen poorly run Minecraft servers. This introduces an interesting set of DNS hijacking and misconfiguration hazards, which I find aesthetically pleasing. Systemd-resolved machines, for example, are prone to acting up due to resolved's well-known oddities around split-horizon systems. Of course, in all truth I completely agree with Poettering that split-horizon DNS is sin, but why live if we can't sin a little?

On a rack shelf below is a 5-bay NAS made by company called Kobol that doesn't exist any more. I like it because it's a simple arrangement of an ARM SBC (running Fedora of course) with a lot of SATA controllers and yet they made an unreasonably nice aluminum enclosure for it. I use btrfs because every time I use ZFS I end up having to tune it, and for how much I appreciate the inanities of computers tuning ZFS is actually somewhere near dental surgery in my list of favorite activities. I follow btrfs development just closely enough to figure that there is about a 10% chance of massive data loss, which is why I back the entire thing up to a cloud provider. What I really want is to back it up to LTO tape, just for appearance, but LTO drives stay expensive until they're several generations old and I have a hard time getting excited about LTO7 when I know that LTO9 exists.

One day the NAS will probably die or I will get annoyed with how slow it is CPU-wise, but I really don't know what I'll do to replace it. Maybe the NVR is an omen of things to come.

And right, the NVR, or network video recorder, which records the surveillance cameras. It's a small-form-factor Dell workstation I bought used off a friend to replace a failed NUC. Neither the NUC nor it have reasonable internal storage capacity (on account of their small size), so it has most of its storage in a Startech 2-bay USB3.0 enclosure that I am surprisingly in love with. It's fast and reliable, and has no-fuss RAID0/1 in hardware. It even comes apart to install the drives in a pleasing way. It has 8TB of storage which is enough for around a month of history. I do have 2TB of SSD storage in the NVR which is used for live recording so that a less performance-sensitive batch job can move older recordings to the slow platter drives in the enclosure.

When it comes to software, the NVR runs a commercial package called Blue Iris on Windows. I am not particularly interested in defending this choice, other than to explain that I have been using Blue Iris for years. Well, I will be a little argumentative. Open-source NVR packages suck. All of them are just incredibly bad. For some reason all of the replacements for Zoneminder either almost single-mindedly target Raspberry Pis with barely the performance for a single UHD camera or are nodejs monstrosities. Most are both. If you get cameras on the cheap and sometimes from surplus auctions like I do, you need support for a lot of video and PTZ protocols, and Blue Iris is mature enough to have out-of-the-box support for every bit of hardware I've come up with. It has both a reasonably good web interface and the ability to run the full desktop console remotely. Although it's not open source, it has simple but functional HTTP and MQTT APIs that have made it easy to integrate with my broader tangled mess, and CodeProject AI server support for object classification to boot. It definitely seems like there should be a suitable open-source replacement at this point but I just haven't found one. Maybe growing up on Milestone VMS just ruined my taste the way growing up on Perl did.

Jammed below the NVR and next to its drive enclosure is a NUC. This is the warranty replacement for the one that failed. There's a whole story here, I wasn't expecting to get a warranty replacement, but then it showed up in the mail. I hooked it up so that I can WoL it when needed to run longer, more performance-intensive tasks like video encoding that I don't want to have to keep my laptop plugged in for. In this regard it replaces my old laptop, which used to be shoved into the rack with its screen always on for some reason.

Also sharing the lower rack shelf is an HDHomeRun TV tuner cabled to a nice active antenna on the roof. Would you believe that I can get some 60 channels of infomercials and televangelism, completely free? My favorite part is just how heavily compressed it all is, now that DTV broadcasters realized they can cram something like eight SD channels onto one carrier. There's also a Davis WeatherLinc back there somewhere, it's sort of an IP gateway for Davis Vantage weather instruments also mounted on the roof. A small service I wrote on the Home Assistant machine loads data from it into Prometheus for use elsewhere. There's also a second, separate wireless weather instrument system elsewhere in the house that also goes into Prometheus. That one is by Ecowitt and it's just for temperature and soil moisture sensors in the small heated greenhouse (Home Assistant controls the heater and irrigation via Z-Wave).

At the bottom of the rack is a not-great-but-okay Cyberpower UPS. I have a slight bias against Cyberpower because another of their products I own has twice taken down the computer plugged into it due to what seemed to be a software bug that could only be resolved by leaving it unplugged for long enough for the battery to die... a long time since it stops producing output in that state. Admittedly it's done this twice in about five years and that issue hasn't stopped me from buying a new battery for it occasionally. This rackmount one doesn't seem to have that problem, or at least hasn't so far, but it's really just the cheapest rackmount UPS I could find with readily replaceable batteries.

On the left side, a Ubiquity AP-AC-Lite. This thing, along with its compatriot in the living room, is showing its age. The problem is that I have been holding out for TP-Link to release their Omada-managed WiFi 6E AP in the US, which keeps getting pushed back. I own three of these total, and one of my favorite things about them is that one of the three is an older hardware revision that only supports 24v PPoE, and the other two support 802.3af. Guess how good I am at not mixing them up.

To facilitate all this junk, I have installed a power outlet in the closet and ethernet runs from various parts of the house and exterior. Most of the ethernet runs land at the patch panel at the top, but not all of them for reasons of laziness.

Most ethernet is run through the attic, although the extremely low overhead in the attic (due to a very shallowly pitched roof) makes many areas difficult to access. For this reason I own my friend, Mr. Longarm, a 35' telescopic fiberglass pole. I have found that a great many practical problems in cabling can be solved with the use of a long enough pole. Fiberglass pushrods and a magnet fishing set are invaluable. In some cases I have had to open sections of wall, but I try to avoid it because drywall repair becomes tedious. An inventory of "installer bits," semi-flexible drill bits several feet long, can minimize the need for opening drywall but come with hazards when used blind. Sometimes you can achieve a medium of drilling small pilot holes into each stud bay, inspecting with a borescope to locate electrical wiring and whatever else, and then driving an installer bit through several stud bays at a time. The exploratory holes are fairly quick to repair and paint.

Some aspects of my home technical infrastructure are more whimsical, or perhaps more directly reflect my personal neuroses. I have always been tremendously frustrated at the lack of time synchronization in modern clocks considering the several different technical approaches available. I run an NTP server on one of the APU4s and all of the wall clocks in the house synchronize to it. For the most part these are used/surplus clocks from Primex's now discontinued SNS series, which used to be easy to get in both battery-powered analog and mains-powered LED versions. The supply of these seems to be drying up, but the Primex OneVue series is also NTP-over-WiFi capable. Unfortunately I'm less confident that the OneVue clocks can be configured to use a local NTP server without the Primex enterprise management system, which makes them less appealing for small systems.

clock

Personally I prefer the LED versions for their over-the-top size, although unfortunately the six-digit (seconds-indicating) version seems hard to get in the larger 6" digit height option. This one, a 2.5" model in the bedroom, has had a couple of layers of neutral gray theater gel added to the lens since the lowest brightness setting will still illuminate a room in red.

I have a similar bent when it comes to "smart home" control. I find the industry's focus on phone apps and voice controls infuriating. It's nearly always faster and more convenient to press a button, but the industry as a whole has apparently deemed buttons to be too expensive. Architectural lighting controls used to universally offer "scene controllers," panels with a few buttons that each select a scene, but these are oddly hard to find in the modern home automation market. I make my own.

buttons

This is a programmable keypad scanned by a little Python program running on a cheap SBC with WiFi. Right now it actually hits the Hue controller API directly, but I have been planning for months to re-implement it to send MQTT messages instead. The most obvious (and probably best) choice for a keypad would be X-Keys, but this Genovation ControlPad is popular in warehouse and picker automation so there's a good supply of used ones on eBay. The major disadvantage to Genovation is uglier programming software and no backlighting (the X-Keys models have individually-addressable two color backlighting). I'd highly recommend everyone try these out and help bring physical buttons back to the industry. You could even make it look a lot nicer if you put in even slightly more effort than I did.

And I think that's the grand tour. I'm not sure that I would say that I am completely proud of any of this because it is all so cobbled together and I change things frequently, but that's kind of why I wanted to respond to the genre of "my homelab" or "my home network" posts. I always sort of cringe at these because the focus on aesthetics, with modified Ikea furniture or whatever, is going to make modification down the road much more difficult. There is a big advantage to the 19" rack as a form factor, and wall-mount units are easy to come by. If you're especially space-constrained you might even consider a swing-down vertical one. Whatever you do, just make sure you run a lot of cables. Cables everywhere!