Now that I have time for hobbies again I wanna dive right in to building up a home lab. Right now I just have a Netgear router that reached end of life, and a NAS i’d like to make accessible remotely, and a DIY router seems like the correct next step for me. I’m very interested in ad-blocking network wide and someday I’d like to host my own little webpage, but for now I’m really just barely getting started.
I’m information gathering, I don’t know what I don’t know and my networking skills are pretty weak anyway. But even if your needs and desires have nothing to do with mine, I also just like to hear people talk about their set ups. Please? 🌞
The layout I’m designing around is inline: ISP router, then a small dual-ethernet box doing firewall and DNS filtering, then your LAN. For a start, I’d put OPNsense on a cheap dual-NIC mini PC. You get network-wide blocking through a plugin and a VPN to reach your NAS, and VLANs can wait until you need them. I’m biased, since I’m building a box with that exact layout. What I can’t settle: for someone starting out, is an all-in-one box less scary, or does it hide the stuff you’d want to learn?
OpnSense running on an ex-lease HP i3-something SFF desktop. Added several cheap half-height GigEth PCIe cards for the LAN bridge.
It is way over spec for what OpnSense needs but it was cheap and extensible.
UPS power supply.
There’s a tangle of PoE injectors which makes me sad. PoE PCIe cards appear to be ludicrously expensive here for no apparent reason.
I’m gonna ask you the same question I’m asking everyone using office mini PCs, do you know what kind of power draw? I’m really into the idea of cheap, but I’m also getting super into green energy.
No idea, I’m not monitoring power consumption at all.
I’m just assuming that the Intel CPU based system will idle at X Watts, peak as Y Watts, and that an ARM based appliance would idle at X and peak lower than X.
If any thing, I noticed that my UPS batter didn’t last as long during an extended ded outage once I deploy the SFF PC.
Also, TCO, if I save $100 on the price of the device (that draws more power) that is a lot of power savings required to break even.
Different person. But I have proxmox on multiple mini PC (Lenovo and HP600). They max out at 65w but pull less most of the time. I suppose I should check the weekly average, but my power monitor is on the other side of the UPS and there are multiple mini PC, devices, switches, modem, wireless AP… All on the monitor. If I had to guess I’d say the router/pihole pulls <30w and the media rig with its HDDs pulls ~100 but only when it’s read/writing. The full stack idles <100 including AP/switches, way less then my gaming PC just on idle (>700 when the GPU is GPing).
(Power efficiency is nice, but the ROI for a new ARM / RISK system is probably not great now. Better to use a recycled rig pulling 40-100w now and upgrading in a few years. It’s not like you need a server rack pulling over a kilowatt, and the efficiency gain from 5-10 year old x86 vs ARM/RISK is less then swapping out an old incandescent light to an LED.
Thank you that’s EXTREMELY helpful! I agree that maxing out at 65w and pulling maybe 30 the rest of the time isn’t so bad, but I really wasn’t sure what to expect.
That also means if I find I actually hate it and in a year I’m like “fuck it I’m done tinkering” I didn’t spend a ton of extra money on equipment.
OpenWRT all the way.
Skynet firewall
PiHole
Stable IP addresses and a private interface for family phones/laptops/pcs
Guest network for nasty Roku TV/peleton
I never even thought about a guest network for different services. Why do you do that, if you don’t mind? Privacy?
You could configure different settings, such as preventing the device from connecting to ad servers. You also prevent it from scanning your network to learn about all your devices, since all it would be limited to that isolated network.
Does peloton do that!? 🥲
I don’t know. Is it internet connected? If so I wouldn’t trust it…
Many home grade routers allow you to set up a guest WiFi network though so you may be able to isolate it that way if you’re concerned/can determine what telemetry it’s sending from the privacy policy or other reports.
This article talks about their website’s standard creepy tracking cookies and stuff, which depressingly everyone does. https://blog.avast.com/what-peloton-knows-about-you-avast
But if it has Bluetooth and WiFi it has a lot of potential for tracking all sorts of things. I don’t have one and don’t know if they’re using that potential, but it’s there. It would be interesting to investigate.
I only got into privacy stuff like a year ago, but I was really focused on thinking about my phone and not as much my home network (unless we’re talking like, cloud services that are also accessed on desktop, I was already worried about those). That’s a spooky but totally valid thought imo. Agree, would be a good new rabbit hole to explore.
It’s a big rabbit hole for sure, with how many household devices are internet connected and the concerning capabilities they offer. And an expensive one if you seek to replace devices at home.
Guest Network in OpenWRT prevents inter-device communication on the LAN. So the thing can talk to its cloud service or whatever but can’t see my phone/laptop/PCs.
This means that untrusted devices which get pw0ned (or are surveillance-by-design) can’t then get into your more important machines. Most people use guest networks for IoT devices like Ring cams and smart washing machines but I don’t even own shit like that.
I bought a firewall appliance based on an Intel CPU, with 4x 2.5gbps Ethernet ports, 8gb ram. Installed opnSense on it and been super happy even since!
A bit pricey the hardware, but it’s been totally rock solid for almost two years now.
I have configured unbound with ad blocking on opnSense, and it has every single option I would ever want on it. Amazing stuff.
I’m eyeing a similar one, but with 2x sfp+ ports too.
I’ve seen a lot of people on YouTube talking about this or that cast off networking device that they turned into a router. I do find the idea of repurposing especially old tech appealing. Or did you buy it brand new?
I bought it new, but of course you can repurpose old hardware!
OPNsense on used dell optiplex. Very easy but it can do whatever you want to grow into as you learn more.
This is what I did before I went to all unifi gear.
I was Unifi gear and then went back to Opnsense :P. That shit is way too expensive to replace.
dell optiplex
How many NICs can you cram in these? I added a second one to my Lenovo Tinys but some more are nice for a FW…
Two NIC. I’m usually using the SFF. I went 10G last year and it’s been so great. Single, used, low profile, dual port SFP+ module. One port has SFP+ to RJ45 to my modem. Other port is SFP+ passive DAC to a used Brocade ICX 2750.
OpenWrt on Raspberry Pi 4 with UE300 dongle for the WAN side. 7 years of trouble-free operation so far.
What’s the throughput on that bad boi?
I’ve only tested up to 1Gb. It does 1Gb no probs, with SQM (QoS) which burns a lot of CPU.
Okay cool, thank you
I build a Linux system with multiple NICs. It’s heavyweight for a router, and probably not very efficient, but it does the job very well and I love it. I’m using foomuuri to manage the NFTables and NAT and it was very easy (although it’s a very obscure piece of software and the documentation and community isn’t excessively comprehensive, you might have better luck with something more common but it works nicely for me). It can pull in all the adblocking lists and all that fun stuff. I find it pretty intuitive to configure, at least compared to the horrors of most networking/routing configuration nonsense.
For DHCP I tried using the ISC’s new “kea” in the hopes it would be modern and clean alternative to the venerable old isc-dhcp-server but it turned out to be a horrible mistake. They made some awful, insane, unjustifiable design decisions in that software. It gets confused about IP addresses already being claimed (by the exact same host/MAC) address because some arbitrary value changed in how it identifies itself, and then does absurd things with static leases, it just randomly decides to give it a dynamic lease instead, and then will fixate on that dynamic lease and never let it have the static lease back, I fought with it for months, it kept breaking my network for absolutely no reason and it was a nightmare.
Back to trusty old isc-dhcp-server for me. And of course bind for DNS, I’m a bit old-school but it’s very comfortable.
dnsmasq might be a lighter-weight and easier alternative you could consider, I think it does DHCP as well, convenient.
And of course, being a full Linux server means you can put all kinds of other stuff on it too. Stuff that needs to be both publicly visible and privately visible on the LAN seems like an obvious fit, but in fact it’s probably the worst place to put those services, because you run into issues with NAT hairpinning and other nonsense. For services that are strictly intended for the private LAN/VPN, or services that are strictly public facing, it can handle those just fine by restricting them to a specific interface. But you should avoid stuff that needs to listen on both sides of the connection, it can be done, but it’s tricky and annoying and not ideal.
If you need both internal and external access, port-forwarding from the outside through the NAT to another dedicated internal host is a much better solution if possible, it’s still a bit tricky (you may have to override the DNS for that domain internally to use the private IP instead of the public one) but it’s quite reliable and functional once you have it set up.
None of this is really for the faint of heart, it’ll likely require a fair bit of trial and error and physical access to the machine to resolve issues. But it’s a fun project and I think the end result is worth it.
I have a Flint 2 with OpenWRT installed, it handles all my home networking. I have a handful of vLANs and I think the count is still 5 APs. I am currently considering adding a 6th for HA to TV vLAN, one that cannot access the WAN just HA. People have disagreements over OpenWRT and pfsense, or opensense not sure which is the “FOSS” version, but OpenWRT is not going to try and upsell you.
I am sure OpenWRT offers some ad blocking but I already had two finally tuned piholes. There are arguments between pihole and adguard home. I started with pihole I have tried adguard but went back to pihole.
Can I ask why you didn’t like adguard as much? :o
That was years ago so I do not remember. Maybe it was that I was just used to pihole? If I tried now I think the answer would be the same, but it would be because I have a finally curated list and regex.
I had a virtualized PFsense box for a while doing my inherent. Was actually pretty solid even when having to wrangle the quirks having a virtualized firewall/router caused.
I have something similar it’s been working great.
I have been thinking of trying to set up a “router on a stick” as a live backup but once I started researching it, it got daunting.
Opnsense on whatever. Plus pihole for DNS/adblock.
I put Opnsense on an Intel Pentium G3220 and it’s been rock solid for gigabit.
I got a spiffy little fanless micro PC with five Ethernet ports, installed Ubuntu, and wrote my own iptables rules and installed pihole.
Now THAT is diy
Does it run cool enough?
Yeah, it’s barely doing anything except shifting packets and doing DNS. Load is around 0.08 on a four core processor.
My setup consists of an x86 “network appliance”. They’re low-cost, low-power, sometime headless units that can run most any open-source software solution. Even with RAM as bad as it is, you can find these for less than $200 from multiple sources if you’re okay with 2.5Gbps. If you did need more than that, you’d have to build something custom likely, unless it’s a small-business class device.
If you’re just getting into the network software side of things, I recommend experimenting off-Internet for a while with a few different operating systems. Find which one has a user-interface you enjoy and can wrap your head around since in a personal homelab, they almost all meet the requirements. My personal choice for software ended up being OPNSense.
I run all of my other services that aren’t network related on separate hardware via a managed switch just to keep things isolated and the rest is up to you.
That actually sounds fantastic, low power, low cost, repurposed tech meets a trifecta I didn’t really know was possible and is exactly what I’d want.
I’m not really sure what anyone would need > 2.5 for tbh. If anyone wants to tell me, I’d appreciate it, I see so many network guys on YouTube talking about 10gig, and I don’t know why 😅
it’s worth mentioning mine is over 10 years old and was that cheap when I bought it. If you can find a fanless design, these will last ages since there’s a few simple non-moving components that do the same kind of work all day every day and don’t take a ton of abuse. Even after all these years, the CPU on mine sits mostly idle even with heavy traffic. Modern software is just very efficient.
Another reason the router/firewall lasts is because I use a dedicated WiFi antenna. This is another part that gets the same work every day. I’ve upgraded mine twice, but it was almost double the speed each time, and I didn’t have to spend extra money on routing hardware that was still plenty powerful.
Right now I’m using a Fortigate 101E, but eons back I used a simple PC built as cheaply as possible, installed Linux, ip_forward = 1, and iptables.
I am, however, tempted to build something semi-professional based on a banana pi.
Can I ask why you need something custom for work? I’m myself curious how I’ll make my DIY set up play nice with my job’s VPN 😅
Highly niche requirements not serviced by available hardware without costing waaaay too much. For example, we’re currently using Cisco Meraki MX85, and even they’re not quite there.
I just put OpenWRT on a (at this point quite old) TP-Link non-DIY home router. The only thing wrong with it is that it isn’t a rack-mountable rectangle, which offends my perfectionism.
I don’t care if TP-Link thinks it’s EOL as long as OpenWRT doesn’t. You should check the supported devices list for your current hardware.
Oh that’s something I hadn’t thought about at all! Hmmm
I’m running pfSense on an old Intel Atom fanless mini PC with 6 ethernet ports. I use pfBlocker for ad blocking. I also installed a GPS module in it. That’s the clock source for the NTP server. I was planning on replacing it with something newer, but that’s not happening until memory prices come down.
I would not recommend using pfSense anymore. My next router will be running OPNsense. I will probably install Adguard Home on that for ad blocking.
Why wouldn’t you recommend pfsense anymore? Also what’s an NTS server for?
There are a number of issues with pfSense. The updates for the community edition are slow. They are trying to push everyone to the paid version. You can’t even download an ISO to install it offline anymore. You have to make a Netgate account to download an installer that needs an internet connection to run.
NTP is network time protocol. It’s used to get the time over the network. Using a GPS receiver as the clock source makes it more accurate and allows it to work without an internet connection.
