I don't have the definite date yet, but it seems I'm getting to start moving into my new place somewhere in march. I'm in the middle of a divorce, and from the most part I'm starting from zero on many many fronts, including my network and home lab that quite frankly had too much ancient legacy to be worth trying to salvage much from. I'm leaving all of my 10 Mbps and 100 Mbps stuff behind, as well as my WiFi only stuff like all of my old C.H.I.P stuff. I'm also leaving behind all my shit take consumes too much power for comfort at the other side of the spectrum. So what's left? Not much really.
My old shit
I can't affort to get everything new so I'm taking some of my salvageable shit with me, basicly everything that has 1 Gbps and for what taking it isn't desruptive to the infrastructure my estranged wifa end my girls need at my old address. I'm leaving the switches and WiFi access point, as well as the PS5 that my kids use all of the time. They will be more with their mom than with me, so it seems fair to leave the PS5 there. I'm taking my NVIDIA Shield TV, my old QNAP SAN and my work gear.
My old NAS
My NAS is an aging QNAP TS-253A, a 4 core celeron based NAS with 8GB RAM and two 3TB drives. Currently it's main purpose is to run as server for smallish docker containers. It's too limited to even run GitLab on, but it runs a few python based servers, an nginx instance a small FlureeDB instance and a HIVE bot. It has two 1Gbps ethernet connections and I think I want too repurpose it, for now, to serve as backup disk for my new virtualization and containers solution.
My work NUC
I do most of my work on a little NUC PC that I carry with me wherever I go. It's an not that old i7 with 64 GB of RAM. I have a decent 40" screen and ZSA Voyager mechanical eargonomic keyboard at home, and almost the same setup complete with power brick at my main place of work, so I just take my NUC, plug in the power and USB-C connector and I'm docked. And if I need to go to work somewhere else, I take my powerbrick with me and my Voyager, and I'm usually fine.
This system has a 2.5 Gbps ehternet connector that I currently don't use because my network at bith locations is topped at 1 Gbps, and my screen has a 1GBps integrated that just works when I dock over USB-C to my monitor. When I'm at home, in my new setup, it should be worth the trouble to plug in a cat6 cable and go for that 2.5 Gbps in my new home.
My old work mini PC
My previous work PC is a somewhat aged mini PC from Gigabyte. I'm not sure about the CPU but it only has 16 GB of RAM. It Currently serves no purpose, but as I'll be needing a management station for my homelab setup, I'm taking it with me as my new management station. It has a 1 Gbps thernet port.
Nvidia Shield TV
I'm not taking my PS5 with me, but for my entertainment needs I do have a NVIDIA shield TV stick. While already an oldish model, its got quite a punch for it's age, so much so that there seems to be no justification yet for getting something new. Basicly this thing is for watching Netflix, Youtube and Tubi TV (through NordVPN), and for ocasionaly playing some games. Not at PS5 level, but prety decent for an android stick of this age. In fac for an android stick of any age.
My new shit and almost new shit
I've started ordering some new shit for my lab. Some components already came in. I received 96 GB of RAM, A 4 TB 4th gen x4 M.2 SSD and a 2TB 3th gen x4 M.2 SSD, I'm still waiting on the actual devices they are going to go into.
Minisforum MS-01
Eventualy I want my lab to have three of these machines, but for now I could only affort to start of with one. I orderes myself the cheapest version of the MS-01 from Minisforum.
An i5 system with a set of network connections that make it very suitable for a redundant setup. I know its just a home lab, but I want to have a realistic setup for when I start picking up my open source crypto projects again, especialy when I start working on my Proof Of Concept L2 for HIVE to see if I can make a proof of concept L2 that proves the viability of my CoinZdense post-quantum signatures efforts.
As you see in the picture, the device comes with two 10 GBps SFP+ ports and two 2.5 GBps ethernet ports. Oh, plus two USB4 ports potentialy suitable for networking, but I have no clue as to how I could use these in a usefull way in the setup I envision right now.
A system like this with 96 GB of RAM and a 4TB SSD sets me back about € 950 what is quite decent for the specs but still quite a lot in this phase of my life given that I need to get three of them for my setup. I know I'm taking a risk with availability, but for that reason I'm ordering them one at a time slowly growing my home lab.
Jetson Orin Nano Super
At first I was planning to either put a RX 6400 GPU into this system, or an Hailo-8 M.2 AI accelerator card, and while I was trying to make up my mind NVIDIA dropped the price of their Jetson Orin Nano by half and without thinking I ordered me one of them. This made for a change of plans. The 2TB SSD that I was planning to put into the MS-01 as second disk is now going into the Jetson, and this being an ARM board pretty much makes it fall outside of the admin stack I was planning for.
The Jetson Orin Nano Super if I understand correctly is like an overclocked Jetson Orin Nano for half the price of the original. It comes with 8 GB of RAM , a 6 core ARM CPU, 1024 CUDA cores and 32 tensor cores. Basicly its like a stand alone RX 6400 GPU but faster and with less power consumption with tensor cores to run some extra fun stuff with. The drawback of this setup is that I won't be able to pool the functionality with the same ease with what I could have done it if I used a GPU card for the MS01 systems.
This little card will be running some minor transcoding workloads and likely some fun hobby trading stuff once my finances recover enough to actualy have something to trade with.
While advertised for $250, with taxes, shipping cost and a 2TB SSD this little system sets me back about € 425 per node. I want some failover but not at the same level as for the XCP-NG nodes, so I'm "eventualy ordering a second one of these for fail over, but not yet, I'm on a budget.
Stuff I need to order before march
The above stuff I already ordered and hope to receive soon so I can start playing with it a bit.
QNAP QSW-M408S
I am not taking any of my 1 GBps switches with me, but I'll still be using at least some systems that don't need 2.5 Gbps or more, so my first switch is going to be a relatively affordable QNAP switch that is primaraly 1 GBps in how I will use it for now. But at the same time it is a switch that fits perfectly into my redunadant setup plans for the future.
The QNAP QSW-M408S is a managed switch with VLAN support that has 8 1 GBps ethernet ports plus four 10GBps SFP+ ports. In my first setup I will use only one of the SFP+ ports to connect it to the MS-01
This switch is currently priced at €235.
Some WiFi access point
I haven't looked into this yet, but I am going to be needing a decent new WiFi access point. I'm currently using a TP-Link that is quite aged and doesn't support the newest standards. I'm not sure which AP I'm getting yet, but I know I need one with sufficient WiFi capacity for an 1GBps to be insufficient. For now I'm planning the WiFi AP to be connected to a 2.5 Gbps port of the MS-01
I'm currenly projecting spending about €200 on an AP.
Delta internet with static IP
Now comes a tricky part. Currently I have 100MBps fiber internet from KPN/Xs4all and it just works. I have a static IPv4 address and an IPv6 range and with some port forwarding everything just works. On my new address KPN isn't available as a provider and I need to switch to another provider, most probably Delta. Delta's consumer subscriptions don't come with static IP, I'll need Business Plus, then I can get a static IP, and if I fork out 20 euro per month extra, I get a few more. For now I son't think I'll need more unless I want to start running my own DNS again but I have no direct plans for that.
Problem is with buying shit is that the documentation available from Delta is really limited and old and new info seems to be mixed. I know I should get a Business Plus subscription and that I'll get an ONT, but what type, and what type of port it has, I can't find it. Maybe I can buy my own ONT that comes as a SFP+ module so I can plug it into the MS-01 (for now) and into a switch later for full redundancy.
If I can't get an ONT that plugs straight into an SFP+ port, I likely need to use whatever ONT provides and will need to buy an SFP+ 10Gbps ethernet adapter. It's shocking how impossible it is to get info on this. What I do know is that Delta allows customers to upgrade and downgrade their internet speed on a per month basis, so I'm starting off at 500MBps for a while intill my lab redundancy is there and I;m confident I can run some real services on my new infrastructure. I'll switch to 1GBPs or 2GBps after I get my second MS-01 and to 5GBps when I have my whole setup complete.
Depending on if I need to get a tranceiver or an ONT, I expect this bit will set me back somewhere between €100 and €300.
Tech choices
Before I show my roadmap for my home lab, I want to share a few technical choices. I have zero experience with anything abouve 1GBps networking right now, and my hypervisor knowledge is somewhat limited and partially aged. As is my Linux networking and firewalling knowledge. My aged network had some BSD router filrewall apliances running at 10 GBps and that should say enough.
XCP-NG
The first choice I had to make for my setup was about the hypervisor technology. I've seen a lot of people getting very happy about Proxmox all over the place and many VMWare homelab folks make the jump and be happy about it. I've only had experience with VMWare through its CloudFoundry platform, way to far from the mettal to be of any relevance, and after looking at about a hundred hour of Proxmox youtube videos, things still didn't feel like something I would be comfortable with using. I've used Xen quite a lot though in the past, and after watching a few videos on XCP-NG and XO, I started reading up on XCP-NG and quite frankly I didn't understand how Proxmox could make sensewhen XCP-NG is available, XCP-NG just checks all the boxes, at least it did untill I decided to buy a Jetson Orin Nano Super. For a day or so I was tempted to revise all my plans completely and find out if I could run both the MS-01 and the Jetson Nano with Proxmox instead so everything would be under one management platform. But while reading up, the prospect of implementing my own workload failover and related stuff from scratch kept looking better and better if the price for one platform was going to be having to learn all of Proxmox as opposed to the much simpler and friendlier XCP-NG / XO setup.
I'm going to use XCP-NG, a clean hypervisor without the overhead of a host/guest OS setup. An environment where pools make everything make so much more sense when the hardware, storage and network connections are homogenous.
One price I'm possiby paying is that in Proxmox it might be more doable to use the USB4 ports for some kind of internal 20GBps ring between the nodes, what might make sense, but the simplicity of the pools makes so much more sense to me right now.
Pfsense in a VM
While I have Linux firewall and routing experience from way back, I need to pick things back up on that front. The first VM I'll need to get up and running on XCP-NG after setting up the trunk line and vlans between the MS01 and the first switch will be a Pfsense VM.
XO in a VM
The second VM I'l need to get up and running will need to be a Xen Orchestra VM running on the admin vlan.
Docker
Im not sure yet how I'm going to be running my docker containers. Do I go for the recipees that allow me to administrate docker from within XO, or do I go full K8S, or maybe somewhere in between and see if docker swarm can fit neatly with the recipees. This is a bit that I still need to figure out to see what is convenient in terms of administration and fail over.
My initial single node setup
This is the setup I think I'm starting out with. The fat purple lines are coper SFP+ pathch links running at 10Gbps. The blue line is a 2.5 Gbps connection to the WiFi access point, and all the black lines are 1GBps lines. I'll be running either 5 vlans:
- Internet VLAN
- ONT
- Media VLAN
- WiFi AP
- Shield TV
- Worker VLAN
- NAS
- Jetson Orin Nano
- Dev VLAN
- NUC PC
- Admin VLAN
- Gigabyte PC
From a VLAN perspective, with some VMs int there, things would look roughly like this:
In total, phase one of my setup should cost me somewhere around € 2000. I can't really spent any more than that right now, so the other hardware is going to have to wait a while longer.
Second buying round
There is zero redundancy in the initial setup. When I have sufficient funds for an upgrade, I am going to buy an second identical MS-01 system that will set me back another €950, and to factor in redundancy I'll also need to buy a few extra switches.
Ubiquiti Flex Mini 2.5G
For redundancy on the 2.5 GBps side of the MS-01 nodes, I plan to buy two Ubiquiti Flex Mini 2.5G switches. These are cheap (currently about 60 euro) managed switches with VLAN support and five 2.5 Gbps ports. We reserve one 2.5 Gbps port to each of the MS-01 nodes on each on the switches. The WiFi AP and the NUC each move to their own switch and the NAS moves to each of the two switches.
Two of these switches will set me back €120
MikroTik Cloud Router Switch 305-1G-4S+IN
For redundancy and fail-over on the internet side, we get the MikroTik Cloud Router Switch 305-1G-4S+IN. This managed switch costs a little over 140 euro right now and comes with four SFP+ ports and one 1GBps port. We reserve one SFP+ for each MS-01 node and the last one for the fiber internet link.
This switch is currently priced at €140
First upgrade round
When we patch in the new hardware, the setup will look roughly like this. There will be some redundancy and fail over. Enough to start playing with providing beta level public services and it's probably a good idea to upgrade from 500MBps to 1GBps or 2GBps for the internet connection.
For this first upgrade round I expect I'll need to spent abour €1300 in total (including cables).
Third buying round
In the third buying round we don't need any more switches, but we need to get a third MS-01 and a second Jetson Orin Nano.
We will need to figure out a solid home made failover setup for the services the Jetson's provide as neither XO nor whatever we choose for managing docker containers will be on any help.
Second upgrade round
After the second upgrade round things will look roughly like this. At this point I think I'll be ready to move to 5GBps fiber, eventhough my storage will be limited for some of my plans.
This upgrade round will sum up to about €1400 in total.
Delayed upgrades
Depending on the L2 Proof Of Concept that I end up building with CoinZdense and AioHiveBot, there is a big chance that I will need to eventualy upgrade my NAS. Possibly just my disks, but it's likely a good idea to move to bigger and faster. A QNAP TS-464 with 16 GB RAM and four 8 TB disks plus chaching SSDs seems like something the setup can banafit from. It's however also somthing that would set me back about €1500, and at this point I feel I will need to prioritize some other expenses first (a simple home gym that I would realy like to get in my attic), so it might be a while before I can afford a new NAS.
A NAS like this though would really complete the lab, bringing the total cost of my setup to about €6200, what is a lot of money, but worth it if I manage to use it to provide the ecosystem with a working and fuly operational least authority L2 storage solution that showcases post-quantum signatures as L2 for HIVE. I think that if I manage to show it working on L2, the community is going to buy into the need to get the technology into L1 and L1.5 as I like to call it.
Thoughts ?
I think that when my lab is up and running after upgrade two, it should be quite capable of running whatever Proof Of Concept setup I come op with for CoinZdense and AioHivebot to make a usefull layer 2 for HIVE that demonstrates post quantum signatures, least authority key management, and something of direct practical use. I'm aiming for a least authority file storage system based on my earlier work on MinorFS, although I realize that might be a tad bit too ambitious, especialy without any funding. Maybe at some point when I have soe decent half working stuff to demonstrate I might give a throw at a DHF proposal, especialy if I need to buy the new NAS that is currently completely outside of my projected budget.
But even without the NAS, this setup will take up quite a bit of my limited funds, so if you have some wise advise for me or if you see me making rookie mistakes in my redundant setup with XCP-NG and PfSense, please drop me a comment before I throw money away at something stupid. And if you think the roadmap is spot on, please also let me know so I know I'm on the right track.