Tootfinder

When others are sleeping.. #linux #linuxserver #homelab

Nomad cluster powered down.
Everything is now running in a k3s monocluster on a Radxa Rock5b
What feels insane is I started this project in December.... But lots of little edge cases and such to solve for just like any real migration.
I'm using fluxcd and all my configurations are pretty decent. I could do a big-bang redeploy on a new machine and everything would work.
#homelab

Finally getting around to updating all the servers in the #homelab to fix the `sudo` vulnerability. Kinda of a pain, but almost done. It really helps hosting the repos locally...lol

Things to remember: if running your website in a container, it might be a good idea to ensure that the `restart: always` line resides somewhere in the compose file. Also, I really need to add the `dice.jamesthebard.net` website to monitoring which would've catch the issue.
#homelab #containers

One of the main setbacks to finish my homelab migration from nomad to k8s has been cert-manager compatibility with some of my domains. I didn't want to migrate everything to cloudflare. I've got 2 domains on easydns.com, which isn't supported by cert-manager, but is supported by dns-lexicon.
I found a cert-manager-webhook wrapper around dns-lexicon. Then I forked a fork so that I could tune the build pipeline to produce multi-arch images quickly.
Anyway here…

Number of #bitnami container images in the #homelab cluster: 4
Number of #bitnami container images to remove: 3

Things I find slightly annoying: tracking down how to create a local #CachyOS mirror from information on the internet. The only bits of information I have is that (1) it's possible, and (2) you can use either `rsync` or `syncthing`...the search continues.
#homelab

OMG the comments…
https://www.reddit.com/r/homelab/s/vRA4fuIRR6

When you wake up in the morning after a thunderstormy night and find the one electric fuse which cares for your homelab has triggered. The USV must have screamed as long as it could .. It's strange I didn't hear it.
After I switched power back on one of the proxmox nodes didn't come back. Dead.
Good morning everyone! It's monday!
#homelab
PS: It's…

I needed a Debian 13 LXC container, but proxmox doesn't have one available yet, even though Proxmox 9 in based on Debian 13.
Anyway, I wrote a post explaining how to make a new LXC template for Debian 13 to use with proxmox
#proxmox #lxc #debian13

Me rebooting my #archlinux #zfs #nas box this morning after updating the kernel.
Both zpools came back up, and all containers 😀

Finally got a chance to document something that was hard and may help an entire two people 🙂 Enjoy.
https://blog.faucher.live/how-to-expand-an-asustor-flashstor-raid0-volume/

Only Microsoft could find a way to uninstall vscode from #windows without being asked 🤷‍♂️
#homelab

Anyone else just disliking this heat?
In the UK and my homelab probs not liking this haha.

Alright, all of the Pis are now in use. There were quite a few services I didn't want to run in Proxmox and that was the main reason to have a "few" Raspberry Pis in the rack.
1: Primary DNS
2: Secondary DNS
3: PiHole
4: VPN (Tailscale)
5: Power/UPS status
#homelab

I always feel soooooo slow when I'm starting a new IaC project or doing a major refactor.
In this case there's a lot of orchestration that I'm the tying to leave room for future enhancements. Anyway it always takes long time
I'm re-working my lanecloud Ansible provisioning to better use netbox as a source of truth... And also switching to provisioning VMs with libvirt instead of nomad

Nothing like updating the homelab on a Thursday night. The Proxmox live migrations went perfectly, and no issues with any of the servers. Verified the backups, too. Gonna call everything good to go and relax.
#homelab

Time to punish the new servers. Also, muxed the `m2ts` to `mkv` files so there are functional progress bars this time.
#homelab #ffmpeg #encoding

Okay, finally got CTFreak back up and running so that I don't have to deploy my website manually anymore. When I did the homelab updates/consolidation, the old CTFreak server ended up not making the cut...however, we're back.
#homelab #ctfreak

The homelab has been stable for a while now so it’s time to mix things up. Moving DHCP and DNS from the Synology NAS onto a primary/secondary set up in a pair of #raspberrypi that I had left over from the old cluster.
Today DHCP is moving to #Kea and once the dynamic DNS entries have be…

Ight, I need some help picking an approach for my homelab.
Right now I've got 3 raspberry pis (1gb RAM) always on and 2 optiplexes running proxmox which are only on when needed.
Trying to work out whether I should move everything to one of my proxmox machines and just leave a single pi somewhere running a thing to act as a tiebreaker or just leave as is rn.
I run docker on pis with HA, pihole, some discord bots, status things but am kinda limited rn with RAM.
…

Got the fan grills in which significantly lowers the chances of me putting a finger or cable into one of the rear ventilation fans which totally hasn't happened.
#homelab #minirack #fans

The first MS-A2 node is up and it only took the entire day. After fixing a ton of networking issues across the homelab I've got all the VMs running on the new node. The old nodes have been decomissioned which kinda marks a milestone. Still need to get the other two new nodes into the cluster, but should be fairly straightforward and can wait until tomorrow.
#homelab

I'm in the middle of migrating my HA from docker to an HAOS VM. As part of the cleanup, I did a fresh install of Node-RED to a separate #docker container so I could use tailscale to proxy it.
I wrote up how at #homelab #nodered #NodeRedInstallation #HomeAssistant #Tailscale @…

Hrmm, gonna see if I can't cut down the MS-A2 AV1 encode times. I'm hoping that changing the performance profile will push the CPU a bit harder as I'd like to get it close to the 5950X encode times.
#homelab #minisforum

It's a tight fit, but finally got the PiKVM Switch stack sitting nice and flush in the basement.
#homelab #minirack

Alright, got UPS monitoring and info via `nut` on the homelab network. The three Proxmox nodes are all net clients, and I took the fifth RPi4 that wasn't doing anything and tossed Raspbian on it to be the net server. After a bit of configuring: all of the servers are getting the information without issue.
As a bit of a note: Ubuntu 24.04 LTS _does not work_ on ARM64 because [insert reason here]. I think you can compile it yourself and it's fine, but the packages in the repos…

Okay, everything is buttoned up again, only almost stripped a rack hole out. However, time to leave things alone and see if it catches fire.
#homelab #minirack

Wooo, got the PiKVM on the OOB VLAN. That took way longer than it should've, but finally tracked down the small issue I had with the `systemd-network` files.
#pikvm #homelab #minirack…

The top MS-A2 server is very, very warm as it's ingesting heat from the RPis above along with the two switches. Grabbed a couple of fans, a 12V power supply, and a few fan screws. Give it an hour or two and see if I'll need to add something else or change the airflow.
#minirack #homelab

So, went ahead and connected the HDMI cables to the PiKVM switch and at least got the video routed to the PiKVM v4. While these were pricey, the software and capabilities more than make up for it.
#homelab #pikvm

The last thing to install into the minirack: some nice remote access. Still need a few HDMI cables and I need to figure out a way to integrate the power button control into the PiKVM stuff, but baby steps.
#homelab

Alright, bumped the encoder VM from 8 to 12 cores, let's see if this helps out with the encode times. I have a sneaking suspicion it will not.
#homelab #ffmpeg

Alright, gonna do a quick test. I have the encode numbers for the `encoderXX` workers running Ubuntu 24.04 LTS and they're not great. Gonna build a node using CachyOS and see if that changes things.
#linux #homelab

Hrmm, it seems like there aren't many sources on how to put a PiKVM onto a proper VLAN. It seems like it's using `systemd-network` for networking, might have to put together an article on how to do it properly assuming I can remember correctly.
#pikvm #homelab

Started running some cables into the basement for the PiKVM. Got the network patch keystone installed in the patch panel, and ran the network connection along with the first two MS-A2 server HDMI cables. Unfortunately, I'm still waiting on the third HDMI cable, all of the USB-A to C cables, and a few replacement cables for the PiKVM KVM Switch. Things progress though...
#homelab

Pretty happy, first episode went from ~6GB to 250MB. Settings are still good, quality is outstanding, and the chapters, subtitles, and fonts all muxed in beautifully. Gonna let these encodes finish up on the cluster while I listen to some music and drink a few beers outside.
#homelab #encoding

Went ahead and upgraded the Proxmox cluster from kernel version `6.8` to `6.14`. Seemed like a good idea, didn't have any issues with VM migrations so that was nice.
#proxmox #homelab

AV1 might take a ton of time to encode, but the results are awesome. I do need to take a look at the VMs though as I'm seeing a ~33% increase in encode times when compared to the previous nodes I was using. Either way, the minirack is still quiet and taking everything that's been thrown its way.
#homelab

First real test of the new cluster: encoding AV1 video. I created three Sisyphus encoding workers and decided to run some Blurays across all the servers. Wrote some Ansible to build the encoding workers because it's a pain in the ass, but right now I'm working through the first 3 videos in a queue of 12. Also, these are so much more efficient...
#homelab

Multipathing: it makes fixing cabling issues so much easier. Swapped out the 2x2m fiber patches to the NAS with 2x5m ones because the old ones were just a smidge strained. Had absolutely no issues swapping them out one by one and no downtime.
#homelab #multipath

Old servers are prepped and ready for their new owner and new home.
#homelab

Eh, final home for the time being. Once the second nameserver finishes updating I can get the cables managed a bit better. Either way, officially done.
#homelab #minirack

Did a smidge of cable management because the fiber runs to the QNAP needed a bit more slack. Things I love: being able to disconnect the connections to the iSCSI storage and have absolutely no loss of connectivity.
#homelab

Hrmm, it seems that my Proxmox migrations are going over the 2.5GbE links. I'll have to fix that later so it uses the 10GbE links but it's definitely a low priority.
#homelab #proxmox

Got `acme.sh` installed on all the Proxmox nodes and managed to get the certs issued and installed to the right location without too much hassle. All of the UIs now no longer anger the Firefox gods. Also all of the servers are updated and nothing is on fire. Time to relax and debate on what to eat for supper.
#homelab

Welp, gotta transfer these images over to the NAS. However, after that it's time to figure out networking.
#homelab #minirack

Networking is finished, and every port on the QNAP switch is in use along with every port on each MS-A2. Also tracked down an `interfaces` file typo. The next step is cleaning up the "server room" and getting this situated.
#homelab #minirack

So, might've gotten a few things in the mail today...MUHAHAHA. The Minisforum MS-A2s have arrived and I've started getting the RAM upgraded and tested.
#homelab #minirack

Almost done with RAM testing, the third box is underway. Still need to Clonezilla it, but figure I can just enjoy a few beers and watch some YT videos.
#homelab

Power bricks fit perfectly in the bottom of the minirack. Might have to run a fan or two in the bottom, but we'll see how things turn out there.
#minirack #homelab

Backing up the image on the MS-A2's SSD via Clonezilla. It's so nice plugging in the IODD ST400, booting off of the Clonezilla ISO and also saving the NVMe image to the same device (the IODD ST400). It's not the fastest way of doing it, but it is very, _very_ convenient.
#homelab

I now officially have all the RAM...now just waiting for the boxes to toss the RAM into.
#homelab

New shelves came in, went ahead and pulled the shorter ones out, installed the full length ones, installed the supports, and cleaned up the power cables. Also adjusted the hooks on the back so they make a bit more sense.
Do the new shelves seem crooked? Yes. However they should support the MS-A2s a bit better.
#homelab

In one of the more painful purchases I've done in a bit: just ordered 3 kits of Crucial DDR5 RAM for the MS-A2 servers. This will take them from 32GB -> 96GB which should work out a bit better for Proxmox. Unfortunately, my wallet is crying. If everything goes well, the RAM should get here just before the servers do.
#homelab

Wooo, Jellyfin cache and metadata directories moved over to the slow storage via NFS. Not too much of a performance difference, but this will ensure that if I lose the VM then I won't lose the library.
#jellyfin #nfs

All three nodes are installed. The "server room" is an absolute mess, but figured I'd snap a picture of the back of the minirack. I'll be cleaning up tomorrow, but for now: we are 100% back online here. Migrations work, iSCSI works...well, everything works. Very happy with how this has turned out thus far.
#minirack

Did a quick test fit of one of the Minisforum MS-A2s and ran some of the cabling. I think this should turn out pretty clean and everything fits better than expected which is a good thing. The PSUs will go in the bottom 2U "compartment" without issue which I didn't run.
#minisforum #homelab