KVM

Replacing Multiple Spinning Disks Simultaneously or Serially…

Posted on by dk

So, with a 6-drive RAIDZ2, I faced a drive failure over a year ago with a “hung” Windows host (hosting the Ubuntu Server LTS Hyper-V VM with pass-through, direct access to the 6 physical HDDs used for the RAIDZ2 array) – the Windows UI was still responsive but any drive reads (e.g. Windows Explorer navigation, starting an app) “hung” the offending app attempting the drive reads (even if the dying drive was not the drive being read from)… With 2x 6TB “spares” on hand purchased over time (2017, 2018) for just such an event, a VM-and-host shutdown, HDD swap and a quick zfs replace <pool> <old GUID> <new /dev/sdx> and a “quick” resilver brought everything back to normal.

Then, three months back, I then started facing 2 failed drives – I had the one remaining 6TB “spare” replacement drive for the first, but after a 2nd failure in the span of these three months (without purchasing another set of standby replacements), it was time to start considering replacing all the drives (slowly).

Not too shabby, with ~7+ years’ lifespan of near 24/7 powered-on, low-drive write loads, with some pretty bad temperatures (near constant 50°+C to 60°C, no matter how I tried to force air flow when these were still in the DS380):

  • 2x Seagate ST6000DX001:
    • from March 2015
      • 1x failed in August 2016; RMA/replacement still running
  • 2x Seagate ST6000DM001:
    • from November 2015
      • 1x failed in November 2022
      • 1x failed in November 2023
  • 4x Toshiba X300 HDWE160:
    • 2x from July 2016
    • 1x from November 2017 (spare)
      • 1x (surprisingly the replacement drive from November 2017 that was “just” plugged in in November 2022) (just) failed in February 2024
    • 1x from November 2018 (spare)

I therefore purchased 2x Seagate Exos X18 16TB HDDs, with another still on the way… Wanting to minimise the number of resilver attempts (straining the surviving 6TBs), I attempted to pull a working drive from the degraded 5-drive RAIDZ2 array and plugged both new 16TBs in, fingers crossed that none of the remaining 4 drives give up the ghost while resilvering (confident I had important data backed up elsewhere).

I gave the replacement commands one after another:

2024/03/03 Update: Don’t assign the whole disk, manually create a partition instead and assign that as replacement instead!

And that seems to work… So, 11+ hours later, nearing the end of the resilver process, I was eagerly checking the status…

Wha..?!? Resilvering only completed on one drive (and was only now starting on the other)!

Continue reading →

Adding 4G/LTE Back Up Internet Link to pfSense VM…

Posted on by dk

Updates Fartdates…

So, my Ubuntu LTS 18.04 decided to have a brain fart during a “routine” system update just past midnight on Saturday morning… Rebooted the modem, switches, VM, VM host… nada…

Wither Thou Internet…

With the ‘net down, I could not seem to see the list of update details, nor try and roll anything back… Worst yet, I was actually doing work (which needs a ‘net connection)… So the troubleshooting ensued…

Troubleshooting using my work laptop via my handphone hotspot was no fun… So, four-and-a-half hours later, I retired, disgruntled at not solving the issue (and also having to do three rounds of laundry, get woken up a mere 15 minutes later by my young daughter who wet her bed, and get awakened again 30 minutes after that due to one inconsiderate neighbour’s noisy pet birds – but that’s a totally different story and I digress)…

Saving Grace…

Just a few days ago, I had applied for a free 12-month trial from TPG (Singapore’s fourth telco), so at 10AM in the morning, I dragged myself out of bed, went to church, and then picked up the TPG SIM card… All this to use in a Huawei E3372-607 USB LTE/4G modem (together with a high-gain indoor antenna) purchased nearly two years ago that was meant to fix this exact situation (i.e. be a back-up Internet link).

Continue reading →

Software Firewall…

Posted on by dk

The Problem

I have been using an Asus RT-AC68U, followed by an RT-AC87U, running Merlin’s firmware with customised firewall scripts for the longest time. However, both units had a persistent issue with some (not all) sites being inaccessible, total resets and re-configuration from scratch regardless.

Having confirmed it was an issue with the router(s) and not the firmware nor firewall rules nor server-side blocks, and not being able to find a solution, I decided to just utilise a software firewall. One that I knew well and trusted was/is pfSense.

The Other Problem

At the very same time, I finally discovered that the boot failures of my server was actually due to the PSU (read other Amazon reviews citing similar fan-spin-up-then-dies failures). Having not had time to look at the frequently (and randomly rebooting server), I finally purchased whatever SFX module that was in stock at the local “IT complex” – another Silverstone SST-SX600-G unit… Crossing my fingers that the PSU was the culprit…

2018/06/04 Update: Nope, false hope again… Server is still rebooting rather “randomly” despite using a brand new Corsair SF600…

Continue reading →

KVM: Installing Windows…

Posted on by dk

So, I had a spare, official Windows 7 Pro key that was never installed on the intended laptop. I was thinking that it was a good chance to install it on KVM…

So, what was supposed to be a straight-forward “new VM” + “install Windows 7” + “Windows 10 upgrade” turned into another headache…

Fortunately (and probably yet another reason to stick with the “tried-and-tested”/popular VM solutions), KVM has a “large enough” community, with lots of help online…

Continue reading →

There Is No Spoon…

Posted on by dk

So, attempting to set up a virtual machine on Ubuntu now leaves me some choices (again, which is mostly a good thing).

Attempting to set up a secure Windows environment is never easy. Maybe one of the better/best ways to do this is to simply use VMs and virtualised software…

First, I need virtualisation host software. VMware ESXi and any other hypervisors are out of the question, because we already have an OS. Besides, despite being comfortable with ESXi (and also have somewhat generous “limits” on their “free” version from v5.5 and up), ESXi is pretty strict in terms of supported hardware.

Having looked at some of the “popular” ones out there, including Oracle’s VirtualBox, Citrix’s Xen, and Red Hat’s KVM (not to be confused with the common abbreviation KVM), I finally decided on KVM.

Even with VirtualBox’s ability to use “integrated mode“, I still believe that having low-level integration with the kernel and open source is more important than reliance on a specific kernel version (note: linked search only shows results from past year to show “current” reported issues as at time of search).

Continue reading →