Upgrading my home server, part II

After experiencing two near catastrophic loses of data with two servers, I had to take some time to reevaluate my server configuration and make some changes.

Some of you might remember from my earlier post back in December 2013, I started out building a simple Ubuntu server to replace my old Windows Home server with the idea of 'retiring' it and using the Ubuntu server as my primary server. The Ubuntu server I built ran sock-solid for over a year with no problems or issues. That is, until I started adding drives - I'll get to that in a few.

Fast forward to late September of 2015 - one of the 2TB drives in my Windows Home server started showing read/write errors in the system log. I knew I didn't have much time to pull the data off, so I started backing up what I could to my Ubuntu server. There was about 7.5TB's of data to transfer and I needed to add a drive to my Ubuntu server to make some room. At the time, the only drive I had was a Western Digital 3TB that I removed from a USB enclosure - so I threw it in, partitioned it as GPT, formatted it and added to the LVM logical drive I had setup on my Ubuntu server called 'Storage'. Everything completed without errors, so I continued copying files from my Home server.

Well, using a 3TB drive on an old server turned out to be a huge mistake. Once the server starting writing files to the new drive the entire logical drive became corrupt. As I continued to copy files, all of the data I had assumed was safe on my Ubuntu server was now in jeopardy of being lost. I didn't notice it at first, and it wasn't until I ran through some weekly Ubuntu system updates and rebooted the server that there was a problem. Once the server rebooted, it refused to mount the LVM called 'Storage' because the partition tables on 3 of the 6 hard drives were corrupt.

Now, not only was 7.5TB of data from my old server now at risk, but so was the 10TB+ on my Ubuntu server. 

I spent the next few days trying everything to fix the partition tables and mount the LVM - but nothing worked. In the end, the only thing that did work was loading a spare hard drive with Windows 7, installing R-Studio from R-Tools Technology and slowly copying the data off to three 5TB external hard drives. Nothing I tried in the Linux environment would solve the problem and it took a Windows program to recover files from a Linux LVM. Go figure.

Once I had the majority of my data back, I decided that I had to rethink my server configuration from the ground up. First, I upgraded the CPU's to quad core 2.83Ghz Intel E5440 Xeon's with 24MB cache. Next, I upgraded the RAM to 32GB of ECC DDR2 and added a dual Intel gigabit network card. Finally, the biggest upgrade was installing an HP Smart Array P410 PCI-E SAS controller (w/512MB cache and battery backup) and setting up a hardware RAID 5 array.

After trying out various Linux server operating systems, I decided to go back to Microsoft and install Windows Server Standard 2012 R2. I wanted to mess with virtualization and Hyper-V is built-in to Server 2012. I setup the six 2TB drives in a hardware RAID 5 configuration using the HP P410 Smart Array controller and installed two 240GB SSD drives; one for the OS and another for web server duties. I also connected two 5TB USB 3.0 external drives for nightly backups. The RAID 5 gives me some measure of safety in case one of the drives decides to die, but just to be sure, I have backup copies on the external drives as well.

I don't like Microsoft's built-in web server and I much prefer web servers in the Linux environment, so I opted to run Ubuntu server as a Hyper-V virtual machine that starts and runs automatically at boot up. That gives me the familiarity, speed and simplicity of Ubuntu for my web server while keeping it contained in a virtual machine that could be easily backed up and moved to a new server, if need be. I also have the freedom to try out and test different operating systems in a virtual environment using Hyper-V without dragging out a spare PC. 

The new server is extremely fast. The 10TB RAID 5 array benchmarks over 500MB/sec reads and writes with large files and gets over 1GB/sec with small files. The Intel S5000PSL server motherboard I'm using has 8 SATA II ports and I'm averaging 230MB/sec reads and writes to the 240GB SSD drives - which is fast enough for my needs. 

I also bonded the four gigabit network cards into one 'virtual' network card which gives Windows Server 2012 a huge pipeline to send and receive data across the network to all of my computers and devices with blinding speed. In addition, I also upgraded my network switch to take full advantage of the speed. Now, streaming media to my phone, tablet, TV, stereo or other devices is instantaneous. 

With my new server up and running, I turned my attention back to my old Windows Home Server. I had that server running 24/7 since 2008 and all of my YouTube videos and vinyl rips were stored on in. Unfortunately, the Toshiba 2TB drive that had the majority of my critical files was too far gone to recover any data. I lost almost all of my vinyl rips and some of the source files for over half of my YouTube videos - including 3 new videos I was working on at the time. 

It's been a long and arduous journey to get everything recovered (almost), configured and running smoothly and I learned a valuable lesson about keeping multiple backups of critical files so I don't have to go through this painful recovery process in the future. 

2017 update: Since my last update I added a QNAP TS-451+ quad core 4-bay mini server with four 2TB Toshiba server hard drives (6TB available in RAID 5), another 5TB USB 3.0 Seagate and a Seagate 2-bay 'cloud' storage drive with 8TB's of space.

So far, the Xeon server is working flawlessly with six enterprise level Seagate drives in a RAID 5 (no more mixing and matching desktop drives!). I use KLS Backup Professional to automate the nightly backups to two 5TB external drives and my Seagate cloud server. If you run a Windows Server, I highly recommend KLS Backup Professional.

I also setup a Windows 7 virtual machine as my torrent client with all the traffic going through a VPN for anonymity.