I am documenting here some work I have been doing over the last couple of weeks to get together a mini low power data centre (lab or private cloud if you like). What I wanted to do at the start was to have something I could run up Vmware ESXi on to get to to know this software a bit better as we use it extensively at work and I have not had that much hands on. I wanted something that I could leave running and wasn’t going to use my main PC (Dell XPS 27) so Vmware Workstation (or equivalent) was out, I didn’t want this on my desktop.
I was also looking for something low power as electricity is not cheap here (32 xpf per kilowatt hour, close enough to 32 cents), it had to be low power enough that it could be left running 24/7 and I really wanted to keep power costs under US$10 a month if possible. My aim was to keep it all under about 50 watts (comes out to about $11 a month to run), have enough grunt to run up at least 5-10 VM’s and enough storage to not worry about running out.
However as the project evolved it became a bit more than just a lab. I realised I could run my current web site and services I was paying a lot more than that to run from home and have some fun at the same time.
As I already had a Synology NAS in place that pretty much was going to the storage, it also does iSCSI and NFS so I was probably going to be using iSCSI with the Synology. So, my Synology is a DS1512+ 5 bay model. I have it loaded with 3 x 3Tb drives in the Synology recommended configuration which they call SHR (Synology Hybrid RAID). This gives me about 5.4Tb of storage that I already leave running 24/7 as I use it for all sorts of things in the house. I have a bunch of mapped drives to it from Windows that I use for all important work as it uses RAID and I back it up to another 2 Bay Synology every so often so I have something I can walk out of the house with when need be or when I go on holidays I leave this copy somewhere else.
http://www.synology.com/products/product.php?product_name=DS1512%2B&lang=us You will see it has Vmware, MS and Cirtix compliance logos with links back to respective vendors.
I was a little bit worried about power consumption so I did some tests with a power meter with this model in single drive configuration right through to 5 drives. I was pleasantly surprised to see it only ranged from about 25 watts with no drives to about 30 with a single drive. Of course when you plug a drive in there is a peak while it spins up but then it settles down fairly quickly. So about 32-33 watts with 2 drives and about 35 with 3 drives. I will have to go back for 4 and 5 drives as I didn’t save those figures. So, this was already in place with a UPS connected and automated shut-down on power outage (standard feature with compatible UPS). As I also back this up from time to time I felt pretty good about having all my data on here.
Synology with 3 hard disks settled down somewhat showing 35 watts. So with the storage part sorted now on to the host hardware.
Just to note however that of course your choice of hard disk here is extremely important. I am using Western Digital Green 3Tb. They are the WD30EZRX model which are SATA 6 Gb/s 3.5 Inch using IntelliPower 3 TB drives with 64 MB cache, see here http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-771438.pdf I have three spindles going here so throughput is not an issue, even with 5400 RPM drives I can flood the 1Gb connection if need be, have seen this get up to 800Mbits.
In part II Host Hardware
Here my choice was pretty limited, I had a pretty good idea what I was looking for at the outset. I didn’t want Atom, they are good for certain things but I wanted something with a bit more grunt and at least 4 logical cores. I had seen references to the Intel NUC around and came across them on a French site where I usually buy gear from. Indications where that it would run well under 20 watts and you could cram 16Gb of memory into it. OK, processor is only an i3 but for what I was wanting I reckoned that would suffice.
So off the order went for one of these. http://www.intel.com/content/www/us/en/motherboards/desktop-motherboards/nuc.html In particular I got one of these models which comes with no SSD or no memory as I thought I had some memory I could put in it. http://www.intel.com/content/www/us/en/motherboards/desktop-motherboards/desktop-kit-dc3217iye.html
I was pretty impressed from day 1, well once I got some memory in it anyway 🙂 Turned out the memory I was going to use had disappeared and I had to buy some locally, unfortunately that was the easy part. Asking around town for mSATA SSD was a different story, no-one even knew what they were. So back to the web to order 2 mSATA SSD, a 32Gb and 64Gb, turns out for what I was doing this was way too large and I could have gotten away with 8Gb or less. However the smallest one I saw for sale was an Intel 30Gb. Remember this will just host the Hyper-visor and that’s it, all VM storage will be on the NAS.
Thats me holding the NUC in it’s final resting place headless. It’s pretty small. You can see the size of it next to the dusty ADSL modem and Gb switch.
So, this much pretty much decided what was going to be the first Hyper-Visor on this and out the 5 I tried (ESXi, Hyper-V, Xenserver, XCP and ProxMox) ESXi is the easiest to install on a USB thumb drive and in fact out of the first 4 the only one that is designed to be installed like that. So ESXi it was, well to start with anyway.
The next part of this article pretty much came from here. I was happy to see someone else doing the same thing. http://majornetwork.net/2012/12/installing-vmware-vsphere-hypervisor-5-x-on-a-usb-stick/ So, this outlines a way to install ESXi on a USB stick, it’s really cool because you don’t need an actual CD, it’s all done on Workstation or Player. From there I went to here which was really helpful because although ESXi installed OK it came up with no networking.
From this page here I was able find out how to modify the ESXi install CD to inject the NIC drivers. http://www.tekhead.org/blog/2013/01/nanolab-running-vmware-vsphere-on-intel-nuc-part-2-2/ Pretty nifty really. It uses the following software.
· ESXi-Customizer (created by Andreas Peetz)
· The ESXi driver for and Intel® 82579V Gigabit Ethernet Controller (created by Chilly)
Once I had this done and recreated my ESXi CD and re-installed ESXi (5.1 by the way) it came up just fine with it’s one and only NIC. (Intel 82579V). So next step was connecting ESXi to the Synology with iSCSI. This was one of many configurations I was to try over coming weeks. I found I could install a whole bunch of Vm’s on this little box and it would pretty much take whatever I threw at it.
NUC booted into ESXi running of a USB stick. Hand for size comparison !
Here I have a bunch of stuff running (OS X didn’t actually work) and you can see this is ESXi 5.1 on a i3 with 16Gb RAM. You will have to take my word this is running on the little black box :). While I had this running I did a few tests booting into various Linux distributions and also Windows Server 2003 and Windows 8 Pro. I was interested in power consumption and boot times. But more about that later. In a nutshell, expect the NUC to go up to about 18 watts on an OS boot up then drop back once boot is under way to about 14 watts, average around 13 watts. Then once the OS is booted then around 11-12 watts depending on what it is doing. So all in all it works pretty well. I was running a Windows 8 Pro VM on there with 2 vCPU and 2Gb of memory and it was very usable via RDP. Any Linux (command line only) would be happy with 256-512Mb of RAM and 1 CPU per VM.
In it’s final setting I have right now 3 x Debian 7 Wheezy VM are running, one for Varnish, one for this web site and one shell. Also I have a Windows Server 2003 instance I use for general testing and a Windows 8 Pro VM I use for stuff I don’t want to do with my desktop. I also have a Nagios XI VM running monitoring everything and just recently added an instance of Windows Server 2012. All that sits at about 10% CPU and would be about 12-13 watts of power most of the time. In the next few days I will ad a bunch more Debian 7 VM for Awatst, Syslog etc etc. And at this point in time I am only using 7Gb of RAM.
I will add this to article this over the next week or so to get to where I am now, which went something like this. ESXi (free version) to Hyper-V (free version) to XenServer (free version) to XCP (Open Source) which is what I finally settled on. I also moved my web site from the USA back to this gear here in New Caledonia and is currently working out of my home ADSL and I am saving about $70 a month in the process, not mention learning heaps and having some fun.