h0bbel.p0ggel.org

I initially bought a HP Proliant ML 115 server as a cheap test/lab server for VMware vSphere and miscellaneous rollout projects at work, but all of a sudden I needed it for some other project that required that I install Windows Server 2008 directly on the hardware itself.

As is the story with most HP Proliant servers, you should install it with the tools that HP provides. In the case of the ML 115, you can't use the normal SmartStart setup, but it's little cousin Easy Set-up CD.

The installation started fine, after running through the initial HP wizard, but when the time came to actually get the installation started it went all blue screened on me, complaining about nvstor.sys.
I knew that the Windows 2008 installation medium doesn't include support for the built-in nVidia NFP3400 SATA storage controller in RAID mode, but I wasn't running a RAID based setup on it anyway so that shouldn't cause the problem.

Next I tried installing Windows Server 2008 without using the Easy Set-up CD, in other words just plain old booting of the Windows Server 2008 installation CD and initially it seemed like it was running ok. Thats until it just stopped at 0% progress at the "Expanding files" section of the installation.

So, there I was. Using the HP tools, the installation ends in a big old BSOD, using "native" Windows Server 2008 installation it just stops without any indication on what might be wrong.

As it turns out, the solution was pretty weird. The HDD shipped with the server causes the problem (160GB NHP SATA). I have no idea how, but replacing it with another SATA drive and starting the installation again, with the Easy Set-up CD, fixed it.

The HDD shipped with the server makes the installation of Windows Server 2008 crash, replacing it with a "generic" Western Digital AV-GP 1.5TB SATA drive lets me install without problems.

Obviously the nvstor.sys driver shipped with Windows Server 2008 has problems with some drives, but not all. Imagine that a cheap server, that can run VMware ESX/ESXi right out of the box, can't run Windows Server 2008 with the HDD it came shipped with.

Now, how weird is that? Note that that wasn't tested with Windows Server 2008 R2, so the nvstor.sys file shipped with that version might not have the same problem. Also, I did not try loading newer nVidia drivers during the Windows installation procedure, because a) when using the Easy Setup CD you don't get the option to load third party drivers, and b) because after I figured out that changing the HDD helped I didn't want to try another manual installation.

Remind me again, why don't we just virtualize everything? In this instance, it would actually be easier (and quicker!) to install ESXi on the bare metal hardware, create a VM and install Windows Server 2008 in that instead of installing Windows Server 2008 on the hardware directly. How the world has indeed changed.

Update 10. March 2010:

After finishing the installation, I did run into another problem that quite possibly is also related to the nvstor.sys driver. Windows would fail in creating partitions, of the amount of space used by the partitions exceeded approximately 1TB in total.

Upgrading the server to Windows Server 2008 R2 fixed this issue, and I was able to utilize the full disk. This leads me to think that had I installed Server 2008 R2 from the get-go I would not have seen the installation issues with the original drive at all.

Over time the boot partition on a Windows Server 2003 installation might just turn out to be too small. There can be various reasons for this, but the fact remains that over time you will accumulate data on the boot drive that you didn't take account for when you set it up initially.

Luckily I run almost all of my servers in a VMware based virtualized environment, where it's easy to expand the the virtual disks. The problem is that Windows Server 2003 doesn't let you easily expand the boot volume, at least not without downtime. I've previously talked about using tools like GParted to expand the boot volume but there are easier ways to do it and prevent downtime at the same time!

All you need is love. No,wait, that's something else entirely! All you need is ExtPart. ExtPart is a lovely little 36KB tool that Dell has provided to expand partitions on Dell based servers and storage systems. It is a little known fact that ExtPart can do the job in any 32 bit Windows Server 2000 or 2003 based install (no 64 bit support, sadly), and in Server 2008 there are other methods of doing this.

Enough talk, lets get down to the business at hand.

1. Download ExtPart from the Dell download site
2. Expand your boot volume, either via the Virtual Infrastructure Client or via vmkfstools
3. Run ExtPart inside your VM to expand your boot volume to the new size

Thats it. The following screenshots outline the process very well, without having to guide you through each step. Have a look!

It can't get much simpler that this, honestly.

VMware has released Update 3 for their enterprise ESX(i) platform. Read the release notes for more details.

VMware ESX/ESXi 3.5 Update 2 users around the globe are experiencing huge problems with their infrastructure. A bug in the latest version of the VMware enterprise virtualization software might render thousands, if not more, virtual machines unbootable.

This incredible situation is caused by a licensing bug, that basically expires all licenses on August 12th, 2008. When a VMware ESX lisense expires like this the implications are that VMotion and DRS no longer works and perhaps more serious in the short term is that you will not be able to power on any VMs that are powered off. Rebooting a running VM from inside the guest OS should work fine, but do not try running a virtual power cycle on any running VMs.

VMware are working on a fix that should be be available for download by noon, PST on Aug 13th.

VMware KB 1006716 should have more info on the issue, but is not available at the time of this post. Perhaps the VMware KB is running on a VM hosted on a VMware ESX/ESXi 3.5 Update 2 itself? Most likely it's due to traffic issues though.

A suggested workaround for the issue is to disable NTP on the ESX hosts and manually set the date to some date before the 12th. I would not recommend this unless you really have to, as you might get into a world of hurt if your VMware Tools are set to sync time with the Host server. Imagine a Windows 2008 Server Domain Controller that suddenly gets it clock set to an earlier date. Active Directory does not enjoy this.

For now, if you are running ESX/ESXi 3.5 Update 2 keep your VMs running and do not power them off. Hopefully you don't have critical VMs turned off anyway, so the impact is likely to be small unless you have a critical need for DRS/HA. For those of you that haven't upgraded to ESX/ESXi 3.5 Update 2 yet, wait until the new installation media is available. I downloaded the media kits yesterday in preparation of my upgrade here at work, I'm very happy I didn't do the upgrade last night like I planned to.

Update 1:

VMware has issued a statement outlining their current status and what to expect in the time to come. They also promise an update ever two hours.

The problem is caused by a build timeout that was mistakenly left enabled for the release build.

as well as an express update

VMware Engineering has isolated the root cause and is working to produce an express patch for impacted customers today. The target timeframe is 6pm, August 12, 2008 PST.

For full details, read the whole update at the source.

Update 2:

The promised express patches has now been released. More details and downloads available at vmware.com.
New iso and tarballs are to be released sometime this week, so of you are planning an upgrade hold your horses a few more days.

VMware CEO Paul Maritz has also issued an official statement on the problems customers have experienced, and how VMware is handling it.

Update 3:

New tarballs and ISO's has now also been released and are available at the link above.

VMware has announced that ESXi will be made available for free on the 28th of July. Not very surprising as the release version Microsoft Hyper-V is now also available. I've been anticipating this move for a while now, and so has pretty much everyone who follows the world of virtualization.

But what does this mean? We all know that VMware ESX is a far superior product than the current rendition of Hyper-V, but we have also seen what Microsoft can do when they provide a "close-to-good-enough" product. Hyper-V is good enough for small SMB market customers who don't need/want enterprise features like Live Migration and High Availability/DRS.

In reality, this move by VMware moves the competition away from the hypervisor/virtualization layer, and over to management. By offering ESXi free of charge, hardware vendors can start shipping it with their servers. Some vendors already do this, but I think we'll see more pre-fabricated servers built with virtualization "built in", ready for deployment.

VMware has the edge today with their enterprise offerings, but they sure need to keep momentum up and drive management/deployment solutions into the enterprise. I wouldn't be surprised to see VMware start supporting management of Hyper-V based VMs as well as other solutions like Xen. Microsoft has already announced multi-hypervisor management in Virtual Machine Manager 2008 and VMware is sure to follow suit.

In a few words, this basically means Game On. Interesting times.