Inaccessible datastore

A strange issue hit a remote cluster in the environment the other day. A new ESXi host (physical machine, not blade) was commissioned the other day and somehow or the other resignatured a VMFS volume with a Windows type header. The effect wasnt felt at the time because it didnt affect the guest OS’s in the VM’s running on that LUN. But here’s how it showed in the Datastores view

Inactive datastore

It says (inactive) now, but when the problem actually happened it was (inaccessible). It says inactive because it’s been unmounted but because there was a template on that LUN registered with vCenter, and vCenter remembers that.

See below for the strange signature:

Capture1

After many run-arounds by VMware Support, we resorted to using vCenter Converter to V2V as many VM’s as we could in the hope of saving them. We saved 5 our of 8, not bad considering the dire situation. The other 3 couldnt be saved because they had dynamic disks. Guess why the VM’s had to be recovered this way? NO BACKUPS!

VMware couldnt tell us why the LUN was resignatured and nor could they change the header back to VMFS without formatting it. Because this was a remote site, we depended on an onsite contact (IT guy, btw, but not a person with VMware skills) to build it and hook it up to the network. To date, no one know why/how the volume got resignatured. What’s likely is that there a Windows OS was installed on the host first then ESXi was installed and the Windows partition was somehow kept. What happened after is the great unknown, no logs to pinpoint anything. Investigations continue!

Moral of the story – BACKUP YOUR VM’s!!! Or if you cant, ensure the relevant people know. Protect your own backside with email trails.

Leave a Comment

Your email address will not be published.