Fix: VMware – Quiesced Snapshots failing – Unexpected error DeviceIoControl

I ran into an interesting problem that took a bit of digging around to both find the root cause and also to find the final fix. When running backups on Vmware 5.5 running on NetApp storage I could see some, but not all VMs, failing and throwing up the below errors in the event logs

Event ID 57 ntfs Warning
The system failed to flush data to the transaction log. Corruption may occur.

Event ID: 137 ntfs Error
The default transaction resource manager on volume \?Volume{806289e8-6088-11e0-a168-005056ae003d} encountered a non-retryable error and could not start. The data contains the error code.

Event ID: 12289 VSS Error
Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\?fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 00000000000004A0,0x00560000,0000000000000000,0,0000000000353B50,4096,[0]). hr = 0x80070001, Incorrect function.

The key alert here is Event ID 12289. It was also the most off-putting. It initially looked like a floppy drive issue but there was no floppy drive attached to the VM nor were there any floppy drivers installed on the VM. A look around the VMware community forums led me to this posting – https://communities.vmware.com/thread/309844?start=0&tstart=0 It was focused more on vSphere 4.1 however and most of the advice was around installing an older version of VMware Tools. Comment 27 was the jackpot winner. The System Reserved partition was causing the issue.

So what does the System Reserved partition do?

The System Reserved partition contains the Boot Manager and Boot Configuration data that are read on start up of the virtual machine. The VM boots from the boot loader n the System Reserved partition and then boots Windows from the System drive. It is also used as a location for the start up files for BitLocker Drive Encryption. If you need BitLocker then you’ll need to have a System Reserved partition. For Windows client OS’s then that’s a great feature to have but from a server OS perspective where BitLocker just isn’t used then it’s superfluous. The System Reserved partition is created by default on OS installation so there’s two options to remediate.

1. Remove the partition manually post installation
2. Remove the partition from your Windows OS templates

I won’t go into the details on how to remove the partition from your templates here but you can find more information over on mydigitallife.info which can be used. I ran through the steps myself to do this for all of our Windows templates following finding the root cause of the initial error.

As per one of the links mentioned in Comment 27 in the VMware communities post it’s possible to change the location of the boot files so that the partition can be removed. This information can be found over on geekshangout.com. However the steps didn’t include how to re-claim that partition so that there isn’t an unallocated disk partition sitting in front of the C drive (disk 0). While I haven’t tested backups in this configuration I wouldn’t be surprised if it cause other issues during backup. So below I’ve listed the steps to follow so you can successfully remove the partition as per the steps on geekshangout and then re-claim the space on gparted.

Delete System Reserved partition and reclaim space

The first preparation step in this whole process is to download the gparted software and upload it to your ISO datastore. Gparted can be downloaded from here –http://gparted.org/download.php. Next you’ll need to take a snapshot of the server you are going to modify within vCenter. This is your fallback should there be any issues. Modifying the System Reserved partition and moving the boot records is a very disruptive procedure and can render your virtual machine unusable afterwards so make sure you have had a successful recent backup that resides within your Recovery Point Objective (RPO) and that you have a snapshot of the virtual machine.

Step 1 -Deactivate and delete parition

1.1 Log into the VM as the local administrator or an account that has local admin access or domain admin account

1.2 Open a command prompt with admin privileges

1.3 Type the command bcdboot c:windows /s c:

1.4 You should get a message similar to: Boot files successfully created.

1.5 Open the Disk Management GUI (you could use diskpart for scripts), locate the C: partition right-click and select “Mark Partition as Active”, select yes to the “do you want to continue message”

1.6 Reboot the server and confirm that it can reboot successfully.

1.7 Log into the server again and open the System Manager console. In Disk Management right-click on the System Reserved partition and select Delete Volume. Click Yes on the warning.

1.8 Now comes the fun part, using GParted to re-claim the space. Shut down the server again.

 

Step 2 – Re-claim unallocated space using GParted

2.1 Edit the settings of the VM and attached the GParted ISO to the CD drive

2.2 You will also need to edit the BIOS Options under VM Options so there is a wait period where you can select to boot from the CD. I’ve set it to 3000 ms. There is a bit of cat and mouse that can take place on boot up so you might find that you have to restart the VM a couple of times so you can catch the boot delay and hit ESC.

2.3 Boot the VM and press ESC at the prompt and select to boot from CD-ROM. If prompted to press any key please do so. This will bring up the gparted screen. Leave the default settings and click enter.

2.4 Select Don’t touch keymap

2.5 Move down to Ok and press enter. Press enter for the language

2.6 Press enter again to leave the defaults for the mode

2.7 You will now see the GParted console with the unallocated 101MiB disk in front of the primary disk. There is also another 2Mib unallocated but don’t worry about this one.

2.8 Select the primary disk and click on Resize/Move. Drag the highlighted green bar to the left. You will see a slight grey box to the left of the green one, this is the system reserved partition removed earlier.

2.9 You cannot have 0 Mib leading in front of the disk so this needs to be set to 1. The free space following size will most likely become 102Mib. Add this to the New Size space, i.e. change New size from 102207 to 102399 and the free following will become 0. Then click on Resize/Move

2.10 It will then show you the new disk layout

2.11 Nothing has really been committed yet so click on Apply and in the pop-up click on Apply also

2.12 The reconfiguration of the partitions will now run and can take anywhere between 5 and 30 minutes depending on the size of the disk.

2.13 Once the action has completed you can now power down the server.

 

Step 3 – Recreating boot records

At this stage the server will not boot. It will give a message that the boot files are missing or that the required device isn’t connected. At this point we need to use diskpart to recreate the boot records. In this environment the SCSI controllers are set to Paravirtual but when running diskpart as part of the boot up the server cannot see those disks are the driver for Paravirtual is only loaded on boot up time. In order to get around this problem the SCSI controller needs to temporarily set to Paravirtual. If the server already has LSI Logic SAS already then you can ignore the steps to convert the controller and change it back afterwards.

3.1 Once the server has been shutdown from the previous step Edit the Settings and change the attached CD drive to the Windows OS ISO file.

3.2 Next the SCSI Controller needs to be set to LSI Logic SAS. Click on SCSI Controller and select Change Type. Select the LSI Logic SAS and click Ok

3.3 Normally this is fine but in this instance I got an error message as I was modifying the vCenter server itself. For all other servers I didn’t get the below error message or need to perform any addition steps. If you don’t get an error that “Access to resource settings on the host is restricted to the server that is managing it” then you can move onto step 3.4. This issue is caused by a lock being on the ESXi host to vCenter and is reset by restarting the required services.

3.3.1  Enable SSH on the host. You can do this from Configuration-> Software -> Security Profiles -> Services. SSH into the ESXi host Restart the hostd service and the vpxa service by running the commands below. You will need to log into the vSphere client again once completed.

/etc/init.d/vpxa stop
/etc/init.d/hostd restart

3.3.2 Log into vCenter and now edit the settings again and change the controller type to LSI Logic SAS and the error should go away

3.4 Press ESC on boot up and select CD-ROM from the boot options

3.5 At the initial Windows screen press Shift+F10. This will open a command prompt. Type diskpart to begin the procedure and then type LIS DIS to get a list of disks

3.6 Note down the Disk number and then type SEL DIS <Disk#> in this case SEL DIS 0. Then type LIS PAR to get a listing of the partitions

3.7 Note down the Partition Number and then enter SEL PAR <Partition#> in this case SEL PAR 1 and then type ACTIVE, when you get a result as shown below, type EXIT to leave Diskpart.

3.8 We must rebuild the BCD because it was stored on the partition we just deleted. At the command prompt type BOOTREC /REBUILDBCD. Press Y when requested. If this option doesn’t appear then the server will need to be recovered from backup or snapshot.

3.9 Once that is completed you can shut down the server again and for those that need to change the SCSI controller type backup to Paravirtual. Also set the boot delay to 0 and set the CD drive to Client Device.

3.10 Now boot the VM. You may get a notification that the drive required a CHKDSK to be run, let it run to completion and it will reboot the server automatically.

3.11 Log into the server when it boots, go to the System Manager and check Disk Management. The Disk 0 should now appear as one drive without any partitions.

Next time you run the backup you won’t get those errors anymore. It can take some time to retro-fit these changes to all servers in your infrastructure. My recommendation would be to gather a full list of VMs that exhibit this issue and space the changes out over a course of months following your local change processes.