What are VMware Validated Designs?
I’ve been banging my head against the screen for the past few weeks looking at storage issues and finding orphaned volumes with reams of snapshots using up valuable disk space. In some cases it was due to manual intervention and a snapmirror or snapvault relationship was broken, in others it was caused by DFM creating new instances of the volume but not cleaning up old volumes and associated snapshots and in other cases, well I’ve no idea how they occurred. Hence why I’ve been slapping my brain around the inside of my skull. I’d be interested to know if this is still an issue with OCUM, answers on a postcard.
There’s no pretty way to clean up orphaned snapshots that are essentially owned by DFM. It’s messy, convoluted and requires that you’re very careful and precise about what you’re removing otherwise you’ll make things worse. There are a number of reasons why orphans can occur. One is down to the way SnapProtect and the DFM work together. If a VM is deleted or moved to another volume and no other VM’s that are a part of that same backup subclient exist on that volume the snapshots will not age and will require a manual clean-up process. This seems to limit the use of automated DRS in VMware, but that’s a separate issue really. Another reason, and what looks to be the cause in my case, is that DFM has intermittent issues communication to the storage controller and thinks the volume doesn’t exist so it DFM may create a FlexClone of the volume and index it to have a new suffix while still being able to access the snapshots that were already captured. This can be caused by network drop outs out by the controllers or the CPUs maxing out and not being able to reply to DFM. I’m still investigating the cause of this. If a new storage policy was created in SnapProtect with these volumes assigned it would clear out the orphans but that would involve re-baselining the backups which is not something you’d want to do, unless of course that data had to value to you.
Oh SnapProtect, how you taunt me! It’s one of those products that’s been OEM’d from another vendor so it’s missing some functionality and also means that the documentation specific to it can be sparse. Commvault documentation most likely will be sufficient but ideally there would be documentation would exist on how to perform Service Pack upgrades specifically for SnapProtect. I have a few issues with SnapProtect but I’ll leave that rant for another time. When it came to recently upgrade SnapProtect I had the issue of not finding documentation that would clarify the process so I thought I’d capture it so I can at least return to it in the future if I need to. Below are a list of steps carried out to perform the upgrade. I understand that the media agent upgrade may be flawed, and in my case I couldn’t get it to work correctly, so if someone knows what I’ve done wrong please feel free to leave a comment. I’m not an expert in either SnapProtect or its cousin Commvault. For the vast majority of SnapProtect admins this document may be superfluous but hopefully someone finds it useful.
1. Open a preemptive support case with Netapp
2. Download the software from NetApp support site, copy the installation file to the local drive on the server. The software can be access here with a NetApp login – mysupport.netapp.com/NOW/cgi-bin/license.cgi/download/software/snapprotect/10.0SP11/download.shtml
3. Open the SnapProtect Administrative console on the CommServe/SnapProtect Server. In the console right-click on the commserve, select All Tasks and take a backup of SnapProtect using Disaster Recovery backup.
Select the option as a Full backup and click Ok
4. Find the SET_XXX folder, in this case in the SnapProtectDR folder and zip it.
Following on from installing vROPS a few month back I finally made the jump to install the Blue Medora management packs for both Cisco UCS and NetApp to get greater visibility into my virtual environment and the underlying physical infrastructure. I’m really looking forward to seeing what these management packs have to offer. While I’m not going to cover off the dashboards provided by the management packs in this post it is something I plan on revisiting once it’s been in use for a while and I’ve done a bit more playing around with it. The reason I’m posting this deployment process is that despite Blue Medora having decent installation guide it’s not always 100% clear, so I’ve done this to hopefully help guide a few others through the process a bit easier.
Before you begin this deployment you can download trial versions from Blue Medora and if you want a permanent installation purchase some licenses from Blue Medora.
1: In vRealize Operations Manager go to the Administration -> Solutions
A while back I upgrade my vCenter and vSphere environment to 5.5 Update 2. As part of this upgrade VMware Tools was upgraded on most servers. Except that is of vCenter itself. This wasn’t a major issue but other issues began to arise where alerts came for disk consolidation problems. On investigation of this most KB articles were pointing towards upgrading the VMware Tools and that should fix the problem. So that’s what I tried. When running the VMware Tools installation on the vCenter VM I got an error that the VMwareTools64.msi was not a valid installation package and to find the correct package to install. I tried a number of things to get this to work but it would just not run the VMwareTools64.msi. I also couldn’t update the VM through Update Manager either.
The first step was to get the correct VMware Tools version as a standalone ISO. Since I performed the upgrade VMware have released a new version of VMware Tools, now it’s version 10, and that’s the only one that can be downloaded from the support site. The version I’m looking for is 9.4.5 and I don’t want to install version 10 without doing prior deployment to the test environment. And this all led me to Vladan’s website article called Manual Download of VMware Tools from VMware Website. Thanks to this article I was quickly able to get the VMware Tools package that I needed. You can go to http://packages.vmware.com/tools and select the VMware Tools version you need for download. The ISO was added to the ISO Datastore and mounted to the VM.
Following this I tried a number of different VMware KB articles but the one I finally found to work was KB1012693. This involved opening a command prompt, changing directory to the CD drive where VMware Tools was mounted and running the command:
Once that completed I re-ran the VMTools installation and it completed successfully. Following the server reboot the VMTools are showing as up to date in vCenter.
I had the honour, and I use that sarcastically, of having some backups failing recently following a TSM upgrade. While the reason is not clear as to why a newer version of TSM failed my guess is that how TSM sends API or other calls has changed and that’s why the error came up. The new TSM version can make API calls based on a specific version of vSphere. As the environment was upgraded from vSphere 4 to 5 etc. the original license key edition was at the top of the license chain and this is what was being interrogated by the APIs so it failed to capture a valid backup.
What we were seeing was the the backup software connecting and taking the snapshot as per vCenter GUI but the transmission aborts with the following error;
08/12/2015 17:32:39.321 : vmvddksdk.cpp (1168): VixDiskLib: Error occurred when obtaining NFC ticket for: [DATASTORE_NAME] VM_NAME/VM_NAME.vmdk. Error 16064 at 3707. 08/12/2015 17:32:39.321 : vmvddksdk.cpp (1024): vddksdkPrintVixError(): VM name 'VM_NAME'. 08/12/2015 17:32:39.321 : vmvddksdk.cpp (1054): ANS9365E VMware vStorage API error for virtual machine 'VM_NAME'. TSM function name : VixDiskLib_Open TSM file : vmvddksdk.cpp (1669) API return code : 16064 API error message : The host is not licensed for this feature
While it was not the exact issue I did find a VMware KB article which mentions removing the license from vCenter MOB (Managed Object Browser). The details however were not clear. Thankfully the community came to the rescue and I found the real solution in GSparks response from the Community thread. The overview was there but not the intimate detail which is why I’ve documented the process here.
Step 1: Read More
There’s always talk about finding a work-life balance. I think in some Utopian life that may exist but for the vast majority of us that’s not the case. Everyday life can be stressful. Work can be stressful. Supporting your family can be stressful. And depending on how things are faring out in home or work areas the scales are tipped decidedly in one direction or the other. Usually it’s never balanced. Travel is one of the best ways to re-evaluate what’s important in life and to re-assess how to best find that balance, are at least get it as close to what matters within your life. I was lucky enough to take a substantial break from work to travel with my family recently and introduce my parents to another grandchild.
I’ve been guilty in the past of putting my work in front of my home life, to the detriment of family relationships. Once we started to have children things changed but work still took preference. My ambition to succeed in my career was put ahead of most other aspects of my life. I had an issue of not wanting to let anyone down and not being able to say no and be assertive to ensure my family needs and that of my employer could both be fulfilled. I had put myself into that position by largely starting out eager to prove my worth and then getting caught out as a power-dynamic then existed that I was unable to get out of. The lesson was learnt the hard way. I’m happy to say that now I’m in a role where the correct power-dynamic exists and I have managed to hit the nice work/life balance which is something I truly believed didn’t exist before.