While off on annual leave recently I had a few minutes to spare to look through twitter and came across a tweet from Adam J Bergh (@ajbergh) about a remote code execution vulnerability in Cisco UCS Central. You can read more about the threat over on threatpost.com but the synopsis is that “an exploit could allow the attacker to execute arbitrary commands on the underlying operating system with the privileges of the root user”. UCS Central version 1.2 and earlier are affected by this so it’s time to upgrade. Particularly since the vulnerability score is at the highest severity of 10. So before I go on I want to thank Adam for his tweet and highlighting the issue in the first place.
There are different steps to perform during the upgrade depending of whether UCS Central is in standalone mode or is part of a cluster. You can find more information about both methods over on the UCS Central Install and Upgrade Guide. Some of the key things to keep in mind are the supported upgrade paths and the pre-requisites before beginning the upgrade.
- UCS Central 1.3 requires a minimum of 12Gb RAM and 40GB storage space (otherwise the upgrade will fail)
- Use the ISO image for an upgrade to UCS Central
- After the upgrade clear the browser cache before logging into the Cisco UCS Central GUI
- Make sure UCS Manager is 2.1(2) or newer
- Make sure to take a full state backup before starting the Upgrade Process
- From 1.1(2a) to 1.3(1a)
- From 1.2 to 1.3(1a)
Note: I’m running version 1.1(2a)
Some of the new features in version 1.3 include:
- HTML5 UI: New task based HTML5 user interface.
- KVM Hypervisor Support: Ability to install Cisco UCS Central in KVM Hypervisor
- Scheduled backup: Ability to schedule domain backup time. Provides you flexibility to schedule different backup times for different domain groups.
- Domain specific ID pools: The domain specific ID pools are now available to global service profiles.
- NFS shared storage: Support for NFS instead of RDM for the shared storage is required for Cisco UCS Central cluster installation for high availability.
- vLAN consumption for Local Service Profiles: Ability to push vLANs to the UCS Manager instance through Cisco UCS Central CLI only without having to deploy a service profile that pulls the vLANs.
- Support for Cisco M-Series Servers.
- Connecting to SQL server that uses dynamic port.
- Support for SQL 2014 database and Oracle 12c Database.
I’m really looking forward to seeing what the new HTML 5 UI is like. The initial screenshots I’ve seen are awesome. There’s a nice little introduction from Cisco over on their support site. Also, Jacob Van Ewyk has written a really informative article over on Cisco Communities with details about the UCS Central User Interface Reworked with UCS Central 1.3.
A recent comment highlighted that i was missing a step during Step 8: UPGRADE THE FABRIC INTERCONNECTS AND I/O MODULES. This was to manually change the primary status for the Fabric Interconnects to be the recently upgraded Fabric Interconnect. This way your environment remains accessible while the second FI is being upgraded. I’ve updated this step now to include the manual failover steps on the FI. It is a step I always follow when performing upgrades but I can’t think of why I didn’t originally put it into the post.
Recently I was tasked with performing an upgrade to our UCS environment to support some new B200 M4 blades. The current firmware version only supported the B200 M3 blades. As part of the process I performed the below steps to complete the upgrade. I split the upgrade into a planning phase and an implementation/upgrade phase. Upgrading firmware of any system can lead to potential outages, things have definitely improved on this over the past decade, but it’s imperative that you plan correctly to make the implementation process go without any hitches. With UCS firmware there is a specific order to follow during the upgrade process. The order to follow is:
- Upgrade UCS Manager
- Upgrade Fabric Interconnect B (subordinate) & Fabric B I/O modules
- Upgrade Fabric Interconnect A (primary) & Fabric A I/O modules
- Upgrade Blade firmware by placing each ESXi Host
During the upgrade process, and particularly during the Fabric Interconnect and I/O module upgrades you will see a swarm of alerts coming from UCSM. This is expected as some of the links will be unavailable as part of the upgrade process so wait until all the steps have been completed before raising an issue with support. As a caveat however, if there is anything that really stands out as not being right open a call with support and get it fixed before proceeding to the next step. During my upgrade process some of the blades showed critical alerts that did not clear and the blades needed to be decommissioned and re-acknowledged to resolve it. This problem stood out and wasn’t hard to recognise that it was more serious that the other alerts.
You will need to check the release requirements from Cisco regarding the upgrade path from your current firmware version to the desired version and also the capabilities that the desired firmware version contains. The upgrade process takes some time so it’s best to review everything in advance and not have to do this on the day of the upgrade itself. For instance during this upgrade process I needed to ensure capability to support B200 M4 blades and VIC 1340 modules as they had been purchased as part of an order for a new project. The steps to carry out in the planning phase are:
Step 1: Verify the Upgrade requirements
Check the release version notes on Cisco’s website for the version you want to upgrade to. For this example we are upgrading to version 2.2
Recently during some prep work for a UCS firmware upgrade I noticed that there was a major alert showing for keyring certificate being invalid. At first I was a bit concerned but since it didn’t affect my login to UCS Manager I assumed it wasn’t too serious. After a bit of searching around the internet I found from Cisco’s site the CLI Configuration Guide for UCS (page 6) which shows the quick and easy fix to the problem.
Open an SSH session to the IP address/hostname of UCS Manager. It will connect to the primary Fabric Interconnect. Enter the commands in the order of steps below:
Step 1 UCS-A# scope security
Step 2 UCS-A /security # scope keyring default
Step 3 UCS-A /security/keyring # set regenerate yes
Step 4 UCS-A /security/keyring # commit-buffer
After the commit-buffer command has been issues all GUI sessions will be disconnected and you will need to log in again. When you log in next time you’ll be prompted to accept the new certificate. Once accepted UCSM will open and the alert will now be gone.
All fairly quick and painless!