How To: Cisco UCS Firmware Upgrade

Edit: 22-Jan-2106

A recent comment highlighted that i was missing a step during Step 8: UPGRADE THE FABRIC INTERCONNECTS AND I/O MODULES. This was to manually change the primary status for the Fabric Interconnects to be the recently upgraded Fabric Interconnect. This way your environment remains accessible while the second FI is being upgraded. I’ve updated this step now to include the manual failover steps on the FI. It is a step I always follow when performing upgrades but I can’t think of why I didn’t originally put it into the post.

---

Recently I was tasked with performing an upgrade to our UCS environment to support some new B200 M4 blades. The current firmware version only supported the B200 M3 blades. As part of the process I performed the below steps to complete the upgrade. I split the upgrade into a planning phase and an implementation/upgrade phase. Upgrading firmware of any system can lead to potential outages, things have definitely improved on this over the past decade, but it’s imperative that you plan correctly to make the implementation process go without any hitches. With UCS firmware there is a specific order to follow during the upgrade process. The order to follow is:

1. Upgrade UCS Manager
2. Upgrade Fabric Interconnect B (subordinate) & Fabric B I/O modules
3. Upgrade Fabric Interconnect A (primary) & Fabric A I/O modules
4. Upgrade Blade firmware by placing each ESXi Host

During the upgrade process, and particularly during the Fabric Interconnect and I/O module upgrades you will see a swarm of alerts coming from UCSM. This is expected as some of the links will be unavailable as part of the upgrade process so wait until all the steps have been completed before raising an issue with support. As a caveat however, if there is anything that really stands out as not being right open a call with support and get it fixed before proceeding to the next step. During my upgrade process some of the blades showed critical alerts that did not clear and the blades needed to be decommissioned and re-acknowledged to resolve it. This problem stood out and wasn’t hard to recognise that it was more serious that the other alerts.

PLANNING PHASE:

You will need to check the release requirements from Cisco regarding the upgrade path from your current firmware version to the desired version and also the capabilities that the desired firmware version contains. The upgrade process takes some time so it’s best to review everything in advance and not have to do this on the day of the upgrade itself. For instance during this upgrade process I needed to ensure capability to support B200 M4 blades and VIC 1340 modules as they had been purchased as part of an order for a new project. The steps to carry out in the planning phase are:

Step 1: Verify the Upgrade requirements

Check the release version notes on Cisco’s website for the version you want to upgrade to. For this example we are upgrading to version 2.2

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/notes/ucs_2_2_rn.html

Check the required minimum software version as this should be the one that you should update to. This depends on what new infrastructure you are adding such as new blade type or VIC module.

Also check the Capability Catalog support to ensure that the CPU and RAM. In this instance we are using UCS-CPU-E52670D CPU.

Check the VIC type support also, we are using the latest VIC 1340 which is support in release 2.2 (3a). You may notice that that version also supports the UCS B200 M4 with the CPU listed above so we could go to 2.2 (3a) but it’s best to move to the minimum suggested software release from Cisco.

Review the resolved caveats to see if any issues from a previous release (such as 2.2(3a) has been resolved in the most recent release, in this instance 2.2(3d))

Step 2: Verify Interoperability

Check the interoperability matrix site also to check if there are any specifics relating to your version of ESXi

Note 19: Cluster-on-Die feature is not supported. For VmWare, please see http://kb.vmware.com/kb/2087032

Note 37: B200 M4 and C220/C240 M4(SFF/LFF) requires ESX 5.1 U2 patch P05 Cisco Custom ISO

Read the VMware guide to see if there are any extra items required to get the system working:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2087032

 

Step 3: Review Faults and Open Case with Cisco TAC

Launch a browser and log into UCSM with your Administrative account. Once you login you will be presented with the UCS Manager screen. By default it will open on the Equipment tab from where you can expand each component to get a more in-depth view of the infrastructure

Review the faults on each UCS Chassis or just click on one of the links in fault summary in the top left. This will open up the list of faults for the fabric.

To view the full alerts go to Admin -> Faults, Events and Alerts. It is important to remove critical and major alerts before completing the UCS upgrade. In this instance the critical alert is due to grace license period expired on UCS Central. For the critical issues it is best to open a support call with TAC to advise that a firmware upgrade is going to be performed to ensure that there are no recommended steps missing. In this instance the error for UCS Central could safely be ignored.

 

Open a support call with TAC to ask them to quickly review the upgrade procedure you are going to follow and it’s good to clarify that there are no other issues to be faced. Also, it gives you a support case to reference when you perform the upgrade in case there are issues and you need to quickly escalate any problems.

The major alert is due to Keyrings certificate is invalid. I showed how to resolve this issue in a separate blog post

Another alert that has appeared is for FCoE membership down which has shown to be a bug but it’s worth checking with TAC if this is the case. More information on the bug can be found here:

https://supportforums.cisco.com/discussion/11743596/ucs-fcoe-vsan-1-down

 

UPGRADING PHASE:

STEP 4: DOWNLOAD FIRMWARE 

Download the Cisco UCS Firmware from cisco.com. You will need a valid Cisco Support login to download this firmware.

1. Open a Web browser to software.cisco.com. Select Upgrade and Update.

2. In the Products window select Servers –Unified Computing, then select Cisco UCS B-Series Blade Server Software

1. In the Download Software screen select Unified Computing System (UCS) Server Software Bundle

4. A new window will appear displaying the versions of software for the Cisco UCS B-Series Blade Server Software. Select the highest version of suggested firmware, which in this example is 2.2(3d).

5. There are three binaries to download, an Infastructure Software Bundle (ucs-k9-bundle-infra.2.2.3d.A.bin),  a Software for the UCS B-Series blade server products (ucs-k9-bundle-b-series.2.2.3d.B.bin) and a Software for the UCS C-Series server products (ucs-k9-bundle-b-series.2.2.3d.C.bin). Click Download for each binary file.

1. Accept the End User License Agreement

 

STEP 5: UPLOAD FIRMWARE BINARIES to UCS MANAGER

1. Select the Equipment tab -> select the Equipment tree ->then select Firmware Management tab.

2. Select the Download Tasks tab and press the Download Firmware button.

1. Select location of the file as Local File System.

2. Press the Browse button and locate the three binary files downloaded earlier.

3. Select the first binary (A.bin) and press the OK button.

1. Click ok to view the Transfer State

1. Click Ok on the Download Task dialog to continue

1. Under Download Tasks you can now see the new file has been uploaded. It is confusingly labelled as having a Transfer State as Downloaded

1. Do the above steps to import the B and C bin files.

2. Next go to Firmware Management -> select the Packages tab. The three binaries should now be available

1. Select the Installed Firmware tab. Expand all the items to view the Running, Startup and Backup Firmware versions

 

STEP 6: TAKE BACKUP 

1. Take a backup of the UCS configuration before continuing. Select the Admin tab -> select the General tab -> then select Backup Configuration.

1. Select Download to Local System, select type as Full State and ensure the file extension on the local file system is set to .xml.

2. Select Browse and go to c:/Temp on your local drive and enter the file name. Click ok

3. Click ok to complete the backup

STEP 7: UPGRADE UCS MANAGER

1. Select the Equipment tab -> select the Equipment tree -> select Firmware Management tab -> then select the Installed Firmware tab.

1. Select UCS Manager at the top and click the Activate Firmware button.

2. In the Activate Firmware dialog select the new version of firmware from the Version to Be Activated drop-down menu and click ok (Note: The screenshot below shows 2.2(1d) but the version selected to activate was 2.2(3d))

1. UCS Manager will now upgrade. You will lose access to UCSM during the upgrade. Sit tight for a few minutes and it will be available again.

2. Log in again to UCSM and accept any Java warnings.

6. Select the Equipment tab -> select the Equipment tree -> select Firmware Management tab -> then select the Installed Firmware tab and verify that the Running Version has changed (2.2(3d))

Note: You may see some new alerts and warnings after you upgrade UCSM firmware versions. I’d recommend investigating them to ensure there are no obvious issues before continuing with the FI upgrades. Some issues I encountered involved CIMC errors on the blade and also critical alerts for ethlanflowmon. You can find out more on both issues here:

Cisco UCS – CIMC did not detect storage controller error

Cisco UCS – FSM:FAILED: Ethernet traffic flow monitoring configuration error

 

STEP 8: UPGRADE THE FABRIC INTERCONNECTS AND I/O MODULES

1. Select the Equipment tab -> select the Equipment tree -> select Fabric Interconnects

2. Select Fabric Interconnect A (primary) -> select High Availability Details on General Tab to expand

3. Verify that HA settings are as follows: Ready: Yes, State: Up, Leadership: Primary, Cluster Link State: Full.

4. Select Fabric Interconnect B (subordinate) -> click Activate Firmware in Actions.

5. From the drop down menus select the desired Kernel and System Versions and then click OK.

6. Fabric Interconnect B will now upgrade including the I/O modules. This process will take approximately 20 to 25 minutes. You can monitor the progress of the firmware upgrade from the FSM tab.

1. Use the FSM tab to monitor the progress of the firmware upgrade. Once the progress reaches 100% you can continue with Fabric Interconnect A.

1. Connect via SSH directly to Fabric Interconnect IP address. Once connected you can run the below commands to make the newly updated fabric interconnect the lead FI within the environment. This stops your systems from going offline during the upgrade of the A fabric interconnect. The change over will be pretty much instantaneous.

UCS-A# connect local-mgmt
UCS-A(local-mgmt)# cluster lead b
UCS-A(local-mgmt)#

1. Now that Fabric Interconnect B has taken over the primary role, back in the UI select Fabric Interconnect A (primary) -> click Activate Firmware in Actions.

10. From the drop down menus select the desired Kernel and System Versions and then click OK.

11. Fabric Interconnect A will now upgrade including the I/O modules. This process will take approximately 20 to 25 minutes. You can monitor the progress of the firmware upgrade from the FSM tab. You may lose connectivity to UCSM during this upgrade process as it’s the primary Fabric Interconnect that you are connected to and it will need to reboot. If this occurs just log into UCSM again and it will connect to the remaining Fabric Interconnect.

12.  Select the Equipment tab -> select the Equipment tree -> select Firmware Management tab -> then select the Installed Firmware tab.

1. Check that the Fabric Interconnects and I/O Modules have a Running Version of 2.2(3d).

2. Now that both Fabric Interconnects have been upgraded you can go ahead and return to primary function back to Fabric Interconnect A. Please note that this is an optional step. Connect via SSH directly to Fabric Interconnect B IP address. Once connected you can run the below commands to make the newly updated fabric interconnect the lead FI within the environment. This stops your systems from going offline during the upgrade of the A fabric interconnect. The change over will be pretty much instantaneous.

UCS-B# connect local-mgmt
UCS-B(local-mgmt)# cluster lead a
UCS-B(local-mgmt)# 

STEP 9: UPGRADE BLADE SERVERS

1. Select the Servers tab -> select the Policies tree -> expand Host Firmware Packages.

2. Enter the Name as per your companies naming convention e.g FWP2.2-3d. Enter a description and select Simple for How would you like to configure the Host Firmware Package?

3. From the drop down menus  select the desired firmware version  for Blade and Rack packages and then click ok.

1. Select Maintenance Policies and verify that the Reboot Policy is set to User Ack. Otherwise the ESXi hosts will reboot automatically once the firmware upgrade process begins

 

5. Now you are ready to start upgrading the Blade firmware of your ESXi hosts. Go into vCenter and select the ESXi host that you want to upgrade. Place the ESXi host into Maintenance Mode

6. For Service Profiles that are bound to a service profile template the change can be made across the board quite easily. Select the “Servers” tab, select Service Profile Template, select root, select Sub-organizations and then select the Service Profile Template Name. Select Service Profile Template -> select root -> select Sub-organizations -> select the Service Profile Template Name of the associated ESXi hosts that you want to update.

1. Select the Policies tab and then expand Firmware Policies.  From drop down menu select the newly created Host Firmware policy and click Save Changes.

1.  Select the General tab and press Reboot now from the Pending Activities section. You can select multiple hosts to reboot

2. As part of the blade upgrade the following will be upgraded – BIOS, CIMC Controller and Board Controller. This can take between 35 and 60 minutes per blade

9. Use the FSM tab to monitor the progress of the firmware upgrade.

1. Once the FSM progress shows 100% you can exit Maintenance Mode of the upgraded ESXi host.

2. Verify that all firmware has been upgraded. Select the Equipment tab -> select the Equipment tree -> select Firmware Management tab -> select the Installed Firmware tab. Ensure that the same version appears for all components