post

Cisco UCS – CIMC did not detect storage controller error

by Posted in Cisco, UCS Tagged , , , , 2 Comments

During a recent UCS firmware upgrade I had quite a few blades show up with the error “CIMC did not detect storage”. Within UCSM I could see that the blade had a critical alert. It initially started after I upgraded UCS Manager firmware as documents in a previous post I wrote about UCS Firmware Upgrades. I did some searching around to find what may be causing the issue and the best answer I could find was to from the Cisco community forums to disassociate the blade, decommission and reseat within the chassis. I later spoke to a Cisco engineer and he advised of the same steps but that it was also possible to do without reseating the blade. This also looks like its a problem when upgrading from 2.2(1d) to other versions of UCSM but I haven’t been able to validate if it’s only that version or if it also affects others.

The full error I saw was for code F1004 and for Controller 1 on server 2/1 is inoperable. Reason: CIMC did not detect storage

cimc error

Within UCSM I could see there was an issue with the Blade

cimc error server blade

Before proceeding with the upgrade of the FIs, IOMs and Blades themselves I wanted to clear any alerts within UCSM, particularly critical alerts. The steps I followed to bring the blade back online were to go to the blade and select Server Maintenance
 cimc server maintenance

Read More

post

How To: Cisco UCS Firmware Upgrade

by Posted in Cisco, How-To, UCS Tagged , , , , 33 Comments

Edit: 22-Jan-2106

A recent comment highlighted that i was missing a step during Step 8: UPGRADE THE FABRIC INTERCONNECTS AND I/O MODULES. This was to manually change the primary status for the Fabric Interconnects to be the recently upgraded Fabric Interconnect. This way your environment remains accessible while the second FI is being upgraded. I’ve updated this step now to include the manual failover steps on the FI. It is a step I always follow when performing upgrades but I can’t think of why I didn’t originally put it into the post.

Recently I was tasked with performing an upgrade to our UCS environment to support some new B200 M4 blades. The current firmware version only supported the B200 M3 blades. As part of the process I performed the below steps to complete the upgrade. I split the upgrade into a planning phase and an implementation/upgrade phase. Upgrading firmware of any system can lead to potential outages, things have definitely improved on this over the past decade, but it’s imperative that you plan correctly to make the implementation process go without any hitches. With UCS firmware there is a specific order to follow during the upgrade process. The order to follow is:

  1. Upgrade UCS Manager
  2. Upgrade Fabric Interconnect B (subordinate) & Fabric B I/O modules
  3. Upgrade Fabric Interconnect A (primary) & Fabric A I/O modules
  4. Upgrade Blade firmware by placing each ESXi Host

During the upgrade process, and particularly during the Fabric Interconnect and I/O module upgrades you will see a swarm of alerts coming from UCSM. This is expected as some of the links will be unavailable as part of the upgrade process so wait until all the steps have been completed before raising an issue with support. As a caveat however, if there is anything that really stands out as not being right open a call with support and get it fixed before proceeding to the next step. During my upgrade process some of the blades showed critical alerts that did not clear and the blades needed to be decommissioned and re-acknowledged to resolve it. This problem stood out and wasn’t hard to recognise that it was more serious that the other alerts.

PLANNING PHASE:

You will need to check the release requirements from Cisco regarding the upgrade path from your current firmware version to the desired version and also the capabilities that the desired firmware version contains. The upgrade process takes some time so it’s best to review everything in advance and not have to do this on the day of the upgrade itself. For instance during this upgrade process I needed to ensure capability to support B200 M4 blades and VIC 1340 modules as they had been purchased as part of an order for a new project. The steps to carry out in the planning phase are:

Step 1: Verify the Upgrade requirements

Check the release version notes on Cisco’s website for the version you want to upgrade to. For this example we are upgrading to version 2.2

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/notes/ucs_2_2_rn.html

Read More