Cisco UCS – CIMC did not detect storage controller error

During a recent UCS firmware upgrade I had quite a few blades show up with the error “CIMC did not detect storage”. Within UCSM I could see that the blade had a critical alert. It initially started after I upgraded UCS Manager firmware as documents in a previous post I wrote about UCS Firmware Upgrades. I did some searching around to find what may be causing the issue and the best answer I could find was to from the Cisco community forums to disassociate the blade, decommission and reseat within the chassis. I later spoke to a Cisco engineer and he advised of the same steps but that it was also possible to do without reseating the blade. This also looks like its a problem when upgrading from 2.2(1d) to other versions of UCSM but I haven’t been able to validate if it’s only that version or if it also affects others.

The full error I saw was for code F1004 and for Controller 1 on server 2/1 is inoperable. Reason: CIMC did not detect storage

Within UCSM I could see there was an issue with the Blade

Before proceeding with the upgrade of the FIs, IOMs and Blades themselves I wanted to clear any alerts within UCSM, particularly critical alerts. The steps I followed to bring the blade back online were to go to the blade and select Server Maintenance

I then selected Decomission and clicked Ok

I clicked yes to verify I wanted to decommission the blade.

Next I received a notification to advise that the decommission process has started successfully.

Once the decommission had completed an alert appeared to advise that the blade needed some intervention.

I right-clicked on the blade and selected Re-acknowledge Slot.

I selected Yes to verify that it is what I wanted to do

A notification then popped up to advise that the server had been re-acknowledged.

A KVM session onto the blade showed that it went through a full detection process again

Once the blade had completed the discovery and booted into ESXi again it appeared within UCSM without any issues. The re-discover and reboot took just over 10 minutes. One thing to note is that the rediscovery will strip any labels and naming you’ve added to the blade previously so you’ll need to add them again separately.

Once the CIMC error had been dealt with I was able to continue with the firmware upgrade