Last weekend I had a small task to update BIOS and other firmware on a non-mission critical DELL PowerEdge T310 server. To safe myself a site visit I decided to to do this remotely via iDRAC.
I initially planned to run the updates from DELL Lifecycle Controller, but this was failing with error "The updates you are tying to apply are not Dell-authorized updates" due to out of date version of the Lifecycle Controller itself.
To get around this, I decided to use DELL firmware update LiveDVD, that can be downloaded from here. I mapped the Centos65-OM74-Firmware-LiveDVD.x86_64-1.1.0-Build6.1.iso file to the server via iDRAC and booted from it. Few minutes later, the firmware update utility loaded and gave list of components eligible for an update.
I selected all components that were eligible and started the process. The update was running happily for about 10-15 minutes. Then I left my desk for a few minutes and when I came back the iDRAC client seemed to be frozen. I tried to PING local iDRAC interface IP address from another machine on the same network, but there was no response... At that point the only option left was to drive to the site and check what had happened.
When I got to the site I found the server stuck on LiveDVD boot screen:
All the fans were spinning at the maximum speed and server's little LED screen was completely blank. After a quick reboot the server spent around 1 minute sitting at completely blank screen. When POST messages finally appeared, under the boot options, instead of F10 - System Services, there was text "System Services Disabled".
After another minute or so trying to initialize iDRAC, the server displayed error message "Alert! iDRAC 6 not responding. Rebooting" and rebooted. I then power cycled the server by unplugging it from mains for a few minutes. After this, the server booted normally with no error messages.
Since I wasn't watching server's display when the failure happened, it's a bit of a mystery what exactly caused it. Almost all firmware updates that I had initiated via LiveDVD succeeded, with the only exception being BIOS, which was still on the original version. This can probably be explained by the fact that BIOS update requires a successful reboot to complete. It's also possible that BIOS update failed causing iDRAC to freak out...
The most bizarre thing was that even though I wasn't trying to update iDRAC vith LiveDVD (since it was already up to date), during the crash iDRAC firmware somehow managed to downgrade itself from v1.99 to V1.54... It was actually upgraded to v1.99 just couple of days ago, I suspect this could have been related to the failure...
DELL PowerEdge T310
