The old ONID email service (mailboxes which were not migrated to Google Mail) was unavailable from 11:23am to 2:25pm today.
During a planned upgrade of Fiber Channel switch firmware, connectivity was lost to the disk storage array. FC switch firmware upgrades are normally non-disruptive to traffic, and these are redundant FC switches. However, we did not realize there was a problem until after both switches were upgraded and both redundant paths were offline. At all times, the switches themselves indicated that links were up and the zoning was correct. Only the disk storage array was effected, not any of the other ports on the switches. We finally fixed the problem by hard rebooting each switch.
In the many years we have been doing these switch firmware upgrades, we have never seen a failure case like this. We have opened a support case with the vendor in an attempt to understand why this happened.
This FC switch problem was also the root cause of the Blackboard outage today.
Andy Morgan Enterprise Computing Services