Over the last couple of months, I have received several calls from a client whose main output breaker for one of their UPS systems kept tripping for no known reason. There are five 750kVA systems at this site that feed the critical load.
The suspect breaker was an old-style type with no electrical meters or anything on it to give us visibility of what was causing the breaker to trip. Although we have meters downstream at the PDU level that show no faults, we were very curious as to why this breaker would randomly trip – sometimes on Sunday night, sometimes Saturday, sometimes Friday. The load automatically shifted over to the backup UPS, not affecting the Data Center. Still, it was certainly disconcerting to have an unidentified problem like this keep happening.
We installed monitoring on the breaker to figure out if it was indeed the breaker that was tripping or possibly a fault downstream somewhere. With this much energy, it is very difficult to have something cause a large 1600amp breaker to trip without causing other problems (like lots of sparks etc.)!
We decided to replace all the 22-year-old breakers after our testing showed faults. We made calls all over the country, and guess what, you cannot get any new breakers anywhere! Everybody is quoting eight-month lead times (OUCH)! We had to resort to the next best plan: take our load bank breaker (same type and size) and replace the defective main UPS output breaker. This bought us some time to get the new breakers and matching cabinets on order.
So, I loaded up all my electrical testing gear and tools and flew up to Sacramento to meet with the client and their contractors to get the breaker changed out on a Tuesday night. While we were inside this electrical output board cabinet, we found some loose buss connections, not really loose, but loose enough to reveal no one has done a Preventative Maintenance on it in 22 years!
As you may have guessed, we did the Preventative Maintenance while we were there, too! While there, we also found some pieces of metal left from the installing contractor years and years ago. We removed those pieces because they will create problems down the road. After vacuuming and cleaning inside the cabinet to get the dirt out and make it look like new, we load banked the new breaker, and everything worked fine! They have a completely new output breaker and have not had any more nuisance trips like those that occurred in the last six months.
So, note to self, try to order breakers way in advance so that this does not happen again. I know you really cannot foresee this stuff, but it does happen. The customer is happy, and I am not getting those repetitive evening calls that something else has failed. As soon as the new breakers and boards are delivered, we will put them in on a weekend. The customer will have new breakers and cabinets that will last them another 25 to 30 years! (I hope)
In the action photos provided, you will notice that it shows me with some hand tools – to create the illusion that I am working! I airbrushed those in! (Just kidding) I really have tools in my hand. Anyways, it was a lot of fun. Have a great day, and I look forward to seeing you all soon.
Arnie
