When redundancy isn't

[kengr]

I'm reminded of a joking phrase used in WAN operations: backhoe fade

That's when you lose your connection because some idiot dug up the cable with a backhoe.

A rather infamous incident shut down the Internet in New England back in the late 70s.

Seems that while the customer had specified separate routing for the pair of T-1(?) lines that carried the Internet, the provider had routed the connections via separate cables... in the *same* trench.

Needless to say the customer had words with the provider. And the provider revised their rules so the "separate routing" meant different cables *routed* differently so that one accident couldn't take both out...

A manufacturing place I used to work at got separate power feeds from two different power companies because they had processes that didn't take well to sudden power loss.

One power line came in from the north, one from the south. Only single point of failure was the company's substation that they both connected to.

And they had a *large* room full of batteries to enable shutting down those critical processes gracefully.

Sadly, we still lost power a few times in the dozen years I worked there.

Comments

[siliconshaman]

It's the same reasoning behind having off-site backups... sometimes the single point of failure is where stuff is. One lightning strike, and pfft! all your important stuff is gone!

Although, when you think about it...the same reasoning could apply to humanity in general. We need an off-planet back-up.

[stickmaker]

I worked in the Division of Planning in the Kentucky Transportation Cabinet, Department of Highways. Among other important tasks, the highway maps were made there. That meant multiple, expensive Intergraph graphics machines.

Everyone was glad when we moved out of a ratty old building (originally constructed in the Thirties (!) as a garage (!)) into a brand new one. As specified in the contract, in case of power failure we had a building-wide battery backup designed to keep the power stable until the natural gas fueled generator came up to speed.

The first time the power went off the battery backup came on with a bang. Actually, several of them. There were burn spots on the carpet where single-use surge protectors overloaded.

The problem was that the building contract was awarded to the lowest bidder. Afterwards, there were no funds to correct the deficiencies. We couldn't even get the elevators working right until the Highway Commissioner came over for a meeting and got stuck.