![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Entry tags:
When redundancy isn't
I'm reminded of a joking phrase used in WAN operations: backhoe fade
That's when you lose your connection because some idiot dug up the cable with a backhoe.
A rather infamous incident shut down the Internet in New England back in the late 70s.
Seems that while the customer had specified separate routing for the pair of T-1(?) lines that carried the Internet, the provider had routed the connections via separate cables... in the *same* trench.
Needless to say the customer had words with the provider. And the provider revised their rules so the "separate routing" meant different cables *routed* differently so that one accident couldn't take both out...
A manufacturing place I used to work at got separate power feeds from two different power companies because they had processes that didn't take well to sudden power loss.
One power line came in from the north, one from the south. Only single point of failure was the company's substation that they both connected to.
And they had a *large* room full of batteries to enable shutting down those critical processes gracefully.
Sadly, we still lost power a few times in the dozen years I worked there.
That's when you lose your connection because some idiot dug up the cable with a backhoe.
A rather infamous incident shut down the Internet in New England back in the late 70s.
Seems that while the customer had specified separate routing for the pair of T-1(?) lines that carried the Internet, the provider had routed the connections via separate cables... in the *same* trench.
Needless to say the customer had words with the provider. And the provider revised their rules so the "separate routing" meant different cables *routed* differently so that one accident couldn't take both out...
A manufacturing place I used to work at got separate power feeds from two different power companies because they had processes that didn't take well to sudden power loss.
One power line came in from the north, one from the south. Only single point of failure was the company's substation that they both connected to.
And they had a *large* room full of batteries to enable shutting down those critical processes gracefully.
Sadly, we still lost power a few times in the dozen years I worked there.
no subject
At that company I worked for, the plumbing contractor had to do that. Seems they'd ignored the *explicit* instruction that they were to use some special stuff (teflon tape? something else?) to seal the joints in the DI water lines.
Instead they used regular "pipe dope" or some such. So the hyper pure water was contaminated and messed stuff up.
Even better, the lines run *thru* the concrete floor slabs. So to fix them they had to snake "liners" thru them.
Many years later they had a *different* problem with the DI lines. Since they couldn't be chlorinated, they developed bacterial colonies. Which caused stains on the silicon wafers (they went *nuts* trying to figure out what was causing the stains until somebody noticed that in cleaning the hoses for the lapping machines this "sludge" came out (bacterial mats).
That led to having to shut down stuff one day and flush the lines with 20% (or was it 40%?) hydrogen peroxide. Nasty stuff, but since it breaks down to water and oxygen (and converted the organics to things like water, CO2 and nitrogen) it did the job without contaminating things.
I don't know what the long term solution was, but me, I'd have bought an ozonator like some places use for water purification, and treated the DI water with that.
no subject
Ozone treatment of water is an idea which should have been pursued sooner.
H2O2 breaks down into water, oxygen and *heat*. Great for cleaning out organics and some other materials, with little chance of contamination. Just mind the evolved heat.
Like Ozone, it's pretty neat stuff, but its potential has never been reached, due to certain problems.
Here are some of Dr. John D. Clark's comments on the attempts to use high-test peroxide as a storable oxidizer. "The cleanliness required was not merely surgical - it was levitical. Merely preparing an aluminum tank to hold peroxide was a project, a diverting ceremonial that could take days. Scrubbing, alkaline washes, acid washes, flushing, passivation with dilute peroxide —it went on and on. And even when it was successfully completed, the peroxide would still decompose slowly; not enough to start a runaway chain reaction, but enough to build up an oxygen pressure in a sealed tank, and make packaging impossible. And it is a nerve-wracking experience to put your ear against a propellant tank and hear it go "glub" - long pause - "glub" and so on. After such an experience many people, myself (particularly) included, tended to look dubiously at peroxide and to pass it by on the other side."
no subject
The Me 163 Komet used peroxide (80-85%!!) & hydrazine. Both would do nasty things to anything organic.
Oh yeah, one of thing things that will act as a catalyst to breakdown peroxide is blood. So be sure not to bleed around the stuff... :-)
Still, those are tame compared to chlorine tri-flouride (ClF3) and FOOF.