(A couple of regular commenters will recognize this one...)
Every working research lab has a sort of rhythm to it. There's always a collection of background sounds, in a particular pattern, that indicates that the lab is functioning properly. When I was a post-doc, the pattern was something like three mintues of white noise (the humming of the fans on the various electronic instruments, and the cooling water running through the magnet coils), followed by a sharp click (a mechanical shutter closing), then a loud clunking sort of noise (as the high-current supply for the magnetic trap switched on), followed by a high-pitched whining noise that started loud and got softer (the rotating magnetic field for the TOP trap). So, when things were working well, every few minutes you heard:
...mmmmm-click-CLUNK-EEEEEEEEEEEEEEEeeeeeeeeeeeeeeeeeeeeeeee...
Except every now and then, you got:
...mmmmm-click-BANG-SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS...
Followed by a lot of cursing, and an incredibly awful smell. (Explanation below the fold...)
The "BANG-SSSSSSSSSS" noise came from the high-current MOSFETs we used to switch the magnetic field blowing up. We were pushing somewhere around 50 amperes of current through the coils, and needed to switch it on and off in a very short time (milliseconds), so we had these special semiconductor devices that were designed to act as fast switches in the circuit. When they would fail, they would suddenly act like big resistors, and get very hot, very fast. The "BANG" was one or more of the chips exploding, and the "SSSSSS" was a cheery little orange fire that would burn for thirty seconds or so before all the flammable material in the chips was consumed.
The smell was burnt semiconductor. If you've ever broken a computer in the right way, you have some idea what this is like, but when you completely burn up a couple of one-inch square power MOSFET's, it's a whole lot worse. It's a very sharp, acrid smell, and also somehow kind of greasy. It would fill pretty much the whole basement (where the lab was), and even extend up to the first floor. One time, the chips blew when I was out at lunch, and I knew what had happened well before I got into the Physics building.
"Wait just a minute," you say, "'one time?' You mean this happened more than once"
Well, yeah. Five or six times total, if I remember correctly.
You see, we were in data-taking mode at the time, and were running around the clock. With a five-minute experiment cycle, it takes a long time to get enough data to write a paper, so there were a couple of extended periods when the grad student on the project and I would rotate eight-hour shifts in the lab, taking turns staying up all night running the experiment. The longest we went in twenty-four-hour mode was about a week, and we ran over the weekend one time.
One of those stretches was cut short by the first MOSFET explosion, and when the fire had burned itself out, we considered our options: either we could completely re-engineer the system to allow for a more robust current-switching system, which would've taken a couple of days, or we could clean up the MOSFET block, swap in some new chips, and be running again in a few hours.
So we did what any dedicated experimentalists would do, and swapped the chips. We were back in operation the next day.
The worst part of swapping in new chips was cleaning up the mess from the fire. The chips get fairly hot even when operating normally, so we had them mounted on a water-cooled copper block with some plexiglass pieces to protect the electrical connections, and every time we had a fire, we had to scrub this disgusting, greasy silicon soot off the mount before putting in the new chips. That involved half an hour with a Brillo pad, and the smell would get in your nose so far that you'd taste burned semiconductor for a couple of days. I pushed that task onto the grad students as much as possible...
We did eventually do a little bit of re-engineering-- after the third time the MOSFETs blew up, we added a third chip to the block (we had been running two in parallel, each rated for 25A, and pushing 45-50A through the coils), which slowed down the rate of fires, but didn't eliminate the problem-- I think we had three-chip fires twice.
I don't know if they ever did a more comprehensive re-engineering of the switching system. They must've, because Motorola stopped making those particular chips-- I remember being told that by an electronics supply company, and buying up their entire remaining stock (something like 15-20 chips, at something like $20 each) as a hedge against future fires. We were down to about six chips when I left the group.
Then again, some other company may have come through with a drop-in replacement. For all I know, that particular apparatus may still blow up from time to time...
I'll never forget that smell, though. Damn, that was nasty.
- Log in to post comments
More like this
I like stories like this. Before grad school I'd see scenes in Star Trek where Kirk or Scott claim that the ship doesn't sound or feel right, and I'd think they were bs. Then I started really working in a lab. I'm a believer. To this day I can't hear the sound of a big piston machine (paChoooka-paChooka....) without flashing back to my days working with a helium liquifier.
For a theorist, its usually a blue screen of death, and the noise is in the rebooting!
I'd say usually the reason I know something's gone wrong is that my laptop's fans stop whirring in less than a half hour. "Wow it's quiet... Only read one blog article ... dammit, what is it this time?!"
Ah, exploding MOSFETs. Had a demo in the robotics lab I was in some years ago, and was using a standard H-Bridge circuit with MOSFETs to control motors. The actual logic in the circuit was all done with a PAL(programmable arry logic) chip, which was nifty from a techie point of view and easier than using discrete components.
Unfortunately, you can't type a plus rather than a minus in the code if you're using discrete components rather than a programmable chip. The typo basicly shorted out two car batteries across two MOSFETs and something had to give - in this case the ceramic MOSFET case, rather explosively. More unfortunately, the MOSFETs were bolted to two heatsinks mounted in such a way that it looked a bit like this from the top:
[ ]
Which looks a little like a barrel of sorts, and when you hear an odd noise from your robot and lean over to get a good look to try to find the problem with the same sense of timing that turns mallards into duck a l'orange, it starts to look a lot like a shotgun barrel...
Happily, I only got a minor cut under one eye. Unhappily, my supervisor then decided to get all pissed off at me because the demo later that day wasn't done with the robot rolling around on the floor, but staticly. Ah, the joys of science...
I love the smell of burnt doped semiconductors in the morning!
Ahh, dead MOSFeTs. Didn't encounter them in a lab, but I have encountered them dying in a fire during many a performance. Nothing like that smell.
The best dying semiconductor trick I ever pulled off was accidentally running about 10A through a 7808. It didn't die -- but it started putting out 5V. The best comment on that was "I didn't know the 78 series regulators were fuse programmable..."
There was the little 2222A in a TO-92 case that popped so hard that it flung a glowing red piece of itself over my left shoulder. It really wasn't that impressive a bang, until you look at how small that die was in the first place.