Here be dragons: Preventing static damage, latchup, and metastability in the 386
righto.com100 points by todsacerdoti 4 days ago
100 points by todsacerdoti 4 days ago
But as high-power transistors were developed, SCRs fell out of favor. In particular, once an SCR is turned on, it stays on until power is removed or reversed; this makes SCRs harder to use than transistors.
SCRs, also known as thyristors, are still widely used in very high power applications.
Thyristors in the strict sense of the word, i.e. SCRs, are still used only in a very small fraction of the applications that were dominated by them a few decades ago.
Classic thyristors are useful only for controlling 50/60 Hz AC consumers.
Many applications that were done in the past with normal thyristors, for lack of alternatives, like electric motor control, are done now with GTO (gate turn-off) thyristors or with IGBTs (insulated gate bipolar transistors) or with silicon carbide transistors.
Even the use of GTO thyristors is regressing. If any thyristors will still be used in the future, they will use silicon carbide, with which they can be much faster than the silicon thyristors.
in AC power network, the power is turn off 100/120 times every second, so using SCR is perfectly fine.
“Intel recommends an anti-static mat and a grounding wrist strap when installing a processor to avoid the danger of static electricity, also known as Electrostatic Discharge or ESD.1”
You know back when I built my computers, not once did I ever use any kind of static electricity discharge “system”. No wrist strap, no mat, no anything. And I don’t know anybody who did.
Has anybody ever actually destroyed a chip with static electricity?
(Of course it could be the climate I lived in as well)
I have had to deal with several electronic cards that were unusable upon arrival due to being shipped in unsuitable packaging.
The life of a shipping container can be complicated and include being struck by lightning, the sea breeze, and 50°C in the sun in a car park.
Intel is covering its ass because there's always a subcontractor from an oil drilling company who will set up equipment in the middle of a sandstorm and say he didn't know it was a bad idea.
Yes, although it's not very likely.
But keep in mind that final assembly and packaging is typically happening in large, air-conditioned halls with vinyl floors, conveyor belts, plastic office chairs, disposable coveralls, etc. There's more static zaps in places like that than in a home with wooden floors, reasonable humidity, and casual clothing.
And then, as Ken notes, there's the question of scale. If you statistically kill one chip in 200, you might not even notice that in a home lab. But for a manufacturer, that's more faulty devices shipping to customers than they want.
> reasonable humidity
Maybe that's the biggest factor? Some places are very dry though
Risk of ESD increases exponentially as humidity decreases. 40% is the recommended minimum. But, humidity is not sufficient to ensure safety.
Equipotential bonding is required (commonly "grounding", but the important bit is that all conductors are at the same electrical potential). For personnel, the bonding device MUST include a 1 MOhm series resistor.
Items which have not been in sufficient humidity for at least 48 hours are at increased risk.
Assuming your most sensitive component has a nominal ESD survivability of 250V, the maximum charge difference between items must be less than 100V.
Just crossing your legs or standing from a chair can generate very high static voltages. Removing a plastic film near exposed components is not recommended.
Charge generation happens whenever surfaces separate, including when they slide or roll.
If you want to discharge non-conducting materials, you need a balanced stream of ions, but that's beyond most home or field-service setups.
Sure, I've seen enough motherboards with fried USB controllers caused by an ESD while plugging in USB memory sticks.
This is in a climate with fairly cold winters (-40°C and below isn't unheard of), so layers of wool clothing and very low humidity. It's been less of a problem recently because modern motherboards come with ESD protection, but 10-15 years ago shared computers with most USB ports no longer working were the norm.
I always touch ground before working on electronics, and often get zapped. It's a fairly common practice here AFAIK.
One thing I remember from my time in the CPU industry is that ESD damage can be cumulative and also can have a delayed effect. So just because you handle a device without precautions today, doesn't mean it won't fail at some time in the future as a result. That said, I've never used precautions in home/hobby projects.
Cumulative damage is correct. Damage may at first only consume some of the ability of the protection devices, leaving the product more vulnerable to future events. Of course functionality testing will not reveal this. Once the product exhibits a failure, even with STM examination there's no way to differentiate ESD from lightning strike.
I damaged an embedded development board by walking across a carpeted room before touching it. When I touched it I heard, felt, and saw the zap and one of the IO ports was stuck after that.
Part of it is that the incentives are different for you and for the chip manufacturer. You're not going to notice if, hypothetically, one in a hundred processors gets fried from careless handling. But a 1% return rate is a huge cost to Intel that they would want to avoid.
At one company I worked for that was making embedded devices we had a period of unusually high rate of USB hardware failures in new devices. It was not really conclusively investigated but from what I've heard it was likely a period of low humidity and the people working on assembly not wearing wrist straps consistently.
I never have, and I've been in embedded for ages. So I've dealt with my fair share of chips, consumer electronics and not.
But one vital thing to understand is that a lot of those "vendor recommendations" exist to cover for rare 1% to 0.1% edge case failures.
You can put together 20 PCs, with none of them dying from ESD, and conclude that ESD "isn't a real issue". But if you have a company that puts together tens of thousands of PCs per month? Then those ugly 0.1% edge case failures WILL pop up and they WILL cost you. And if you employ enough people, one of them might be a son of Zeus with a wild Wimshurst machine hairstyle - capable of emitting two high power ESDs, complete with an audible crackle and a visible spark, per minute. So ESD straps it is.
The same applies to things like humidity control or reflow profiles for electronics. Not an issue ~99% of the time. The remaining ~1% can fuck you over in mass manufacturing, so disrespect the vendor at your own peril.
You will fry something if you don't use anti static measures and work on enough boards.
Moisture, clothing, habits play a role so it's highly variable.
It's funny, back in an earlier job, I'd keep a board plugged into a programmer, which in turn was plugged into a USB port on my laptop. Whenever I left the restroom, my hands would be slightly wet, and touching the laptop would give me a small shock. Somehow, this shock would reset the board every time, even through the indirect connection.
I'm surprised that nothing ever actually got fried in that job. (Except for a company laptop that mysteriously bricked itself after I tried rebooting it.)
That's actually fine because you were probably going straight to ground. When things are properly grounded and the small components are covered, the current will flow through the thick ground conduit (the laptop body, thick copper on the PCB, eventually into the power cable).
What you'd like to avoid is releasing that static charge through a tiny component on the board that couldn't handle the surge.
Yeah I'm pretty sure I've seen processors and memory both get eaten by ESD. Of course its impossible to prove but techs that didn't use protection had higher RMA rates on components so it gots to a point where the conclusion drew itself.
I have heard that circuits under construction are more susceptible to ESD damage than finished circuits. My understanding is that it has something to do with having all the grounding hooked up.
Coincidentally, I have a blog post on that subject :-)
Yes, during the construction of an IC, a wire can act as an antenna and pick up charge that can destroy the chip through ESD. Specifically, if a wire is connected to a transistor gate but the other end is not yet connected, it can pick up charge when the chip is plasma etched, and then blow up the gate. The solution is an "antenna diode", a diode that connects the wire to the substrate to drain off the charge. When the chip is completed and in use, the diode is reverse-biased so it has no effect.
https://www.righto.com/2024/11/antenna-diodes-in-pentium-pro...
When you’re digging around in tens to hundreds of PCs each day, the odds of zapping something are higher. I’ve killed a few chips and boards.
Yep, it's a numbers game. There are things that can increase your risk on single computers, like working on carpet with dry air of course. But when you have to build a ton of PCs moving fast things like anti static mats and ground strips make a huge difference.
Decades ago I was in the PC manufacturing and repair business. I religiously used anti-static mats, straps, And diffusers, but I still destroyed several thousand dollars of equipment over those years from equipment/ grounding failures, or even picking a PC up (accidentally touching a port) to put it on a bench.
It was interior Alaska, where humidity is low enough that an orange turns into a passable golf ball in a week and a half, so that was definitely a factor.
TLDR static electricity is bad for electronics, and damage does not necessarily show up as failure but often manifests as flaky behavior.
I work in electronics and we have steam humidifiers in the manufacturing area and labs despite the fact that it's very humid here 90% of the year. Annoying have to maintain them while we're running dehumidification most of the time but we can't afford not to have them ready and working for those couple of weeks a year!
Have cooked a stick of RAM. Not fully, but made some bad cells where there hadn't been any before (stick 1 tested good, removed and marked good, stick 2 tested bad, removed, stick 1 re-inserted and tested bad, and then stick 1 tested bad again in another machine)
> Has anybody ever actually destroyed a chip with static electricity?
Piling on, but yes, you very well may have: https://www.youtube.com/watch?v=tcRqj9FhgcE
Is wonder if there's a strong correlation between whether you experience static zaps in your daily life and your propensity to fry chips with ESD.
When you fry a chip is it obvious because you experience a zap?
If so, then that would make all this "make sure you're grounded" ceremony less mysterious — because unless you feel a zap (even a tiny one) you probably haven't fried a chip, and it doesn't generally happen in environments where you don't feel zaps.
Obvious caveat: "feeling" a zap is not a precise measurement. But perhaps "zaps fry chips" is a lie which reveals a greater truth.
Many common board components have ESD-withstand ratings of 250V, far below what you can feel. The assembled product may be somewhat robust (including case), but most ESD events are not detectable without specialised equpment.
Roughly, 3000 volts is what you can feel and 2000 volts is what can zap a chip. So you can zip chips without feeling it.
You built a few computers a year? Max? It becomes a much bigger issue when you're handling dozens of systems a day.
As far as I know, same here. The only thing I do is grab a ground lug from a electrical outlet before handling chips and boards. Which may be superstition and ineffective. I may be doing the right thing or I may be using up my luck and one day fry something expensive.
I doubt this is doing anything. Static electricity is the difference in latent charge between two things, and if neither of those things is attached the the actual ground, touching the mains ground (which is attached to the actual ground) isn’t doing much.
A common recommendation is to touch your plugged in power simply occasionally.
i just go touch a metal faucet before. Probably doesn't work, but never ruined anything either
Historically this was a huge concern because not every manufacturer implemented their ESD protection properly; or, on occasion, the process technology meant that ESD protection would hinder the functionality of the device. This happened a lot in RF circuits, and still to this day many RF instruments are extremely sensitive to ESD events. Board assembly was also a lot less automated in the early days of integrated circuits, so more human handlers and more opportunities for ESD events were anticipated.
Modern IC ESD protection is very effective against a few moderate energy events distributed on different pins, and there's a few industry standards that help determine the required amount of caution for dealing with a particular IC (HBM or human-body model, and CDM or charged-device model, are common - targeted toward human assembly procedures and things like triboelectric or inductive charge buildup). In the right climate, a single high energy event is sometimes enough to degrade functionality or (rarely) completely destroy the device, so board assembly and semiconductor manufacturing facilities still require workers to use wrist straps, shoe grounders, mats, treated floors, climate control, etc. Some high voltage GaN work I did years ago required ionizing blowers (basically a spark gap with a fan) because GaN gates are easy to destroy with gate overstress, and there are risks involved with unintended high voltage contact with typical ESD protective solutions. In another embedded-focused lab, the only time I've ever seen someone put on a wrist strap was for handling customer hardware returns. It really depends what you're working with, and in what environment.
I've more frequently (once or twice a year) had devices which exhibit symptoms of something being wrong at the inputs or the outputs, but only on a specific pin or port. For outputs, some symptoms include the output slew rate is inadequate, or the output appears stuck sometimes, or the output has higher than expected voltage noise (though this is a non-exhaustive list). For inputs, the symptoms are more complex - sometimes there's a manifestation at the outputs for amplifiers or other linear circuits, but for feedback systems or digital systems they might behave as though an input is stuck, toggling slowly, etc. which is difficult to distinguish from other, more common errors. I've also directly been the cause of several ESD failures, but in these cases the test objective was to determine the failure thresholds for the system, so I'm not sure that counts.
I've had a customer hardware failure that was eventually traced back to electrical overstress damage on a single pin of an IC near the corner of a board, right where someone might put their thumb if they were holding the board in one hand. In the absence of a better explanation, I suggested this was an ESD failure due to handling error. I never heard about it again, which is weak evidence in favor of a one-off ESD event.
The higher the feature density, the likelier a discharge of a given voltage will cause physical damage?
Damage is at the pad, so probably no. (the ESD protection structures that you're proposing to zap are not teeny tiny).
Both of these assertions are false, unless you can provide some appropriately limiting context. On-chip ESD countermeasures are microscopic, and often ablative — that is, they can become less effective with each exposure. Damage occurs most commonly at or near corners of conductors, and secondly between adjacent traces. The dielectric breakdown or material migration that occur, when not immediately catastrophic, may induce increased susceptibility, instability, or subtle changes in behaviour that are not easy to detect.
Fair point about the protection structures lagging non-linearly in equivalent process node. Though wouldn't CDM events still correlate with feature density since the charge distribution across the die (assuming it correlates with the total gate capacitance)? More gates (higher density) means more to discharge safely?
Actually, for latchup specifically - even with oversized protection, don't the triggering conditions get worse with scaling since the parasitic SCR structures in the core have tighter spacing?
Author here if you have questions about some obscure circuitry in the 386.
I think the metastability can't destroy a chip thing is not true, you can get a flop into a state where it's oscillating at whatever freq the internal feedback path is (maybe up to GHz) rather than resolving to a stable 1 or 0. That can propagate to adjacent flops resulting in a bunch of flops pulling too much current.
Like anything to do with metastability this is a statistical thing - it can do this, but it's highly unlikely.
I worked on a chip in the mid 90s where we were very careful about our clock crossings, dropped in special high-gain anti-metastability flops, designed logic to reduce synchronised signal frequencies etc etc all the good stuff - we calculated that we'd see a failure (and mostly that would be a pixel burble on the screen) every year or so - at the time Win95 couldn't stay up a week so management decided to ship it
Windows 95 could stay up for a maximum of 49.7 days, provided that no horribly-written driver or application took down the system sooner.
In the late 90's, I administered the software test lab including UNIX workstations and Windows beige boxes I had build in addition to some servers, the LAN, and WAN for a nuclear engineering consultancy. There were a few Windows 95/98 boxes that were imaged for testing and contained 4 handy-dandy clones that could be copied (prior to widespread usage of Ghost). It would regularly stay running for a month at a time when disused, but I believed it was set to simply reboot to the first primary partition on BSOD. I bet at least one of event logs probably contained entries like these because centralized syslog/log shipping and monitoring weren't set up.
CS/EE here. Malfunction still leads to dragons overall. The problems of sequential logic looping to itself and talking to the outside world under various conditions include (but aren't limited to): getting predictable initialization, predictable durability, matching input impedance, and creating chips with characterizable and reliable setup/hold/delay/etc. times.
These days, I leave chip design to chip designers and barely do silly things like create seven (7) total 4 pin to 3 pin Arduino PWM fan controllers with MOSFETs, MOSFET protection and noise reduction circuitry. See, I have to keep the fans fed with over 4 volts so the tach signal continues and the storage array's BMC doesn't freak out. (The fans characteristics I needed aren't/weren't available in 4 pin PWM.) I try not shock myself like ElectroBOOM or release too much magic smoke from gear or vintage gear that might not be replaceable.
Just a big thanks from someone lurking on your blog for a while now. Every article has been fascinating deep insights!
Are the techniques described in the article still in use today or have they been superseded?
My understanding is that modern techniques are similar, but the tradeoffs have changed as chip voltages become lower and transistors become smaller. (Admittedly, I don't know a lot about modern techniques.)
This article, from a company that designs ESD circuits, describes various modern techniques: https://monthly-pulse.com/2022/03/29/introduction-esd-protec...
Yes, using 2 or three flip flops in series to prevent metastability is still common practice today. As is using diodes in that configuration for esd protection.
Question for @kens -- this circuit you found, how would Intel have designed and tested this?
There are various test procedures. For ESD, there are official tests like the Human Body Model, where a capacitor is charged to a specific voltage and discharged into the circuit. Similarly, for latchup, there are official tests that apply specific pulses. Metastability can be tested by applying asynchronous signals, but I'm not sure how that scales when the MTBF is very large.
Human Body Model: https://en.wikipedia.org/wiki/Human-body_model
Latchup testing: https://www.ti.com/lit/wp/scaa124/scaa124.pdf?ts=17555138993...