Unpowered SSDs slowly lose data
xda-developers.com96 points by amichail 5 hours ago
96 points by amichail 5 hours ago
So on the off-chance that there's a firmware engineer in here, how does this actually work?
Like does a SSD do some sort of refresh on power-on, or every N hours, or you have to access the specific block, or...? What if you interrupt the process, eg, having a NVMe in an external case that you just plug once a month for a few minutes to just use it as a huge flash drive, is that a problem?
What about the unused space, is a 4 TB drive used to transport 1 GB of stuff going to suffer anything from the unused space decaying?
It's all very unclear about what all of this means in practice and how's an user supposed to manage it.
SSD firmware engineer here. I work on enterprise stuff, so ymmv on consumer grade internals.
Generally, the data refresh will all happen in the background when the system is powered (depending on the power state). Performance is probably throttled during those operations, so you just see a slightly slower copy while this is happening behind the scenes.
The unused space decaying is probably not an issue, since the internal filesystem data is typically stored on a more robust area of media (an SLC location) which is less susceptible to data loss over time.
As far as how a user is supposed to manage it, maybe do an fsck every month or something? Using an SSD like that is probably ok most of the time, but might not be super great as a cold storage backup.
Typically unused empty space is a good thing, as it will allow drives to run in MLC or SLC mode instead of their native QLC. (At least, this seems to be the obvious implication from performance testing, given the better performance of SLC/MLC compared to QLC.) And the data remanence of SLC/MLC can be expected to be significantly better than QLC.
>as it will allow drives to run in MLC or SLC mode instead of their native QLC
That depends on the SSD controller implementation, specifically whether it proactively moves stuff from the SLC cache to the TLC/QLC area. I expect most controllers to do this, given that if they don't, the drive will quickly lose performance as it fills up. There's basically no reason not proactively move stuff over.
I assume this blog is a re-hash of the JDEC retention standards[1].
The more interesting thing to note from those standards is that the required retention period differs between "Client" and "Enterprise" category.
Enterprise category only has power-off retention requirement of 3 months.
Client category has power-off retention requirement of 1 year.
Of course there are two sides to every story...
Enterprise category standard has a power-on active use of 24 hours/day, but Client category only intended for 8 hours/day.
As with many things in tech.... its up to the user to pick which side they compromise on.
[1]https://files.futurememorystorage.com/proceedings/2011/20110...
With 1 year power-off retention you still loose data, so still a compromise on data retention
> I assume this blog is a re-hash of the JDEC retention standards[1].
Specifically in JEDEC JESD218. (Write endurance in JESD219.)
What about powered SSDs that contain files that are rarely read?
My desktop computer is generally powered except when there is a power failure, but among the million+ files on its SSD there are certainly some that I do not read or write for years.
Does the SSD controller automatically look for used blocks that need to have their charge refreshed and do so, or do I need to periodically do something like "find / -type f -print0 | xargs -0 cat > /dev/null" to make sure every file gets read occasionally?
what is the exact protocol to "recharge" an ssd which was offline for months?
do I just plug it in and let the computer on for a few minutes? does it needs to stay on for hours?
do I need to run a special command or TRIM it?
We really don't know. One thing I wish some of these sites would do is actually test how long it takes for the drives to decay and also do a retest after they have been left powered for say 10 minutes to an hour, read completely, written to a bit etc and see if they can determine what a likely requirement is.
The problem is the test will take years, be out of date by the time its released and new controllers will be out with potentially different needs/algorithms.
There was one guy who tested this
https://www.tomshardware.com/pc-components/storage/unpowered...
The data on this SSD, which hadn't been used or powered up for two years, was 100% good on initial inspection. All the data hashes verified, but it was noted that the verification time took a smidgen longer than two years previously. HD Sentinel tests also showed good, consistent performance for a SATA SSD.
Digging deeper, all isn't well, though. Firing up Crystal Disk Info, HTWingNut noted that this SSD had a Hardware ECC Recovered value of over 400. In other words, the disk's error correction had to step in to fix hundreds of data-based parity bits.
...
As the worn SSD's data was being verified, there were already signs of performance degradation. The hashing audit eventually revealed that four files were corrupt (hash not matching). Looking at the elapsed time, it was observed that this operation astonishingly took over 4x longer, up from 10 minutes and 3 seconds to 42 minutes and 43 seconds.
Further investigations in HD Sentinel showed that three out of 10,000 sectors were bad and performance was 'spiky.' Returning to Crystal Disk Info, things look even worse. HTWingNut notes that the uncorrectable sectors count went from 0 to 12 on this drive, and the hardware ECC recovered value went from 11,745 before to 201,273 after tests on the day.Note that the SSD that showed corrupted files was the one that had been worn out well beyond the manufacturer's max TBW rating (4× the TBW or so). There was a difference of two-to-three orders of magnitude in the ECC count between the "fresh" and the worn-out SSD; I'd call that very significant. It will be interesting to see if there's any update for late 2025.
I'd imagine full read of the whole device might trigger any self-preservation, but I'd also imagine it's heavily dependent on manufacturer and firmware
I would run something like CHKDSK, or write a script to calculate a hash of every file on disk.
No idea if that's enough, but it seems like a reasonable place to start.
I don't use my drive much. I still boot it up snd write some data, just not the long term one. Am I in risk?
AIUI, informal tests have demonstrated quite a bit of data corruption in Flash drives that are literally so worn out that they might as well be about to fail altogether - well beyond any manufacturer's actual TBW specs - but not otherwise, least of all in new drives that are only written once over for the test. It seems that if you don't wear out your drive all that much you'll have far less to worry about.
Flash is programmed by increasing the probability that electrons will tunnel onto the floating gate and erased by increasing the probability they will tunnel back off. Those probabilities are never zero. Multiply that by time and the number of cells, and the probability you don’t end up with bit errors gets quite low.
The difference between slc and mlc is just that mlc has four different program voltages instead of two, so reading back the data you have to distinguish between charge levels that are closer together. Same basic cell design. Honestly I can’t quite believe mlc works at all, let alone qlc. I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
All the big 3D NAND makers have already switched from floating gate to charge trapping. Basically the same as what you describe but basically the electrons get stuck in a non-conductive region instead of on an insulated gate.
> I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
You can run an error-correcting code on top of the regular blocks of memory, storing, for example (really an example; I don’t know how large the ‘blocks’ that you can erase are in flash memory), 4096 bits in every 8192 bits of memory, and recovering those 4096 bits from each block of 8192 bits that you read in the disk driver. I think that would be better than a simple “map low levels to 0, high levels to 1” scheme.
> I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
Loads of drives do this(or SLC) internally. Though it would be handy if a physical format could change the provisioning at the kernel accessible layer.
the market demands mostly higher capacity
tlc/qlc works just fine, it's really difficult to consume the erase cycles unless you really are writing 24/7 to the disk at hundred of megabytes a second
I have a MLC SSD with TBW/GB much higher than the specified TBW/GB guarantee of usual qlc SSDs
> Honestly I can’t quite believe mlc works at all, let alone qlc. I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
Manufacturers often do sell such pMLC or pSLC (p = pseudo) cells as "high endurance" flash.
Powering the SSD on isn't enough. You need to read every bit occasionally in order to recharge the cell. If you have them in a NAS, then using a monthly full volume check is probably sufficient.
Isn't that the SSD controller's job?
It would surely depend on the SSD and the firmware it's running. I don't think you can entirely count on it. Even if it were working perfectly, and your strategy was to power the SSD on periodicially to refresh the cells, how would you know when it had finished?
NVMe has read recovery levels (RRLs) and two different self-test modes (short and long) but what both of those modes do is entirely up to the manufacturer. So I'd think the only way to actually do this is to have host software do it, no? Or would even that not be enough? I mean, in theory the firmware could return anything to the host but... That feels too much like a conspiracy to me?
Huh. I wonder if this is why I'd sometimes get random corruption on my laptop's SSD. I'd reboot after a while and fsck would find issues in random files I haven't touched in a long time.
If you're getting random corruption like that, you should replace the SSD. SSDs (and also hard drives) already have built-in ECC, so if you're getting errors on top, it not just random cosmic rays. It's your SSD being extra broken, and doesn't bode too well for the health of the SSD as a whole.
It's quite possible. Some SSDs are worse offenders for this than others. I have some Samsung 870 EVOs that lost data the way you described. Samsung knew about the issue and quietly swept it under the rug with a firmware update, but once the data was lost, it was gone for good.
I ran into this firmware bug with the two drives in my computer. They randomly failed after a while -- and by "a while" I mean less than a year of usage. Took two replacements before I finally realized that I should check for an fw update
Huh, I thought I got some faulty one, mine died shortly after warranty ended (and had a bunch of media errors before that)
Unless your setup is a very odd Linux box, fsck will never check the consistency of file contents.
But metadata is data too, right? I guess the next question is, would it be possible for parts of the FS metadata to remain untouched for a time long enough for the SSD data corruption process to occur.
I had to search around and feel like a dork not knowing this. I have my data backed up, but I keep the SSDs because it's nice to have the OS running like it was... I guess I need to be cloning the drives to ISOs and storing on spinning rust.
I learned this when both my old laptops would no longer boot after extended off power time (couple years). They were both stored in a working state and later both had SSDs that were totally dead.
I could be wrong, but I believe the general consensus is along the lines of "SSDs for in-use data, it's quicker and wants to be powered on often. HDDs for long-term storage, as they don't degrade when not in use nearly as fast as SSDs do.
I'd imagine HDDs also don't like not spinning for years(as mechanical elements generally like to be used from time to time). But at least platters itself are intact
I've been going through stack of external USB drives with laptop disks in them. They're all failing in some form or another. I'm going to have to migrate it all to a NAS with server-class drives I guess
At the very least, you can usually still get the data off of them. Most SSDs I've encountered with defects failed catastrophically, rendering the data completely inaccessible.
or you could power them on 1-2x /year.
Power them on and run something to exercise the read function over every bit. Thats why a ZFS filesystem integrity check/scrub is the useful model.
I'm unsure if dd if=/the/disk of=/dev/null does the read function.
why would it not? it's a low level tool to do exactly that. you could "of" it to somewhere else if you're worried it's not. I like to | hexdump -C, on an xterm set to a green font on a black background for a real matrix movie kind of feel.
good to know but apart from some edge cases this doesnt matter that much
This is why I would rather pay someone a couple of dollars per year to handle all this for me. If need be pay two providers to have a backup.
Who do you pay for this? (To rephrase : which cloud storage vendors do you use?) interested in the $2/month price point :)
I assume "couple of" was figurative, to indicate the cost is substantially less than managing your own bank of SSDs and ensuring it is periodically powered etc.
[Edit: LOL, I see someone else posted literally the same example within the same minute. Funny coincidences.]
That said, they could also be storing relatively small amounts. For example, I back up to Backblaze B2, advertised at $6/TB/month, so ~300 MB at rest will be a "couple" bucks.
Backblaze B2 is $6TB/mo, so if you have around 300GB... stuff like restic or kopia backups nicely to it
Recently started fiddling with restic and B2, it worked fairly seamlessly once I stopped trying too hard being fancy with permissions and capabilities (cap_dac_read_search). There were some conflicts trying to have both "the way that works interactively" [0] versus "the way that works well with systemd". [AmbientCapabilities=]
One concern I have is B2's downloading costs means verifying remote snapshots could get expensive. I suppose I could use `restic check --read-data-subset X` to do a random spot-check of smaller portions of the data, but I'm not sure how valuable that would be.
I like how it resembles LUKS encryption, where I can have one key for the automated backup process, and a separate memorize-only passphrase for if things go Very Very Wrong.
[0] https://restic.readthedocs.io/en/latest/080_examples.html#ba...
> Even the cheapest SSDs, say those with QLC NAND, can safely store data for about a year of being completely unpowered. More expensive TLC NAND can retain data for up to 3 years, while MLC and SLC NAND are good for 5 years and 10 years of unpowered storage, respectively.
This is somewhat confused writing. Consumer SSDs usually do not have a data retention spec, even in this very detailed Micron datasheet you won't find it: https://advdownload.advantech.com/productfile/PIS/96FD25-S2T... Meanwhile the data retention spec for enterprise SSDs is at the end of their rated life, which is usually a DPWD/TBW intensity you won't reach in actual use anyway - that's where numbers like "3 months @ 50 °C" or whatever come from.
In practice, SSDs don't tend to loose data over realistic time frames. Don't hope for a "guaranteed by design" spec on that though, some pieces of silicon are more equal than others.
Any given TBW/DWPD values are irrelevant for unpowered data retention. Afaik, nobody gives these values in their datasheet and I'm wondering where their numbers are from, because I've never seen anything official. At this point I'd need to be convinced that the manufacturers even know themselves internally, because it's never been mentioned by them and it seems to be outside the intended use cases for SSDs