ECC RAM vs non-ECC: which do you actually need?

You need ECC RAM for a NAS, a ZFS pool, a virtualization host or any machine whose wrong answers can outlive its uptime, and you do not need it for a gaming PC or an everyday desktop. ECC does not make memory reliable. It makes memory honest: the eight extra bits on a 72-bit bus correct every one-bit error in a word, detect every two-bit error, and report each event against a slot, so a module that is starting to fail says so weeks before it takes data with it. On a desktop a flipped bit usually costs you a crash you blame on a driver. On a file server it can cost you a file that was already wrong when it was written. The extra DRAM costs 12.5 percent; the real cost is a CPU and a board that support ECC, so the platform decides the purchase before the module does, and putting ECC modules in a machine does not prove that anything is correcting them.

Who needs ECC, and who does not

The deciding question is not how often memory fails. It is how long a wrong answer survives. A memory fault is unlikely on any given machine and near-certain across many machines or a long enough time. It is usually a hard fault in one module that announces itself with corrected errors first, and a non-ECC machine gets none of that warning and no sign afterward that anything went wrong.

The value of ECC is that it converts an unbounded, unattributable, silent failure into a bounded, located, logged one. If a bad bit produces a crash you reboot from, ECC is not worth a platform change. If it produces a number that gets written to disk and read back next year, it is. How long data sits in memory on a file server is the subject of RAM caching.

What the field data says

The large public studies - Google’s fleet in 2009, Facebook’s in 2015, and high-performance computing systems at different altitudes - agree on four things, with the figures in the field-study section. The median module never produces an error; errors concentrate in a few failing modules, and the cause is mostly manufacturing and wear, not radiation. Errors predict errors, strongly, so the most valuable output of ECC is not the corrected bit but the counter that says which slot to replace, weeks before the machine falls over. And nobody has published a rate for DDR4 at scale or for DDR5 at all.

The ZFS question

The strong form of the argument is that ZFS requires ECC because a scrub with bad memory will propagate corruption across the pool. Matt Ahrens, one of ZFS’s original authors, has said publicly that there is nothing about ZFS which requires or encourages ECC more than any other filesystem does. Both halves are true. ZFS checksums catch corruption on the way to the disk and back, but not data that was already wrong when the checksum was computed: ZFS will faithfully record the wrong checksum for the wrong data and report the pool as healthy. That is an argument for ECC on a machine holding data you cannot re-create, not an argument that ZFS is unusually vulnerable, and the “scrub of death” story - that a single scrub with bad RAM destroys a whole pool - is not supported by the way scrub works. The cheapest boards with real ECC for such a box are in the platform section, and the drives that go in it are the subject of drives for a NAS on the drive site. How much memory such a box needs, and what each NAS vendor says about ECC, is in how much RAM for a NAS or homelab.

The checklist

1. Decide on the lifetime of a wrong answer, not on the error rate. The mean error rate per gigabyte is a fleet statistic that predicts nothing about one machine.
2. Check the CPU first, then the board, then the module. The memory controller is in the CPU, so the chipset never settles it. Start from the platform table or the CPU pages; how to check what RAM fits covers the rest of the identification.
3. Get the buffering right before the code. ECC says nothing about whether a module will POST. On DDR4 a registered module fits a desktop slot and will not start the machine; on DDR5 it will not enter the socket.
4. Count the chips. Nine, eighteen or thirty-six devices on DDR4, or ten x8 devices per rank on DDR5, is ECC. Eight and sixteen are ordinary memory. Reading ECC off a listing has the other signals.
5. Ignore “ECC” on a consumer DDR5 listing. On-die ECC is on every DDR5 die, reports nothing, and is not what a server board means.
6. Prefer x4 if the platform will take it. 2Rx4 is the difference between surviving a bit flip and surviving a dead chip (the chipkill section), and DDR4 registered ECC is where that stock is.
7. Compare the two markets before assuming ECC costs more. DDR4 registered ECC against DDR4 desktop, by price per gigabyte. Server decommissioning supplies one of those markets by the pallet.
8. Read dimm_edac_mode on the machine you own. It is the only place anything tells you which code the controller is running; proving ECC is running walks through it.
9. Install rasdaemon and label the slots on the day you build the machine, so that a future error names a slot you can find with a flashlight.
10. Turn patrol scrubbing on and leave it on. It costs about 0.03 percent of one channel.
11. When errors appear, count distinct addresses before you buy anything. One address means one broken module. Thousands across every DIMM behind one controller means the memory is fine and something else is not.
12. Do not trust a memory test on an ECC machine. Reset the counters, run the test, then read the per-DIMM counters. A clean pass with a rising dimm_ce_count is a failing module that the test corrected on its way past.

The 12.5 percent is not the cost of ECC and the correction is not the benefit. The cost is the platform, and the benefit is a counter with a slot name on it.

Which CPUs and boards support ECC

ECC needs the CPU, the chipset, the board firmware and the module to agree. The memory controller is inside the CPU, so a chipset name never settles it on its own, and the board firmware can refuse what the silicon permits.

Five rows in that table routinely cost people money.

“Xeon W” spans two incompatible memory worlds. A W-1300 on an LGA1200 board takes ordinary unbuffered ECC DDR4. Intel’s support documentation for the W-2400 and W-3400 says they take “ECC RDIMMs & 3DS RDIMMs” and that “ECC UDIMMs and non-ECC UDIMMs will not be supported.” Same brand, same decade, parts that will not run in each other’s boards.

Intel does sell ECC on consumer silicon, through W680 and W880. A Core i9 on a W680 board runs ECC UDIMMs with full error reporting. It is the only sanctioned route to ECC on a mainstream Intel chip, and it hinges on the board vendor having implemented it - check the board’s own memory support list, never the chipset name.

AMD’s desktop position is “it works, we will not say so.” On AM4 the silicon has ECC enabled across the line and only the PRO parts are validated for it. The non-PRO APUs are the exception: AMD confirmed that the Ryzen 8000G APUs do not support ECC despite early specification sheets suggesting otherwise. On AM5, DDR5 ECC UDIMMs run with live reporting on some boards and not on others with the same chipset and a different firmware, so treat it as a per-board fact you verify after purchase.

Threadripper stopped taking unbuffered memory. With the 7000 series AMD went registered-only: TRX50 and WRX90 boards take ECC registered DDR5 and nothing else, and DDR5 ECC UDIMMs will not enter that socket.

A NAS is the case where the cheap route exists and nobody mentions it. Intel’s Atom C3000 and P5000 parts take ECC, and so do the AMD embedded Ryzen parts. Boards built on them, such as Supermicro’s A2SDi-8C-HLN4F, are the least expensive way to get real error reporting under a filesystem, on ordinary DDR4 ECC UDIMM stock: the DDR4 ECC listings filtered to unbuffered parts.

In-Band ECC: correction without an ECC module

Some Intel SoCs implement In-Band ECC, which carves a fraction of ordinary non-ECC DRAM out of the visible address space and uses it to hold check bits for the rest. The memory controller does SECDED using that reserved region, so the machine gets real, reported, logged error correction on plain non-ECC memory.

The family most NAS builders meet is Intel’s N-series. Intel’s product brief for the Alder Lake-N line lists In-Band ECC for parts including the N97, N200 and Core i3-N305, with the caveat that not every feature is on every SKU. The N100 is not among the parts that brief names. Intel’s spec pages for the N100 and the N200 alike say “ECC Memory Supported: No”, so that field does not settle In-Band ECC either way. It must be enabled in firmware, sometimes with a choice of protecting a range or everything, so whether a given board offers it is a per-board fact. On Linux the igen6_edac driver binds to it, and its source lists Alder Lake-N among the SoCs it supports; when the firmware has In-Band ECC switched off, the driver logs a line ending In-Band ECC is disabled and registers no memory controller, so /sys/devices/system/edac/mc/ stays empty.

The costs are structural: capacity disappears into the check region, and every protected access becomes two accesses at the controller, one for data and one for check bits, which a real x72 bus does not pay. But on a board whose firmware exposes it, ECC is a firmware setting and a capacity trade, not a shopping trip.

Unbuffered ECC and registered ECC are different products

“ECC” on a listing tells you about the code and nothing about the buffering, and the buffering decides whether the machine POSTs. An unbuffered ECC UDIMM wires command, address and clock straight from the controller to every DRAM device, which caps a channel at one or two modules and a module at 32GB for most of DDR4’s life (48GB on DDR5 with 24Gb dies); it is the memory a workstation takes. A registered ECC RDIMM puts a register in between, so the controller sees one load per module instead of nine or eighteen or thirty-six, which is why servers take three modules per channel and 128GB parts. A board specified for one will generally not run the other: the full rules are in registered vs unbuffered, the per-socket limits in server memory capacity limits, and why an RDIMM’s CL is not directly comparable with a UDIMM’s in CAS latency explained.

The used-market pricing is upside down

On new consumer parts ECC costs more. On the used market it frequently does not, because decommissioned servers shed registered ECC modules by the pallet while decommissioned desktops shed one or two UDIMMs at a time. Open DDR4 registered ECC and DDR4 desktop side by side and compare the price per gigabyte at 16GB and 32GB; both pages rank by it, the methodology page documents how listings are recorded, and the size of the gap moves week to week. The trap is the buffering: registered memory is cheap because it only fits machines that take registered memory.

What happens when you get it wrong

• Non-ECC memory in a machine that requires ECC does not POST: no video, a memory fault LED or a beep code. Mixing ECC and non-ECC modules in one system fails the same way.
• ECC memory in a machine that does not support it usually boots and runs as ordinary memory with the extra devices idle; ASRock’s consumer Z690 documentation says ECC UDIMMs “operate in non-ECC mode”. That is the harmless case, and it is not universal, so check the board’s memory list.
• Registered memory in an unbuffered board, or the reverse: DDR4 UDIMMs and RDIMMs share a key notch, so a server RDIMM slides into a desktop board and refuses to POST, while DDR5 UDIMMs and R/LRDIMMs are keyed differently and will not enter the wrong socket.
• ECC modules in a supporting board with ECC disabled in firmware is the silent one, and the reason the proving section exists.

Reading ECC off a listing

Sellers write titles from whichever sticker they can read, so the signals below are ranked by how hard they are to get wrong. The most reliable needs no sticker at all.

Count the chips

Count the devices in the photograph or on the module, ignoring the small register, SPD and PMIC packages near the center. Nine, eighteen or thirty-six on DDR4 means ECC; eight, sixteen or thirty-two means it is not. On DDR5 the ECC count moves up by one device per rank, because every DDR5 ECC module on sale is the eighty-bit EC8 kind, unbuffered parts included (the DDR5 section explains why):

On DDR5, ten x8 devices per rank is an ECC module. Five packages a side against four, and it works from a seller’s photograph. The count says ECC; it does not say registered. Only the register says that, and the register is the wide chip alone in the middle of the card.

Micron’s DDR5 part numbers carry the count in the digits after the prefix. A 16GB 1Rx8 non-ECC SODIMM is MTC8C1084S1SC56BD1, eight devices. The ECC SODIMM at the same capacity and organization is MTC10C1084S1TC48BA1, ten. The unbuffered ECC UDIMM is MTC10C1084S1EC48BA1, ten again, and its registered sibling MTC10F1084S1RC56BR is also ten; the letter that changes between those last two is the buffering, not the width. A 32GB 2Rx8 ECC UDIMM is MTC20C2085S1EC56BR, twenty, the same ten per rank.

The codes in titles and part numbers

The Micron convention is the one worth memorizing, because the two halves of the part number agree with each other and with the photograph. MTA18ASF2G72PDZ is eighteen packages and 72 bits wide: a 16GB 2Rx8 registered ECC module. MTA9ASF1G72AZ is nine packages and 72 bits: an 8GB 1Rx8 unbuffered ECC module, a very different part for a very different board. MTA16ATF2G64AZ is sixteen packages and 64 bits: ordinary desktop memory. SK hynix encodes the same distinction one character apart, which is why HMA81GU7AFR8N-UH is an ECC UDIMM and HMA81GU6AFR8N-UH is not, and why a typo in a listing title is a real hazard. DDR5 part numbering is less settled across vendors, and the device count is the check that does not depend on knowing a convention.

The rank code in front of the x4 or x8 is decoded in full in ranks and 3DS modules, and the rest of the checks for a used module are in buying used RAM on eBay. The part number pages group every listing carrying the same MPN, the fastest way to see whether a price is normal for that exact module, and the compatible server pages for Dell, HPE, Lenovo and Supermicro start from the machine instead.

What each generation offers

DDR3 ECC is the cheapest real error correction available, on a platform whose power draw is its own argument. DDR4 is where the used market is richest and nearly all the x4 registered stock lives: work from DDR4 ECC and DDR4 server, and use speed facets such as 2666MHz only when the platform pins the speed (RAM speed, MHz and MT/s explains why). For DDR5, start from DDR5 ECC, DDR5 registered and DDR5 unbuffered. DDR2 ECC and DDR ECC run the same 72-bit code.

DDR5’s on-die ECC is not ECC memory

Every DDR5 die has on-die ECC. Listings mention it constantly, it sounds like server protection in a consumer part, and it is not the same thing at all.

On-die ECC pairs 128 data bits with 8 check bits into a 136-bit codeword held entirely inside the DRAM die. On a read the die fixes a single bad bit and hands over corrected data across the normal pins. Run the Hamming bound on it and the code’s strength is forced:

SEC over 128 data bits needs r with 2^r >= 128 + r + 1
  r = 7   ->   128 >= 136   fails
  r = 8   ->   256 >= 137   works
SECDED would need a ninth bit.  The die has eight.

On-die ECC is single-error-correct with no double-error detect, and that is arithmetic rather than a vendor choice. It exists because the cell geometry DDR5 needs does not yield without it: a manufacturing necessity that lets the vendor sell dies which would otherwise fail incoming test. It reports nothing - no pin, no log entry, no counter the operating system can read - so a module degrading toward failure looks exactly like a healthy one until it stops. Patel and Mutlu’s device-level work found the mechanism “undocumented, proprietary, opaque, and possibly unique per product”. It protects nothing outside the die, and Facebook’s fleet study found that failures in the memory controller and the channel accounted for the majority of observed errors. And it repairs weak cells before a memory test can see them, so a die with a hundred weak cells and a die with none both pass. DDR5’s Error Check and Scrub, defined in JESD79-5, adds a background scrub inside the die whose counts sit in DRAM mode registers that nothing in a normal operating system reads.

Module ECC - what people mean by “ECC memory” - protects the data traveling between the module and the CPU and reports every event to the operating system. A DDR5 module with on-die ECC and no extra devices is an ordinary non-ECC module. If a listing says “DDR5 with ECC” at consumer prices, that is what it is describing, and counting the devices in the photograph settles it in ten seconds.

Proving ECC is running, not merely installed

Buying ECC modules and putting them in a board that supports ECC does not prove ECC is on. Firmware has settings, defaults change between BIOS revisions, and “Auto” is not an answer. There are four levels of evidence and only the last one is proof. The command output in this guide is an illustrative example, not a capture from any particular machine.

Level one: what is in the slot

The device count from the listing section proves the module can do ECC and nothing else.

Level two: what the firmware and the module claim

SMBIOS carries two separate assertions. Type 16 describes the memory array as a whole; type 17 describes each device:

$ sudo dmidecode -t 16
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: Multi-bit ECC
	Maximum Capacity: 2 TB
	Number Of Devices: 16

$ sudo dmidecode -t memory | grep -E 'Total Width|Data Width|Size|Part Number'
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Part Number: M393A4K40DB3-CWE

Total Width must exceed Data Width. 72 against 64 is DDR4 or earlier, buffered or not; 80 against 64 is DDR5, buffered or not. Equal numbers mean non-ECC, whatever the sticker says. But both are the BIOS asserting something about the hardware, and neither is evidence that the memory controller has the ECC logic switched on: a board can report Multi-bit ECC because ECC modules are present while the controller runs them unprotected.

The module’s SPD EEPROM declares its width too (on DDR4, SPD byte 13 carries a bus width extension of zero on a non-ECC module and eight bits on an ECC one), and with the SPD EEPROM driver loaded first, decode-dimms reads it on Linux. On most server boards the BMC owns that bus or the OS cannot reach it, so the command returns nothing on a healthy machine, and SPD is writable, so a relabeled module can claim whatever its reseller wanted. The device count in the photograph does not lie and SPD can.

Level three: what the memory controller admits

The Linux subsystem that turns the CPU’s machine-check records into per-slot counters is EDAC, and an EDAC driver generally only binds when the memory controller reports that ECC is enabled. The evidence is a controller registered under /sys/devices/system/edac/mc/, not a loaded module: igen6_edac stays loaded with In-Band ECC off and registers nothing.

$ lsmod | grep edac
amd64_edac             49152  0
edac_mce_amd           40960  1 amd64_edac

$ ls /sys/devices/system/edac/mc/
mc0  mc1

$ cat /sys/devices/system/edac/mc/mc0/dimm0/dimm_edac_mode
S4ECD4ED

An empty mc/ directory is a finding, not an inconvenience. On AMD the kernel says plainly whether ECC is on:

EDAC amd64: DRAM ECC enabled.
EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.

The second line is the most useful diagnostic in the subject, because it is the machine telling you that you bought ECC memory and are not using it. On AM5 the setting is usually buried under the AMD CBS submenu rather than in the memory page, and on several boards it defaults to “Auto”, which resolves to disabled. The driver is chipset-specific:

On a server with firmware-first error handling, the chipset driver may not load and ghes_edac reports instead, with slot labels from SMBIOS. Everything still works; it just looks different.

Under /sys/devices/system/edac/mc/mc0/ each controller keeps ce_count and ue_count, and each dimm directory keeps dimm_ce_count, dimm_ue_count, dimm_dev_type (x4, x8, x16) and dimm_edac_mode. dimm_edac_mode is the most useful file in this guide. It is the kernel naming the code the controller is actually running:

Read S4ECD4ED and you have confirmed, from the kernel rather than from a datasheet, that the machine is running device-level correction on x4 modules. Read SECDED on a board you bought x4 registered memory for and you have found a firmware setting somebody left alone, unless dimm_dev_type beside it reads x8, and then the listing lied.

Level four: proof, by injection

Everything above is an assertion by some component about itself. The only way to close the loop is to inject an error and watch the counter move. On servers with ACPI APEI support the kernel exposes an injection interface documented in Documentation/firmware-guide/acpi/apei/einj.rst:

$ sudo modprobe einj
$ cd /sys/kernel/debug/apei/einj
$ cat available_error_type
0x00000008	Memory Correctable
0x00000010	Memory Uncorrectable non-fatal
0x00000020	Memory Uncorrectable fatal

$ echo 0x8            | sudo tee error_type
$ echo 0x12345000     | sudo tee param1      # physical address
$ echo 0xfffffffffffff000 | sudo tee param2  # address mask
$ echo 0x2            | sudo tee flags       # bit 1: address and mask valid
$ echo 1              | sudo tee error_inject

Then read ce_count again. If it moved, the chain from DRAM through the controller and EDAC to your counters is intact. Many boards lack the firmware support, and the interface also needs a kernel built with CONFIG_ACPI_APEI_EINJ and debugfs mounted. On some platforms the firmware emulates the injection, which proves the reporting path and says nothing about the correction hardware. Inject the correctable type, never the fatal one. Where injection is not available, the substitute is patience: enable patrol scrubbing, leave rasdaemon running, and accept that a counter still at zero after six months is weak evidence and better than none.

Windows, and why the BMC matters more there

Windows exposes much less. The firmware claim:

Get-CimInstance Win32_PhysicalMemoryArray | Select-Object MemoryErrorCorrection

The value is an SMBIOS enumeration: 3 None, 4 Parity, 5 Single-bit ECC, 6 Multi-bit ECC, the same type 16 assertion dmidecode reads. Corrected hardware errors are logged by WHEA into the System event log as event ID 47, “A corrected hardware error has occurred”:

Get-WinEvent -FilterHashtable @{LogName='System'; ProviderName='Microsoft-Windows-WHEA-Logger'}

There is no per-slot counter equivalent to EDAC’s, the records are throttled, and an empty log proves nothing either way. That is why the baseboard management controller matters more on Windows. The BMC watches the same errors from below the operating system and keeps its own log: Dell’s iDRAC raises MEM0701, correctable memory error rate exceeded, against a named DIMM location; HPE’s iLO writes threshold entries into the Integrated Management Log with the slot; Supermicro and most white-box boards use the IPMI system event log, and Redfish exposes the same over HTTP for a monitoring system:

$ ipmitool sel elist
  12 | 09/14/2026 | 03:11:52 | Memory #0x53 | Correctable ECC | Asserted

Reading the logs before the module dies

This is what you bought. A DIMM that is going to fail usually says so first, and the Google figures in the field-study section put the strength of that signal far past marginal. Corrected errors are not noise to be filtered out. They are the product.

rasdaemon, and the counters you will want in six months

rasdaemon consumes the kernel’s tracepoints and keeps the history in SQLite at /var/lib/rasdaemon/ras-mc_event.db. Install it on the day you build the machine, because it is the difference between “this machine has errors” and “this slot has had 4,213 errors, all at one address, since Tuesday”.

$ sudo ras-mc-ctl --error-count
Label                    CE      UE
CPU0_DIMM_A1              0       0
CPU0_DIMM_B1           4213       0
CPU0_DIMM_C1              0       0
CPU0_DIMM_D1              0       0

$ sudo ras-mc-ctl --summary
Memory controller events summary:
	Corrected on DIMM Label(s): 'CPU0_DIMM_B1' location: 0:1:0:0 errors: 4213
	No Uncorrected Errors.

Labels are not automatic. ras-mc-ctl --guess-labels compares the controller’s topology against SMBIOS and suggests a name for each slot; ras-mc-ctl --register-labels writes a board-specific mapping from /etc/ras/dimm_labels.d/. Do that before you have a failure, because working out which physical slot location: 0:1:0:0 means at two in the morning is a job nobody enjoys. And watch dmesg for this pair:

mce: CMCI storm detected: switching to poll mode
mce: CMCI storm subsided: switching to interrupt mode

A CMCI storm line is one of the highest-signal messages a server can produce. It means corrected errors arrived fast enough that handling them one by one was hurting the machine.

The anatomy of one EDAC line

EDAC MC0: 1 CE memory read error on CPU0_DIMM_B1 (channel:1 slot:0
  page:0x89a2f4 offset:0x180 grain:32 syndrome:0x1c4 -
  err_code:0001:0091 socket:0 channel_mask:2 rank:1)

CE against UE is the whole severity question: corrected, or not. A memory scrubbing error in place of memory read error means the patrol scrubber found it, so the address was cold. page and offset locate the error, and they are the two fields that separate a soft error from a broken cell once you have more than one record. Two errors sharing a syndrome came from the same bit position, which on a symbol-code controller means the same device.

The query that makes the diagnosis

$ sudo sqlite3 /var/lib/rasdaemon/ras-mc_event.db \
    'select label, count(*), count(distinct address) from mc_event group by label;'
CPU0_DIMM_B1|4213|1

Four thousand two hundred and thirteen errors at one distinct address is a single failing cell or row being read repeatedly: a hardware fault that no reseating will change. The same count spread over four thousand addresses would point at the controller, the socket or the board. Column names have moved between rasdaemon releases, so run .schema mc_event first on an unfamiliar machine.

The triage table

The sixth row is the one that saves money. Facebook’s study found that failures in the memory controller and the channel accounted for the majority of observed errors, and their symptom is errors on every module behind one controller. Replacing four good DIMMs because the channel is broken is a mistake that verifies itself: the errors come back.

When it becomes an uncorrectable error

If the error is found but not consumed - typically by the patrol scrubber - Linux poisons the page, removes it from allocation, signals any process mapping it, and the machine stays up. If it is consumed by a load, Linux sends SIGBUS with BUS_MCEERR_AR to the process; one that handles it survives, one that does not dies, and a kernel access to a poisoned page panics. The running total is one line away:

$ grep HardwareCorrupted /proc/meminfo
HardwareCorrupted:    64 kB

Anything other than zero there means the machine has already had an uncorrectable error and kept going. Almost nobody checks it.

Why a memory test is weaker on an ECC machine

On an ECC machine with ECC enabled, a cell that flips gets corrected before the test sees the value, so the test passes and the fault is invisible. A module with a hard single-bit fault in one row will run a full pass of any test cleanly while generating a corrected error on every access to that row. So the procedure for qualifying used modules is:

1.  Reset the counters:   echo 1 > /sys/devices/system/edac/mc/mc0/reset_counters
2.  Run the memory test for as long as you are willing to.
3.  Read the counters again, per DIMM, not per controller.

A pass with a non-zero dimm_ce_count is a failure the test could not see, which is why a machine with ECC and no counter monitoring is in some ways worse off than one without ECC: it is hiding faults and telling nobody.

Scrubbing, and what a server does when errors keep coming

Correction happens on read. A word nobody reads is a word nobody corrects, and single-bit errors that sit long enough for a second one to land in the same word become uncorrectable. Demand scrub writes corrected data back when a read hits a correctable error. Patrol scrub is a background engine in the memory controller that walks the whole address space at a configured rate, so a machine can sit idle for a year without accumulating double-bit errors in its cold pages. In firmware it is usually an enable setting and an interval, with 24 hours the common default. People disable it out of a vague fear of overhead, so here is the cost:

1 TiB scrubbed once per 24 hours
  1,099,511,627,776 bytes / 86,400 s  =  12.7 MB/s

one DDR5-4800 channel
  4800 MT/s x 8 bytes                 =  38.4 GB/s

12.7e6 / 38.4e9                       =  0.033 percent of one channel

A third of a tenth of one percent of a single channel, on a machine that has eight or twelve of them. Patrol scrubbing is free, and it turns latent errors into logged ones. Leave it on.

When scrubbing is not enough, server platforms climb a ladder, ordered by cost. Post Package Repair remaps a failing row onto one of the die’s spare rows (JESD79-4 introduced it, JESD79-5 kept it); server firmware commonly runs it at boot for rows that crossed a correctable error threshold during the previous run, which is why a machine that threw errors last week can be silent this week with no hardware changed. ADDDC, from the chipkill section, retires a device and pairs ranks in one region. Rank sparing holds a spare rank per channel, at a rank of capacity. Memory mirroring writes everything to two channels, at half the installed capacity. Every rung is invisible unless somebody is reading the counters.

How often does a bit actually flip

This is the question the purchase turns on and it is the one with the weakest answer. The measured literature is good and public, and it says something more awkward than either camp in the argument wants.

Google, 2009: the number everyone quotes

Schroeder, Pinheiro and Weber published “DRAM Errors in the Wild: A Large-Scale Field Study” at SIGMETRICS 2009, covering two and a half years of Google’s commodity server fleet:

A FIT is one failure per billion device-hours, so 25,000 FIT per Mbit is about 0.2148 errors per GiB per hour, or 1,883 per GiB per year, and 70,000 is 5,272: the range Google themselves quote, roughly 2,000 to 6,000 correctable errors per gigabyte per year. Now apply it to a desktop, which is exactly what people do with it:

32 GiB desktop at the low end of Google's fleet rate
  0.2148 errors/GiB-hour x 32 GiB  =  6.9 correctable errors per hour

Seven an hour is a preposterous answer for a desktop, and it is arithmetic correctly applied to the wrong statistic. Google’s fleet mean is not a prediction about a machine. It is a fleet-wide average dominated by a small number of modules with hard faults, and the median DIMM in that fleet contributed nothing to it at all.

The two Google findings worth acting on get quoted far less. A DIMM that saw a correctable error was 13 to 228 times more likely to see another in the same month than a DIMM that had not, the range running across the platforms in the fleet. And correctable errors raised the odds of an uncorrectable error by a factor between 9 and 400: 27 times at the best-behaved platform and over 400 at the worst when the correctable error was in the same month, 9 to 47 times when it was in the month before. That is the finding that turns ECC from insurance into instrumentation.

Facebook, 2015: the distribution, not the mean

Meza, Wu, Kumar and Mutlu published “Revisiting Memory Errors in Large-Scale Production Data Centers” at DSN 2015, fourteen months across Facebook’s entire fleet. They measured a lower per-machine rate than Google - 9.62 percent of servers with a correctable error over twelve months, against Google’s roughly 32 percent per machine - and the top 1 percent of servers accounted for more than 97.8 percent of all correctable errors, the mean overstating the median by more than 55 times. Averages of memory errors actively mislead. They also found 4Gb devices showing roughly 1.8 times the failure rate of 2Gb devices, and failures in the controller and the channel accounting for the majority of what they observed, which is why the triage table has a row for “every DIMM on one channel”.

Altitude, and what it says about causes

The JEDEC JESD89 standards measure soft errors against a reference flux of 13 neutrons per square centimeter per hour above 10 MeV, at sea level in New York City, and the flux roughly doubles for every 1,500 m (about 4,900 feet) of altitude. Sridharan and colleagues compared high-performance computing systems where the altitude differs: Cielo at Los Alamos sits at roughly 2,200 m (about 7,200 feet), the comparison systems near sea level. Their work at SC 2012 and ASPLOS 2015 found that DRAM fault rates did not rise with altitude while SRAM fault rates did. Particles break SRAM; DRAM mostly breaks for other reasons. Hwang, Stefanovici and Schroeder’s ASPLOS 2012 paper says it in its title, “Cosmic Rays Don’t Strike Twice”: a large fraction of errors land at addresses that have produced errors before. That is also why buying used ECC memory is reasonable and buying used memory with no way to see errors is less so - the subject of buying used RAM on eBay.

The memory nobody has measured

Google measured DDR1, DDR2 and FBDIMM parts. Facebook measured DDR3. Sridharan measured DDR2 and DDR3. There is no comparable public field study of DDR4 at scale and none at all of DDR5, and on-die ECC now stands between any future study and the raw device behavior it would need to measure. A per-bit error rate for a DDR5 desktop kit is an extrapolation across two generations, a process node change, a storage-cell change and a correction layer that did not previously exist. This site will not put a figure on it.

How ECC works: the code under the ninth chip

The rest of this guide is the mechanism, for readers who want to know why the rules above hold.

An ECC module carries one extra DRAM device for every eight: nine instead of eight, eighteen instead of sixteen, thirty-six instead of thirty-two. Sixty-four data bits become seventy-two. The dies are ordinary dies. A 16GB 2Rx8 ECC UDIMM and a 16GB 2Rx8 non-ECC UDIMM from the same vendor in the same quarter are usually the same DRAM part number, sixteen of them on one board and eighteen on the other. What you are buying is a ninth lane, and a memory controller that is obliged to use it. The overhead is exactly one eighth:

capacity   72 bits carried for every 64 delivered   = 12.5% more DRAM
power      9 devices switching instead of 8         = 12.5% more DRAM power
board      one more device, one more DQ byte lane   = a wider module

The eight bits are not arbitrary. A single-error-correcting Hamming code over 64 data bits needs seven check bits, and detecting a second error needs an eighth, so on a 64-bit bus one extra bit per byte is precisely the SECDED budget, where on the older 32-bit bus the same overhead could only buy parity: detection, and a halted machine.

What SECDED promises, and where it breaks

Single Error Corrected, Double Error Detected. Read it as a promise about error weights in one 72-bit word, not about causes, chips, addresses or time. Two bits wrong in different words are two separate single-bit errors and both are corrected; two bits wrong in the same word take the machine down. Whether a physical defect produces one or the other depends on how the bits of a word are spread across devices, which is the subject of the chipkill section.

Memory controllers use a Hsiao code, from M. Y. Hsiao’s 1970 paper in the IBM Journal of Research and Development, “A Class of Optimal Minimum Odd-weight-column SEC-DED Codes”: a parity-check matrix with 8 rows and 72 columns, every column distinct and of odd weight, chosen to use as few ones as possible. The lightest 72 columns carry 216 ones, 27 inputs per check bit on average, a five-level XOR tree that hides inside a pipeline stage that already existed. On a read the controller recomputes the eight check bits and XORs them with the stored ones; the result, the syndrome, either is zero (clean), matches one of the 72 columns (flip that bit), or matches neither (detected, uncorrectable).

Push error patterns through it and the folklore turns out wrong. Two bits wrong always give an even-weight, non-zero syndrome, so every double-bit error is detected, with no exceptions and no probability. Three bits wrong give an odd-weight syndrome, which is never zero, but if it happens to equal one of the 72 columns the controller flips a fourth bit and reports a successful correction. Assuming the syndrome is uniform over the 128 odd-weight patterns, that happens 72 times in 128, or 56.25 percent; the exact split depends on a controller’s matrix, which no vendor publishes, but a triple-bit error is more likely to be reported as a successful correction than as a failure. Four bits wrong can produce a zero syndrome, because a SECDED code has minimum distance four and some weight-four patterns are themselves codewords.

The practical reading is that ECC is a guarantee about one bit and a probabilistic argument about everything else, which is precisely why a machine that is throwing corrected errors is not a machine that is fine.

What the code cannot see

The word is the unit and the word is where the guarantee ends. The command and address bus carries no ECC: a flipped address bit writes a good word to the wrong place, which is why JESD79-4 added command and address parity, with the receiver raising ALERT_n so the controller retries (on a registered module the register checks it). The data bus in flight has write CRC, introduced in JESD79-4 and extended to reads in JESD79-5, which is why a marginal trace or a dirty contact usually shows up as a CRC retry storm rather than as ECC errors. The CPU’s caches have their own protection and reporting banks. And Rowhammer can flip more than one bit in a word: Cojocar and colleagues showed in 2019 (“Exploiting Correcting Codes: On the Effectiveness of ECC Memory Against Rowhammer Attacks”, IEEE Security and Privacy) that the timing of a correction leaks which words were corrected. Treat ECC as a reliability feature that happens to have security side effects, not as a mitigation.

Chipkill: why x4 modules survive a dead chip

A dead DRAM device is the failure SECDED was never built for, and it is the common one. A chip does not lose one bit. It loses a column, a bank, an I/O driver or all of itself, and every bit it contributed to every word goes wrong at once: eight bits per beat on a x8 module, sixteen on x16.

The fix is to count symbols instead of bits. Read the same 72 bits as eighteen 4-bit groups, arrange the module so that each group comes from exactly one device, and use a code whose alphabet is the group. Then “this device is wrong” is “this symbol is wrong”, and one wrong symbol is what a symbol code corrects, however many of its bits are bad. IBM shipped it first as Chipkill. Timothy Dell’s 1997 white paper “A White Paper on the Benefits of Chipkill-Correct ECC for PC Server Main Memory” put the idea in front of the industry in an earlier form that was not a symbol code at all: it spread a x4 device’s four bits across four separate SECDED words. Modern controllers use the symbol form, which Intel calls SDDC, Single Device Data Correction. Correcting one symbol needs at least two check symbols (the Singleton bound), and detecting one needs one. That decides every case:

symbol size 4 bits, word 72 bits
  18 symbols total.  2 check symbols = 8 bits.  16 data symbols = 64 bits.
  Corrects one 4-bit symbol.  A x4 device contributes exactly one symbol.
  => full single-device correction.                                  FITS EXACTLY

symbol size 8 bits, word 72 bits
  9 symbols total.  Only 1 check symbol available (8 bits).
  One check symbol detects a symbol error and cannot correct it.
  => a dead x8 device is detected, not corrected.                    DOES NOT FIT

symbol size 8 bits, two channels in lockstep, word 144 bits
  18 symbols total.  2 check symbols = 16 bits.  16 data symbols = 128 bits.
  => full single-device correction for x8.                           FITS EXACTLY

symbol size 4 bits, two channels in lockstep, word 144 bits
  36 symbols total.  4 check symbols = 16 bits.  32 data symbols = 128 bits.
  4 check symbols give distance 5, which corrects two symbols.
  => double-device correction for x4.                                FITS EXACTLY

A 72-bit bus built from x4 devices has exactly the redundancy needed for chipkill and not one bit spare. A 72-bit bus built from x8 devices does not, and no amount of firmware makes it. Device correction on x8 needs two channels in lockstep, moving 128 bytes where you wanted 64, and double-device correction needs lockstep and x4; Supermicro’s memory RAS guide says the double-device case “is not supported for x8 DRAM device-based DIMM”. Because lockstep makes every access pay for redundancy it usually does not need, Intel moved from Skylake-SP onward to ADDDC, Adaptive Double Device Data Correction: run ordinary SDDC until one device crosses a corrected error threshold, then retire it and pair two ranks for that region only. ADDDC requires x4 devices too. So the module code on a listing is not decoration:

2Rx4 in a listing title is the difference between surviving a bit flip and surviving a dead chip. It is also, usually, a registered module, because building 36 devices onto an unbuffered board is an electrical problem nobody wants, so on this site the x4 stock lives almost entirely in DDR4 registered ECC rather than DDR4 unbuffered. Whether a particular controller implements chipkill is a property of the CPU and the firmware and is usually undocumented. The arithmetic tells you what the module permits; dimm_edac_mode, in the proving section, tells you what the machine did with it.

DDR5 moved the ECC width: EC4 and EC8

DDR4 and everything before it put 64 data bits and 8 check bits on one channel. A DDR5 DIMM instead presents two independent 32-bit subchannels, each with its own command and address bus, and the burst length doubled from 8 to 16 so that one subchannel still delivers one 64-byte cache line per access. That is a clean swap for bandwidth and not for ECC, and JEDEC standardized two answers.

EC4 puts four ECC bits on each 32-bit subchannel, which makes a 72-bit module. EC8 puts eight, which makes an 80-bit one. Both are standardized - JEDEC even publishes EC4 raw cards for unbuffered modules, JESD308-U4-RCE and JESD309-S4-RCE - and exactly one of the two is on the market. Every DDR5 ECC module you can buy is EC8: eighty bits wide, ten x8 devices per rank, unbuffered parts included.

The Micron part numbers in the listing section settle it without trusting anyone’s marketing copy, and Samsung’s M321R4GA3BB6-CQK is sold as an “EC8 RDIMM”. With the doubled burst, EC4 would have handed the controller exactly DDR4’s budget, 64 check bits per 64-byte line or 12.5 percent; EC8 hands it 128, or 25 percent. So why does nobody build the cheaper one? Run the symbol arithmetic against the narrower subchannel, one beat at a time:

DDR4, x72 beat, x4 devices
  72 bits = 18 symbols of 4 bits.  16 data + 2 check.
  Two check symbols correct one symbol.       CHIPKILL

DDR5 EC4, 36-bit subchannel beat, x4 devices
  36 bits = 9 symbols of 4 bits.   8 data + 1 check.
  One check symbol detects and cannot correct.  NO CHIPKILL

DDR5 EC8, 40-bit subchannel beat, x4 devices
  40 bits = 10 symbols of 4 bits.  8 data + 2 check.
  Two check symbols correct one symbol.       CHIPKILL RESTORED

Halving the channel width halved the check budget per beat, which would have destroyed device-level correction, and EC8 buys it back. An EC4 module can detect a dead device but not correct one. The four-bit card is legal, cheaper and worse, and the industry declined it on every module type. (A real controller computes its code over the whole 64-byte transfer, so its correction may be better than the per-beat framing implies; no CPU vendor publishes what it does with the 128 bits.)

MRDIMM, the other new DDR5 module, multiplexes two ranks behind a buffer to present a doubled pseudo-channel data rate on Intel’s Xeon 6 platforms (DDR5 MRDIMM listings). It is an ECC registered module like any other. The wider generation comparison is in DDR4 vs DDR5.