[GTALUG] nvme SSD: critical medium error, dev nvme0n1

[D. Hugh Redelmeier]

| From: Stewart C. Russell via talk <talk at gtalug.org>

| I just wish the old drive had the smarts to remap bad sectors. There
| were a couple of files that *always* caused kernel errors, but enough
| space that files could be moved/made lost+found.

I'm pretty sure that the drive can remap those bad "sectors".  What
you have to do is write to the bad sectors.

Here's the logic of remapping, as I understand it:

- if a write fails and the controller knows this at the time of the
  write, that block is marked as bad AND the write is attempted at
  another block.  This is done with no indication to the OS (except
  S.M.A.R.T. counts are changed).  This is fine since there is no
  observable discrepancy (except for time).

  This is called "remapping" in the HDD world.  It should not be
  called this in the SSD because all blocks are mapped in the SSD
  world.

- if a read fails, even on retry, the controller must report this to
  the OS.  After all, the information has been lost.  No remapping
  is done because that would be hiding the loss of information.

- as a user, if you notice a bad block, and give up on trying to
  recover its content, just overwrite that block.

  + on an HDD, that will likely trigger a remapping

  + on an SDD, that will just cause the block to be mapped somewhere
    else (as always with a write).  One hopes that the controller
    is smart enough to consider the bad block permanently bad and
    never use it again.

Complications:

- bad blocks are probably part of a bigger bad unit.  For SSDs, I'd
  guess that the unit is an erase block (perhaps 128k or larger).  For
  HDDs it is surely 4K these days.

  For HDDs, that may mean that a cluster of blocks (adding up to 4k)
  is bad, not just one.  They may not be contiguous within one file
  but they probably are (I think that most Linux file systems allocate
  4k or more at a time).

  For SDDs, many files may have blocks within a single erase block.

- the normal way of finding bad blocks systematically is to use
  smartctl(8).  You should be able to get a list of bad blocks on the
  drive.

  You probably want to be able to find out which file contains each bad
  block, and at what offset.  It seems that the debugfs command can
  help for extN filesystems.  See
<https://linoxide.com/linux-how-to/how-to-fix-repair-bad-blocks-in-linux/>

- e2fsck -c almost does what you want, but not quite.

| What really annoyed me
| was that fsck would show clean, but these files still existed with errors.

fsck only checks metadata.  Errors are probably in plain old data.

But do see e2fsck's -c option.

smartctl --test=long is the right way.  Not surprisingly this can take
quite some time on a large HDD.  Probably not on an ordinary SSD.