[GTALUG] long war story: growing the ESP (/boot/efi)

[Lennart Sorensen]

On Thu, Jul 15, 2021 at 10:37:03AM -0400, D. Hugh Redelmeier via talk wrote:
> The following is based on knowledge plus a fair bit of superstitious
> extrapolation and interpolation.
> 
> NAND flash and NOR flash are different.  You are surely talking about
> a NOR flash
> 
> Lennart is talking about an unknown system.  I suspect it used NAND
> flash, if the description is right.
> 
> NOR flash:
> 
> - allows update of each byte separately (most are 1 byte wide)
> 
> - each update only wears the byte updated
> 
> - are small (eg. 8M is large).  Older consumer routers used NOR flash
>   and that is why their flash sizes were so limited.
> 
> - it is reasonable to run programs directly out of NOR flash (it looks
>   like RAM)
> 
> NAND flash
> 
> - allows updates in pages, but only to previously erased pages
> 
> - erases can only be done in large blocks.  The size of these blocks is
>   generally secret but I think that they are in the range of 128KiB to
>   1MiB.
> 
> - are so alien to deal with that they have a complicated controller
>   between them and your system.  The controller emulates a disk drive.
>   This does wear levelling.  There is some work at moving this into
>   the OS for performance reasons, but it is not common and only for
>   data-centres.
> 
> - density is everything in NAND flash systems.  The original ones used
>   a single bit per cell (SLC) but now 3 bits per cell is common.  To store 3
>   bits per cell, 8 charge levels must be distinguished.  This makes
>   them much slower and less reliable (shorter-lived).
> 
> - each cell can only be erased a small number of times (perhaps 1000)
>   but the wear-levelling usually prevents this being a big problem.
>   It is not part of published specifications.
> 
> - consumer routers now use NAND flash for firmware and allow orders
>   of magnitude larger firmware.  For reasons I haven't investigated,
>   it took OpenWRT years to support NAND flash routers.
> 
> - it is unreasonable to run programs directly out of NAND flash.  They
>   can, however, be paged in, just as one would do from a HDD.
>   This wrecks real-time performance, so routers would not do this.
>   They copy the contents of flash to RAM as part of booting.
> 
> Normal PCs:
> 
> - have used a battery-backed-up CMOS RAM module for firmware settings.
>   Everything is CMOS now, I think, but when this dedicated memory was
>   added to the PC, CMOS was notable.  Especially for its low standby
>   power requirement.

They did for BIOS systems.  UEFI systems tend to have a lot more
settings than could fit in the ram of the RTC chip.  Certainly the
UEFI boot entries are much bigger and not stored in ram.  A bug in the
SPI flash driver in the linux kernel that shipped with ubuntu 17.10
turned some intel SPI flash chips read only which made it impossible
to change any settings in the UEFI on those systems until you somehow
booted a kernel that had a fix to undo that read only flag on the chip.
This included settings for secureboot, usb booting, etc.  You couldn't
change any settings at all on those systems.  The RTC ram is no longer
used for system settings with UEFI.

I believe most systems in fact use one SPI flash chip for both the
firmware and the settings and partition it and keep the firmware area
write protected except when being updated.  They often seem to even use
one SPI flash chip to store the UEFI firmware, the management controller
firmware, the settings, and whatever else in the system needs some flash
for persistent storage all partitoned into chunks of the right size for
each device that needs it.

> - use flash memories with sequential access for firmware (SPI).  The
>   power-on sequence fetches the firmware and dumps it into RAM so that
>   it can be accessed randomly.  Dedicated programs are needed to
>   update the flash through the SPI interface.
> 
> - uses RAM, just as we are used to
> 
> - uses SSD, with its own controller to emulate HDD but more quickly.

-- 
Len Sorensen