[GTALUG] war story: fixing a Fedora installation that got broken during update

[Nick Accad]

On Sun, Oct 20, 2024 at 18:10 D. Hugh Redelmeier via talk <talk at gtalug.org>
wrote:

> Many people don't update their software.  They claim that it only brings
> grief.  I'm the opposite: I take any update offered in the hope it is an
> improvement.
>
> One of my computers hung during a Fedora update.  I don't know why
> it happened because I wasn't there (I also don't watch paint drying).
>
> Symptom: booting would not get the the log-in screen (GDM?).
>
> Step 1: have booting log to the display
>
> Normally Fedora booting is quiet so as not to disturb the serenity of the
> user.  In this case we want to see the log.  It might tell us what's wrong
> and it might tell us how far it got before hanging.
>
> - at the GRUB boot menu, type "e" to be given the chance to edit the
>   commands that GRUB will execute to boot Linux.
>
> - cursor to the end of the "linux" line
>
> - remove the last two parameters ("rhgb quiet")
>
> - type CTRL-x to get the modified commands executed
>
> Now lots of logging information will scroll by on the screen.
>
> In my particular case, it didn't tell me much but I could see that booting
> got a long way.
>
>
> Step 2: boot from a live Fedora USB and examine the system's HDD/SSD.
>
> I assume that you know how to boot from a live installation USB flash
> drive.
>
> It is easy from that desktop to examine the broken system.
>
> - SMART scanning the system's disk might by worthwhile, if only to
>   eliminate that source of doubt.
>
> - file system checks are always a good idea
>
> In my particular case, I didn't see anything obvious.
>
> Complicated brain surgery can be done by starting up the broken system in
> a chroot environment.  I started down that road but was interrupted before
> success or failure.
>
>
> Step 3: boot the system console mode, without a GUI.
>
> This had promise because it looked to me as if the failure was starting
> the graphical system
>
> Proceed like step 1, but after erasing "rhgb quiet", add " 3"; hit CTRL-X.
>
> Log in.  Now you have a shell.
>
> Since the system failed after a software update, I decided to do more of
> them.  You could instead back out of them.
>
>   sudo dnf update
>
> Failure:
> There was a set of kernel packages that seemed to be half installed.
>
> I removed the half that was installed and asked again for an update.  It
> worked.
>
> I could reboot the system and it would come up as expected.
>
> Success!
> ---
> Post to this mailing list talk at gtalug.org
> Unsubscribe from this mailing list
> https://gtalug.org/mailman/listinfo/talk


I love war stories

In the future, add “verbose” and “systemd.debug-shell=1” to the grub line,
the first is obvious, the second gives you unsecured root shell on tty9
(alt f9).

-nick

> <https://gtalug.org/mailman/listinfo/talk>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gtalug.org/pipermail/talk/attachments/20241020/787c55b6/attachment.html>