Fixing SSD, Encrypted, Dead Grub, Failed Kernel

We have all been there: sudo apt upgrade...DEATH. And the Linux distro comes crashing down. We immediately need to start thinking about the options: Do I have backups? What is the big problem? Is there an easy solution? Today I will talk about issues with the Kernel failing to properly install on an NVME SSD with a Luks encrypted LVM...all this meant the regular tutorials were not going to help me to get my system back, nor does the Boot Repair utility that is on the Linux Mint Live CD.

The Problem We Solve

• Broken Kernel and Grub
• NVME SSD
• LVM
• Luks Encryption

Our Final Result

Success! The computer is back to normal

I Could Have Saved This

Yes, I was dumb. I actually saw that the kernel did not successfully install, and like a fool, I rebooted the system anyway! If you are reading this while passively watching the update process, Linux CAN be saved if a Kernel does not update correctly, but NOT as easily once the computer is turned off.

This is because when updates to a Linux system are performed, the new version does not take over until rebooted. This means that the Kernel will keep being used in the old form. So I could have mitigated this by (before rebooting), going into the Kernel manager and removing the one that failed to install. But life lessons learned, and we got an article and video to help others instead!

Getting The System Back

Our steps here are going to require booting the computer with a Live CD. It is best to use whatever is on the computer in a live install if possible. We just want to make sure software versions are compatible. We will be mounting the borked up mess into the /mnt directory of the live distro and running some grub updates in chroot.

NVME Is a Problem

Grub does not see NVME SSDs as well as your traditional sda, sdb, etc. This poses a problem for us. Additionally, my system is encrypted, so there are extra steps to get to the mount point. We will start with some helpful guides on AskUbuntu for some basics with mounting encrypted files and fixing grub, but we will need to make some modifications.

To start, we need to open up and mount our drives, but we need to know what we are working with. Run this command to see all the disks and their partitions:

sudo fdisk -l

Here is my readout:

The disk we are working with is labeled nvme0n1 and it has three partitions:

• nvme0n1p1 – This will get mounted to /mnt/boot/eft
• nvme0n1p2 – This will get mounted to /mnt/boot
• nvme0n1p3 – This is the actual distro with files, it will need decrypted, then mounted to /mnt

Decryption

Decrypting a Luks drive requires the cryptsetup package. Most live CDs will have it, but if you are unsure or you run into a problem, you can simply install it with:

sudo apt install cryptsetup

Once you are ready to decrypt, use this general format:

sudo cryptsetup luksOpen /dev/{drive} {name}

{drive} is the drive partition we are working with. In this case, that is “/dev/nvme0n1p3”

{name} is any name you want to give it; I used “emint”

Mounting

Mounting an encrypted drive will require you to use the LVM mapping, not the partition name or the name you gave the decryption. This info is in the fdisk we used earlier, or you can get it with:

sudo lvscan

In my case, it shows two logical volumes on my decrypted partition:

We will be using the /dev/vgmint/root and mounting it to /mnt

sudo mount /dev/vgmint/root /mnt

We also need to mount some other things important to make sure everything is mapped for the CHROOT to be successful. This command will mount some of the points we need:

for i in /sys /proc /run /dev; do sudo mount --bind "$i" "/mnt$i"; done

This command is a simple script that will mount each of these points from the original /mnt that we just made. Now we need the /boot and the /boot/efi. These both come from our drive:

sudo mount /dev/nvme0n1p2 /mnt/boot

sudo mount /dev/nvme0n1p1 /mnt/boot/efi

CHROOT

CHROOT is a Linux command that will lock your working directory into a child. In this case, we are mounting our old system into /mnt which will allow us to make system changes in that directory instead of the parent (in this case, the live CD). We are ready for grub to be fixed, so use this to lock yourself into your poor, suffering operating system:

sudo chroot /mnt

You will now be at a super user prompt inside your OS (note the # instead of $).

Here we will need to check that the fstab file is loading the correct partition. This file determines what partitions are loading. Run this:

sudo nano /etc/fstab

Look for the EFI line and note the UUID next to “boot/efi” We need to make sure this line matches the UUID of the partition, so run this:

blkid

The output will give you the UUID of each partition. Just check that the UUID of the partition mounted to /mnt/boot/efi is the same. If not, edit the file to match.

Once the UUID matches, install grub and update (We may not have to install, but it will not hurt anything to do so)

grub-install /dev/nvme0n1

update-grub

Now your system should be ready to reboot.

Wrap Up

In my case, the chroot gave me back my grub menu, but something else was still messed up from the failed Kernel update. I used my grub menu to load an older kernel in recovery mode, then I cleared out the broken Kernel, and updated correctly this time.

I also needed to make one more change to the fstab file as we did earlier to fix the UUID one more time. Once I did that, my system was working perfectly.