Precautions

• Have a rescue disk or flash drive:
  
  Either a live-session copy of your current distro, on USB, or:
  Rescatux & Super Grub2 Disk
  Boot-Repair-Disk
  Ubuntu's "Boot-Repair"
  GParted Live (login as "user" / "live")
  SystemRescueCd
  LinuxAndUbuntu's "Top 5 Linux System Rescue CDs"
  Martins D. Okoi's "12 Useful Rescue and Recovery Tools for Linux"
  Hard-core: CAINE (Computer Aided Investigative Environment)
  
  The time to make rescue disks is before you have a problem, not after. If your hard disk is encrypted, test how you would access (decrypt) the disk when you've booted from a live-session on USB stick. Also test booting from hard disk into GRUB recovery.
  
  You may have to disable Secure Boot to run some of these.
  
  
  
  
• Have SSH access to the system:
  
  If you set up SSH access ahead of time, then you have a way to connect to a system even if GUI login fails, and you can examine log files and maybe fix things.
  
  
  
• Have backups of system configuration details:
  
  I run a script: SaveConfig.sh
  
  
• Disconnect extra disks any time you're going to do something dangerous:
  
  If you have removable disks A and B, and you want to format B, first disconnect A.
  
  

Problems and troubleshooting

• If the system won't boot:
  
  If system boots to an "(initramfs)" prompt (BusyBox), see article1, article2. "Booting into initramfs usually means that the system couldn't mount the root filesystem."
  
  To have system show everything it does as it boots, get to menu that shows list of kernels, select a kernel, press "e" to edit that entry. Remove 'rhgb quiet' [Red Hat specific] or 'quiet' [other Linux] and then boot from that kernel. Note: the edits will affect just the next boot, they won't be permanent. Note: you may not see a prompt for the FDE password, the output may just stop at that point until you type the password.
  
  System normally boots to systemd "graphical.target" (run "systemctl get-default" to see that). To boot to a different target, edit the line containing 'rhgb quiet' [Red Hat specific] or 'quiet' [other Linux] to add 'systemd.unit=NAME.target'. Then boot from that kernel. Alternative targets include "multi-user.target" (no GUI, just CLI), "emergency.target" (root is mounted but RO, no services started), and "rescue.target" (no network services).
  
  [Older instructions:] While booting, hold down Shift key to get into GRUB, then choose Options - Recovery, or some older kernel, or select current kernel and press "e" to edit its boot options. [If you edit the "linux" line, you can add "1" to the end of the to boot into TTY1.]
  
  
  Make sure the disk is not throwing hardware errors. Maybe use a SMART utility. article
  
  Maybe you have filesystem damage:
  Marin Todorov's "How to Use 'fsck' to Repair File System Errors in Linux"
  
  Maybe you have RAM problems; run memory test from GRUB menu:
  Karim Buzdar's "How to Run Memtest in Ubuntu 20.04"
  Or from OS: "sudo memtester 14G"
  
  
  Maybe the bootloader has been removed; see Bootloader section
  
  Boot into a live session (USB), and see if hard disk filesystems are full, fsck them.
  
  Boot into a live session (USB) of same OS, and see if there is a "rescue" option.
  
  
  If you boot a live environment, to look at the installed OS's journal:
  "journalctl --root=/path/to/mounted_root"
  or
  "journalctl --directory=/path/to/journal"
  
  
  If you want to restore from a Timeshift backup:
  [NOT SURE OF THIS] Boot from USB stick into a live session, then:

  
  timeshift --help
  sudo timeshift --list
  

  
  
  If you did an update, writing initramfs failed, then you rebooted:
  Probably old kernels have filled up /boot and it's out of space. Boot from an old kernel, then

  
  df -h                            # to see space
  dpkg --list | grep linux-image   # to see the kernel images
  sudo apt purge IMG1 IMG2 IMG3    # to purge several of the oldest images
  

  
  
  
  
  
• If dual-booting and some OS's won't boot:
  
  See Bootloader section
  
  
  
• If you can login but immediately get kicked out again ("login loop"):
  
  Could be due to GPU drivers, home partition full, bad configuration under home directory ?
  
  Boot from USB, see if hard disk filesystems are full, fsck them.
  Mount root and then "journalctl --root=/path/to/mounted_root".
  
  Nick Congleton's "How to Fix the Ubuntu Login Loop"
  Tiwo Satriatama's "3 Ways To Fix Ubuntu Gets Stuck in Login Screen Loop"
  
  
  
  
• If the system won't boot into GUI:
  
  Look in $HOME/.xsession-errors file.
  
  On Ubuntu, maybe "sudo ubuntu-drivers autoinstall" ?
  
  See if "sudo systemctl start gdm3" works.
  
  While booting, hold down Shift to get into GRUB, then choose Options - Recovery.
  If you want to restore from a Timeshift backup:

  
  timeshift --help
  sudo timeshift --list
  

  
  
  
• If a filesystem can't be mounted:
  
  Xiao Guoan's "How to Fix 'can't read superblock' Error on Linux (ext4 & Btrfs)"
  
  
  
  
• If all of your files are read-only:
  
  Do "mount" to see if the filesystem was mounted with read-only flag.
  
  This might mean that the OS detected filesystem errors it couldn't repair.
  
  Check the health of your drives.
  
  Maybe dismount filesystem and use "fsck".
  If it's the root filesystem, do "sudo touch /forcefsck" and then reboot.
  Or, hold down Shift key while booting, get into GRUB, select Advanced Mode / recovery / fsck.
  Or, in GRUB, edit a kernel command line to add "fsck.mode=force".
  
  
  
  
• If some application is locking up:
  
  Alt+F2. If you get a small window, type "xkill" and click on a window you want to kill.
  
  If it's a GUI app, launch it from CLI, and see if any error messages appear in the terminal.
  
  
  "sudo gnome-system-monitor" to see various processes ?
  
  
  While app is running, do "top -H -p PROCESSID" to see what the app's threads are doing.
  
  "ps -ax | grep APPNAME" to find the app's PID and then "strace -p PID" to see what system call(s) it's doing. This also will show you if the app really is stuck (waiting in one system call) or just is doing lots of work.
  
  Launch app via "strace APPNAME" to see execution from start. "strace -o FILE.txt APPNAME" to capture the strace output in file "FILE.txt".
  
  To trace what files an app is using, "strace -e trace=open,openat,openat2 -f YOURCOMMANDANDARGS".
  
  "ltrace" is similar to "strace" but includes calls to library functions, such as C stdio functions.
  
  
  
• If the GUI is locking up:
  
  Right-click on a blank area of the System Tray, select Troubleshoot, then Looking Glass. Or on CLI, run cinnamon-looking-glass. It will launch Melange, the Cinnamon DE debugger. Select Actions/Quit to exit. Pressing Esc also exits, but not completely ?
  
  Or if it's the X Server that's freezing, do Ctrl+Alt+F3, get login prompt, login, do "sudo systemctl restart gdm3". Logout, Ctrl+Alt+F7 to get back to GUI.
  
  Instructions from someone on reddit:
  Do Ctrl+Alt+F6, get login prompt, login, do command "w", FROM column will show a display number such as ":0". Do "export DISPLAY=:0; cinnamon &". Logout, Ctrl+Alt+F7 to get back to GUI.
  
  See Magic key-sequences section.
  
  In Linux Mint 19.0 Cinnamon I had "UI freezes" where I'd still see occasional disk activity, so the underlying OS still was running. Stopped using Synaptics touchpad driver, and the freezes went away. See https://www.linuxmint.com/rel_tina_cinnamon.php for some info about touchpad drivers.
  
  "less /var/log/Xorg.0.log"
  
  
  
• If the system boots but then soon crashes:
  
  Make sure the disk is not throwing hardware errors. Maybe use a SMART utility. article
  
  Boot again, then do:
  sudo journalctl -b -1 >lastboot.log
  tail lastboot.log (maybe take it to a working machine)
  
  
  
  
• If the whole OS is freezing or crashing:
  
  Look for info in output of:

  
  sudo dmesg -xT
  sudo dmesg --level=emerg,alert,crit,err,warn -T
  sudo journalctl --pager-end
  sudo journalctl --since=today
  sudo journalctl --since=2019-12-31
  less /var/log/Xorg.0.log
  

  Thomas Stringer's "Using journalctl Effectively"
  
  Look for crash dumps in System Reports application.
  If you want to get hardcore and analyze a crash dump yourself, there are utilities such as "crash" for that. Do a search for articles about "Linux analyze crash dump". Stephan Avenwedde's "Creating and debugging Linux dump files"
  
  Did you change anything recently ? "dnf history | less"
  Are you doing something such as overclocking the system ? Don't.
  
  Check the health of your drives
  Run fewer apps, does crash still happen ?
  For UI freezes, try removing Synaptics touchpad driver.
  Remove anything that is optional: USB devices, for example.
  Run the System Monitor application and watch it for any strange behavior.
  Install something (maybe Psensor) to measure CPU and GPU temperatures.
  Run BIOS diagnostics for RAM and disk.
  
  Maybe you have RAM problems; run memory test from GRUB menu:
  Karim Buzdar's "How to Run Memtest in Ubuntu 20.04"
  
  From someone on reddit:

  When I have inexplicable hangs I ssh in from another box and run logs (sudo dmesg -w, tail -F -s 0.1 --retry /var/log/lots_of_log_files etc) and then replicate the problem. Often I can see some error reported before the box croaked.

  
  Nick Congleton's "4 Easy Ways to Get Out of a Ubuntu Crash"
  
  Are you running out of RAM and don't have swap, or don't have much swap ?
  Maybe see hakavlad / nohang
  
  From someone on reddit 3/2021:

  Linux tends to freeze on Ryzen CPU's from time to time. Either modify your kernel or add a few parameters to GRUB. I personally use:

  
  idle=nomwait acpi_backlight=vendor rcu_nocbs=0-7
  # https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
  

  The first and the last are usually the most important, the second is there because I'm on a laptop. Without these parameters I have random system-wide hang ups that can only be resolved through hard power-off.
  
  
  
• System testing utilities:
  
  Ubuntu: "sudo apt install checkbox-gui" (maybe deprecated ?)
  Ubuntu: CheckboxNG
  Phoronix Test Suite
  Hardware Probe
  
  
  
  
• If you have no network access:
  
  Nice little script for checking your network connectivity: hakerdefo / check-my-net
  
  Maybe you have duelling firewalls ? You can't run firewalld and ufw/gufw at the same time. firewalld and ufw both use iptables.
  "whereis -b -B /usr/bin /usr/sbin -f gufw ufw firewalld"
  
  Some VPN clients also use iptables. "sudo iptables -L -v"
  
  Check status of network interfaces. "ip addr"
  
  
  Any services failing ? "sudo systemctl list-units --state failed"
  
  
  
  
• If your Wi-Fi stops working:
  
  Does same Wi-Fi network still work on other machines in the house ?
  Have you turned off Wi-Fi with a switch on the outside of the laptop ?
  Have you turned off Wi-Fi in BIOS ?
  Have you turned off Wi-Fi in software ? "rfkill"
  Does the option to turn on Wi-Fi exist in the Settings ?
  Does a Wi-Fi device (probably named "wSOMETHING") exist in output of "ip addr" ?
  Does a Wi-Fi device (maybe "Network controller") exist in output of "lspci" ?
  Check Wi-Fi device and driver in output of "inxi -n".
  Run the "Drivers" GUI app for your distro. Maybe switch between "proprietary driver" and "don't use device". "Don't use device" may be a lie; try that option.
  
  
  
  
• If you really screw up the system configuration somehow, before re-installing you could try resetting:
  
  Resetter:
  FOSSMint's "Resetter - Reset Ubuntu and Linux Mint to Default Settings"
  Jack Wallen's "Set Ubuntu Derivatives Back to Default with Resetter"
  VITUX's "How to reset Ubuntu"
  
  To reset all GNOME settings: "dconf reset -f /"
  
  You could try creating a new user, which would get all the default settings.
  
  
  
• If you delete some system files somehow:
  
  nixCraft's "Accidentally overwrote a binary file on Linux? Here is how to restore it"
  
  
  
  
• Do not delete any versions of Python:
  
  Much of the system GUI is written in Python, apparently.
  
  Other things to not delete: gcc, lightdm, boost, mysql-*, mariadb-server, /run/timeshift, /var/timeshift.
  
  
  
  
  
  
• If you "sudo apt remove" a lot of critical system files but your system still is running:
  
  

  1. Quickly "sudo apt install" what you removed.
  2. Close down running applications.
  3. Do backups of important files.
  4. Make sure you have an installer image on USB in case the system won't boot.
  5. Think: what will you do if the system won't boot at all, what will you do if it won't boot to GUI, what if you have to re-install, etc ? Do you have passwords, etc ? Take your time.
  6. Take a deep breath and shut down, then power on.
  
  
  
• If you delete a few files under your home directory somehow:
  
  Martin Brinkmann's "How to use PhotoRec GUI to recover lost digital photos and files"
  Aaron Kili's "PhotoRec - Recover Deleted or Lost Files in Linux"
  Falko Timme's "Recover Deleted Files With foremost" (ancient ?)
  sleuthkit / scalpel
  Kali Linux's "Scalpel -- Recover Permanently Deleted Files"
  Alex Clemmer's "Recovering deleted files using only grep" (text files)
  Recover onto a different drive than the files were deleted from.
  File names probably will be lost, but the contents may be recoverable.
  
  For low-level surgery on a damaged ext* filesystem: "man debugfs", "man ext4magic".
  
  On the CLI, it's safer to delete routinely using "trash-cli" or "gio" or "rm-protection" instead of "rm". But none of them are a same-name drop-in replacement for rm; you have to remember to use them, and they don't have the same options as rm.
  
  
  
  
• If you rm -fr lots of important files under your home directory:
  
  Recommended recovery tools:
  ReclaiMe (paid, easiest)
  R-STUDIO (paid)
  DMDE (paid, cheapest)
  TestDisk and PhotoRec (free)
  
  From someone on reddit:

  1. Power off ASAP.
  2. Get an unused external drive. It must be larger than the partition you're trying to recover.
  3. Boot from a live USB image.
  4. Connect the backup drive to the system.
  5. Use dd to copy the entire partition to a file on the backup drive:
     "dd if=/dev/sda4 of=/mnt/usb/myhomepartition conv=fsync"
     or
     "dd if=/dev/mapper/VolGroup-home of=/mnt/usb/myhomepartition conv=fsync"
  6. Install testdisk within the live USB environment, and use that to try to recover your data:
     "testdisk /mnt/usb/myhomepartition"
  7. Repeat steps 5 and 6 if you damage the image of your partition.

  
  

  
  maketecheasier's "6 Linux Tools to Help Recover Data from Corrupted Drives"
  VITUX's "How to Recover deleted files in Ubuntu through TestDisk"
  CGSecurity's "TestDisk Step By Step"
  extundelete ?
  UFS Explorer
  
  
  
  
• If you delete an important file but some application still has it open: article
  
  
• Damaged VeraCrypt volume:
  
  Keep a backup copy of the volume header (made via VeraCrypt / Tools / Backup Volume Header).
  
  If you don't have such a header backup file, you could try VeraCrypt / Tools / Restore Volume Header and select "Embedded backup header".
  
  
  
  
• If OS is destroyed and you need to copy files from an encrypted volume:
  
  If you have a home directory encrypted with eCryptfs ("df -khT | grep /home/" will show a /home/USERNAME/.Private partition with type "ecryptfs"): Linux_Lobo's "HOWTO: Recover files from encrypted home directory"
  
  
  
  
• If the hard disk is damaged:
  
  MakeTechEasier's "7 of the Best Data Recovery Tools for Linux"
  Carla Schroder's "GNU ddrescue - The Best Damaged Drive Rescue"
  
  "Don't fool around with half-measures for a while, trying to copy off files using the file explorer or dd or such: every use of the disk risks more and worse damage. Go straight to ddrescue and copy the whole thing to a new drive.".
  
  After you get a copy onto a new drive, then you can try to recover files from the new drive using tools such as:
  ReclaiMe (paid, easiest)
  R-STUDIO (paid)
  UFS Explorer (paid)
  GetDataBack (paid; runs on Windows)
  DMDE (paid, cheapest)
  PhotoRec
  foremost
  sleuthkit / scalpel
  
  If drive seems totally dead:

  
  sudo hdparm -i /dev/sda
  sudo hdparm -I /dev/sda
  

  
  
  
• There are hidden sections of the hard disk:
  
  HPA (Host Protected Area) and DCO (Drive Configuration Overlay) are "hidden areas" used by some hard drives. It's possible that they could get wiped or get malware in them, maybe. Or maybe you just want to use every single byte of space.
  
  

  
  # show areas:
  sudo hdparm -N /dev/sda
  sudo hdparm --dco-identify /dev/sda
  
  # VERY dangerous:
  sudo hdparm -N pSSSSSSSSSSS /dev/sdx  # remove HPA, set visible area/max sectors to SSSSSSSSSSS
  sudo hdparm --dco-restore /dev/sdx    # revert DCO back to factory defaults; may brick drive
  # Also, GParted has a feature to remove HPA ?  But I don't see it.
  

  
  
  
• If a password won't work:
  
  If a password you're sure is correct stops working, maybe your keyboard is in a mode such as caps-lock or function-lock, or maybe a key has failed, or somehow you changed keyboard layout. Test it by typing the password into a document or some visible field. If you're copying and pasting the password, make sure you don't have a space at one end.
  
  
  
• If you forget the login password:
  
  Note: things differ between encrypted and not-encrypted situations.
  Ubuntu's "LostPassword"
  Mint Guide's "Reset the password for ROOT or any user in Linux Mint"
  Easy Linux tips project's "Forgotten password: how to recover from this in Linux Mint"
  
  
  
  
• If your whole disk is LVM/LUKS-encrypted and something goes wrong:
  
  This really isn't "whole disk" encryption: partition table and boot partition are not encrypted.
  See LUKS encryption section of my Linux Storage page.
  
  
  
  
• If you really screw up the system and need to re-install:
  
  Easy Linux tips project's "How to re-install Linux Mint easily"
  
  If you're going to re-install, maybe see if there's a BIOS update available from your PC's manufacturer, it's a good time to update. To see the current BIOS version, do "sudo dmidecode -s bios-version".
  
  There is a firmware service that some vendors use: Linux Vendor Firmware Service.
  Arch Wiki's "fwupd"
  fwupd / fwupd
  gnome-firmware
  "man fwupdmgr" and "fwupdmgr --show-all-devices get-devices" and "fwupdmgr refresh" and "fwupdmgr get-updates"
  
  Do "fwupdate --help" or "/usr/lib/fwupd/fwupdate --help". But it seems to be EFI-only, and mostly a test-tool. fwupdate.1 man page
  
  
  
  
• If you forget the password for a ZIP or RAR archive:
  
  Use fcrackzip or RarCrack.
  
  
  
  
• If something is keeping a file/directory open when you want to delete it:
  
  Use fuser, lsof, lslocks to find out what is keeping the file open.
  
  
  
  
• If you're running out of disk space:
  
  Jesse Smith's "What to do when the disk acts full"
  
  See Clean up space on disk.
  
  
  
  
• If you're running out of RAM:
  
  Run "System Monitor" application.
  Some applications have internal memory-monitor features (Firefox has "about:memory").
  
  
  
  
• If your Wi-Fi cuts out every now and then:
  
  Check power-management settings; maybe something is turning off Wi-Fi when it's been idle for a while, or when battery is low.
  Maybe install TLP.
  Check if same thing happens with wired connection, or on a different LAN, or to other devices on same LAN.
  Turn off all other devices using Wi-Fi on same LAN.
  
  
  
  
• If some app or process gets killed every now and then:
  
  Maybe you're running out of memory, and an "OOM killer" is acting ? Some such killers are: systemd-oomd, oom_reaper, earlyoom.
  
  

  
  sudo ps -ex | grep oom
  
  sudo journalctl | grep 'Out of memory'
  # To see more, take the timestamp when that occurred, and do:
  sudo journalctl | grep 'TIMESTAMP'
  
  systemctl status systemd-oomd
  
  # to turn it off, persistently across reboots:
  sudo systemctl mask systemd-oomd.service
  

  
  How much memory allocated by all the processes spawned by an application:

  
  # https://opensource.com/article/21/10/memory-stats-linux-smem
  smem -c pss -P APPNAME -k -t | tail -n 1
  

  
  There are some ways to control OOM-killing, but it seems there is no simple way to control it per-application (as in "never kill Firefox"). You can control it per-service, or per-process once a process is running and has a PID.
  article1
  "The way the systemd OOM killer behaves is to monitor the usage of the entire cgroup, and when the cgroup exceeds its limit, the whole cgroup is killed, not just the most offensive process within the cgroup."
  
  

  
  # Which processes have adjustments so less likely to kill:
  export pids=""
  export sep=""
  for pid in `grep '-' /proc/*/oom_score_adj | cut -d / -f 3`
  do
  pids="$pids$sep$pid"
  sep=","
  done
  ps -p $pids
  grep '-' /proc/*/oom_score_adj
  
  # Set application to be harder to kill:
  for pid in `pidof keepassxc`
  do
  sudo echo '-200' >"/proc/$pid/oom_score_adj"
  done
  # Some processes seem to have hardcoded values,
  # will give "permission denied".
  

  
  
  
  
  
• If some process is making the whole system slow and causing GUI timeouts and such:
  
  Try "nice" or "ionice" commands on that process to give it lower priority.
  
  
  

Magic key-sequences

• Alt+F2 (Ctrl+Alt+Esc on Fedora 34 KDE) to get a small command window, if Xorg is still working. If you get the small window, type "xkill" and click on a window you want to kill.
  
  
• Ctrl+Alt+F1 to get out of GUI and to console-login;
  then Ctrl+Alt+F7 to get back to GUI.
  
  May have to enable this via:

  
  grep 'ACTIVE_CONSOLES=' /etc/default/console-setup	# see "/dev/tty[1-6]" or similar
  sudo sed -i "s/#NAutoVTs=/NAutoVTs=/" /etc/systemd/logind.conf
  # then reboot
  

  
  While in console, "sudo service lightdm restart" will kill all GUI apps and restart the GUI system, Ctrl+Alt+Del will abruptly restart the system.
  
  
• Alt+Shift+F12 might restart the window-compositor.
  (Works on Kubuntu 20.10)
  
  
• Ctrl+Alt+Esc might kill current GUI and start a new one.
  (Works by default on Mint 19)
  
  
• Ctrl+Alt+Backspace might kill current GUI and take you to login screen.
  (Not by default on Mint 19 or Ubuntu 20)
  
  To enable on Mint 19 or Ubuntu 20: run "sudo dpkg-reconfigure keyboard-configuration", press Enter 5 times to get through 5 screens with current settings, then see dialog about "Ctrl+Alt+Backspace terminate the X server", choose Yes.
  
  
• Ctrl+Alt+Del might restart the system. Also, try typing it 4 times in rapid succession.
  
  
• Alt+SysRq+ or Alt+PrntScr+ keys (on Mint 19 and Ubuntu 20, they work only in console):
  • H: Show help.
  • Space: Show available Alt+SysRq keys.
  • R: Switch the keyboard from raw mode to XLATE mode. (Disabled on Mint)
  • E: Send the SIGTERM signal to all processes except init. (Disabled on Mint)
  • I: Send the SIGKILL signal to all processes except init. (Disabled on Mint)
  • S: Sync all mounted filesystems.
  • U: Remount all mounted filesystems in read-only mode.
  • B: Immediately reboot the system, without unmounting partitions or syncing.
  • 0: Change the console loglevel to 0 to reduce error messages.
  • K: Kill all processes on the current virtual terminal.
  • F: Invoke the OOM-killer, to kill a memory-hogging process.
  See /etc/sysctl.conf
  To enable transiently: "sysctl -w kernel.sysrq=1" ?
  To enable persistently: edit /etc/default/grub and add "sysrq_always_enabled=1" to GRUB_CMDLINE_LINUX ?
  kember's "REISUB - the gentle Linux restart"
  Wikipedia's "Magic SysRq key"
  Fedora Wiki's "QA/Sysrq"
  Kernel.org's "Linux Magic System Request Key Hacks"
  

Clean up space on disk

df -khT
df -khT -x squashfs -x tmpfs		# don't show snap devices

sudo du -h --summarize --exclude=/home --exclude=/sys --exclude=/proc --exclude=/run /
sudo du -h --summarize -c --exclude=/home --exclude=/sys --exclude=/proc --exclude=/run /*

sudo du -h --summarize -c /home/*
du -sh $HOME/* | sort -h
du -sh $HOME/.[0-9a-z]* | sort -h

sudo df -i

• Check to see how many Timeshift snapshots have been saved.
  
  But apparently subsequent snapshots are incremental, so the 1st will be big (15-20 GB ?) and the rest will be small (2 GB ?).
  
  
  
• Space occupied by system journal: "journalctl --disk-usage".
  
  You can delete old entries in the journal by doing "sudo journalctl --vacuum-time='2d'". Saves a surprising amount of space.
  
  Put a permanent cap on number of entries in journalctl: Sudo edit /etc/systemd/journald.conf to set "SystemMaxFiles=7" (saves info from last 7 boots).
  
  
  
• Space occupied by system log files: "sudo du -sh /var/log".
  
  

  
  grep logrotate /etc/crontab /etc/cron.*/* /etc/anacrontab /var/spool/cron/crontabs/*
  cat /etc/logrotate.conf
  man logrotate
  
  # As a quick fix:
  sudo rm /var/log/*.gz
  

  
  
  
  
• If you are using Btrfs or ZFS snapshots, deleting files may not free up any space until the snapshot(s) referencing those files are deleted. So delete some snapshots. Btrfs info
  
  
  
• If you installed a lot of applications just to try them out, go back and remove the ones you don't use. If you installed something through Software Manager, remove it through Software Manager. Scroll through list of installed applications by opening Software Manager, clicking on "hamburger" icon in upper-right, check "Show installed applications".
  
  

  
  # List installed packages sorted by size:
  dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -rn | less
  rpm -qa --queryformat '%10{size} - %-25{name} \t %{version}\n' | sort -n
  pacgraph     # Arch
  

  
  
  
  
• Apps based on Flatpak take a lot of space. Run Software Manager and click on the Flatpak button in lower-right to see list of all Flatpak-based apps. But sometimes that doesn't show all of them, so: at CLI, do "flatpak list --app". Consider removing any that you have installed.
  
  Lots of Flatpak cache files can pile up in /var/tmp/flatpak-cache* directories. Do:

  
  sudo bash -c "! pgrep -x flatpak && rm -r /var/tmp/flatpak-cache-*"
  

  Lots (325K files, 5.8 GB) of Flatpak stuff under /var/lib/flatpak, but I don't know if any of it can be deleted.
  
  It's possible that an app is removed but its "runtime", or a "runtime" that it and others once depended on, remains behind ? Compare outputs of "flatpak list --app" and "flatpak list -d --app --runtime" ? Run "flatpak uninstall --unused" to remove them. On my machine, it reduced that 5.8 GB to 47 MB !
  
  
  
• Apps based on Snap can take some space. At CLI, do "snap list". Consider removing any that you have installed. Do "apt search APPNAME" to see if a non-snap version is available.
  
  By default, last 3 versions of each snap image are cached. Change to minimum via "sudo snap set system refresh.retain=2".
  
  
  
• See what Docker images are installed: "sudo docker images".
  
  
  
• See what AppImage apps are installed:
  "sudo find / -name "*.AppImage" -type f -print | less"
  
  
  
• Run Software Manager and search for "font". Remove any you don't need. But you'll probably save only 10 MB or so.
  
  
  
• To see what directories are consuming space, use "baobab" or "ncdu" or "kDirStat" or "du -h".
  
  
  
• See list of all installed packages, sorted by size:
  "dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | less"
  
  Also: "du -sh /var/cache/apt/archives" and then if the number is large, "sudo apt clean".
  
  
  
• Do you have a big swapfile ? "ls -l /swapfile"
  
  
  
• Search for big files:
  "find / -type f -size +10240k"
  or
  "find / -size +20M -exec ls -lahg {} \;"
  
  
  
• Clear caches in browsers and email client apps.
  
  
  
• Caches etc under your home directory:
  "sudo du -sh $HOME/.cache/* $HOME/.local/share/* $HOME/.var/app/* >tmp.txt; less tmp.txt"
  There may be garbage left over from things you uninstalled.
  
  
  
• Delete crash reports/dumps. See "System Reports" application, and Firefox's "about:crashes".
  
  
  
• Install some things under your home directory instead of on /, if they are separate partitions and you are low on space on /. Flatpak option "--user" when installing a flatpak application.
  
  
  
• Run something like BleachBit. But I would turn off all the apt-related options.
  SK's "How To Clean Up Junk Files In Ubuntu Using Ubuntu Cleaner"
  Also "apt clean" (clears local repository cache).
  
  
  
• Removing old kernel images:
  
  In Mint, open Update Manager, do View / Linux Kernels.
  
  On Ubuntu, if using "mainline", do "mainline --clean-cache".
  
  Debian-family, through CLI:

  
  # show current kernel:
  uname -r
  
  # show all kernels except the one currently running:
  sudo dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -v `uname -r`
  
  # For each image you want to remove, do:
  sudo apt purge linux-image-NNNNNNNNNNNNN-generic
  
  # These shouldn't be necessary:
  sudo apt autoremove
  sudo update-grub
  

  ipbastola's "Safest way to clean up boot partition - Ubuntu 14.04LTS-x64"
  
  RedHat-family, through CLI:

  
  # show current kernel:
  uname -r
  
  # show all kernels:
  dnf list kernel --quiet
  rpm -q kernel
  
  # For each kernel you want to remove, do:
  sudo dnf remove kernel-NNNNNNNNNNNNN
  

  
  Manjaro Kernels
  
  
  
  
  
• I greatly distrust global cleanup operations such as "apt autoremove" (some people say it's okay, just don't use wildcards). Other global operations such as "sudo apt clean" just empty cache, much safer but soon much of the contents will come back.
  
  
  

Performance

• Run "System Monitor" application to see what is taking up most CPU or RAM.
  
  
• Look in system logs to see if there are errors, or tons of logging by some app.
  
  
• Probably worth doing: set "noatime" on all mounted physical volumes.
  Jim Hall's "Improve Linux system performance with noatime"
  
  
• If you have extra RAM, maybe use no swap at all.
  
  
• If you have extra RAM, maybe put temp filesystems in RAM. Check typical size with "watch sudo du -hs /tmp". Edit /etc/fstab to add:

  
  tmpfs  /tmp      tmpfs  noatime,nodiratime,nodev,nosuid,mode=1777,size=1G,defaults  0  0
  tmpfs  /var/crash  tmpfs  noatime,nodiratime,nodev,nosuid,mode=1777,size=1G,defaults  0  0
  

  Close all apps, use CLI to remove contents of those dirs, then reboot.
  
  Maybe do same for /home/USERNAME/.cache ? There's around 300 MB of stuff under there.
  
  Maybe do same for /var/log ? But after a system crash you'd have no logs to look at. I don't do it.
  
  "echo $TMPDIR" may show another place used for temp files.
  
  
• If you have extra RAM, maybe use "vmtouch" to lock some key files into RAM ? Probably not worth doing.
  
  
• Check: CPU governor.
  CPU section
  
  
• Very minor tweak: "grep CRON /var/log/syslog /var/log/syslog.1" to see how often cron jobs are running, maybe dial some of them back. See Cron.
  
  
• If system boot is slow:
  
  "sudo systemd-analyze"
  "sudo systemd-analyze blame"
  "sudo systemd-analyze critical-chain"
  [But see Thomas Stringer's "Understanding systemd Critical Chains"]
  "journalctl -xp 0..3 -b"
  Look in /etc/fstab for any volumes that don't exist any more.
  Failing login to VPN or connection to network shares ?
  
  
• If system is extremely slow after booting:
  
  RAM or disk full ?
  Some process taking 100% CPU ?
  Look in system logs for repeated errors.
  Disk: "sudo smartctl -a /dev/sda | less" or "sudo smartctl -a /dev/nvme0n1p2 | less"
  Some driver or hardware problem ? Remove any external or optional hardware.
  
  
  

Run Performance Tests

# Download and extract the test package.
# Then:

sudo ./install-sh
sudo apt install php-cli php-xml pkg-config

phoronix-test-suite --help
# have to accept some license terms etc

# can run via cli or GUI
# GUI will give a CLI-type interface; type 'quit' to get out

# https://www.phoronix-test-suite.com/documentation/phoronix-test-suite.html

phoronix-test-suite list-available-suites
phoronix-test-suite list-available-tests | less

# I want: simple, few dependencies, run in a few minutes, test CPU RAM disk.

# Best to run tests after a fresh boot, no other apps running.
# Run each test 3 times and average them.

phoronix-test-suite benchmark pts/sysbench
# My 2011 Dell Inspiron N5010 with Kubuntu 20.10:
# (Intel Core i3 370M @ 2.39GHz, 8 GB RAM, 320 GB WD3200BEVT-7 with ext4/LUKS)
# gave CPU 2100, RAM 9.8 M
# My 2021 KDE Slimbook 15 with KDE Neon 5.20.5:
# (AMD Ryzen 7 4800H @ 2.9GHz, 16 GB RAM, 256 GB NVMe SSD with ext4)
# gave CPU 17600, RAM 13.6 M

phoronix-test-suite benchmark pts/leveldb
# My 2011 Dell Inspiron N5010 with Kubuntu 20.10:
# (Intel Core i3 370M @ 2.39GHz, 8 GB RAM, 320 GB WD3200BEVT-7 with ext4/LUKS)
# gave seq fill 12 MB/S, hot read 1.76 usec/op, random read 1.71 usec/op.
# My 2021 KDE Slimbook 15 with KDE Neon 5.20.5:
# (AMD Ryzen 7 4800H @ 2.9GHz, 16 GB RAM, 256 GB NVMe SSD with ext4)
# gave seq fill 23 MB/S, hot read 7 usec/op, random read 7 usec/op.

phoronix-test-suite benchmark pts/fio
# Choose sub-test 3 (sequential read), option 4 (Linux AIO), Buffered, Not Direct,
# 4 KB block size.
# My 2011 Dell Inspiron N5010 with Kubuntu 20.10:
# (Intel Core i3 370M @ 2.39GHz, 8 GB RAM, 320 GB WD3200BEVT-7 with ext4/LUKS)
# gave 48 MB/S.
# My 2021 KDE Slimbook 15 with KDE Neon 5.20.5:
# (AMD Ryzen 7 4800H @ 2.9GHz, 16 GB RAM, 256 GB NVMe SSD with ext4)
# gave 1132 MB/S.

Speedcheck

My 2011 Dell Inspiron N5010 and 100 mbps chip with Kubuntu 20.10:
Without VPN on 600 mbps service: ping 75 down 95 up 100.
With Windscribe VPN (OpenVPN) on 600 mbps service: ping 80-90 down 55-75 up 45-80.
Without VPN on 100 mbps service: ping 70-80 down 95 up 100.
With Windscribe VPN (OpenVPN) on 100 mbps service: ping 85-125 down 85-90 up 75-80.
My 2021 KDE Slimbook 15 and 1000 Mbps chip with KDE Neon 5.20.5:
Without VPN on 600 mbps service: ping 45-55 down 365-445 up 105-130.
With Windscribe VPN (OpenVPN) on 600 mbps service: ping 65 down 15-20 up 50 [fast.com = 65/120].

Apparently "iperf3" is a better way to test (requires 2nd machine on LAN) ?
Also "speedtest-cli" ?

Apparently some games, and Steam, will display a "frames per second" as you play the game. You can use this to compare performance before and after changes.
Also, at CLI, run "glxgears" to see frame-rate.
Test suites: Novabench, Heaven UNIGINE.
Is direct rendering enabled ? "glxinfo | grep "direct rendering"

Sequential read: "sudo hdparm -t /dev/nvme0n1p2"
KDiskMark
bonnie++

Tweaking/fixing video

• Intel CPU with integrated graphics: drivers built into kernel.
• Intel CPU with Intel GPU (HD, Iris, UHD).
• AMD CPU with integrated graphics: driver built into kernel.
• AMD CPU with AMD GPU (Radeon): closed-source AMDPRO driver, or open-source AMDGPU / AMDVLK driver, or open-source Mesa RADV Vulkan driver built into Mesa stack.
• Any CPU with NVIDIA GPU (GeForce, Quadro, NVS): closed-source NVIDIA driver, or open-source Nouveau driver.

Tweaking/fixing mouse/trackpad/scrolling/video latency/tearing:

Definitely back up the config files before you start messing with stuff, it's easy to go wrong.
Check what drivers you are using.
Check what resolution and refresh you are using: "xrandr".
Compare behavior in various apps; maybe change a "smooth scrolling" setting in an app.
For KDE, maybe try kwin-lowlatency package.
Maybe try a different DE, distro, or GPU hardware.


"The addition of input lag is always going to be the trade-off for most solutions that fix screen tearing."

"... tearing happens when the screen has to draw the next frame before its done drawing the current one. If you stay below [maximum rated Hz] your monitor will always have the time to draw frames to the screen. So, just capping the game's fps to, say, [maximum minus 4] fps, should do the trick."

Tweaking/fixing audio

• Hardware.
  "hwinfo --sound"
  
  
  
• ALSA is the driver level.
  "aconnect -l"
  "cat /proc/asound/cards"
  "alsa-info.sh --stdout | less"
  rendaw's "ALSA, exposed!"
  ArchWiki's "Advanced Linux Sound Architecture"
  
  
• PulseAudio is an audio server used by applications. Also emulates hardware to handle legacy apps that try to access hardware directly.
  "pactl list short clients"
  "less /etc/pulse/daemon.conf"
  "pavucontrol"
  Freedesktop's PulseAudio
  
  
• PulseAudio-compatible apps: Chrome, Firefox ?
  
  
• JACK is a way to connect devices and software processors/mixers/applications together.
  JACK
  
  
• JACK-compatible apps: Carla, Hydrogen ?
  
  
• PipeWire is a (new) audio/video server used by applications. It tries to unify handling of cases handled by JACK and PulseAudio.
  Ahmed S. Darwish's "PipeWire: The Linux audio/video bus"
  Architecture diagram (from Confluence)
  Venam's "PipeWire Under The Hood"
  PipeWire
  
  "Pipewire has compatibility plugins (pipewire-pulse, pipewire-jack, pipewire-alsa) which allow clients from all the major audio back-ends to interface with it. Providing you have the proper plugin installed, it should be transparent to the application."
  
  "systemctl --user status pipewire"
  "pw-cli info all | grep process.binary"
  "pw-cli info all | less"
  "pw-record --list-targets"
  
  Can set environment variables such as "PIPEWIRE_LATENCY=128/48000".
  
  

default-sample-format = float32ne  
default-sample-rate = 44100  
alternate-sample-rate = 48000  
default-sample-channels = 2    
default-fragments = 2  
default-fragment-size-msec = 125  
resample-method = soxr-vhq      
nice-level = -11    
realtime-priority = 9  
rlimit-rtprio = 9

Reporting Bugs

• OS bugs:
  
  Check release notes or "known issues" to see if the problem is known:
  Ubuntu 18.04.2 LTS (Bionic Beaver) Release Notes
  Release Notes for Linux Mint 19.1 Cinnamon
  
  First, search for existing reports about the bug. If you find one, add your data to it.
  
  If no existing report is found, you'll have to open a new bug report. Tricky: you have to figure out if the problem is something Mint-specific, or if it is inherited from Ubuntu or Debian.
  
  Linux Mint Troubleshooting Guide - Reporting Bugs - Where
  
  (For Linux Mint) Log in to GitHub. Report to one of these:

  • Cinnamon issues
  • Nemo issues
    (If you're having nemo problems, they want you to run nemo with debug symbols, "sudo apt install nemo-dbg" and then what ? But it turned out I was already running that, from the default installation of Mint.)
    (Also give list of nemo extensions: output from running "/usr/lib/*/nemo/nemo-extensions-list" on CLI. Extensions can be disabled using nemo menu item Edit/Plugins; then "nemo -q" and "nemo -n".)
  • Linux Mint Troubleshooting Guide
  • linuxmint

  
  Include these files and output from these commands (as appropriate):

  • $HOME/.xsession-errors
  • $HOME/.xsession-errors.old
  • "sudo dmesg -T"
  • "inxi -Fmpxz"
  • /tmp/mintreport/crash.tar.gz
    Which contains many things, including output from:
    • Trace from System Reports app.
    • "inxi"
    As far as I can tell, there is no private data in it, but the coredump may contain private data.

  
  From someone on reddit:
  If the OS is core-dumping, install any relevant "*-dbgsym" packages, then run "coredumpctl list", then "coredumpctl debug PID", then in GDB use command "bt" to generate a stack trace.
  
  Canonical Ubuntu's "Bugs"
  Ubuntu mailing lists
  Ubuntu developer mailing list
  Ubuntu Community Discourse
  xubuntu's "Reporting"
  
  See previous Ubuntu crash reports from this system:

  
  xdg-open https://errors.ubuntu.com/user/$(sudo cat /var/lib/whoopsie/whoopsie-id)
  

  
  Shirish's "Complete Guide To Bug Reporting In Debian Linux"
  Debian bug tracking system"
  
  KernelNewbies: FoundBug
  The Linux Kernel's "Reporting issues"
  
  
  
  
• Application bugs:
  
  Check to see if you're running latest version of the app.
  
  Run it again from CLI, to see if any error or debug messages appear there.
  
  Run it again without add-ons or plug-ins, to see if the problem goes away.
  
  Run it again without VPN or firewall or any other relevant modules, to see if the problem goes away.
  
  While app is running, do "top -H -p PROCESSID" to see what the app's threads are doing.
  
  Some apps have special logging or debugging options, usually on the CLI.
  Low-level GTK debug options (not very useful IMO): "man 7 gtk-options"
  KDE debug logging into system log: kdebugsettings (rule "*=false" to turn all off).
  
  In Ubuntu 20.04, if you know the package name, the best way to report is to do "ubuntu-bug PKGNAME" in terminal, which will collect some info from your system and then send you to Ubuntu One web site to log in and complete the bug report.
  
  Fedora Docs' "How to file a bug"
  
  Look in the application's Help menu or on developer's web site for how to report. Some apps have special bug-reporting menu items (KeePassXC has Help / Report a bug), or report crashes automatically (Thunderbird has Preferences / Privacy & Security / Crash Reporter).
  
  If it's a system application such as Nemo, look in the "OS bugs" section (above) to see where to report it.
  
  Some apps (Firefox) have a "feedback" (happy/sad) mechanism. Don't use it. You will have no way of tracking the report. Instead use the main bug-tracking system.
  
  Some apps have special crash-info features (Firefox has "about:crashes").
  
  
  
• CLI command bugs:
  
  Look in the command's man page for bug-reporting info. If none, maybe:
  FSF's "Coreutils - GNU core utilities"
  "All GNU packages" section of FSF's "GNU Software"
  
  
  
  
• If asked to provide a stack trace with debug symbols, not a core dump:
  
  GNOME Wiki's "Getting Stack Traces"
  Mint's "Software Sources" application has a switch to add PPA with debug symbol packages.
  

  
  apt install PACKAGENAME-dbgsym
  # or:
  apt install PACKAGENAME-dbg
  
  gdb NAMEOFAPP 2>&1 | tee $HOME/gdb-NAMEOFAPP.txt
  (gdb) run ARG1 ARG2 ARG3
  # get app to crash/fail
  (gdb) thread apply all bt
  (gdb) q
  

  
  
  

• If you're reporting a clear crash or failure: expect friendly reaction.
  
  
• If you're reporting something not useful or done wrong: expect acceptance and put on a list.
  
  
• If you're asking for a new preference or feature: maybe acceptance, maybe "why don't you implement it yourself", maybe "go away".
  
  
• If you're asking for a change to the way things work now: expect "go away" or something less polite.
  
  

Devices

Webcam

Are you using "HD Webcam" or "HD IR Webcam" ?
IR camera is for Windows-Hello-style facial recognition.

ArchWiki's "Webcam setup"

"The current state of video capture, usually webcams, handling on Linux is basically the v4l2 kernel API."

"sudo apt install v4l-utils qv4l2" and launch "qv4l2" (note: that's 4 L 2).
"sudo apt install guvcview" and do "guvcview --control_panel" ?
How many processes using webcam ? "lsmod | grep '^uvcvideo' | awk '{print $3}'"
See what's using webcam: "sudo fuser -v /dev/video*"

Christian F.K. Schaller's "PipeWire and fixing the Linux Video Capture stack"

Keyboard

X and Wayland do keyboard layout differently, but overlap ?
setxkbmap, localectl ?
"sudo localectl set-keymap MAPNAME"
"sudo localectl set-x11-keymap MAPNAME"

If you suspect a stuck/disabled key or bad touchpad or similar:
Do "xev -event keyboard" (works in both Wayland and X; little GUI window must be in foreground to get events).

"xbindkeys_show" ?

The Super key stopped working on my system. It turned out I had pressed Fn+F2, which has a "padlock" symbol on it. Pressing that again made Super key work again.

Online test: Key-Test

"man sxhkd"

Battery / Power Management

Maybe install TLP ? I don't.

See System Hardware Monitoring And Control section of my "Linux Controls" page.


Ctrl blog's "Fix Linux not suspending on low battery levels, and prolong your battery life"
thermald
auto-cpufreq
powertop
GNOME extension "CPU Power Manager"
On KDE, the system tray widget to show temperatures seems to be called "Pie Chart" ?

Display

See what resolution and refresh you are using: "xrandr".
Dead pixel test
JScreenFix - Pixel Repair (didn't seem to work for me)

If you have a high-resolution display, things (icons, text, etc) may be displayed too small. There are various ways to fix this: scale the whole desktop, or scale just the fonts ?

Mouse

Special software for advanced features of the mouse:
logiops
piper
Solaar

Touchpad

Turn off "tap to click": usually under something like Settings / Mouse / Touchpad.

For me on Mint: install Synaptics touchpad support via "sudo apt install xserver-xorg-input-synaptics" and then log out and back in. But later I removed it with "apt remove xserver-xorg-input-synaptics" and log out/in; I suspect it was causing UI freezes.

To see what driver you're using:
"grep -i 'Using input driver' /var/log/Xorg.0.log"
"sudo journalctl --since=today | grep 'Using input driver'"

There is another driver: "xserver-xorg-input-evdev". Maybe others: libinput-gestures, Fusuma, Touchegg ?

On X, stop middle-click from doing paste:

# May have to install package xorg-xinput

xinput --list
xinput --list DEVICEID
# Output shows 7 buttons, 2nd is middle button.
xinput set-button-map DEVICEID 1 0
# Test it, then add it as a startup application.

# If ID of device keeps changing, do:
xinput set-button-map $(xinput list --id-only 'DEVICENAME') 1 0

# If full name and ID of device keep changing, do:
xinput set-button-map $(xinput list --id-only "$(xinput list --name-only | grep Touchpad)") 1 0

If you want to run automatically at startup, put in a shell script file and run the script, instead of running the command directly. Needed for proper expansion of expressions, I think.
There was an AMD interrupt problem that made the touchpad not work at all; workaround is "sudo rmmod i2c_hid && sudo modprobe i2c_hid".

Kris Wouk's "How to Fix a Touchpad Not Working in Linux"

USB

lsusb
lsusb -D /dev/bus/usb/BUSNUMBER/DEVICENUMBER
udevadm info /dev/bus/usb/BUSNUMBER/DEVICENUMBER
udevadm info --query=path /dev/bus/usb/BUSNUMBER/DEVICENUMBER
ls -l /sys/`udevadm info --query=path /dev/bus/usb/BUSNUMBER/DEVICENUMBER`

usb-devices

dmesg | grep 'New USB device found'
dmesg | grep 'usb '
sudo usbview

jmtpfs --listDevices	# list MTP devices

USB ID Database
Device Hunt

Note: an Android smartphone's device number in "lsusb" changes when you click on a button in the "Use USB for ..." dialog on the phone screen.

Miscellaneous

Bootloader:

See current state:

To see bootable partitions other than the current one: "sudo os-prober"
To see kernels bootable from a partition: "sudo linux-boot-prober /dev/sdaN"

In Windows, to see/edit bootloader list, run "bcdedit" as administrator. article1, article2

In Linux, UEFI (if you can boot):
"efibootmgr -v"
"sudo bootctl status"
Fedora:
https://docs.fedoraproject.org/en-US/fedora/rawhide/system-administrators-guide/kernel-module-driver-configuration/Working_with_the_GRUB_2_Boot_Loader/
"cat /etc/default/grub" "grubby --info=ALL"
Fedora UEFI:
"cat /boot/efi/EFI/fedora/grub.cfg"
"grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg"

Xiao Guoan's "Use Linux efibootmgr Command to Manage UEFI Boot Menu"

Using GRUB menu:

If your GRUB-menu is hidden, hold Shift when booting, or run "sudo grub2-editenv - unset menu_auto_hide", or edit /etc/default/grub to comment out "GRUB_TIMEOUT_STYLE=hidden" line.

At EFI menu, press Esc or "c" for GRUB.
Get "grub> " prompt.

"help" to get help.
"help testspeed" to get help about that command.
Has ZFS commands.
But output of "help" zooms past and 80% of it can't be seen.
(To add paging: "set pager=1")
Type "exit" to get out and go to BIOS.
Type "normal" to get out and boot current kernel.
(Edit /etc/default/grub, add pager=1, run update-grub But doesn't work.)

GNU GRUB Manual 2.04
Dedoimedo's "GRUB 2 bootloader - Full tutorial"

Kowalski7cc's "Systemd-boot install on Fedora"

Repair:

Bootable repair images for GRUB:
Rescatux & Super Grub2 Disk
Boot-Repair-Disk
Ubuntu's "Boot-Repair"

For GRUB:
Kiran Kumar's "'Boot Repair' for Ubuntu, Linux Mint, and elementary OS can fix Bootloader issues"
Chris Hoffman's "How to Repair GRUB2 When Ubuntu Won't Boot"
Boot Repair
Chris Titus's "Grub Rescue | Repairing your Bootloader"
Jahid Onik's "How To Repair the GRUB Bootloader"
Vivek's "How to Edit & Repair GRUB Boot Menu Using Commands"
Also see: Help section in GParted app.

For systemd-boot:
Probably boot from live USB and repair from there ?
System76's "Repair the Bootloader"

You may have to disable Secure Boot to run some of these.

Add memtest:

[Legacy BIOS:] Should see memtest in GRUB menu.
[UEFI:]
Bootable USB image: Memtest86, or
Add to GRUB menu: Matias's "Installing memtest86 on UEFI Grub2 Ubuntu"
(my /boot/efi is nvme0n1p1, so I used "set root='hd0,gpt1'".
"sudo update-grub2" is for Debian and SUSE;
"sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg" for Fedora;
"sudo grub-mkconfig -o /boot/grub/grub.cfg" for Arch family ?)