Precautions








Problems and troubleshooting



In general:
What did you change before the problem happened ?
What can you remove to simplify the situation ?
What can you vary to test the situation ?

Not Unix's job to stop you from shooting yourself in the foot
Easy Linux tips project's "Solutions for 27 bugs in Linux Mint 19.1"
Easy Linux tips project's "System hacks for advanced Linux Mint users"
Neil Bothwick's "How to fix any Linux problem"
System 76's "Troubleshoot"
kmille's "How to solve hard (technical) problems"

Star Trek Picard: have you tried reversing the polarity of something ?




Magic key-sequences




Eric Simard's "Frozen Linux System? Here are 3 Ways to Deal With It"
superuser's "Does Linux have a Ctrl+Alt+Del equivalent?"
From dmich on StackOverflow:
"Ctrl+Alt+Fn is handled by X server. It can be configured with 'setxkbmap -option srvrkeys:'. Also this functionality can be disabled with DontVTSwitch option. Virtual consoles are switched with Alt+Fn combination, it is handled by virtual console driver."







Clean up space on disk



To see used/free space:

df -khT
df -khT -x squashfs -x tmpfs		# don't show snap devices

sudo du -h --summarize --exclude=/home --exclude=/sys --exclude=/proc --exclude=/run /
sudo du -h --summarize -c --exclude=/home --exclude=/sys --exclude=/proc --exclude=/run /*

sudo du -h --summarize -c /home/*
du -sh $HOME/* | sort -h
du -sh $HOME/.[0-9a-z]* | sort -h

sudo df -ih        # inodes



Easy Linux tips project's "How to clean Linux Mint 19 safely"






Performance



If your performance is bad, or you want to make it better:
ArchWiki's "Improving performance"



People complain about things that don't matter:
+/-
  • Boot is slow:

    How often do you boot ? Once a day ? A few extra seconds don't matter. And when I boot, much of the time is spent by me typing passwords (BIOS, disk decryption, login).

  • Application launch is slow (especially with Snaps):

    For your major applications (browser, IDE, email client, etc), you probably launch them once and use them all day. Launch time doesn't matter. For small, transient apps such as calculator or image-viewer, yes, it matters.

  • System taking too much RAM:

    If system performance is okay, don't worry about RAM usage. In some sense, you want your software to take full advantage of the RAM. Linux Ate My Ram ! (but see PR)

  • Distro is "bloated":

    If some apps are installed that you don't want, just uninstall them. Or just leave them unused; an extra GB or two of disk space used won't kill you.

  • DE is too "heavy":

    So turn off animation effects and 3D effects in the Settings. Turn off features you don't use: maybe multiple desktops, "activities", file indexing, file previews in the file explorer.






Run Performance Tests



Phoronix test suite to check performance.
+/-

# Download and extract the test package.
# Then:

sudo ./install-sh
sudo apt install php-cli php-xml pkg-config

phoronix-test-suite --help
# have to accept some license terms etc

# can run via cli or GUI
# GUI will give a CLI-type interface; type 'quit' to get out

# https://www.phoronix-test-suite.com/documentation/phoronix-test-suite.html

phoronix-test-suite list-available-suites
phoronix-test-suite list-available-tests | less

# I want: simple, few dependencies, run in a few minutes, test CPU RAM disk.

# Best to run tests after a fresh boot, no other apps running.
# Run each test 3 times and average them.

phoronix-test-suite benchmark pts/sysbench
# My 2011 Dell Inspiron N5010 with Kubuntu 20.10:
# (Intel Core i3 370M @ 2.39GHz, 8 GB RAM, 320 GB WD3200BEVT-7 with ext4/LUKS)
# gave CPU 2100, RAM 9.8 M
# My 2021 KDE Slimbook 15 with KDE Neon 5.20.5:
# (AMD Ryzen 7 4800H @ 2.9GHz, 16 GB RAM, 256 GB NVMe SSD with ext4)
# gave CPU 17600, RAM 13.6 M

phoronix-test-suite benchmark pts/leveldb
# My 2011 Dell Inspiron N5010 with Kubuntu 20.10:
# (Intel Core i3 370M @ 2.39GHz, 8 GB RAM, 320 GB WD3200BEVT-7 with ext4/LUKS)
# gave seq fill 12 MB/S, hot read 1.76 usec/op, random read 1.71 usec/op.
# My 2021 KDE Slimbook 15 with KDE Neon 5.20.5:
# (AMD Ryzen 7 4800H @ 2.9GHz, 16 GB RAM, 256 GB NVMe SSD with ext4)
# gave seq fill 23 MB/S, hot read 7 usec/op, random read 7 usec/op.

phoronix-test-suite benchmark pts/fio
# Choose sub-test 3 (sequential read), option 4 (Linux AIO), Buffered, Not Direct,
# 4 KB block size.
# My 2011 Dell Inspiron N5010 with Kubuntu 20.10:
# (Intel Core i3 370M @ 2.39GHz, 8 GB RAM, 320 GB WD3200BEVT-7 with ext4/LUKS)
# gave 48 MB/S.
# My 2021 KDE Slimbook 15 with KDE Neon 5.20.5:
# (AMD Ryzen 7 4800H @ 2.9GHz, 16 GB RAM, 256 GB NVMe SSD with ext4)
# gave 1132 MB/S.




"man lmbench"



Network performance testing

+/-
Speedcheck

My 2011 Dell Inspiron N5010 and 100 mbps chip with Kubuntu 20.10:
Without VPN on 600 mbps service: ping 75 down 95 up 100.
With Windscribe VPN (OpenVPN) on 600 mbps service: ping 80-90 down 55-75 up 45-80.
Without VPN on 100 mbps service: ping 70-80 down 95 up 100.
With Windscribe VPN (OpenVPN) on 100 mbps service: ping 85-125 down 85-90 up 75-80.
My 2021 KDE Slimbook 15 and 1000 Mbps chip with KDE Neon 5.20.5:
Without VPN on 600 mbps service: ping 45-55 down 365-445 up 105-130.
With Windscribe VPN (OpenVPN) on 600 mbps service: ping 65 down 15-20 up 50 [fast.com = 65/120].

Apparently "iperf3" is a better way to test (requires 2nd machine on LAN) ?
Also "speedtest-cli" (uses a public server) ?
gping



GPU performance testing

+/-
Apparently some games, and Steam, will display a "frames per second" as you play the game. You can use this to compare performance before and after changes.
Also, at CLI, run "glxgears" to see frame-rate.
Test suites: Novabench, Heaven UNIGINE.
Is direct rendering enabled ? "glxinfo | grep "direct rendering"



Disk performance testing

+/-
Sequential read: "sudo hdparm -t /dev/nvme0n1p2"
KDiskMark
bonnie++
Codrut's "How to benchmark the speed of your SSD" (fio)



Linux Shell Tips' "How to Create 100% CPU Load on Linux System"





Tweaking/fixing video



Definitely back up the config files before you start messing with stuff, it's easy to go wrong.

Check which GPU and which video driver you're using.
"lspci -nnk | grep -iA2 vga"
"hwinfo --gfxcard"
"lsmod | grep nouveau"
"sudo dmesg | grep gpu"
"sudo tlp-stat --graphics"

"nvtop" (for NVIDIA ? or for all ?)
"intel_gpu_top" (for Intel)
"radeontop" (for AMD Radeon)
# also https://gitlab.com/corectrl/corectrl for AMD
"nvidia-smi"

Maybe switch between X and Wayland to see which works better.
Maybe try a different compositor (picom ?) on X.

[NOT SURE] Video configurations, and drivers for them:
+/-
  • Intel CPU with integrated graphics: drivers built into kernel.
  • Intel CPU with Intel GPU (HD, Iris, UHD).
  • AMD CPU with integrated graphics: driver built into kernel.
  • AMD CPU with AMD GPU (Radeon): closed-source AMDPRO driver, or open-source AMDGPU / AMDVLK driver, or open-source Mesa RADV Vulkan driver built into Mesa stack.
  • Any CPU with NVIDIA GPU (GeForce, Quadro, NVS): closed-source NVIDIA driver, or open-source Nouveau driver.

Some people report better results (more stability, especially in games) if they disable the "dashboard" that comes with manufacturer's drivers (GeForce Experience and the NVIDIA control panel for NVIDIA; Radeon Software for AMD). Maybe this applies to Windows only ?

Some applications use CUDA, which is a parallel programming framework by NVIDIA.

Refresh rate can be limited by many factors, including connection to monitor (HDMI, DisplayPort).

Scaling: some distros have 3 or more scaling adjustments: "GTK scaling", "window scaling", "xrandr scaling". "xrandr" is the most basic type, should affect all apps etc, but might not scale text well.

To fix screen-tearing, maybe: use X11 compositor "picom --vsync", or configure X11 variable "TearFree".

Vivek Gite's "Linux Find Out Graphics Card Installed In My System"
Vivek Gite's "GPU Monitoring and Diagnostic Commands Line Tools"
Vivek Gite's "nvtop"
Mesa 3D Graphics Library documentation
Gentoo Wiki's "AMDGPU"
Christian F. K. Schaller's "PipeWire and fixing the Linux Video Capture stack"
GPU-Viewer

Ctrl.blog's "Finally fixed my PC's persistent graphics and audio stutters"

"my BIOS settings had set my graphics cards to a 'hybrid' format where sometimes my distro would use my integrated and other times the dedicated. To make my computer use only the dedicated card, I booted into BIOS, Advanced, scrolled down to SwitchableGraphics, and then selected dGPU."

To see if you're using hardware video acceleration:
"glxinfo | grep 'direct rendering'"
From someone on reddit:
To check on Chrome: about:gpu
To check on Firefox about:support
To check on VLC, launch it via CLI and it will tell you.

Chrome (and chromium-based browsers) don't come with video decode enabled by default, you need to manually enable it by adding --enable-features=VaapiVideoDecoder and --use-gl=desktop in front of the command that launches your browser, e.g editing its .desktop file.

Firefox, https://wiki.archlinux.org/title/firefox#Hardware_video_acceleration .

If you want video decode hardware accelerated on VLC and others AFAIK they're enabled by default, so no need to worry.

BTW YouTube uses a CODEC that is not accelerated by older versions of CPUs, so you'll need an extension such as "enhanced-h264ify" to force YouTube to use H264.



From someone on StackExchange:
"Graphics driver" can mean any number of things.

The way X (the graphical windowing system) works is that there is a central X server, which can load modules ("X drivers") for different hardware. Such as vesa, fbdev, nvidia, nouveau, amdgpu.

Some of these drivers can work on their own (vesa). Some need linux kernel drivers. Many of these kernel drivers follow the "direct rendering manager API", and therefore they are called "DRM drivers". Others, like the proprietary nvidia driver (which needs both an X driver and a kernel driver), don't.

It gets more complicated: The hardware consists of parts that read out the framebuffer and display it at different resolutions etc. This is called "modesetting". Modern graphics cards also have a GPU, which is used to accelerate 3D drawing (OpenGL). "DRM kernel drivers" provide an interface for both.

"Mesa" is a software library that understands OpenGL, but does the rendering either on the CPU, or on some (but not all) GPUs. So the Mesa library can offer this functionality for graphics cards that do not or do not sufficiently have hardware for this, or can serve as the OpenGL library for a few GPUs.

You could probably make a case to call anything in this complex picture a "graphics driver".



Tweaking/fixing mouse/trackpad/scrolling/video latency/tearing

+/-
Definitely back up the config files before you start messing with stuff, it's easy to go wrong.
Check what drivers you are using.
Check what resolution and refresh you are using: "xrandr".
Compare behavior in various apps; maybe change a "smooth scrolling" setting in an app.
For KDE, maybe try kwin-lowlatency package.
Maybe try a different DE, distro, or GPU hardware.


"The addition of input lag is always going to be the trade-off for most solutions that fix screen tearing."

"... tearing happens when the screen has to draw the next frame before its done drawing the current one. If you stay below [maximum rated Hz] your monitor will always have the time to draw frames to the screen. So, just capping the game's fps to, say, [maximum minus 4] fps, should do the trick."

Chris Titus's "Fix Screen Tearing Linux"



Chris Titus's "Buying Cheap Ebay Graphics Cards"





Tweaking/fixing audio



No sound in space because used Linux

There are various compatibility options. For example, running "apulse" on top of ALSA emulates the API of PulseAudio to the app above apulse. And PipeWire can be run along with PulseAudio, or without PulseAudio ?

If "pw-cli list-objects Device" and "pw-record --list-targets" show no audio devices, "That means PulseAudio is managing the audio devices and not PipeWire. There is nothing wrong with this in and of itself."

The Well-Tempered Computer's "Linux Audio Architecture"
Venam's "Making Sense of The Audio Stack On Unix"
Ahmed S. Darwish's "PipeWire: The Linux audio/video bus"
Mike Royal's "PipeWire Guide" (about many tools)


Definitely back up the config files before you start messing with stuff, it's easy to go wrong.

ArchWiki's "Advanced Linux Sound Architecture / Troubleshooting" ArchWiki's "PulseAudio/Troubleshooting" Dedoimedo's "Linux audio"
System 76's "Audio Troubleshooting"
EasyEffects (formerly PulseEffects)
Gamunu Balagalla's "Enable High Quality Audio on Linux"
Jahid Onik's "How To Fix Sound Issues On Ubuntu"
ojumah's "Enabling Instant Noise Cancellation on Linux"


inxi -A             # to see most of the stack
fuser -v /dev/snd/* # to see what process is running sound system

less /etc/pipewire/pipewire.conf
# example == /usr/share/pipewire/pipewire.conf

sudo apt install pulseaudio-equalizer

# If speaker device disappears:
pulseaudio -k

# If no sound, maybe:
sudo alsa force-reload
# or:
systemctl --user restart pipewire

ps -ax | grep -E 'pulseaudio|pipewire' | grep -v grep
pactl info
apt list | grep alsa | grep installed
less /etc/modprobe.d/alsa-base.conf

# How to software-prevent any use of microphone ?  Maybe:
pactl suspend-source @DEFAULT_SOURCE@ true

# If you decide to give up on PipeWire and switch back to PulseAudio:
sudo dnf swap --allowerasing pipewire-pulseaudio pulseaudio

From someone on reddit 4/2021, about non-PipeWire system:
The default values in /etc/pulse/daemon.conf use very little CPU but make audio sound like complete garbage on decent headphones.

Edit /etc/pulse/daemon.conf and audio will sound great:

default-sample-format = float32ne  
default-sample-rate = 44100  
alternate-sample-rate = 48000  
default-sample-channels = 2    
default-fragments = 2  
default-fragment-size-msec = 125  
resample-method = soxr-vhq      
nice-level = -11    
realtime-priority = 9  
rlimit-rtprio = 9





Reporting Bugs



On Mint, run System Reports application to see any crash reports.

Run "apt show PKGNAME" or "dnf info PKGNAME" etc to get info about a package, including URLs for bug-reporting and source code.

For some apps, it's hard to even figure out what you're running. For example, in Ubuntu 20.04, the default app that plays mp4 video files has no app-name or About menu item anywhere in the UI of the app. From the file-association in file explorer, I found out it's called "Videos". But that name doesn't appear anywhere on disk or in packages or in running processes, that I can find. From the application store, I was able to find out its home web site: https://wiki.gnome.org/Apps/Videos From there, I was able to find that another name for it is "Totem". Then "apt show totem" gives useful info.

For a given problem, check the version number of the software you are running, and what the latest released version number is. Is it possible for you to upgrade and re-test ?



Rocket2DMn's "Improving Ubuntu: A Beginners Guide to Filing Bug Reports"
Brendan Hesse's "How to Submit a Bug Report to Apple, Google, Facebook, Twitter, Microsoft, and More"

My impression of what usually happens:
+/-
  • If you're reporting a clear crash or failure: expect friendly reaction.

  • If you're reporting something not useful or done wrong: expect acceptance and put on a list.

  • If you're asking for a new preference or feature: maybe acceptance, maybe "why don't you implement it yourself", maybe "go away".

  • If you're asking for a change to the way things work now: expect "go away" or something less polite.


Some things that may occur during bug reporting:
+/-
  • Search shows that the bug already has been reported by dozens of people over last 15 years, lots of pleas to fix it, no action on it.

  • Report immediately closed as "probably some other component's bug, you go figure out somewhere else to file it".

  • Report languishes for 2 years, then you're asked for more info. But you no longer use that app or that distro, you forget how that feature works, etc.

  • Report languishes for 2 years, then closed as "can't replicate, maybe something we changed in last 2 years fixed it".

  • Every 3 months, a bot tries to close your bug report because "no activity in last 3 months, it's stale".




A lot of bug-tracking and mailing lists, plus packages and PPAs, are hosted on Launchpad.




Devices



Webcam

+/-
Are you using "HD Webcam" or "HD IR Webcam" ?
IR camera is for Windows-Hello-style facial recognition.
"ls /dev/video*"

ArchWiki's "Webcam setup"
Lan Tian's "Disabling Webcam"

"The current state of video capture, usually webcams, handling on Linux is basically the v4l2 kernel API."


sudo apt install v4l-utils qv4l2
qv4l2

guvcview --control_panel

# How many processes using webcam ?
lsmod | grep '^uvcvideo' | awk '{print $3}'

# See what's using webcam:
sudo fuser -v /dev/video*

# Software-prevent anything from using webcam:
sudo modprobe -r uvcvideo

# Make a fake webcam device:
sudo modprobe v4l2loopback
ls /dev/video*
ffmpeg -loop 1 -re -i MYIMAGE.jpg -f v4l2 -vcodec rawvideo -pix_fmt yuv420p /dev/video5
# Will show at speed of conversion, not speed in file:
ffmpeg -stream_loop 99 -i MYVIDEO.mp4 -f v4l2 -vcodec rawvideo -pix_fmt yuv420p /dev/video5

Christian F.K. Schaller's "PipeWire and fixing the Linux Video Capture stack"
"akvcam, Virtual camera driver for Linux"
Joey Sneddon's "Cameractrls is a Neat Webcam Settings App for Linux"



Keyboard

+/-
X and Wayland do keyboard layout differently, but overlap ?
setxkbmap, localectl ?
"sudo localectl set-keymap MAPNAME"
"sudo localectl set-x11-keymap MAPNAME"

If you suspect a stuck/disabled key or bad touchpad or similar:
Do "xev -event keyboard" (works in both Wayland and X; little GUI window must be in foreground to get events).

"xbindkeys_show" ?

The Super key stopped working on my system. It turned out I had pressed Fn+F2, which has a "padlock" symbol on it. Pressing that again made Super key work again.

Online test: Key-Test

"man sxhkd" (hot-key-mapping daemon)

How to type non-English keys on an English keyboard:
Fsymbols' "Linux keyboard shortcuts for text symbols"
€ == compose C =
¿ == compose ? ?
ñ == compose ˜ n



Battery / Power Management

+/-
"acpi -V"
"battop"
"upower -e"
"upower -i /org/freedesktop/UPower/devices/battery_BAT0"
"sudo tlp-stat --psup"

Aadesh's "Check your Laptop's Battery health on Linux"
Hugo Barrera's "Setting a battery charge threshold"

Maybe install TLP or power-profiles-daemon ? I don't.
See System Hardware Monitoring And Control section of my "Linux Controls" page.


Ctrl blog's "Fix Linux not suspending on low battery levels, and prolong your battery life"

thermald (Intel only).
auto-cpufreq
powertop
On Pop!_OS: pop-os-power ?
Slimbook Battery: a graphical frontend for TLP.
GNOME extension "CPU Power Manager"
On KDE, the system tray widget to show temperatures seems to be called "Pie Chart" ?
Overheating: Switch from NVIDIA graphics to integrated graphics ?

Baeldung's "Linux Power-Saving Tools"
Chris Titus Tech's "Laptop Power Management"
Arcadian Cloud's "Optimizing Ubuntu Battery Life"

AMD pstate (new in 2022):
"sudo cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver"
Normally gives "acpi-cpufreq" or "intel_cpufreq".
If you're running pstate, you'll see "amd-pstate".
"journalctl -b -g pstate"
Enable by putting "amd_pstate.shared_mem=1" on kernel command line.
There also is an energy-efficiency parameter: "amd_pstate.epp_enabled=1"
Since 6.1 you need kernel parameter "amd_pstate=passive" ?
Nik's "Power Management States: What is a S-state and a P-state?"
kernel.org's "amd-pstate CPU Performance Scaling Driver"


How to limit maximum battery charge ?
+/-
Few laptops are supported.
"sudo tlp-stat --battery"
On my laptop, I see that "charge_full" is about 84% of "charge_full_design", so maybe that is enforcing a limit already. But maybe "charge_full" is just showing "how high battery got on last full charge".
"ls /sys/class/power_supply/BAT*/charge_control_end_threshold"

Linux Uprising's "How To Limit Battery Charging For ASUS Laptops On Linux"
Ask Ubuntu thread
man tlp
But: JollyRoger's "Battery Management" (maybe Google-Pixel-specific)




Display

+/-
See what resolution and refresh you are using: "xrandr".
"xrandr --current"

"xdpyinfo | less"

Dead pixel test
JScreenFix - Pixel Repair (didn't seem to work for me)

If you have a high-resolution display, things (icons, text, etc) may be displayed too small. There are various ways to fix this: scale the whole desktop, or scale just the fonts ?



Mouse

+/-
Special software for advanced features of the mouse:
logiops
piper
Solaar



Touchpad

+/-
Turn off "tap to click": usually under something like Settings / Mouse / Touchpad.


For me on Mint: install Synaptics touchpad support via "sudo apt install xserver-xorg-input-synaptics" and then log out and back in. But later I removed it (going back to libinput AKA xserver-xorg-input-libinput) with "sudo apt remove xserver-xorg-input-synaptics" and log out/in; I suspect it was causing UI freezes.

To see what driver you're using:
"grep -i 'Using input driver' /var/log/Xorg.0.log"
"sudo journalctl --since=today | grep 'Using input driver'"

From ArchWiki's "Libinput":
If you have libinput and synaptics installed in parallel with default configuration (i.e. no files in /etc/X11/xorg.conf.d for either), synaptics will take precedence due to its higher numeric order 70- in the default installation directory. To avoid this, you can symlink the default libinput configuration (40-libinput.conf) to /etc/X11/xorg.conf.d/ where directory search order precedence over 70-synaptics.conf will take place instead:

"sudo ln -s /usr/share/X11/xorg.conf.d/40-libinput.conf /etc/X11/xorg.conf.d/40-libinput.conf"

"sudo apt install xinput"
"xinput list"
"xinput list-props 12"

There is another driver: "xserver-xorg-input-evdev". Maybe others: libinput-gestures, Fusuma, Touchegg, xf86-input-synaptics ?


On X, stop middle-click from doing paste:

# May have to install package xorg-xinput

xinput --list
xinput --list DEVICEID
# Output shows 7 buttons, 2nd is middle button.
xinput set-button-map DEVICEID 1 0
# Test it, then add it as a startup application.
# Could put it in ~/.xinitrc or ~/.xprofile instead ?

# If ID of device keeps changing, do:
xinput set-button-map $(xinput list --id-only 'DEVICENAME') 1 0

# If full name and ID of device keep changing, do:
xinput set-button-map $(xinput list --id-only "$(xinput list --name-only | grep Touchpad)") 1 0
If using Synaptics driver, maybe use "synclient" to control it.

If you want to run automatically at startup, put in a shell script file and run the script, instead of running the command directly. Needed for proper expansion of expressions, I think.


There was an AMD interrupt problem that made the touchpad not work at all; workaround is "sudo rmmod i2c_hid && sudo modprobe i2c_hid".

Kris Wouk's "How to Fix a Touchpad Not Working in Linux"



USB

+/-

lsusb
lsusb -D /dev/bus/usb/BUSNUMBER/DEVICENUMBER
udevadm info /dev/bus/usb/BUSNUMBER/DEVICENUMBER
udevadm info --query=path /dev/bus/usb/BUSNUMBER/DEVICENUMBER
ls -l /sys/`udevadm info --query=path /dev/bus/usb/BUSNUMBER/DEVICENUMBER`

# see what speed has been negotiated with each of your USB devices:
lsusb -tv

usb-devices

sudo tlp-stat --usb

dmesg | grep 'New USB device found'
dmesg | grep 'usb '
sudo usbview

# Set a USB port to send out power or charge from external supply:
ls /sys/class/typec
echo 'source' >/sys/class/typec/port0/power_role
echo 'sink' >/sys/class/typec/port0/power_role
# Not in my openSUSE Tumbleweed system; need sysfs-class-typec driver ?
# https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-class-typec
# https://www.kernel.org/doc/html/latest/driver-api/usb/typec.html
grep -i typec /boot/config-$(uname -r)

USB ID Database
Device Hunt

Microchip's "USB Port Power Control in Linux (Ubuntu)"
kernel.org's "Power Management for USB"

Note: an Android smartphone's device number in "lsusb" changes when you click on a button in the "Use USB for ..." dialog on the phone screen.

Seth Kenlon's "An introduction to Udev: The Linux subsystem for managing device events"
Vivek Gite's "How to protect Linux against rogue USB devices using USBGuard"
Daniel Drake's "Writing udev rules"
milosz's "How to disable touchpad when external mouse is connected"

See "Connect Android phone via USB cable to Linux" section


From Ten@mastodon.lol:
What's the difference between USA and USB ?

One connects to your computer and accesses all your data, and the other is an IT industry standard.




PCI

+/-

lspci
lspci -k
sudo lspci -v

sudo dmidecode | grep -i pci

sudo lshw
sudo hwinfo

sudo tlp-stat --pcie
I don't know how to find out the PCIe version supported by the system.

Xenproject Wiki's "Bus:Device.Function (BDF) Notation"
Mateus Rodrigues Costa's "Use sysfs to restart failed PCI devices"





Miscellaneous



Don't clean the inside of your computer, with compressed-air or whatever. There are many stories from people who did some kind of cleaning and then their machine won't boot or their GPU is dead or something.

Don't stress about a few extra packages or apps on your system. Don't bother to remove them, unless you're positive it's okay. Wasting a few extra MB of disk space doesn't hurt anything.

The Linux Cast's "5 Mistakes Every Linux User Makes" (video)



Firmware

+/-
"ls /lib/firmware"
"xz -l FILENAME.xz" to see some file size info.
"xz -k -d FILENAME.xz" to extract compressed file FILENAME (but you'll just get some binary file).

"zypper search firmware"

There is a firmware service that some vendors use: Linux Vendor Firmware Service.
Arch Wiki's "fwupd"
fwupd / fwupd
gnome-firmware
"man fwupdmgr" and "fwupdmgr --show-all-devices get-devices" and "fwupdmgr refresh" and "fwupdmgr get-updates"
"fwupdtool --help"
"sudo fwupdtool get-history"

Do "fwupdate --help" or "/usr/lib/fwupd/fwupdate --help". But it seems to be EFI-only, and mostly a test-tool. fwupdate.1 man page

Michael Larabel's "In A UEFI World, 'rm -rf /' Can Brick Your System"
Dan Goodin's LogoFAIL article

Terms: motherboard firmware can be BIOS or UEFI.


Firmware update that comes only in a Windows/DOS EXE format

+/-
"zypper info FirmwareUpdateKit" (creates a bootable mini-DOS system)

Firmware updates from manufacturers may come only as EXE files. To use one, maybe install FreeDOS on a flash drive and boot from that. Or your motherboard firmware may have a command for updating itself. Or see:
Dell's "Update the Dell BIOS in a Linux or Ubuntu environment"
nixCraft's "How to install/update Intel microcode firmware on Linux"
And from discussions on reddit:
"With Dell, you can even put the .exe file directly onto an USB stick (I think it has to be FAT32 though), and the flash it from the BIOS." and "Plug in flash drive, F12 to bring up a BIOS boot menu and there should be an option for a BIOS upgrade there. I know this works on Dell's business machines, not so sure about their consumer models." and "Try renaming the .EXE to .ZIP and unzipping it." and: use MediCat, a bootable Windows troubleshooting environment on USB or DVD.

If you have some emergency need to run Windows, one possibility is to boot from a USB stick with Windows To Go. But you can't make such an image from the Home edition of Windows. article

Also Linux Uprising's "How To Make A Bootable Windows 10 USB On Linux Using The New WoeUSB"
Jahid Onik's "How To Create a Windows Bootable USB on Linux"


Or put Ventoy on a (8 GB minimum) USB stick, then in a chrom* browser go to Download Windows 10 Disc Image (ISO File).

Very technical:
Ristovski's "Inside the InsydeH2O BIOS"


Motherboard Firmware

+/-
Terms: motherboard firmware can be BIOS or UEFI.

From Adam Williamson's "UEFI boot: how does that actually work, then?":
"If you really want to understand UEFI, it's a really good idea to go and read the UEFI specification. ... There is no BIOS specification. BIOS is a de facto standard - it works the way it worked on actual IBM PCs, in the 1980s."


sudo dmidecode -s bios-version
sudo dmidecode -s bios-release-date
sudo dmidecode -s bios-revision
sudo dmidecode -t bios
sudo find /sys/devices/virtual/dmi/id/ -type f -exec bash -c "echo -n '{}: '; cat {}" \;
sudo biosdecode
sudo vpddecode
EndeavourOS's "ACPI Kernel Parameters and how to choose them"
"Kernel Parameters" section

Aaron Klotz's "New Linux Build Allows BIOS Updates Without a Reboot" (PFRUT)

View contents of ACPI "BGRT" (Boot Graphics Resource Table):
"ls -l /sys/firmware/acpi/bgrt/*"
"image" should contain a BMP; copy it to a file elsewhere and view it.

UEFI open-source implementation: TianoCore

Coreboot:
+/-
coreboot source
coreboot
Wikipedia's "coreboot"
"man nvramtool"

From someone on reddit 5/2022:
Drawbacks are security and [lack of] official support. coreboot doesn't sign any of their firmware updates and, instead, uses a tool on your Linux installation to update firmware and its settings. Their security FAQ.

... a corporate-backed product will tend to last longer than community projects as they have a profit motive to continue. ... coreboot tends to be faster for almost everyone.

Vendors using it 12/2022: Purism, Starlabs, System76, Novacustom.

Libreboot:
+/-
"Libreboot is a coreboot distribution"
Libreboot
A vendor using it: Minifree

PureBoot:
Enabling virtualization features in firmware setup:
May have to switch from Easy Mode to Advanced Mode, maybe via Setup Mode item or F7 or some other key.
Look for "Virtualization" tab or setting.
Maybe look under CPU options.
Names for virtualization support: VT-x, AMD-V, SVM, Vanderpool, Intel VT-d, VTD, AMD IOMMU.
Check in Linux: "grep -q -E 'vmx|svm' /proc/cpuinfo && echo yes || echo no"

If many options are greyed out in firmware settings, try setting a "supervisor password" (don't forget it, there's no way to recover).

Michael Altfield's "Trusted Boot"


Idea for verifying firmware integrity:
+/-
Is the firmware (say, BIOS firmware) readable ? Can an OS or user process read it and compare to the last-installed version, and flag "hey, firmware has changed since the last time you booted !" ? Do any current OS's do that ? It could even be a user-level feature.

Shouldn't all devices (routers, security cams, disk drives, etc) come with a "read out the current firmware contents" feature ? Maybe a very clever malicious firmware could mimic a legit firmware, but it might not be easy if firmware memory is full (excess space padded with random static stuff when legit firmware is generated).

In Linux, do "sudo grep ROM /proc/iomem". If it returns "000f0000-000fffff : System ROM", you can read BIOS via "sudo dd if=/dev/mem of=pcbios.bin bs=64k skip=15 count=1 # 15*64k + 64k" or "sudo dd if=/dev/mem of=pcbios.bin bs=1k skip=960 count=64". Also relevant "sudo dmidecode". Maybe someone could make a little daemon or cron job that uses them to report any changes.

How about Linux's /dev/microcode ? Also would be nice to know if the router/gateway MAC address has changed ("arp" command).

Maybe enhance the "fwupdmgr" command to be able to read/verify existing firmware contents.
Does "fwupdmgr verify DEVICEID" do that ?
"fwupdmgr --show-all-devices get-devices", "fwupdmgr refresh", "fwupdmgr verify DEVICEID", "fwupdmgr get-updates", "fwupdmgr update".
There may be a timer running; see it in "sudo systemctl list-timers".
Maybe do "sudo systemctl disable fwupd-refresh.timer" and "sudo systemctl disable fwupd-refresh.service" ?


There is a way to store kernel log or crash dump in NVRAM: the pstore filesystem.
"man systemd-pstore"
"cat /usr/lib/tmpfiles.d/systemd-pstore.conf"
"man pstore.conf"



See Linux Boot Process page for info about bootloaders.



Sleep / Suspend

+/-
Power states:
    G0 / S0 == powered on == running.
    ?? / S0ix == connected standby / active idle === ???
    G1 / S1 == standby == CPU clock stopped.
    G1 / S2 == freeze == suspend-to-idle == CPU powered off, caches lost, RAM preserved.
    G1 / S3 == suspend-to-RAM == more chips powered off, RAM still preserved.
    G1 / S4 == suspend-to-disk == hibernation == must have enough swap, everything powered off.
    G1 / S3 + S4 == hybrid suspend.
    G2 / S5 == soft powered off.
    G3 / S5 == mechanical powered off.

Check kernel parameters:
"sudo sysctl -a | grep -E 'suspend|hibernation|resume'"

"cat /sys/power/state" (states supported by the kernel):
    standby == standby / S1.
    freeze == suspend-to-idle / S2Idle.
    disk == hibernation / STD / S4.
    mem == do as specified in /sys/power/mem_sleep

"cat /sys/power/mem_sleep" (string in square brackets is current value):
    shallow == standby / S1.
    s2idle == suspend-to-idle / S2Idle.
    deep == Suspend-to-RAM / S2RAM.

"sudo dmesg -HT | grep -i acpi"

As far as I can tell, the kernel has no support for state S3.

How to tell what sleep states your BIOS / ACPI supports ?
"sudo journalctl --since today | grep 'ACPI: (supports'"
"fwupdmgr security --force"
"sudo fwupdtool security --force" (security state of system)

Connection between key/lid events and resulting state:
"cat /etc/systemd/logind.conf"

Flags:
"cat /etc/systemd/sleep.conf"
"man systemd-sleep.conf"

Microsoft's "System Sleeping States"
The Linux Kernel's "System Sleep States"
Tookmund's "Linux Hibernation Documentation"
Aleksandar's "Difference between S1 (POS) and S3 (STR) standby mode in BIOS?"
Homo Ludditus article
Chris Siebenmann's "Stopping an Ubuntu 22.04 desktop from suspending at the login screen"
Nik's "Power Management States: What is a S-state and a P-state?"
Anand Lal Shimpi's "The New Sleep States: S0ix"
Wendy Wang's "How to achieve S0ix states in Linux"

Tell the system to go into various states:
"sudo systemctl suspend" # save to RAM
"sudo systemctl hibernate" # save to disk
"sudo systemctl hybrid-sleep" # save to both RAM and disk
"sudo systemctl suspend-then-hibernate"

See what happened as it changed states:
"sudo journalctl -b 0 | grep -E '\-sleep|suspend|hibernate|resume' | grep -v grep"

Summarized from someone on reddit 11/2021:
Fedora by default deliberately doesn't support hibernate (save to disk) because:

- A copy of RAM left on disk is a security leak.
- Secure Boot does not support Hibernation due to "Linux kernel lockdown policy", according to Fedora.
- Hibernation is unreliable/unstable on lots of hardware/BIOS's.
- You need to have a swap area, larger than RAM. Fedora by default has no swap area on disk.
- Swap to disk is slow for the system, while running normally.
- There are devices/drivers that can't cleanly power off completely, and then suddenly be asked to power back up into the "same state".
- Hibernation can fail if the battery suddenly is exhausted while writing to disk.
[- Hibernation can cause a lot of writes to SSD.]

One person with AMD hardware says:
I have seen that hibernation fails to resume about 20% of the time on openSUSE Tumbleweed. ... And whenever it did "successfully" resume, I often experienced issues due to things like sockets that had died/timed out while the computer was hibernated. Background processes sometimes hung because they had been suddenly frozen/turned off without an internet connection for too long in an unexpected state that the program's code was never written to handle, etc. ... It's not just a Linux problem. Not even Windows or my MacBook Pro does it reliably. ...

If the system won't sleep/suspend/hibernate, try disabling Wi-Fi. Especially likely with Broadcom.

If system freezes after waking up from sleep/suspend, maybe see SSD / IOMMU article.




Idea for a 'Linux Troubleshooter':
+/-
Linux needs a simple standard Troubleshooter application or dialog. This is intended to avoid having new users give up on Linux, improve the experiences of new and older users, reduce support requests, and avoid unnecessary bug reports. It should be a standard thing in all distros and DEs.

This app wouldn't fix anything, but gives a flowchart or check-list you can step through, pointing you to settings that may be wrong or things you could try.

Proabably should be a module under System Settings / Security, maybe called "Troubleshooting". That would keep it a standard part of the system, well-maintained, in a known place, maybe able to link to other settings applets, able to have links to documentation. Some of the items may be distro-specific or DE-specific, so best to keep this as part of the distro or DE.

Should have an URL linking to identical page on distro's or DE's web site, in case user needs to fix an unbootable system.

This is not intended to compete with major web sites such as ArchWiki or the distro's Wiki. It should be as simple and clear as possible.


For example:
+/-
Troubleshooting Check-list
  • Installation fails:
    • Using correct ISO (legacy BIOS vs. UEFI, Secure Boot vs. not, MBR vs. GPT, 64-bit vs. 32-bit) ?
    • Try turning off Secure Boot.
    • Create bootable USB image again, maybe copying failed.
    • Try using Ventoy.
    • Try using a different USB stick.
    • Try installing a different Linux distro.

  • [While your system is working, before any problems:]
    • Back up your important personal files, to multiple places.
    • Save output of something such as sosreport, or save output of fdisk lsblk fstab mount luksDump.
    • Save the bootable USB stick you used to install, don't overwrite it.
    • Make a bootable USB stick with a rescue OS/utility on it.

  • [For all situations:] Did you just change something ?
    • Change it back.

  • System won't boot, fails right after entering BIOS password:
    • Try turning off Secure Boot.
    • Try booting from USB stick. If it works, check HDD or SSD for errors, then check filesystems.
    • See more steps in "System boots to black screen" below.

  • System boots to black screen:
    • In black screen, try Ctrl+Alt+F2 and see if you can get a login prompt and log in. If you can, look in system journal for any error messages. "sudo journalctl --pager-end"
    • While booting, after typing BIOS password, press Esc or Shift to get into bootloader menu. Edit kernel command-line to remove "quiet" and "splash", then boot it. Look for error messages.
    • If this is a new install (never has booted successfully), try adding "nomodeset" to kernel command-line.
    • Boot to an older, saved kernel.
    • If you have bootable snapshots, boot to one of them.
    • Boot to rescue or emergency target.
    • Some distros have a recovery partition you can boot.
    • Try booting from USB stick. Mount filesystems and look in system journal for any error messages.

  • Disk encryption password wrong or forgotten:
    • Make sure keyboard is working and CapsLock is off.
    • VeraCrypt: are PIM and algorithm specified correctly ?
    • Try taking disk out, attaching to another system, and trying it there.
    • Try using CLI commands instead of GUI.
    • If only a few chars of password are in doubt, you may be able to use software to brute-force it.
    • If you don't have any idea what the password is, forget it, the data is lost.

  • User login password wrong or forgotten:
    • Make sure keyboard is working and CapsLock is off.
    • Is the correct user selected ?
    • Try Ctrl+Alt+F2 and see if you can log in there.
    • Boot from Linux on USB stick, mount root from system disk, edit /etc/shadow file to remove password.

  • User login succeeds but then kicked back out immediately:
    • Something broken in .profile or .bashrc ?
    • Changed account to use a shell that doesn't exist or is broken ?

  • Removed user from sudoers, and root login disabled:
    • In boot manager, edit default boot entry to add "init=/bin/sh", then boot it, should get root shell.
    • Boot from Linux on USB stick, mount root from system disk, you'll be root.
    • Once you're root, do "adduser USERNAME sudo", or edit /etc/sudoers file.

  • System runs and then crashes or freezes:
    • Look in system journal for any error messages. "sudo journalctl -b -1 -e"
    • Run RAM and disk diagnostics, preferably in BIOS.
    • Try re-seating RAM.
    • Run hdparm to check HDD.
    • Run SMART utility to check SSD.
    • Remove any optional peripherals and see if problem goes away.
    • AC power or grounding problems ?
    • Nearby equipment that could be causing the problem ?
    • Run no applications for a while, does problem go away ?

  • System is extremely slow:
    • Look in system journal for any error messages.
    • See output of top or htop or systemd-cgtop command.
    • RAM and swap are full.
    • Run hdparm to check HDD.
    • Run SMART utility to check SSD.

  • No internet access:
    • Look in system journal for any error messages.
    • If using Wi-Fi, try wired Ethernet.
    • If using Ethernet, try Wi-Fi.
    • Is there a Wi-Fi switch on outside of laptop ?
    • Turn off VPN.
    • Try "ping 1.1.1.1".
    • Can other devices on same LAN access internet ?

  • All web pages won't load:
    • Turn off VPN.
    • Try a different browser.
    • Try "ping 1.1.1.1".

  • One particular web site won't load:
    • Try a different browser.
    • Flush browser cache.
    • Try "dig DOMAIN" and "ping DOMAIN" and "traceroute DOMAIN".
    • Turn off VPN.
    • Disable blocker extensions (uBlock Origin, Privacy Badger, Disable WebRTC, Canvasblocker, etc) in browser for that site.

  • Audio doesn't work:
    • Look in system journal for any error messages.
    • Your system says it has no audio device.
    • You have the speaker volume set to zero.
    • You have the microphone turned off.
    • Remove or plug in headphones.
    • Try different headphones.

  • Video problems:
    • Look in system journal for any error messages.
    • You're using NVIDIA hardware and NNN driver. Try a different driver ?
    • You're using Wayland. Try switching to X ?

  • Bluetooth doesn't work:
    • Bluetooth turned off in settings ?
    • Bluetooth turned off in BIOS ?
    • Look in system journal for any error messages.
    • Try ...

  • VPN doesn't work:
    • Try a different VPN server.
    • Look in system journal for any error messages.
    • Try ...

  • Some application won't launch:
    • Look in system journal for any error messages.
    • Try launching it from CLI, see if any error messages appear.

  • Some application crashes:
    • Look in system journal for any error messages.
    • Try launching application from CLI, see if any error messages appear.
    • See if application has any command-line options that enable debug or trace mode.
    • Ask for information in some forum specific to that application.
    • Run application under strace or ltrace and send output to someone knowledgeable.

  • [For all situations:] Do you want to ask for help ?
    • If corporate, submit a ticket here: link.
    • Choose the most specific forum or group you can find.
    • Supply this information about your system: XXX hardware, YYY kernel and distro and DE and windowing system, ZZZ application.
    • Put a good title on your post.
    • Tell them exactly what you do to cause the problem, and what error you get.

For more information, see:
  • A copy of this check-list in the documentation site: link.
  • Distro documentation: link.
  • Distro-family documentation: link.
  • DE documentation: link.
  • Linux documentation: man pages, link.


Behind the scenes, I'd like to have this based on a shell script file, so a distro or DE maintainer or corporate IT person could alter it without doing C or C++ coding. The Settings module (C or C++) will call a shell script which does the work and then produces a Markdown file, which the module (C or C++) then displays. Is there any existing System Settings widget that is similar to this one ?



Practice troubleshooting:
SadServers



A common problem: "nut loose on the keyboard".