Layers / variations:
[Mostly from lowest to highest:] A bundle of choices of the last 4-5 items and some apps usually is called a "Desktop Environment" (DE): GNOME, GNOME Flashback, KDE Plasma, Xfce, LXDE, LXQt, Cinnamon, MATE, Enlightenment/E, Pantheon, Trinity, Budgie, Unity, Deepin/DDE, Razor-qt, Sugar, UKUI, PIXEL, Equinox/EDE, etc. article1, article2, article3

From someone on reddit:
"A DE gives you an overall user experience. It has the panels, the system menus, the starters, the status applets. It has a window manager. It might offer a default file explorer and viewer. To streamline, it might even contain default editor, terminal program, or even e-mailer, all made to look alike and work together."

Distributions (distros):
A distro is a set of choices of layers/parts/policies, all packaged together under one label.

Some major distros: Debian, Fedora, Red Hat, Ubuntu, Xubuntu, Linux Mint, openSUSE, etc.

Debian Family Tree (!)
Which is just part of a bigger GNU/Linux Distributions Timeline (!!)

Other pieces that usually vary by DE:

Other pieces that often vary by distro or distro-family:
Things that vary less often, or vary only from one distro-family to another:

Some other variations or modifications, some of them user choices:


Newbie questions:

More-complicated things you can do:

See my "Moving to Linux" page

Linux Myths

From 4/2000 interview of Rob Young, founder of Red Hat:
There are two big myths about the business, and the first is there is a single Linux operating system. Linux is a 16-megabyte kernel of the 600-megabyte operating systems that companies like Corel and Red Hat make. Linux might be the engine of your car, but if you plunk an engine in your driveway, you're not going to drive your kids to school.

Our job is making people understand this revolution is about open-source software and it is not about Linux at all. Linux is simply the poster boy for this movement. The other myth is that Linux is being written by 18-year-olds in their basements when actually most of it is being written by professional engineering teams.
Dawn Foster's "Who Contributes to the Linux Kernel?" (1/2017)

Another myth is that "free" software and "open-source" software are identical concepts.
Mark Drake's "The Difference Between Free and Open-Source Software"

Linux Truths


If you want to contribute to the Linux community:
You could pick an app or distro and:

Jason Evangelho's "8 Ways To Contribute To The Desktop Linux Community, Without Knowing A Single Line Of Code"
wikiHow's "How to Contribute to Open Source"
KillYourFM / contribute-foss
Open Source Guides' "How to Contribute to Open Source"
LTP - Linux Test Project
See "Reporting Bugs" section of my "Using Linux" page
Shubheksha's "How to find your first open source bug to fix"
Davide Coppola's "How to contribute to an open source project on GitHub"
Elizabeth K. Joseph's "4 ways I contribute to open source as a Linux systems administrator"
Quora discussion "How do I start contributing in Open Source projects?"
Sarah Drasner's "How to Contribute to an Open Source Project"
Command Line Heroes' S2E3 "Ready to Commit" (audio)

For development, start with an app, DE, or distro you use:

First, become a very good user:
  • Use the software, explore the features, read the docs and Help.

  • Update to the latest version, or the beta version.

  • File bug reports, and read the open issues/bugs.

  • Join the user forum or group or reddit sub.

kamranahmedse's "Getting Familiar with Unfamiliar Codebase"

Development things to realize:
  • Probably you will be stepping into a project that has a 20-year history, an enormous base of scripts, code, docs, and issue-tracking, and a big somewhat-changing team of developers.

  • It will take time to learn the code and tools and style.

  • Flitting from one project to another may be difficult, if they all use different tools and languages etc.

  • People-issues and process will be very important; follow the rules, be humble, start very small.

  • If a project seems stalled (2 years since last release, bugs open for a long time, pull/merge requests outstanding for a long time), or going through a huge transition, or very undocumented, or broken in some way, maybe stay away (at least as a first project to join).

Big projects often have extensive guides for developers:

GNOME Wiki's "Building system components"
KDE Community Wiki's "Get Involved"
Linux Mint Developer Guide
Debian Developers' Manuals
What can I do for Mozilla
What can I do for LibreOffice
What can I do for Fedora ?

From someone on reddit:
Anyone suitably capable can become an Ubuntu developer! ...

You start by providing fixes for existing Ubuntu developers to sponsor. Once you have a track record of good work and your existing sponsors are willing to endorse you, you can apply to become an Ubuntu developer yourself. ...
Ubuntu Packaging and Development Guide

Check the plans/roadmap for the unit you have chosen, to see the priorities, and what other people may be working on:
linuxmint / Roadmap

Find out the developer communication channels (mailing list, forum, bug-tracker, IRC, whatever) for the unit you have chosen, and join those channels.


Konrad Zapalowicz's "Three Ways for Beginners to Contribute to the Linux Kernel"
Adam Zerella's "How to become a Linux kernel developer"
Jason Wertz's "Kernel Basics" (video) (2013 but interesting)
torvalds / linux "HOWTO do Linux kernel development"
Linux Kernel Newbies
Arch: KernelBuild
Tamir Suliman's "Beginner's Guide to Writing your First Linux patch"
Sayli Karnik's "A checklist for submitting your first Linux kernel patch"
Kosta Zertsekel's "Who should I sent Linux Kernel patch to?"

Ashish Vara's "Kernel Architecture Of Linux (Part 7/15)"
Wikipedia's "Linux kernel - Architecture"
linux-kernel-labs' "Overview of the Linux kernel"

The Linux Kernel documentation
torvalds / linux (Linux kernel source tree)
Plailect / linux-devel (kernel development using VSCode and libvirtd)

Hcamael's "How to Develop Linux Driver from Scratch"
Corbet, Rubini, and Kroah-Hartman's "Linux Device Drivers, Third Edition"

From someone on reddit:

> I want to take a stab at creating a couple of GUI Linux apps to improve
> the usability of some parts of the system that are locked to CLI.

Honestly, coding probably isn't going to be the lynchpin issue you'll run into.

Most projects are ruled like fiefdoms, some are more reasonable/practical than others and as a result allow for more broader contributions, but you still run into issues where maintainers/developers will not accept or consider certain features because it either violates their vision or ethos for what they think their creation should be, or they don't want to support features they aren't interested in (desktop icons disappearing from Gnome3 is a prime example), whatever the case it's not a coding issue, the biggest problems with OSs are people issues.

Please don't create a new distro:
Please don't create a new distro. We have far too many already. GNU/Linux Distributions Timeline

If you want/need something, do it some other way. A configuration script that modifies an existing distro. A new DE or theme. A new kernel module. Whatever is appropriate for your need/idea.

Of course, if you have some great new world-beating idea that just can't be done in an existing distro, go for it ! But expect lots of work and little audience. Better if you could implement your idea in some major distro such as Ubuntu, and get it out to millions of users.

Linux is suffering from Fragmentation:
See "Fragmentation" section of my "Linux Problems" page.

Standards / projects / groups:

Plenty of huge companies (Intel, Microsoft, Google, Samsung, more) and smaller companies/foundations (Mozilla, Apache, more) contribute to Linux and FOSS.

Unix Family Tree

The CLI (terminal; Command-Line Interface):

Where there is a shell, there is a way.
From procabiak comment on Guide: Migrating to Linux in 2019:

Stop recommending the command line to beginners. Disclaimer/tl;dr: I like the command line, use it every day, but it's not the user-friendly tool you make it out to be. It's a tool for veterans, power users, tech-savvy folk. It's dangerous in the wrong hands.

This is a Linux/nix culture we need to move away from if Linux is to be successful mainstream. There's a reason many distros have a GUI for package management. If everything was indeed easier on a command line for every user demographic, why waste time implementing the GUI package manager, just tell everyone to use the terminal, right? Wrong! (IMO)

There're people who don't speak English or have difficulty with spelling even the simplest words, people with typing disabilities, older aged people, children ... Just to name a few, but all who just want to enjoy a game without needing to learn the complexities of Linux or us vets pushing that need onto them.

With a GUI package manager, you don't need to know anything beyond the ability to explore, click around to find stuff you need, etc. Often times you'll find other software you didn't think about wanting. The guide already states package managers are akin to app stores. These days everyone knows what an app store is (well, I make the assumption - there are people who have never owned smartphones before), so it's not as difficult a concept as it used to be, so we can use fewer words to explain it. The instructions are pretty simple: "open the package manager (app store) from the menu, search for steam and click install, enter your password and click OK". It's not faster than a command line, but it's simpler for many people. The instruction is nearly universal for all distros, even if the GUI looks or behaves slightly differently.

Contrast with apt, yum or pacman. You'd also need to learn to use grep to get anything remotely useful out of their list-package outputs. You can't explore - you just execute a command like sheep and hope you really just installed steam from Valve and not some other kind of steam. And when it comes to instructions, you need more words to explain the command line (that many people are omitting): "open the Terminal, type sudo apt-get install steam, type in your password (it shows up blank but your password is really there, keep typing), press enter, then type Y and press enter" ... Not only that, that command only works for Debian-based distros using apt. It will confuse Fedora and Manjaro novices using yum or pacman (they might even try to install apt, just to be able to run your command!).

You can also very easily mislead a novice to run dangerous "sudo rm -rf /"-like commands. I assume it's backwards to suggest to the user to understand the command before executing it when they are learning the command in the first place, so we can't assume the novice user will know what every command does. Therefore I can maliciously explain that line means "sudo = as a superuser, rm = remind me, -rf = to read from, / = root", for example. If I am their only reading material, I've just tricked a user to wipe their OS, imagine what else you can do to that novice by exploiting shortly-named commands?

The reality is command lines are simpler for the person offering help, but not necessarily for the person on the other end.

From grady_vuckovic comment on Guide: Migrating to Linux in 2019:

I have experience in converting people to Linux and that terminal is absolutely poison. When I show a typical Windows user an OS like Linux Mint, they are interested until they see the terminal, then groan and lose interest immediately.

Even if GUIs change, it is still waaaay easier for a new user to figure out a GUI with obviously labelled buttons, text boxes and tabs, than figure out how to use a terminal.

Try to put yourself in the mindset of a person who has only ever used a computer for homework, Facebook, job-hunting, Amazon, and who mainly uses a smartphone for all their computer needs. A terminal itself is a foreign concept and not a pleasant one.

On most decent software managers for Linux distros, the GUI is mostly pretty self-explanatory, with a simple search box for finding new software. That's pretty easy for a new user to figure out how to use.

But it is almost impossible for them to figure out what 'sudo apt-get steam' means at a glance, (and no, taking the time to explain it doesn't help, it will likely result in a 'wait wait I just want to install an app, why do I need to know all this rubbish?') or figure out where they need to go to type it, because if your user doesn't know where the software manager is, they certainly don't know where the terminal is or how to use it, so for all you know they could end up typing those commands into a search box somewhere. Not to mention it's difficult for them to even remember something like that.

And when I say "how to use it" regarding the terminal, I mean that literally. I've seen new users half type in commands, causing the terminal to enter a mode of waiting for the rest of the input of the command, then they can't figure out why it isn't working as they type in more and more commands, and other frustrations. It is truly a TERRIBLE user experience and one that should only be reserved for veteran Linux users.

Also keep in mind, users like to install and uninstall software, how on Earth is a user meant to figure out how to uninstall software which they installed with 'sudo apt-get'? There's no list of installed software they can look at, no buttons to click, no GUI to navigate, all they can do is hopelessly start Googling for help on how to use apt commands (that's an appallingly bad user experience for someone new to Linux) or (more likely) just get frustrated and decide "Linux is rubbish!" and switch back to whatever they were using before.

On top of that, it looks pathetically antiquated in 2019 to use a terminal to do such a simple basic OS operation as installing software or installing some drivers.

It may be easier for you to give instructions in terminal commands, and the terminal may be easier for you to use than a GUI, but it is scaring away possible Linux converts. For Ubuntu at least you should include instructions on how to navigate to the Software Manager.

I personally wouldn't offer any advice at all for Linux newbies that includes terminal commands unless you're trying to scare them away from Linux.

My thoughts:

"Terminal" is not same as "shell":
Terminals: gnome-terminal, konsole, guake, rxvt-unicode, aterm, xterm, kitty, alacritty, tilix, terminator, PuTTY, more.

Shells: sh, csh, bash, fish, zsh, more.

Expression of Change's "CLIs are reified UIs"

Package formats and managers:
Format Distros Binary Managers
rpm Red Hat, Fedora, CentOS, SUSE, Mageia rpm, yum, dnf, zypper, YaST, urpmi, Drakrpm
deb Debian, Ubuntu, Mint dpkg, apt, apt-*, aptitude, Synaptic, wajig
tgz/txz Slackware, VectorLinux, Zenwalk pkgtools, slackpkg, slapt-get, Swaret, netpkg
pkg.tar.gz Arch, Manjaro ? pacman, pamac ?
tbz2 Gentoo, Sabayon emerge/Portage, entropy, equo
nix NixOS nix
xbps Void xbps
Solus eopkg
Intel Clear Linux swupd
Some front-end managers that work on top of multiple package formats: smart, PackageKit.

Combination package manager / configuration manager ? GNU Guix

Devopedia's "Package Manager"
DistroWatch's "Package Management Cheatsheet"

Dedoimedo's "Broken apt, missing dependencies, what now?"

From /u/Linux4ever_Leo on reddit:
Honestly, there really isn't anything stopping the Linux world from standardizing on one single package management system except for egos. Each distro line deems its own package management systems to be the "best". Red Hat derived distros use RPM packages; Debian derived distros use DEB packages, Slackware uses good ol' tarballs. Gentoo and Arch prefer sources based packages. Still other distros have developed their own proprietary ways of managing packages. In short, it's sort of a mess. ... Personally, I don't care what sort of package manager wins the game so long as SOMETHING is standardized which will help further the widespread adoption of software on Linux and make things easier for developers to port their software to the Linux platform.

There are a couple of other levels of "management":

Disk management and encryption:
  1. Linux standard organization of files and directories in a system: /, /bin, /etc, /etc/passwd, /usr, and so on.
    Chris Hoffman's "The Linux Directory Structure, Explained"

  2. Linux OS API to a filesystem: inodes which represent files (some of which are directories), operations such as read/write/createfile/deletefile/makedir/rmdir.
    Wikipedia's "File system"

    Format on disk (of ext* filesystem):
    - Boot block ?
    - Superblock (maybe run "sudo dumpe2fs -h /dev/sda1 | more" to see info).
    - Inode (index node) table. (Each inode points to data blocks)
    - Dentries (translate between names and inode numbers, and maintain relationships between directories and files).
    - Data storage blocks (actual contents of files and directories).
    Diagram from Simone Demblon and Sebastian Spitzner's "Linux Internals - Filesystems"
    M. Tim Jones' "Anatomy of the Linux file system"'s "ext4 Data Structures and Algorithms - High Level Design - Special inodes"

  3. Plaintext or individually encrypted files: e.g. normal files and directories; app-encrypted files such as password manager databases or encrypted SQL databases.

  4. Filesystem mounted instances: mount-point name, and type, and device name.
    e.g. "/" is the mount location of an ext4 filesystem stored on /dev/sda5.
    Run "df -khT" to see the mounted filesystems.

  5. [I'm told layers 5 through 9 really can be mixed in any order, you can stack any block device on top of any other.]

  6. Upper (stacked) filesystem formats: Veracrypt containers; eCryptfs; EncFS; gocryptfs; Windows' Encrypting File System (EFS).

    Wikipedia's "ECryptfs"
    SK's "How To Encrypt Directories With eCryptfs In Linux"
    I'm told the eCryptfs code is considered old and unmaintained, so Ubuntu has dropped that option.
    Wikipedia's "Encrypting File System" (EFS)
    Wikipedia's "VeraCrypt"

  7. Base filesystem formats: format of data stored inside a partition. E.g. ext4, fat32, NTFS, Btrfs, ZFS.
    Jim Salter's "Understanding Linux filesystems: ext4 and beyond"

    ext4 can have a file/directory encryption module added to it: fscrypt (article1, article2).

  8. Manager: e.g. Linux's LVM (Logical Volume Manager), or some forms of RAID, or ZFS, or Btrfs.

    Presents a "virtual partition" that the layer above can use, but that single "virtual partition" could be stored across multiple physical partitions and devices.

    Ubuntu Wiki's "LVM"
    Wikipedia's "Logical volume management"
    Wikipedia's "Logical Volume Manager (Linux)"
    Sarath Pillai's "The Advanced Guide to LVM"
    terminalblues' "LVM Lab Setup With VirtualBox"

  9. Device-mapper and full-volume/block-level encryption: e.g. dm-crypt (a LUKS-compliant implementation); Veracrypt "full-disk" (really, full-partition) encryption; BitLocker.

    Wikipedia's "dm-crypt"
    Wikipedia's "Linux Unified Key Setup" (LUKS)
    ArchWiki's "dm-crypt / Encrypting an entire system"
    Wikipedia's "VeraCrypt"
    Wikipedia's "BitLocker"

  10. Physical partitions: e.g. /dev/sda5, /dev/sdb1. And a partition table (Master Boot Record (MBR) or GUID Partition Table (GPT)) to list the partitions.

    From someone on StackExchange:
    "Volume implies formatting and partition does not. A partition is just any continuous set of storage sectors listed in some table (e.g. MBR/GPT). A volume is a set of sectors belonging to the same filesystem, i.e. an implemented filesystem.".

  11. Disk hardware striping/mirroring (if any). E.g. some forms of RAID.
    Wikipedia's "RAID"
    Wikipedia's "RAID-Z" (with ZFS)

  12. Intermediate/bridge physical interfaces to media: e.g. USB, SCSI, RAID controller.
    Try doing "cat /proc/scsi/scsi" and "lsusb".

  13. Physical interfaces to media: e.g. IDE, SCSI, SATA.
    Wikipedia's "Hard disk drive interface"

  14. Disk controller, including hardware encryption (if any).

  15. Raw media: e.g. spinning hard disk, SSD, flash drive, CD-ROM, DVD-ROM, floppy disk.
    Appears as e.g. /dev/sda, /dev/sdb.
    Use GParted View/DeviceInformation to see info.

Example 1, my Linux Mint 19.2 system:
$ df -hT
Filesystem             Type      Size  Used Avail Use% Mounted on
/dev/sda5              ext4       33G   25G  7.0G  78% /
/dev/sda6              ext4      259G  182G   65G  74% /home
/dev/sda1              ext4      945M  175M  705M  20% /boot
/home/user1/.Private   ecryptfs  259G  182G   65G  74% /home/user1
  1. Standard Linux organization, except my personal files under /home.
  2. Standard Linux filesystem API.
  3. My password manager database file "KeePassDatabase.kdbx" is app-encrypted.
  4. It is under "/home/user1", which is the mount location of an eCryptfs filesystem stored on "/home/user1/.Private".
  5. "/home/user1/.Private" is using upper filesystem format eCryptfs.
  6. The base filesystem format of "/home" is ext4.
  7. Manager: none; not using LVM or RAID.
  8. Device-mapper and full-volume/block-level encryption: none ?
  9. Physical partitions: /home is on /dev/sda6.
    Partition table on /dev/sda is a Master Boot Record (MBR) table.
  10. Disk hardware striping/mirroring: none.
  11. Intermediate/bridge physical interfaces to media: SCSI.
  12. Physical interface to media: SATAII / Serial-ATA/300.
  13. Disk controller: whatever is on the circuit board attached to the disk (probably mainly some FPGA chip); no hardware encryption.
  14. Raw media: Western Digital model "ATA WDC WD3200BEVT-7" spinning hard disk, 298 GiB (320 GB), 5400 RPM 2.5" diameter, appears as /dev/sda.

Example 2, a Veracrypt container mounted in my Linux Mint 19.2 system:
$ df -ahT
Filesystem             Type             Size  Used Avail Use% Mounted on
/dev/sda6              ext4      259G  182G   65G  74% /home
/home/user1/.Private   ecryptfs  259G  182G   65G  74% /home/user1
/dev/mapper/veracrypt1 ext4             2.0G  1.1G  750M  60% /media/veracrypt1
  1. Standard Linux organization, except my personal files under /home.
  2. Standard Linux filesystem API.
  3. Plaintext file "MyBankInfo.txt" is in a 2.0GB Veracrypt container on /home/user1.
  4. It is under "/media/veracrypt1", which is the mount location of an ext4 filesystem stored on "/dev/mapper/veracrypt1".
  5. "/dev/mapper/veracrypt1" is using upper filesystem format ???.
    Both Veracrypt and ECryptfs are in here somewhere.
  6. The base filesystem format of "/dev/mapper/veracrypt1" is ext4 ?
  7. Manager: none; not using LVM or RAID.
  8. Device-mapper and full-volume/block-level encryption: none ?
  9. Physical partitions: /home is on /dev/sda6.
    Partition table on /dev/sda is a Master Boot Record (MBR) table.
  10. Disk hardware striping/mirroring: none.
  11. Intermediate/bridge physical interfaces to media: SCSI.
  12. Physical interface to media: SATAII / Serial-ATA/300.
  13. Disk controller: whatever is on the circuit board attached to the disk (probably mainly some FPGA chip); no hardware encryption.
  14. Raw media: Western Digital model "ATA WDC WD3200BEVT-7" spinning hard disk, 298 GiB (320 GB), 5400 RPM 2.5" diameter, appears as /dev/sda.

From someone on reddit:

My view on it is that there are no layers. There are just different combinations, abstractions, attachments, slices and mirrors of block devices. Upon which you can either build other block devices, or store raw data which could include filesystems.


The root of it is that the Linux block device is the base unit and since the other entities present block devices as their product, it gets confusing since the system is making block devices from other block devices and parts of block devices.


The first two items in #5 [Veracrypt containers; eCryptfs] are special types of filesystems, but the 3rd thing [Windows' Encrypting File System (EFS)] is referring to something that becomes a block device. Once it is a block device, then it can be used wherever a block device is used.

#6 is talking about filesystems and "partitions". But it's only a partition if it is referred to in a partition table (GPT, MBR, Sun, SGI, BSD). And even then, the OS only sees that data through the lens of a block device. See "man fdisk".

Trying to represent this as layers breaks pretty fast. For example with LVM, the LV is in a VG. And a VG encompasses one more more PVs. An LV can be spread across multiple PVs.

As I say, in the end actual data is on storage that shows up in Linux as a block device.

> [me trying to defend layers:]
> For example, can a Veracrypt container be below (contain) a LVM
> volume ? I don't think so, but maybe I'm wrong.

In Linux, Veracrypt can encrypt a file. That file can contain general data, a filesystem, or a partition table that divides up the file into partitions.

Also as a file, you can attach it to a loop device and then you can use that as an LVM PV (physical Volume) -- the first bullet here:

"sudo blkid" is the best way to see what type a filesystem is if you're using non-standard filesystems. In output of blkid, exFAT displays as 'SEC_TYPE="msdos" TYPE="vfat"'; NTFS displays as 'TYPE="ntfs" PTTYPE="dos"'. Most other commands show them as "fuseblk" or no-type.

"mount" may show an amazing number of mounted filesystems, including snaps, tmpfs's, cgroups, mappers.

nixCraft article
Beencrypted's "How To Encrypt Disk In Linux Securely"
ArchWiki's "Disk encryption"
"man cryptsetup"

ZFS is a somewhat-new-on-Linux [in 2020] system that integrates several layers (logical volume manager, RAID system, and filesystem) into a unit, and includes features such as checksums and snapshots and copy-on-write. Mostly oriented toward server/enterprise. And it requires letting ZFS manage the entire disk; you can't put ZFS in just one partition.

Paraphrased from 2.5 Admins episode 03 podcast:
Btrfs may be fine for non-RAID desktop systems, but there are serious issues with its RAID-type behavior; use ZFS instead for RAID-type situations.

Apparently Btrfs has one key feature that both gives it a bad reputation and makes it attractive to me: it checksums all files to verify the contents. So if your disk gets corrupted in a way that affects only data blocks of some files, other filesystem types will ignore those problems, but Btrfs will report them. In the past, some people have interpreted that as "Btrfs has/causes data corruption". [ZFS also has checksums.]

Magesh Maruthamuthu's "13 Methods to [Identify] the File System Type on Linux"

probono's "Make. It. Simple. Linux Desktop Usability" (series of 6 articles)

Boot process:
Ramesh Natarajan's "6 Stages of Linux Boot Process"
Narad Shrestha's "A Basic Guide to Linux Boot Process"
Sarath Pillai's "Linux Booting Process"
David Both's "An introduction to the Linux boot and startup processes"
Arch Wiki's "Boot loader"

Search my site