VMs and Containers (mostly in Linux)
Linux





Basics



Types:
Wikipedia's "List of Linux containers"
Docker's "What is a Container?"
EDUCBA's "Docker vs VMs"
Weaveworks' "A Practical Guide to Choosing between Docker Containers and VMs"
Mike Coleman's "So, when do you use a Container or VM?"


From someone on reddit:
Musl and glibc are two implementations of libc, as in the standard C library. This is basically the standardized interface between the kernel and userland. The kernel itself actually has no stable interface, the stability is guaranteed by libc which is a bunch of C functions wrapped around binary system calls that make it easy to write programs. So now you can do fork() instead of having to manually copy numbers into registers to communicate to the kernel what you want to do which is what the libc does for you amongst other things.

It also provides basic routine libraries like handling of strings and what-not.

...

The thing is that when applications are compiled, they are compiled against a specific libc for the most part and if you want to use them with another libc you have to recompile them. ...





Diagrams



Virtual machines:
App #1 App #2
VM #1: Linux [^ API is libc ^]
App #3 App #4
VM #2: Windows [^ API is win32 ^]
Hypervisor: VMWare [^ API looks like bare metal: CPU, RAM, devices, etc ^] App #5 App #6
Native OS: Linux [^ API is libc ^]
Bare metal [^ CPU, RAM, devices, etc ^]


Emulators:
App #1
Emulator #1: WINE [^ API is win32 ^]
App #2
Emulator #2: Anbox [^ API is Android ^]
App #3 App #4
Native OS: Linux [^ API is libc ^]
Bare metal [^ CPU, RAM, devices, etc ^]


Docker:
Docker container #1 Docker container #2
Docker Engine [^ API is libc ^] App #3 App #4
Native OS: Linux [^ API is libc ^]
Bare metal [^ CPU, RAM, devices, etc ^]

Docker container #1 Docker container #2
Docker Engine (includes a Linux VM) [^ API is libc ^] App #3 App #4
Native OS: macOS [^ API is libc ? ^]
Bare metal [^ CPU, RAM, devices, etc ^]

[Normal situation on Windows]
Docker container #1 Docker container #2
Docker Engine (includes a Linux VM) [^ API is libc ^] App #3 App #4
Native OS: Windows [^ API is win32 ^]
Bare metal [^ CPU, RAM, devices, etc ^]

[Unusual case: Docker Enterprise on Windows]
Docker container #1 Docker container #2
Docker Engine [^ API is win32 ^] App #3 App #4
Native OS: Windows with Hyper-V (Docker Enterprise) [^ API is win32 ^]
Bare metal [^ CPU, RAM, devices, etc ^]


Flatpaks:
Flatpak app #1
(on freedesktop)
(on bubblewrap)
Flatpak app #2
(on GNOME)
(on bubblewrap)
Flatpak app #3
(on KDE)
(on bubblewrap)
App #4
Native OS: Linux [^ API is libc ^]
Bare metal [^ CPU, RAM, devices, etc ^]


Snaps:
Snapd Snap app #1
(on AppArmor)
Snap app #2
(on AppArmor)
App #3 App #4
Native OS: Linux [^ API is libc ^]
Bare metal [^ CPU, RAM, devices, etc ^]


AppImages:
AppImage app #1 AppImage app #2 App #3 App #4
Native OS: Linux [^ API is libc ^]
Bare metal [^ CPU, RAM, devices, etc ^]


Erika Caoili's "Linux Basics: Static Libraries vs. Dynamic Libraries"
package "apt show libc6"; library "locate libc.so" at /usr/lib/x86_64-linux-gnu/libc.so and /snap/gimp/273/usr/lib/x86_64-linux-gnu/libc.so etc. There also is /snap/gimp/273/usr/lib/x86_64-linux-gnu/libsnappy.so





Virtual Machine



A virtual machine has a complete copy of an operating system in it; a container shares a single underlying OS with other containers, mediated by the container framework/engine. VMs are a much more mature technology and have CPU support, so are more secure in general. An emulator is a VM that has a veneer of a different operating system in it.

I think there are VM providers and then VM managers ? Each provider also includes a manager, but there are managers that can control many kinds of providers ? Not sure.



Providers:
By the way, virtualenv for Python is just a way of running a Python app with a certain set of libraries. Despite the name, it is not a virtual machine, and the app is not isolated from the OS.

WSL2 on Windows 10 is a VM: you run a Linux kernel inside it.
Joey Sneddon's "How to Install WSL 2 on Windows 10"


Crostini (AKA "embedded Linux (beta)") on Chromebook is a VM: you run a Linux kernel inside it.
From /u/rolfpal on reddit 9/2019:
Crostini is the same as the embedded Linux (beta). It runs an instance of Linux in a container, the container is "sandboxed", the beta comes with tools allowing you to run pretty much anything in the Debian distro. It does support GPU acceleration, but you have to set it up. Crostini is an official project of Google and is a work in progress.

​Crouton is an un-official script that allows you to run an instance of Linux in "chroot", meaning it uses the Linux kernel of Chrome as the foundation for the distro of your choice. Crouton is more of a hack, and is suspect from a security point of view, but sometimes you can do more with it, particularly if hardware hasn't been activated in Crostini yet.
From /u/LuciusAsinus on reddit 9/2019:
Crostini can be run by a supported Chromebook as-is. Crouton requires you to put your computer into "developer mode", which is less secure, and requires a dangerous prompt whenever you reboot (dangerous in the sense that it says, essentially, "Something has gone horribly wrong, hit space NOW to make it all better", but if you DO hit space you wipe your computer, including your Linux partition. I lost my Linux 3 times when my kids used my computer; very pleased that Crostini doesn't have that problem, even if it's a bit less powerful than Crouton).
Crostini: Don Watkins' "Run Linux apps on your Chromebook"



Managers:
ZeroSec's "Learning the Ropes 101 - Virtualisation"
Bryant Son's "6 open source virtualization technologies to know in 2020"
da667's "Resources for Building Virtual Machine Labs Live Training"
SK's "How To Check If A Linux System Is Physical Or Virtual Machine"
SK's "OSBoxes - Free Unix/Linux Virtual machines for VMWare and VirtualBox"





Emulator



A virtual machine has a complete copy of an operating system in it; a container shares a single underlying OS with other containers, mediated by the container framework/engine. VMs are a much more mature technology and have CPU support, so are more secure in general. An emulator is a VM that has a veneer of a different operating system in it.

I think WSL1 on Windows 10 is an emulator; it maps Linux syscalls to Windows syscalls.





Container System



A virtual machine has a complete copy of an operating system in it; a container shares a single underlying OS with other containers, mediated by the container framework/engine. VMs are a much more mature technology and have CPU support, so are more secure in general. An emulator is a VM that has a veneer of a different operating system in it.

Wikipedia's "Linux containers"
Alistair Ross's "What is Docker (and Linux containers)?"
Opensource.com's "What are Linux containers?"
Merlijn Sebrechts' "Why Linux desktop apps need containers"

OS building blocks:
Containers on Linux generally use namespaces, cgroups, and (on SELinux) seccomp to confine the app and strip services from its environment.

man namespaces
man cgroups
man seccomp

[I think I'm mixing container systems (Docker etc), app frameworks (Node/Electron/Ruby/Flask/Python etc), and app-packaging frameworks (Flatpak/Snap/Appimage etc) in here. Not sure. They overlap a bit.]

Some comparisons (focusing mainly on differences):

From /u/lutusp on reddit 4/2020:

Flatpaks, Snaps and Appimages solve the "dependency hell" issue by packaging all required dependencies with the application itself in a separate environment. This solves an increasing serious problem (inability to install and run some applications) with another one -- an application's download and storage size and startup time goes up.

By contrast, an application installed from the normal repositories must find all its dependencies (right version and properties) in the installed libraries, which unfortunately is a declining prospect in modern times.

Nitesh Kumar's "Comparison: Snap vs Flatpak vs AppImage"
OSTechNix's "Linux Package Managers Compared - AppImage vs Snap vs Flatpak"
AppImage / AppImageKit - "Similar projects"
Verummeum's "Snap, Flatpak and AppImage, package formats compared"
Merlijn Sebrechts' "A fundamental difference between Snap and Flatpak"

From someone on reddit:
Snap is hard-wired to Ubuntu and does not contain basic libs that exist in Ubuntu.
Flatpak is designed to be cross-distro, and packages everything.
AppImage contains as many libs as its developer decided to put in it.

From /u/galgalesh on reddit 8/2020:

One of the issues with Docker is that confinement is all-or-nothing. You cannot give Docker containers curated access to host capabilities such as audio, serial, hypervisors etc.

Flatpak has the same issue as Docker in that it's very hard to give applications mediated access to specific host features. The xdg desktop portals are an attempt to solve this, but they require applications to be rewritten to use the new api's. As a result, most Flatpaks run unconfined.

...

Snap heavily uses the AppArmor Linux Security Module for confinement. This is on top of cgroups and namespaces. This allows them to give apps fine-grained permissions to access different capabilities of the host system. This makes some cool things possible: Flatpak uses Bubblewrap for confinement, which is a much more traditional container where you have to turn off the confinement completely in order to use advanced features of the OS.

Both Snap and Flatpak use XDG Desktop Portals which is a new api for applications to securely access things like the filesystem and the camera. This, for example, allows Flatpaks to access the camera without turning off the confinement. The downside is that applications need to be rewritten in order to use the secure api. As a result, most Flatpaks have much of the security disabled.

Because Snap uses AppArmor, it can mediate the existing Linux API's for accessing the camera and other things, so applications can run in a secure container without any code modifications. The downside of using AppArmor is that some distributions use a different Linux Security Module and you can only run one at a time. On Fedora, you have to choose: if SELinux is enabled, snaps will not be confined. If SELinux is disabled, snaps will be confined. Canonical is working with many other devs in order to put "Linux Security Module Stacking" into the kernel which will make it possible to turn on Snap confinement together with SELinux. This won't be finished for a long time, though.

...

> I'm really torn about centralization, or the gatekeeper concept

I personally think centralization and gatekeeping are important. Flatpak tried the decentralized approach initially, but they are now pushing Flathub much more because a decentralized approach has a lot of issues. Ubuntu tried the decentralized approach too, btw, with PPA's. Snap was explicitly centralized because of the lessons learned from PPA's.

With snap, there is no gatekeeping for the applications. There is gatekeeping for the permissions, however. Snaps describe which permissions they want to use, but they do not describe which permissions they are allowed to use. The default permissions are part of a Snap declaration. This is additional metadata also hosted in the Snap Store. Users can override the default permissions themselves.

When you publish a snap in the snap Store, it only has the permissions which are deemed "safe". For example, snaps do not have access to the camera by default because that is very privacy-sensitive. If your application needs a webcam, then you can either try to convince the user to manually enable the webcam or you can go to the Snapcraft forum and ask for additional default permissions. The Snapcraft developers then act as a gatekeeper, they decide which additional permissions are allowed based on a number of documented criteria.

I think this is a really good model. The current issue, in my view, is that Canonical is the only ones who creates the criteria for additional permissions. I think this should be done by an independent committee instead, so that it can remain neutral. Right now, the Snapcraft developers are completely independent of the Ubuntu developers, so Ubuntu has no more power over the Snap Store than other distro's. This is not enough, however. We really need an independent committee.

For comparison,(AFAIK; I'm not an expert in Flatpak), the default Flatpak permissions are set by the Flatpak itself. So Flathub without gatekeepers would not be possible: it would allow anyone to have complete control over your system by publishing a Flatpak on Flathub.

...

> Snap and Flatpak are less secure than distribution-supported software

Indeed, the point of Snaps and Flatpaks is that the packages are created by the upstream developers instead of the distro maintainers. Traditionally, the distro maintainers would make sure that apps are "safe", and you lose most of this advantage by using Flatpaks and snaps. The advantage is that a lot more software is available.

But I think the comparison of "snaps" vs "repositories" is a bit misleading. Most users already install third-party software from PPA's, binaries from web sites, installation scripts etc. If you compare snap and Flatpak to PPA's, they are actually a lot more secure. Even if you completely trust the person who created the binary or the PPA, there is still the issue of stability. The worst a broken snap package can do is "not start". The worst a broken PPA can do is "make your system unbootable".

From Luis Lavaire's "The future of app distribution in Linux: a comparison between AppImage, Snappy and Flatpak":
AppImage is not a package manager or an application store ... AppImage is just a packaging format that lets anybody using a Linux distribution to run it ...





Container Managers (orchestration)



Merlijn Sebrechts' "What's up with CRI-O, Kata Containers and Podman?"





Miscellaneous



Most of these containers have a good/bad feature: they allow your system to have N versions of library L at the same time. That's bad if many of those versions have security vulnerabilities. Better hope that the container's sandbox works properly.





Search my site