Basics





Derived from Ubuntu Security Podcast episode 83 7/2020:
+/- LSM == Linux Security Module inside the kernel.

Stacking == using multiple security modules in same system.

As of 5.8 kernel, stacking is limited, but this is being changed.

Current (5.8) stacking rules:
Major modules (SELinux, AppArmor, Smack): can't stack with another major module, because they all try to attach their security data blobs to the same hooks inside the kernel.
Minor modules (TOMOYO, Yama, LoadPin): can stack.
Some modules might be allowed to stack but not make sense to stack on each other, they conflict or duplicate.

Wikipedia's "Linux Security Modules"



Network Control and Firewalls



This section is for tools that generally run unattended. For tools used by a person, see the Network Monitoring section.

Some terms:

Ubuntu's "DoINeedAFirewall"
Adrian Grigorof's "Open Source Security Controls"

You can change your MAC address to any value, either for Wi-Fi or for wired Ethernet, via the Network or Network Manager application.



Firewalls

+/-
"Netfilter is the framework in the Linux kernel, which implements the rule and filters provided by the user, through an interface available to user called iptables."

GUFW and UFW (simplified UIs to use instead of iptables)


Coming in KDE: plasma-firewall, a GUI on top of multiple types of back-end firewalls such as ufw and firewalld.

  • /etc/hosts file:
    +/-
    MintGuide's "Hosts - change and manage the /etc/hosts file"


  • iptables:
    +/-
    "modinfo ip_tables"

    +/- There are 5 "tables": filter, nat, mangle, raw, security. We only care about the filter table, which has these built-in "chains" of rules: INPUT, FORWARD, OUTPUT. You can create new chains if you wish. Each rule ends with an action which can be: ACCEPT, DROP, LOG, or name of another chain. (There is more, but that's all we need to know.)

    To see how much traffic is passing through each section of rules in filter table, do "sudo iptables -L -v" (can reset counters via "sudo iptables -Z"). On my system, with Windscribe VPN active, after doing a bunch of downloading, that gives: 
    
+/-
Chain INPUT (policy ACCEPT 2005K packets, 2860M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy DROP 306 packets, 21425 bytes)
 pkts bytes target     prot opt in     out     source               destination
  486 75925 ACCEPT     all  --  any    any     anywhere             192.168.0.0/16
 1104 72461 ACCEPT     all  --  any    any     anywhere             10.0.0.0/8
    0     0 ACCEPT     all  --  any    any     anywhere             172.16.0.0/12
 495K  860M ACCEPT     all  --  any    any     anywhere             localhost
 1339  139K ACCEPT     all  --  any    any     anywhere             104.20.122.38
 1369  143K ACCEPT     all  --  any    any     anywhere             104.20.123.38
 417K   55M ACCEPT     all  --  any    tun+    anywhere             anywhere
    0     0 ACCEPT     all  --  any    any     anywhere             localhost
 423K   78M ACCEPT     all  --  any    any     anywhere             89.238.nnn.nnn
    So my system is doing no routing ("FORWARDing"). And if no rule is matched, the packet gets DROPped if it is output, ACCEPTed if it is input.

    To block SSH connections from any address, do "sudo iptables -A INPUT -p tcp --dport ssh -j DROP". Can do the same with http and https if you're not running a web server.

    There is "ip6tables" which is separate but mostly has the same syntax as "iptables".

    There is a "conntrack" module which lets you do things such as "ctstate" in rules.

    To save changes so they survive across system restart, on Ubuntu-type systems, do "sudo apt install iptables-persistent", then turn off Windscribe VPN and firewall, then do "sudo su", and then "iptables-save >/etc/iptables/rules.v4" and "ip6tables-save >/etc/iptables/rules.v6".

    You can write commands such as "-P INPUT DROP" into a file such as "iptables.txt" and then run "sudo su" then "iptables-restore <iptable.txt"

    How-To Geek's "The Beginner's Guide to iptables, the Linux Firewall"
    Supriyo Biswas's "An In-Depth Guide to iptables, the Linux Firewall"
    Ravi Saive's "Basic Guide on IPTables (Linux Firewall) Tips / Commands"
    Mitchell Anicas's "Iptables Essentials: Common Firewall Rules and Commands"
    IP sets

    I ran a bash file with commands: 
    
iptables -P INPUT DROP
iptables -I INPUT -i lo -j ACCEPT
iptables -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -P FORWARD DROP
    Ran Windscribe VPN client which resulted ("iptables -L") in: 
    
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  anywhere             192.168.0.0/16
ACCEPT     all  --  anywhere             10.0.0.0/8
ACCEPT     all  --  anywhere             172.16.0.0/12
ACCEPT     all  --  anywhere             localhost
ACCEPT     all  --  anywhere             104.20.123.38
ACCEPT     all  --  anywhere             104.20.122.38
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             localhost
ACCEPT     all  --  anywhere             89.238.nnn.nnn
    That anywhere-anywhere rule actually has "out=tun+" attached to it (can see that via "iptables -L -v"), so all that traffic goes through the VPN.

    Put DROP on all IPv6 chains; my ISP doesn't support IPv6.

    Ran a couple of net-testing apps on my phone (which is on my LAN), targeting my PC, and they show all ports blocked, no response to ping, no services offered.

    If you want to log packets that don't match any rule, and end up getting DROPed by the chain policy, add a rule at the END of the chain via a command such as "iptables -A FORWARD -m limit --limit 1/minute -j LOG".

    Default logfile for iptables is /var/log/kern.log;
    "sudo dmesg -T" command shows same log, with some useful coloring added.

    I would like to log/detect all applications that create output connections. But it seems this is very difficult to do in a human-readable way:
    Super User's "With Linux iptables, is it possible to log the process/command name that initiates an outbound connection?"
    Akkana's "Find out what processes are making network connections"

    You can get instantaneous snapshots (not cumulative logs) by running "ss -tp" or "netstat -A inet -p".

    To see IP address and other details about each network interface, run "ip -d address".

    I want to log any incoming attempts on protocols/ports I don't use.

    Found this: "You can add the --syn flag to make the rule match only packets with the SYN flag set, which is set only on new connection attempts."

    Maybe do this:
    "iptables -I INPUT -p tcp --dport ssh --syn -m limit --limit 1/minute -j LOG --log-prefix "Incoming SSH attempt ""
    Do similar for FTP, Telnet, HTTP, HTTPS.

    I didn't bother to make rules for TeamViewer (port 5938), Remote Desktop (port 3389), SMB (port 139), NFS (port 2049), VNC (port 5900), http-alt (port 8080), RTelnet (port 107), TFTP (port 69), Simple FTP (port 115), rsh (port 514), rsync (port 873), Telnet-TLS (port 992), PPTP (port 1723), SSDP (port 1900), CIFS (port 3020), UPnP (port 5000), Socks Proxy (port 1080), Microsoft-DS (port 445) attempts. You can go a little nuts with this stuff. I think I don't have listeners active for any of it, but it would be nice to log and drop the packets.

    But it turns out you can make one rule for multiple ports, so I made:
    "iptables -I INPUT -p tcp --match multiport --dports 5938,3389,139,2049,5900,8080,107,69,115,514,873,992,1723,1900,3020,5000 --syn -m limit --limit 1/minute -j LOG --log-prefix "Incoming suspicious port attempt ""
    Limit of 15 port numbers per rule; had to split it into two.

    If one of your rules logs incoming HTTP attempts, test it by putting address "localhost:80" into browser's address field, then looking in logs. Or test by running "curl localhost" on the CLI.

    If one of your rules logs incoming HTTPS attempts, test it by putting address "https://localhost" into browser's address field, then looking in logs. Or test by running "curl localhost:443" on the CLI.

    If your rules log incoming SSH and/or Telnet and/or FTP attempts, test them by running "ssh localhost" and/or "telnet localhost" and/or "ftp localhost" in CLI, then looking in logs. Or doing "curl localhost:22" and/or "curl localhost:23" and/or "curl localhost:21" in CLI, then looking in logs.

    But I want to drop the packets after logging them. So I did this: 
    
iptables -N I_LOG_DROP
iptables -A I_LOG_DROP -m limit --limit 4/minute -j LOG --log-prefix "IPTABLES-I-LOG-DROP: " --log-level 6
iptables -A I_LOG_DROP -j DROP

iptables -P INPUT DROP
iptables -A INPUT -p tcp --match multiport --dports 20,21,22,23,80,5938,3389,139,2049,5900,8080,107 --syn -j I_LOG_DROP
iptables -A INPUT -p tcp --match multiport --dports 69,115,514,873,992,1723,1900,3020,5000 --syn -j I_LOG_DROP


    At some point, maybe it's easier to list the ports that should be open, instead of those that should be blocked. But maybe some high port numbers get opened dynamically.
    "iptables -I INPUT -p tcp --match multiport ! --dports 80,443 --syn -m limit --limit 1/minute -j LOG --log-prefix "Incoming suspicious port attempt ""

    Did this (in a shell script file): 
    
iptables -F
iptables -Z

iptables -N I_LOG_DROP
iptables -A I_LOG_DROP -m limit --limit 4/minute -j LOG --log-prefix "IPTABLES-I-LOG-DROP: " --log-level 6
iptables -A I_LOG_DROP -j DROP

iptables -N O_LOG_DROP
iptables -A O_LOG_DROP -m limit --limit 4/minute -j LOG --log-prefix "IPTABLES-O-LOG-DROP: " --log-level 6
iptables -A O_LOG_DROP -j DROP

iptables -P INPUT DROP
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p tcp -m tcp --syn -j I_LOG_DROP
# UDP 1194 for ProtonVPN, UDP 5353 is multicast DNS / Avahi
iptables -A INPUT -p udp --match multiport ! --dports 22,67,68,80,443,1194,5353 -j I_LOG_DROP

iptables -P FORWARD DROP

iptables -P OUTPUT ACCEPT
# port 23189 used by SFTP
iptables -A OUTPUT -p tcp --match multiport ! --dports ssh,http,https,23189 --syn -j O_LOG_DROP
# UDP 1194 for ProtonVPN, UDP 5037 when using Wireshark
iptables -A OUTPUT -p udp --match multiport ! --dports 22,53,67,68,80,123,443,1194,1900,5037,5353,30000:64000 -j O_LOG_DROP


    The whole thing gets more complicated because your VPN probably does iptables stuff too. Windscribe VPN adds rules and changes chain policies. I had to have a big shell script to add rules before the VPN starts, then a small script to do a couple of tweaks after VPN has started.

    And the number of ports keeps growing and growing. Apps such as Firefox and torrent-client open lots of high port numbers. Most/all of them may be on localhost, so maybe you have to start wiring addresses into your rules. The whole thing just gets too complicated.

    If you see an iptables LOG line in /var/log/kern.log that doesn't show source and dest ports, but gives "PROTO=n", look up that protocol number in /etc/protocols.

    Do "sudo netstat -tulpn" to see what ports have listeners. (Also "sudo ss -lptu")

    Around this time, I mostly gave up with iptables. I think it was the wrong approach.
    Instead, concentrate on reducing and understanding the number of listeners you have. It doesn't matter if an incoming packet gets through iptables, as long as no process is listening on that port.

    Let the VPN do what it wants with iptables. Maybe do "sudo iptables -L -v" occasionally to see how much traffic is hitting each rule.

    I think it would be different with a server, running only a few services. There you could allow only 10 or so open ports.

    Maybe there are some "listeners" built into the protocol stack ? Turn off protocols you know you aren't using. Maybe do this to see unusual protocols:
    
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
...
# assuming your INPUT chain policy is ACCEPT, put these at the end of the chain
iptables -A INPUT -p tcp -j ACCEPT
iptables -A INPUT -p udp -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -m limit --limit 1/minute -j LOG --log-prefix "Incoming IPv4 protocol "


  • ufw and gufw:
    +/-
    GUFW and UFW (simplified UIs to use instead of iptables)
    Virdo, Boucheron and Juell's "How To Set Up a Firewall with UFW on Debian 10"
    Vivek Gite's "How To Configure Firewall with UFW on Ubuntu 20.04 LTS"
    Jahid Onik's "How To Configure Firewall with UFW on Ubuntu"
    Daniel Aleksandersen's "How to switch firewalls from FirewallD to UFW"
    "less /etc/default/ufw"
    "less /etc/ufw/sysctl.conf"
    "sudo ufw status verbose"
    Files in /etc/gufw: "sudo less /etc/gufw/gufw.cfg" to find which profile is in use, then "sudo less /etc/gufw/Home.profile" to see that profile.


  • firewalld:
    +/-
    firewalld
    LinuxTeck's "15 basic useful firewall-cmd commands in Linux"
    Brian Boucheron's "How To Set Up a Firewall Using firewalld on CentOS 8"
    GUI for firewalld: firewall-config. Very detailed, a bit overwhelming.

    Avoid duelling firewalls. You can't run firewalld and ufw/gufw at the same time. firewalld and ufw both use iptables.
    "whereis -b -B /usr/bin /usr/sbin -f gufw ufw firewalld"


  • plasma-firewall:
    +/-
    Coming in KDE: plasma-firewall, a GUI on top of multiple types of back-end firewalls such as ufw and firewalld.


  • Nftables:
    +/-
    This is a replacement for iptables, ip6tables, arptables, and ebtables. It is a more procedural language, I think. Should avoid a lot of duplication that you can get in iptables rules, and scale better. Also combines IPv4 and IPv6 in one structure. Not supported by VPNs yet, I think. Still in development 11/2018, I think.

    Debian / wiki / nftables
    Nftables wiki
    Alistair Ross's "Hello nftables, Goodbye iptables"
    "apt show nftables"
    "nft list tables"


  • BPF (Berkeley Packet Filter; AKA cBPF) and eBPF:
    +/-
    Possible replacement for iptables and nftables.

    BPF has a fairly small VM inside the kernel, and byte-code can be placed in it to do network processing.

    eBPF adds attaching code in many more places inside the kernel, adds a Just-In-Time compiler to native code (for some CPU architectures), and adds global state (in arrays, key-value pairs).
    Also has ability to jump out to user-land code, bypassing much of the standard network stack, for example, in cases where reduced features and higher performance make sense. Or go the other way: where there used to be a (costly) jump out to user-land, instead do the operation inside the kernel, in the VM.

    Wikipedia's "Berkeley Packet Filter"
    Filip Nikolovski's "TIL: eBPF is awesome"


Mehedi Hasan's "Firewall Software For Protecting Your Linux System"
Eric Geier's "Little Known GUI Firewall Options for Linux"

IDSs and loggers and file-checkers

+/- List of Open Source IDS Tools
Daniel Berman's "6 Open Source SIEM Tools"

Honeypots and tar-pits

+/-
Vinsloev's "Cybersecurity 101 - What is a Honeypot?"
paralax / awesome-honeypots



Application Control and Security



Maybe decide what is important to you.

Isolate applications from each other and the OS:

+/-
Solutions from lightest to heaviest:
  • No isolation: userid gives every app access to all of your files, X gives apps access to each other's event-queues. Native packaging: deb, rpm, etc.

  • Sandbox created by someone other than app dev: AppArmor, Firejail.

  • Container (and permissions) created by app dev: Snap, Flatpak, Docker. All containers run on top of same, shared OS.

  • Micro-VM or unikernel for each app ? A severely stripped-down server image. If you're going to run a single process, no GUI, no SSH, single user, no command-line shell, then strip lots of stuff out of the Linux kernel-plus. Even run kernel and app in same address space.

  • VM for each app. OS is a "full" server distro such as Ubuntu server or a minimal distro such as Alpine.

  • Separate bare metal for each app. OS is a "full" server distro.

Not sure which is better: security set by app dev (e.g. in Snap), or by a third party (e.g. in AppArmor). App dev knows the app best, but OTOH a third party totally focused on security may do a better job.

Latest software:

+/-
Solutions from fastest to slowest:
  • Container / image created by app dev: Snap, Flatpak, Docker, appimage.

  • Distro maintainer does native packaging: deb, rpm, etc.

  • LTS distro / stable repo.

  • VMs created using LTS software.




Low-level kernel stuff and building blocks:

+/-

Lowest-level stuff



  • Namespaces:
    +/-
    Wikipedia's "Linux namespaces"
    Mahmud Ridwan's "Separation Anxiety: A Tutorial for Isolating Your System with Linux Namespaces"
    Ed King's "Linux Namespaces"

    Namespaces are a key feature used to implement isolation for container systems (Docker, flatpak, snap, LXC), applications and daemons (AppArmor, Firejail), and multi-process applications (some browsers). 

    
man namespaces
man lsns

lsns		# list namespaces in use by current user
sudo lsns	# list namespaces in use by all users

    A program uses the clone() system call instead of fork(), to make a copy of itself running in a separate space. That can mean separate process trees (clone can't see parent process or any pre-existing processes), different views of the network connections, separate mount spaces (/etc/fstab's, I think), pseudo-root privileges, separate IPC spaces, more. There can be sockets set up to communicate between namespaces.

    In the CLI, you can use the "unshare" command to run a process in a separate namespace. The "ps" command has options to display namespace IDs for processes.



  • cgroups:

    Not a security mechanism. Used to control resource contention among groups of processes. Defines 13 resource categories ("controllers"), and lets you specify that process group A can use amount N of resource category R ?

    So, for example, normally if 2 applications start 5 processes each, those 10 processes all contend for resources on an equal footing. But suppose you want the 5 processes for application A to get higher priority for CPU than the 5 processes for application B ? Define each set of 5 processes as a separate cgroup.

    David Both's "Managing resources with cgroups in systemd"


  • Access Control Lists (ACLs):

    A way to set finer-grained permissions than the base UGO permissions.
    Kuldeep Sharma's "Secure Files/Directories using ACLs (Access Control Lists) in Linux"
    
man acl
man setfacl

grep -i acl /boot/config*"	# see if enabled in kernel
mount | grep acl			# see if any filesystems mounted with ACL enabled

    Works on most Linux filesystems, but not vfat, exFAT, and some other non-Linux filesystems.

  • Capabilities:

    man page
    K3A's "Linux Capabilities in a nutshell"
    Good info in Chapter 39 "Capabilities" of "The Linux Programming Interface" by Michael Kerrisk.

    "The goal of capabilities is to divide the power of superuser into pieces".

    There is a set of defined capabilities: audit-control, audit-write, network admin, network broadcast, etc. 
    
grep 'define CAP_' /usr/src/linux-headers-$(uname -r)/include/uapi/linux/capability.h | grep -v \( | grep -v LAST_CAP
man capabilities
    A set of capabilities can be granted to a file, and then a process that execs that file gets those capabilities. Threads in a process can each have different capabilities. Capabilities can be inherited in various ways by daughter processes.

    Most applications probably aren't written to know or care about capabilities, and don't have any capabilities assigned to their executable file. 
    
getcap /bin/* /usr/bin/* /usr/bin/*
    And anything run by root user gets all capabilities.

Mid-level stuff




# all available kernel modules:
find /lib/modules/$(uname -r)/kernel -name '*.ko' -print | sort | less

lsmod | sort | less		# see loaded modules
modinfo MODNAME			# see info about module, incl params when loaded

cat /proc/cmdline		# see kernel launch command line

sudo sysctl -a | less	# see kernel parameters
# https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html

less /etc/sysctl.conf	# see system variables



Service and app-level stuff:

+/-
  • Firejail:
    +/-
    An application that restricts the environments of other apps, running them with access only to certain directories, or to fake copies of system directories, or no network access.

    Firetools/Firejail
    Easy Linux tips project's "Run your web browser (and other apps) in a secure sandbox"
    xenopeek's "Using firejail as security sandbox for your programs"
    "firejail --version" (shows more than just version number)

    You could use Firetools or Firejail Configuration Wizard apps in the GUI. But building a security profile doesn't seem to be persistent, and the Firetools monitor shows a much-too-high number for memory usage. And the UI is really annoying.

    To just run an app while denying network access: 
    
firejail --net=none APPNAME

    "sudo apt install firejail firejail-profiles firetools".
    Profiles stored in /etc/firejail.

    Poked around a bit, quickest way to see the restrictions is to run "mount" in CLI before and after running "firejail bash".

    When trying to debug a Firejail profile, launch the app in CLI ("firejail APPNAME"), not via an icon. Apps often put out error messages on stderr. Also, do "sudo ps -ax | grep firejail" to make sure an app isn't hanging; reboot if there is one.

    Firejail project home: netblue30 / firejail

    Firefox with Firejail:
    +/- [You could run Firetools, right-click, Configuration Wizard, select Firefox, click "Build a custom security profile", Continue. But the changes are not persistent; they're used for only one launch ?
    Instead:]

    1. Do "mkdir -v ~/.config/firejail".
    2. Do "cp -v /etc/firejail/firefox.profile ~/.config/firejail".
    3. Do "xed ~/.config/firejail/firefox.profile". [Following works on FF 63.0; other versions may be different.]
      Key lines I did/didn't change:
      • ~/Downloads is already whitelisted.
      • Add whitelist lines for ${HOME}/Videos or any other directories you want to give access to.
        No way to specify read-only ?
      • "caps.drop all": I didn't change it.
      • "netfilter": I didn't change it.
      • "noroot": I didn't change it.
      • Set the "protocol" line to "protocol unix,inet,inet6,netlink".
      • Set the "seccomp" line to "seccomp.drop @clock,@cpu-emulation,@debug,@module,@obsolete,@raw-io,@reboot, @resources,@swap,acct,bpf,fanotify_init,io_cancel,io_destroy, io_getevents,io_setup,io_submit,ioprio_set,kcmp,keyctl,mount, name_to_handle_at,nfsservctl,open_by_handle_at,personality, pivot_root,process_vm_readv,remap_file_pages,setdomainname, sethostname,umount,umount2,userfaultfd,vhangup,vmsplice" (all on one line, no spaces in the list)
        ["seccomp.drop mount,umount2,swapon,swapoff" also worked].
      • "shell none": I didn't change it.
      • "private-etc" commented out: I didn't change it.
    4. Right-click on the orange Firefox icon near the Start button, click Edit, and change Command from "firefox %u" to "firejail firefox %u".
      Some articles say add "-no-remote", but that prevents my password manager from sending keystrokes to Firefox.
    5. Launch Firefox, and test to see if it works normally.
      • Try Ctrl+O to open a page from disk, and only the allowed directories should be shown.
      • In CLI do "firejail --tree" or "firejail --list" and you should see Firefox there.
      • Test password manager sending info to Firefox.


    If you caused a bunch of FF crashes while debugging, look in "~/.mozilla/firefox/'Crash Reports'/pending" and delete them.

    Using Firefox under Firejail, I'm seeing some cases where FF can't access the internet, some where it doesn't shut down properly. It's not saving my uMatrix settings changes, too. Stopped running it under Firejail routinely.

    Mint 19 has Firejail 0.9.52-2 as of 14 Nov 2018. "sudo apt install firejail" says that is latest version. But the project page says there is a 0.9.56-LTS version. Download page says download DEB file and do "sudo dpkg -i firejail_X.Y_1_amd64.deb". Did that, and got some error "while trying to overwrite /etc/firejail/etr.profile". I don't see anything special about that file. Removed that file, tried again, same error. Firejail still works, still says version 0.9.52-2.

    So brought over just new firefox.profile, firefox-common.profile, and created empty firefox.local. Seems to work better, but still some problems, end up with zombie Firefox processes, etc. Stopped using Firejail on FF for now.

    KeePassXC with Firejail:
    +/- Mint comes with a Firejail profile for it, "/etc/firejail/keepassxc.profile". But it disables all the "click on URL to open in browser" and "auto-type" features of KeePassXC. You'd be reduced to copy-switch-paste to copy info from manager to browser.




  • AppArmor:
    +/-
    Very similar to Firejail, but implemented as a Linux kernel security module.

    Do "sudo apparmor_status" to see info.
    To install more: "sudo apt install apparmor-profiles apparmor-profiles-extra apparmor-utils apparmor-easyprof"
    "man -k apparmor"

    Wikipedia's "AppArmor"
    AppArmor

    Making profiles:
    Ubuntu Tutorials' "How to create an AppArmor Profile"
    Uzair Shamim's "The Comprehensive Guide To AppArmor: Part 1"
    The Debian Administrator's Handbook's "14.4. Introduction to AppArmor"

    In Mint, Apparmor (user-space parser utility) was installed by default.
    I installed Apparmor-utils through Software Manager.
    Profiles are stored in /etc/apparmor.d directory.
    You could turn on the profile for Firefox by doing "sudo aa-enforce /etc/apparmor.d/usr.bin.firefox".
    To turn it off, do "sudo aa-disable /etc/apparmor.d/usr.bin.firefox".
    If you do a disable when enforce is not on, you'll see a "Profile doesn't exist" error.
    To see the list of active profiles, do "cat /sys/kernel/security/apparmor/profiles".
    To see AppArmor activity, do "grep apparmor /var/log/kern.log".


    When trying to debug an AppArmor profile, launch the app in CLI, not via an icon. Apps often put out error messages on stderr. Also, do "sudo ps -ax | grep APPNAME" to make sure an app isn't hanging; reboot if it is.

    Firefox with AppArmor:
    +/- Already had FireFox working in Firejail, so all of the following is with FF running in Firejail.

    Started enforcing AppArmor for Firefox, launched Firefox in Firejail, it showed some "welcome to Mint - file access failed" page, and the kernel log shows a dozen or more AppArmor "denied" messages.

    Quit Firefox, disabled AppArmor for Firefox, launched Firefox, and my Firefox user profile is gone ! Bookmarks, settings, add-ons, everything gone !

    Fortunately, I was able to find it all under a directory in "~/.mozilla/firefox", and was able to restore it by editing "~/.mozilla/firefox/profiles.ini" to point at the old profile instead of the new one. Whew ! Do a backup of that stuff before trying again.

    Looking at "denied" messages in /var/log/kern.log, it looks like access to "/run/firejail/mnt/" is denied, maybe also access to "home/.ecryptfs/USERNAME/...". Edited "/etc/apparmor.d/usr.bin.firefox". Above "# so browsing directories works", I added lines "/run/firejail/mnt/** r," and "/home/.ecryptfs/** rw,".

    Still got "bookmarks and history system will not be functional because one of Firefox's files is in use by another application. Security software can cause this problem." error message.

    Tried AppArmor on, Firejail off, and FF works ! So the problem is somewhere in the interaction of AppArmor and Firejail. Did "sudo xed /etc/apparmor.d/firejail-default /etc/apparmor.d/usr.bin.firefox ~/.config/firejail/firefox.profile". Couldn't fix the problem. Tried adding lines "network unix stream, network netlink stream," and "/run/firejail/mnt/** r, /home/.ecryptfs/** rw," to "/etc/apparmor.d/usr.bin.firefox", didn't help.

    If you caused a bunch of FF crashes while debugging, look in "~/.mozilla/firefox/'Crash Reports'/pending" and delete them.

    KeePassXC with AppArmor:
    +/- Mint has no AppArmor profile for KeePassXC.

    Made a profile the stupid way, trial-and-error, not using the profiling tools. Started with no restrictions.
    "sudo aa-enforce /etc/apparmor.d/usr.bin.keepassxc"
    "sudo aa-disable /etc/apparmor.d/usr.bin.keepassxc"
    "sudo apparmor_parser -r /etc/apparmor.d/usr.bin.keepassxc"
    Got it to work, despite showing "qt5ct: D-Bus global menu: no" error (same error when AppArmor is disabled).
    Tried to turn off networking, got "Could not connect to any X display" error. Adding "network unix stream" fixed that. Very unfortunate that I had to do that; my main goal for this profile was to turn off networking.
    Took a lot of fiddling to get the directory-tree permissions okay; I'm sure they're still too generous.
    Ended up with this.




  • OpenSnitch:

    evilsocket / opensnitch

    But LinuxUprising's "How To Install OpenSnitch Application-Level Firewall In Ubuntu" says it's beta and you have to build from source (compiling in Go). The source project says it edits iptables rules and interacts with netfilter.
    7/2019: Developer says he's dropping it for a while, discouraged by no help.
    nixCraft's "OpenSnitch: The Little Snitch application like firewall tool for Linux"
    Step 3 in TokyoNeon's "Using Ubuntu as Your Primary OS, Part 4 (Auditing, Antivirus & Monitoring)"
    Do Son's "opensnitch: GNU/Linux port of the Little Snitch application firewall"

    gustavo-iniguez-goya / opensnitch (fork that IS active)
    Linux Uprising's "OpenSnitch 1.3.0" (original is active again)


  • Douane:

    Douane
    Douane on GitLab

    Tuxdiary's "Douane: easy firewall with app rules"
    Dedoimedo's "Linux per-application firewalls - Doable? Douane"
    Project-insanity's "Application firewall Douane for ArchLinux"


  • Trickle:

    ArchWiki's "Trickle"
    Reduce bandwidth available to a single process.




My evaluation

+/- The mainstream solutions (at least, in Mint) seem to be Firejail and AppArmor.

  • Firejail better because:
    • Can have different icons that launch an app with/without Firejail.
    • Can have private copies of /dev, /bin, /tmp, /etc.
    • Easy to say nodvd, noshell, noroot.
    • In Mint, there are far more Firejail profiles than AppArmor profiles.


  • AppArmor better because:
    • No way to evade controls when you launch an app.
    • Lots of fine-grained controls on D-Bus.
    • Can set detailed rwxmk permissions on directory trees and files; Firejail just has whitelist/blacklist.
    • Has some kind of controls on "signals".


  • Neither AppArmor nor Firejail supports restricting network access to just X system, or just localhost, or just LAN addresses.

  • Trying to use both AppArmor and Firejail on same app usually fails.

  • In Mint, by default, AppArmor is controlling several daemons, such as dhcp, cups, mysql-akonadi, Clam updater, ippusbxd (USB printer).

  • I think I would try to get my key applications running in Firejail. Leave AppArmor enforcing on the daemons.





Opinion: firewalls / app control / security is a mess:

+/-
Firejail, AppArmor, SELinux, Yama, iptables, ip6tables, nftables, bpf, ufw, gufw, firewalld, netfilter, VPN, IDS, IPS, etc. Why can't we have one integrated solution, with a body of rules that can do filesystem and network access and syscall access and service and app and login control, and take user ID, app ID, MAC address, IP address, IP port, etc as input to the rules ?

Examples:
  • I'd like to control Firefox now. I go to the Security GUI, select Firefox, and I see everything about it: what files/dirs it can access, what syscalls it can do, what devices, what IP ports, domain whitelists and blacklists, what users/groups can access it, everything. All in one place. I don't care if separate kernel modules do the networking and the filesystem control, and some module inside Firefox does the domain whitelist/blacklist.

  • Same for any other app or service or module. I go to the Security GUI and choose my password manager app. In one place, I can control everything about it: filesystem access, network access, syscalls, etc.

  • I don't want my system to do anything with IPv6. I go to the Security GUI and choose IPv6. In one place, I can control everything about it: turn it off for everything, turn off incoming, whatever. Later, maybe I decide only Firefox can do IPv6. I can check a box to allow that.

  • "Networking" should have zones: can/can't access internet, can/can't access router, can/can't access other devices on LAN, maybe also services listening on localhost. And internally X11 does networking, so that has to be allowed.

  • I don't want user "skype" (which owns Skype app and the cron job that updates it) to be able to contact any domains except Microsoft's Skype domain. I go to the Security GUI and choose user "skype" and set the appropriate things.

I think today Firejail and AppArmor already mix some of this. You can set filesystem, syscall and network permissions in a Firejail profile or an AppArmor profile, I think. But Firejail and AppArmor settings are hard to share across many apps, and iptables rules know nothing about applications/processes. Iptables has a module that allows rules based on connection state, but I don't think Firejail and AppArmor have anything like that.



System Hardware Monitoring and Control



Periodically check the health of your drives. Record the stats.


Software Resource Monitoring





Why Did Something Get Installed ?



With deb/apt/dpkg packages: 

ls -lt /var/lib/dpkg/info/*.list | less

zcat -f /var/log/dpkg.log* | grep -i PKGNAME | egrep "\ install\ |\ upgrade\ "
# grab part of datetime from that, and do:
zcat -f /var/log/dpkg.log* | grep -i DATETIME | egrep "\ install\ |\ upgrade\ "

sudo apt-get remove PKGNAME --simulate

apt-cache rdepends --installed --recurse PKGNAME

aptitude why PKGNAME --show-summary



Network Monitoring



This section is for tools used by a person. For tools that generally run unattended, see the Network Control section.

Some terms:

Traffic monitoring and analysis

+/- Mehedi Hasan's "Linux Monitoring Tools For SysAdmin"
Hayden James' "Linux Networking commands and scripts"
Martin Bruchanov's "Linux Network Administration"

Monitor the traffic in/out of your LAN. Best ways probably are custom software in your router, and a Pi-hole doing DNS filtering. From Security in Five Podcast - Episode 746, investigation of traffic volume exceeding data cap found that iCloud was uploading/downloading the entire collection any time one thing was added, and after that was fixed almost 50% of all traffic was due to blockable scripts (ads, trackers).



Endpoint monitoring

+/-
I want a monitoring app on my Linux system that tells me how much free storage/disk each device on the LAN (Windows laptops, Android phones and tablets, printer) has, last time it was updated, is it on the network right now. That's about it. I don't really care about network traffic monitoring.

It looks like all the managers work by using DNS to name and access devices. You'll have to set static IP assignments in the router, and put names/numbers in /etc/hosts on manager machine, then "sudo systemctl restart systemd-resolved", then "ping NAME".

Some people are telling me that there is a market separation between "network monitoring (SNMP)" products [LibreNMS, Icinga, more] and "mobile device management (MDM)" products [Quest Kace, JAMF, Spiceworks, Intune, more], and you will not find one product which can monitor both servers and smartphones, for example.

System Types

  • Custom agent on each client: install a custom "agent" app or service on each client device, get lots of detailed information and control.

  • SNMP agent on each client: install or enable an SNMP agent app or service on each client device, get varying amount of information and control.

  • Agentless: manager does network access (ping, HTTP, whatever) to each client to see if it is alive and seems to be functioning as intended. Little information, probably no remote control.


Agent Types

  • Passive: agent just collects data and responds to requests/orders from manager. So all clients have to have fixed known IP addresses ?

  • Trap: when some trigger happens (e.g. free disk space goes below 10%), agent sends an event to manager. So only manager has to have fixed known IP address ?

  • Both


Network Managers

+/-
  • EventSentry Light:

    Limited to 5 devices ? Maybe limited only for updating/control ?
    EventSentry
    Server runs on ???
    Agents available for ???


  • Zabbix:

    Zabbix
    Server runs on Linux.
    Agents available for Linux, Mac, Windows.
    There is an Android app "Unofficial Zabbix Agent" by dentier.
    "docker pull zabbix/zabbix-server-mysql" and "docker run -p 8080:8080 zabbix/zabbix-server-mysql"
    "docker pull zabbix/zabbix-web-nginx-mysql" and "docker run -p 8080:8080 zabbix/zabbix-web-nginx-mysql"


  • LibreNMS:

    LibreNMS
    Server runs on Linux and Windows/Docker.
    No agents, devices must have static IP address and SNMP.
    LibreNMS was forked from Observium before Observium became a commercial product.
    Installing LibreNMS

    There is a Docker image, "docker pull librenms/librenms" and "docker run -p 8080:8080 librenms/librenms", but it expects you to have a running MySQL database first.

    Tried native install: 
    
# See https://docs.librenms.org/Installation/Install-LibreNMS/
# I followed the "sudo -s" / Ubuntu / Nginx path.
# but starting mariadb and OpenIPMI services failed.

# Removed everything:
# Do NOT remove mysql-server and mariadb-server !  desktop went with them !
sudo rm -r /opt/librenms
sudo deluser librenms
sudo delgroup librenms
# Ended up in a state where neither MySQL nor mariadb will re-install.



  • Icinga 2:

    Icinga
    Server runs on Linux.
    No agents, devices must have static IP address and SNMP.
    Can use Nagios plug-ins.
    There is an Android app "Probe for OMD and Nagios" by Marco Ribero.
    "docker pull jordan/icinga2" and "docker run -p 8080:8080 jordan/icinga2"
    "sudo apt install icinga2"


  • Observium Community:

    Observium
    Server runs on Linux, more ?
    Agents available for xxx
    Tagged as "simple to use" in reviews.
    It seems that a client device MUST have an agent running on it (responding to SNMP queries) before you can define the device in the manager application; there's no way to define the device and then just get traps from it (at least in CE).

    Tried Docker image (FAILED): 
    
docker pull uberchuckie/observium

# See https://hub.docker.com/r/uberchuckie/observium

# paths have to be full paths !
#docker run -d -v /your-config-location:~/projects/observium/config -e TZ="Madrid" -v /path-to-logs:~/projects/observium/logs -v /path-to-rrds:~/projects/observium/rrd -p 8668:8668 uberchuckie/observium
docker run -d -v /your-config-location:/home/user1/projects/observium/config -e TZ="Madrid" -v /path-to-logs:/home/user1/projects/observium/logs -v /path-to-rrds:/home/user1/projects/observium/rrd -p 8668:8668 uberchuckie/observium
docker container list
# Browse to http://localhost:8668
# Log in as observium / observium

# Fail: it just ran and ran, never created subdirs,
# didn't respond to browser.
# Later, the image owner said there are a whole
# lot of undocumented reqts, such as run inside unRAID,
# assumes that there is a "nobody" user with user id 99 and
# group id 100.  I didn't retry.

# When done:
docker stop CONTAINERID

docker images
docker rmi IMAGEID
# if it says there are stopped containers:
docker rm CONTAINERID

    Tried native install: 
    
# See https://docs.observium.org/install_debian/

wget http://www.observium.org/observium_installscript.sh
chmod +x observium_installscript.sh
sudo ./observium_installscript.sh
# choose Community Edition
# Said "no" to installing snapd and agent.
# set MySQL password
# set Observium admin acct name and password
# Observium CE 20.9.10731
sudo systemctl restart apache2
# Browse to http://localhost:8668 or http://127.0.1.1:8668/
# FAIL
# Browse to http://localhost
# Log in; works okay.

# Decided to remove it.
sudo apt remove apache2 php7.4-mysql mysql-server mysql-client rrdtool
sudo rm -fr /opt/observium*
sudo rm /etc/cron.d/observium
sudo deluser observium
sudo delgroup observium



  • MeshCentral:

    MeshCentral
    /r/MeshCentral
    Seems very beta 11/2020.
    Server runs on Linux or Windows.
    Agents available for Linux, Windows


  • Xymon Monitor:

    The Xymon Monitor
    Server runs on Linux.
    Agents available for Linux, Windows.
    Need to have web server (e.g. Apache) installed, and C compiler and make utilities.
    Seems to expect a dedicated server, where you log in as user "xymon" and run a daemon.
    Latest update 2019-09.


  • Linux snmp CLI utilities:

    There is no constantly-running manager command ?
    http://net-snmp.sourceforge.net/wiki/index.php/TUT:MRTG
    https://www.comparitech.com/net-admin/snmpwalk-examples-windows-linux/

    There IS a trap-handler daemon, which will receive messages from clients and maybe send email to notify. First, set up the Linux agent software (see "agents" section). Then: 
    
sudo systemctl start snmptrapd
# Snmptrapd service says:
#		Warning: no access control information configured.
#		This receiver will *NOT* accept any incoming notifications.
man snmptrapd
man snmptrapd.conf

sudo edit /etc/snmp/snmptrapd.conf
# un-comment line "authCommunity log,execute,net public"
sudo systemctl restart snmptrapd
sudo systemctl status snmptrapd
sudo netstat -tulpn

sudo journalctl --unit='snmp*' --pager-end


  • Simple Android manager-type apps, mainly for testing:

    • "Fing - Network Tools" by Fing Limited.

      Under a machine's "Network Details" section, it will show SNMP machine name, contact, location. But in a port scan, it will not show machine's port 161 as open, I think because SNMP is using UDP.

    • "SNMP MIB Browser" by Zoho Corporation.

      When installed, just has one MIB "RFC1213-MIB", stored on phone as three files in "Internal shared storage / mibs" folder. There is a similar file "/var/lib/snmp/mibs/ietf/RFC1213-MIB" on my Linux system. Copied all MIB files from remote device (e.g. /var/lib/snmp/mibs/ from my Linux laptop) to phone, NOT replacing the RFC1213-MIB that was there already.

      When you add a host to the "Host List", the circle to the left of it will turn green if the SNMP connection to it is successfull.

      When polling for an ObjectId value, you can put in an ObjectId such as ".1.3.6.1.2.1.1.1.0", or a name if you've loaded the MIB where that name is defined. Some names and their MIBs are "sysDescr" (SNMPv2-MIB), "sysLocation.0" (SNMPv2-MIB). "hrSystemProcesses.0" (HOST-RESOURCES-MIB). If you put in a name and start polling and get no response at all, probably you have not loaded the MIB needed to translate that name.

      After you've successfully connected to the client system, you can go into "SNMP MIB Browser" and you will find that all the fields have been filled in with the values obtained from the client.



DNSstuff's "Best Linux Network Monitoring Tools for 2020"

All of the above systems seem to be huge overkill, and few of them support Android phones.

SNMP Agents

+/-
  • Linux snmp CLI utilities:

    To get SNMP agent working locally: 
    
# http://net-snmp.sourceforge.net/

sudo apt install snmp
man snmpget
man snmpcmd
man snmpstatus
man snmptrap
man -k snmp

sudo apt install snmp-mibs-downloader
# https://medium.com/@CameronSparr/downloading-installing-common-snmp-mibs-on-ubuntu-af5d02f85425
sudo sed -i 's/mibs :/# mibs :/g' /etc/snmp/snmp.conf	

sudo apt install snmptrapd	# can log incoming SNMP notifications to syslog etc
sudo systemctl status snmpd
sudo systemctl status snmptrapd
sudo systemctl enable snmptrapd
sudo systemctl start snmptrapd
sudo netstat -tulpn

# https://help.ubuntu.com/community/SNMPAgent
# man snmpd.conf
# sudo edit /etc/snmp/snmpd.conf to add:
rocommunity  public default
includeAllDisks 10%		# or e.g. "disk /home %10"
# Note: default frequency for disk-checking is every 10 mins ?
# But I don't see it logging anything, ever.
# Then:
sudo systemctl restart snmpd
sudo systemctl status snmpd

snmpwalk -c public -v1 localhost | less
# notice that lines start with various "DISMAN-EVENT-MIB",
# "SNMPv2-MIB", "IF-MIB", "SNMPv2-SMI", "IP-MIB",
# "HOST-RESOURCES-MIB", more.

snmpdf -v 2c -CH -c public localhost

snmpps -v 2c -c public localhost | less

watch --interval 5 snmpstatus -v 2c -c public localhost

# https://www.debianadmin.com/linux-snmp-oids-for-cpumemory-and-disk-statistics.html
# every 5 seconds show Total RAM free:
watch --interval 5 snmpget -v 2c -c public localhost .1.3.6.1.4.1.2021.4.11.0

snmpdelta -v 2c -Cp 5 -c public localhost .1.3.6.1.4.1.2021.10.1.3.1
# notice that lines start with "UCD-SNMP-MIB"
    To make SNMP agent accessible from network: 
    
# https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-an-snmp-daemon-and-client-on-ubuntu-18-04

# edit /etc/snmp/snmpd.conf to comment out
# "agentAddress  udp:127.0.0.1:161" line and add:
agentAddress udp:161
# or just comment out all the agentAddress lines.
# add lines:    NO !!!
createUser suser1 MD5 SOMETMPPASSWORD DES
rouser suser1 priv

# A different way to do it:    NO !!!
sudo apt install libsnmp-dev
net-snmp-create-v3-user --help

sudo systemctl restart snmpd
sudo systemctl status snmpd

# Run gufw and add rules to allow IN/OUT on ports 161 and 162
# from/to anywhere, with UDP, with IPv4.

# get your LAN IP address (call it THEIPADDRESS):
ip addr | grep 192.

snmpdf -v 2c -CH -c public localhost
snmpdf -v 2c -CH -c public THEIPADDRESS

snmpget -u suser1 -l authPriv -a MD5 -x DES -A SOMETMPPASSWORD -X SOMETMPPASSWORD THEIPADDRESS 1.3.6.1.2.1.1.1.0
# should get machine's "name", which is output of "uname -a"

# Go to manager app, such as Android apps in previous section,
# and try to query values from this system.

# Got stuck at this point, rebooted my Linux system,
# and things started working !  SNMP agent (snmpd) is working.

sudo journalctl --unit='snmp*' --pager-end


  • Android agent apps:

    I don't see a FOSS SNMP agent app for Android. Is SNMP built into Android by default, or just not available ?
    https://github.com/brnunes/Android-SNMP
    https://sourceforge.net/projects/libandroidsnmpsdk/
    https://gist.github.com/issamux/6709513
    http://www.net-snmp.org/
    https://www.javatpoint.com/simple-network-management-protocol
    https://agentpp.com/
    https://www.programcreek.com/java-api-examples/?api=org.snmp4j.Snmp
    https://apkpure.com/snmp-agent-4a/jp.snmp_agent4a
    https://joyent.github.io/node-snmpjs/agent.html

    Android "trap" app that sends to manager when some event happens:
    • "SNMP Trap" by Maildover LLC.
      http://www.maildover.com/
      http://www.maildover.com/joomla/



SNMP manager and client can be on separate parts (Wi-Fi and ethernet) of the same LAN.

Phone can NOT be using VPN-to-internet (at least, strongSwan the way I have it configured), when acting as a manager.

Linux system CAN be using VPN-to-internet (at least, OpenVPN the way I have it configured), when acting as a client.



Mobile Device Managers (MDMs)

+/-
  • Desktop Central:

    Desktop Central
    Server runs on Windows only, I think.
    Agents available for ???


  • Spiceworks "Inventory":

    Spiceworks
    Server runs on Windows.
    Agents available for ???
    Scans by IP address. No mention of mobile devices.


  • Esper:

    Esper
    Server runs on ???
    Agents available for Android only.
    More of a DevOps thing.


  • Miradore:

    Miradore
    Server runs on ??? Not Linux.
    Agents available for all but Linux.


  • Countly:

    Countly
    Server runs on Linux.
    Your app on mobile has to be compiled with their SDK, to talk to server.


  • ITarian:

    ITarian
    Server ("manager") runs on Mac, Windows, Linux.
    Agents available for Android and iOS mobile devices.


  • Relution:

    Relution
    Can use their cloud server (SaaS) up to 5 devices.
    Agents available for Android and iOS mobile devices.


  • AppTec EMM:

    AppTec EMM
    Server runs on ???
    Agents available for Android and iOS mobile and Windows Mobile devices.


  • ManageEngine:

    ManageEngine
    Server runs on Windows, but can use their cloud server (SaaS) up to 25 devices.
    Agents available for Apple, Android, Windows, Chrome.





At this point, I'm thinking of rolling my own solution: An agent on each client that just does an HTTP GET of the manager's web server every hour, passing in some information.
"MacroDroid - Device Automation" by ArloSoft Tools (easy to send IMEI and Battery%, not much else)

Syslog protocol

+/-
Teknikal's_Domain's "Graylog, and the Syslog Protocol, Explained"
Wikipedia's "Syslog"
RSyslog Documentation

Port 514 on UDP is simplest way. 

# See if rsyslog service is enabled and listening:
systemctl status rsyslog
sudo netstat -tulp

# If not, enable and check again:
less /etc/rsyslog.conf
# Edit /etc/rsyslog.conf to enable UDP-514.  Then:
sudo systemctl restart rsyslog
# Then check again.

echo "<29>mymachine mycmd[2]: hello" > /dev/udp/127.0.0.1/514
sudo grep hello /var/log/syslog

# Apparently, if the message starts with a date-stamp, that stamp
# will be used in syslog; otherwise rsyslog will add its own stamp.

# Change data to start with different leading value to end up in
# different log files.  Or omit leading "<nn>" entirely.

sudo apt install sendip

sudo sendip -p ipv4 -is 127.0.0.1 -p udp -us 514 -ud 514 -d 'Hello' -v 127.0.0.1
sudo grep Hello /var/log/syslog

# In router admin, set a fixed address for your computer.  Then:
sudo sendip -p ipv4 -is 192.168.1.131 -p udp -us 514 -ud 514 -d 'Hello Again' -v 192.168.1.131
sudo grep Hello /var/log/syslog

# Now go to Android phone and install app and send messages from there.
# This app works for testing purposes but mainly is logging
# its own internal failure msgs:
# https://f-droid.org/en/packages/sk.madzik.android.logcatudp/

sudo tail /var/log/syslog
# If firewall is running on your computer, disable it or add exception for port 514.
# Probably have to disable VPN on phone.

# System log will aggregate repeated identical messages into
# one "repeated N times" message, which is a real pain while testing.
# Edit /etc/rsyslog.conf to set "$RepeatedMsgReduction off".

# Various rules in /etc/rsyslog.d/50-default.conf

# Client for Windows:
# https://www.solarwinds.com/free-tools/event-log-forwarder-for-windows
# https://en.freedownloadmanager.org/Windows-PC/syslog-ng-Agent-for-Windows-FREE.html

"snap install graylog"
"Edit /var/snap/graylog/common/server.conf to set the admin password and mongodb connection string. Connection information for elastic search is also needed."




Security Testing and Penetration Testing





Tightening Security



Really, it seems that 95% of the vulnerabilities are eliminated if you just don't run a web server on your machine. Also don't run SSH or FTP or other login-type services, and keep software updated, and you're above 99%.

From older version of Easy Linux tips project's "Security in Linux Mint: an explanation and some tips":
"Don't install Windows emulators such as Wine, PlayOnLinux and CrossOver, or the Mono infrastructure, in your Linux, because they make your Linux partially vulnerable to Windows malware. Mono is present by default in Linux Mint; run 'sudo apt remove mono-runtime-common' to get rid of Mono."
[First run 'sudo apt --simulate remove mono-runtime-common' to see what else you'd lose.]

Ask Ubuntu's "What are PPAs and how do I use them?"
But: "One thing to keep in mind about using PPAs (Personal Package Archives) is that when you add a PPA to your Software Sources, you're giving Administrative access (root) to everyone that can upload to that PPA. Packages in PPAs have access to your entire system as they get installed (just like a regular package from the main Ubuntu Archive), so always decide if you trust a PPA before you add it to your system."

It's a good idea to get CLI mail working, and check it regularly, since various services and packages will send failure or security notices to root's email. See "Getting Linux local CLI mail working" section.


Easy Linux tips project's "Security in Linux Mint: an explanation and some tips"
The Empire's "An Ubuntu Hardening Guide"
lfit's "Linux workstation security checklist"
blakkheim's "Linux Security Hardening and Other Tweaks"
Madaidan's Insecurities' "Linux Hardening Guide"
Maybe likely to break things:
SK's "How To Password Protect GRUB Bootloader In Linux"
[But that doesn't protect against booting from USB drive.]

See Anti-Virus and Malware Scanners section of my "Using Linux" page.

See Application Control and Security section.



Tightening Privacy





Accounts




sudo bash

# To see accounts that can log in:
grep -v ':\*:' /etc/shadow | grep -v ':\!:' | grep -v ':\!\!:' | grep -v ':\!\*:'

# To see accounts that will be shown on the login page:
grep -v '/usr/sbin/nologin$' /etc/passwd | grep -v '/bin/false$' | grep -v '/bin/sync$' | grep -v '^root'

# To see accounts that can do "sudo":
grep sudo /etc/group



My understanding of accounts:

+/-
  • There are "local" accounts (defined in /etc/passwd), but in a corporate environment there could be "network" accounts that are defined in a server somewhere (e.g. LDAP).

  • Generally there is a "local" home directory for each account (named "/home/USERNAME"), but in a corporate environment the home directory could be on some server, and just mounted onto the local mountpoint.

  • Login can occur on the system desktop GUI (display and keyboard and mouse), through system virtual console (TTY1; Ctrl+Alt+F1 to open, Ctrl+Alt+F7 to close), through Telnet/SSH from a network, through remote-desktop software such as VNC or TeamViewer.

  • The account you specified at installation time is a user that belongs to the "sudoers" group and thus can use the "sudo" command to do super-user things. You can login as that user.

  • There is a "root" account, but it is locked so you can't log in as root. No password can be typed which will match the value in root's password hash field.

    BUT: from Easy Linux tips project's "Security in Linux Mint: an explanation and some tips":
    "In Linux Mint 18.3, the root password is unfortunately no longer set by default. This means that a malicious person with physical access to your computer, can simply boot it into Recovery mode. In the recovery menu he can then select to launch a root shell, without having to enter any password. After which your system is fully his. So, set a password for root (preferably identical to your own password)."

  • There may be a "guest" account, and you can log in as that user. [Disabled by default on Mint 19.] There is no password on that account. Can it login through SSH ?

    Definitely disable guest login, and you have to reboot after doing so. In Mint, run "System Settings", go to "Administration" section, click on "Login Window", click "Users" tab, probably everything should be turned off.

    Abhishek Prakash's "How To Disable Guest Account In Ubuntu"
    Ubuntu's "CustomizeGuestSession"

  • If you create more users, you can choose to add each new user to the "sudoers" group or not.

  • If you are logged in as a user who belongs to the "sudoers" group, you can do any administrative operation, and the password to use when "sudo" asks is the password for that user.
    Sudo make me a sandwich

  • I guess it is okay to routinely log in and do all your daily computing as the user you specified during installation, which belongs to the "sudoers" group. Any malware you run accidentally would have to know your password in order to do "sudo" and then do administrative stuff, and I think sudo accepts password only from keyboard. So it's not quite like routinely running as Administrator under Windows.

  • On a single-user system, the security distinction between root and normal user is not so important. All of the interesting personal files probably are owned by the normal user. So if an attacker can get in as that normal user, they get all the good stuff, no need to escalate to root.

    Escalating to root might let the attacker do a few more things, such as access network hardware at a low level to attack other machines on the LAN.

    Escalating to root on a multi-user system is much more serious/important than on a single-user system.

  • From someone on reddit:
    "Normally, all keyrings should get signed into automatically upon user login to the system. If suddenly you have to log in to a keyring when you never did before, what's gone wrong is that you changed the password with passwd instead of using the account manager built into the GUI. The GUI will automatically change your keyring password too. The command line won't."

Ubuntu's "RootSudo"

Some command-line ways to list all users: "getent passwd", "compgen -u", "cat /etc/passwd".

List users with no password set: "sudo awk -F: '($2 == "") {print}' /etc/shadow"

List users with UID set to 0 (superuser): "sudo awk -F: '($3 == "0") {print}' /etc/passwd"

List info about a user: "id user1"

Set limits on users or groups: /etc/security/limits.conf

Login security can be defeated if attacker has physical access:
+/-
Alarming article about (a hole in) account security:
Abhishek Prakash's "How to Reset Ubuntu Password in 2 Minutes" (boot into Recovery mode)
Maybe there is some way to password-protect GRUB, or maybe this doesn't work if /home is encrypted ?
SK's "How To Password Protect GRUB Bootloader In Linux"

Another way to change passwords if you have physical access: boot the machine from a Live system on USB or CD, do "sudo -i", do chroot to the main system disk, do "passwd $username".

Ask Ubuntu's "How do I reset a lost administrative password?" (boot into Recovery mode)
SK's "How To Reset Root User Password In Linux"

Not sure, but I think these methods work even if user's home is encrypted. Access to the disk encryption passphrase is controlled by the user permissions, so once you login as the user (with any or empty password), software can decrypt the user's home.

PAM (Pluggable Authentication Modules):
+/-
Applications (including the "login manager" and sudo) can be written to be PAM-aware. Then the application doesn't have to invent its own authentication mechanism.

The authentication process can be complex and multi-step. But as far as I can tell, all it does is give a yes/no for access to the application; there's no way to have it do fine-grained control of access to permissions within the application ? To do that, you'd have to define separate users with separate permissions ?

Configuration files are in /etc/pam.d directory. Some are for applications, others for events.
"apt list | grep libpam | less"
PAM can limit resource usage via settings in /etc/security/limits.conf
PAM can enforce password strength rules via libpam-cracklib and /etc/pam.d/system-auth
PAM can lock an account after N failed login attempts via pam_tally2 and /etc/pam.d/system-auth
Mokhtar Ebrahim's "Configure and Use Linux-PAM"
Good info in Chapter 23 "Understanding Advanced Linux Security" of "Linux Bible" by Christopher Negus.
Old but some useful stuff: Debian Reference's "Chapter 4. Authentication"

To enable TOTP on desktop logins:
If you're going to enable this, I would save a copy of "/etc/pam.d/lightdm", then create another user account, login to that account, and enable TOTP on that account, to make sure everything works.

Chris Hoffman's "How to Log In To Your Linux Desktop With Google Authenticator"
Daniel Pellarini's "How To Configure Multi-Factor Authentication on Ubuntu 18.04"
nixCraft's "Secure Your Linux Desktop and SSH Login Using Two Factor Google Authenticator"
Linux Uprising's "How To Login With A USB Flash Drive Instead Of A Password On Linux Using pam_usb"

"sudo apt install libpam-google-authenticator".
"man google-authenticator".


Types of keys and certificates:
+/-
  • RSA / SSH public-private key-pair.

    Usually generated by a client, which authenticates to a server some other way and then sends the public key to that server, for future use in authentication.
    File id_rsa.pub is the public key and file id_rsa is the private key.
    Can these files hold multiple key-pairs, or only one pair ?
    ~/.ssh directory.

  • Identity certificate.

    X.509, usually generated and signed by a trusted authority.
    Wikipedia's "X.509"
    Personal cert file types .pem .p12 .pfx.
    Authority cert file types .cer .cert .crt. Also .der ?

  • PGP public-private key-pair.

    ~/.gnupg directory.



From "man ssh":
"The idea is that each user creates a public/private key pair for authentication purposes. The server knows the public key, and only the user knows the private key. ssh implements public key authentication protocol automatically ..." and "A variation on public key authentication is available in the form of certificate authentication: instead of a set of public/private keys, signed certificates are used. This has the advantage that a single trusted certification authority can be used in place of many public/private keys."
Also relevant "man ssh-keygen".

Steve Cope's "SSL and SSL Certificates Explained For Beginners"

Keyring / GnomeKeyring / ksecretservice:
+/-
setevoy's "What is: Linux keyring, gnome-keyring, Secret Service, and D-Bus" (also Arseny Zinchenko (setevoy)'s "What is: Linux keyring, gnome-keyring, Secret Service, and D-Bus")
GNOME Keyring
Keyrings(7) man page
Arch Wiki's "GNOME/Keyring"
Nurdletech's "Gnome Keyring"

There is a Linux kernel keyring (see "man 7 keyrings"), and a GNOME Keyring (GNOME Keyring).

Is integrated with ssh, sftp, scp, PAM, Chrome, chromium. Can be integrated with Git, GnuPG, Firefox.
swick / mozilla-gnome-keyring (extension for Firefox and Thunderbird)
From Gnome Keyring - Security FAQ:
"Gnome Keyring is integrated with PAM, so that the 'login' keyring can be unlocked when the user logs in.".
LZone's "Using Linux keyring secrets from your scripts"

On CLI, do "cat /proc/keys" to see some of the keys in the Linux kernel keyring.
On CLI, do "man keyctl".

GNOME keyring stored under ~/.local/share/keyrings

Apparently Thunderbird has its own internal keyring.

"Passwords and Keys" application (AKA "Seahorse"):
+/-
Accesses GNOME Keyring.
AKA Seahorse

Under Passwords - Logins, it seems to have a bunch of placeholder entries for web sites, and a couple of things for apps (Chrome, Skype). There's nothing (for me) under Certificates (I do have certs installed in FF, Chrome, Thunderbird, but they don't show up here), and under Secure Shell (OpenSSH = ~/.ssh). But there are several keys under PGP Keys (maybe stored under ~/.gnupg directory ?). Hover mouse over each item to see tooltips.

SSH logins:
+/-
Ubuntu's "SSH / OpenSSH / Installing Configuring Testing"
Chris Hoffman's "How to Secure SSH with Google Authenticator's Two-Factor Authentication"
Linuxaria's "Add security to your ssh daemon with PAM module"
nixCraft's "Top 20 OpenSSH Server Best Security Practices"

From Ravi Saive's "How to Setup Two-Factor Authentication (Google Authenticator) for SSH Logins":
"Important: The two-factor authentication works with password based SSH login. If you are using any private/public key SSH session, it will ignore two-factor authentication and log you in directly."

SK's "How To Configure SSH Key-based Authentication In Linux"
Alistair Ross's "How To Set Up SSH Keys"
Carla Schroder's "5 SSH Hardening Tips"

Testing your SSH from outside:
InfoByIp's "SSH server connectivity test"
Rebex SSH Check
But really you need to try to connect from an outside machine and see what happens.

Jesus Vigo's "How to join a Linux computer to an Active Directory domain"

Trusted certificate stores:
+/-
Security certificates can be stored in a number of places ?
  • In some browsers. /usr/share/ca-certificates/mozilla/
  • In other mini-browser-containing apps such as Thunderbird email client ? ~/.thunderbird/PROFILENAME/cert9.db and key4.db ?
  • Electron apps contain the Chromium browser engine; what certificate store is used, or none ?
  • Node.js comes with a built-in store of CA's ?
  • Java has its own store ? Also each Java app could have its own ? /etc/ssl/certs/java/cacerts, jre/lib/security/cacerts ? "man keytool" article
  • LDAP or OpenLDAP ?
  • System store used by openssl (/etc/ssl): 
    
# find the base directory:
openssl version -d
# list the certs:
ls -R `openssl version -d | sed -E 's/OPENSSLDIR: "([^"]*)"/\1/'`/cert*
# or maybe
sudo ls -lAR /etc/ssl
# or
sudo find /etc/ssl -name '*.pem' -print | xargs -I{} openssl x509 -subject -noout -in {}
sudo find /etc/ssl -name '*.crt' -print | xargs -I{} openssl x509 -subject -noout -in {}
sudo find /etc/pki -name '*.pem' -print | xargs -I{} openssl x509 -subject -noout -in {}
# but my personal certs (installed in browsers etc) don't show up in there
man update-ca-certificates
ls -l /etc/ssh
ls -lAR ~/.pki
ls -lAR /etc/pki
# while user is logged in:
sudo ls -lAR /run/user/USERIDNUMBER/keyring

From someone on Stack Exchange:
Most distros put their certificates soft-link in system-wide location at /etc/ssl/certs.
  • Key files go into /etc/ssl/private
  • System-provided actual files are located at /usr/share/ca-certificates
  • Custom certificates go into /usr/local/share/ca-certificates
Whenever you put a certificate in one of the above mentioned paths, run update-ca-certificates to update /etc/ssl/certs lists.

From someone on reddit 11/2019:
Applications that utilize the system cert store: Chrome on macOS/windows [but 11/2020 Chrome is changing to have its own store: article]. Safari on macOS. Edge on windows [old Edge, I assume; don't know about new Edge]. Linux support depends on the distribution. RHEL is probably better than others.

Firefox uses its own key store ...

Java applications will vary in support. It really depends on the implementer.

[Certs can be stored in a hardware device:] A Yubikey with certs provisioned acts as a pkcs#11 device which is an industry standard interface to cryptographic devices. It has good support for all applications that utilize the system cert store. There are plugins to utilize pkcs11 devices for Firefox.

Amit N. Bhagat's "Digital Certificates Explained"
Federal Public Key Infrastructure Guides' "Trust Stores"

Places passwords are stored:
+/-
GNOME networking passwords are stored in plaintext in files in /etc/NetworkManager/system-connections

MEGA password discussion
MEGAchat: Technical Security Primer

libsecret-based clients via the Freedesktop.org secret storage DBus API ?
KeePassXC 2.5.x can be used as a vault service by libsecret: https://keepassxc.org/blog/2019-10-26-2.5.0-released/ KeePassXC as "secret service"

KeePassXC password manager can supply SSH keys to an SSH agent: KeePassXC and SSH.

Run "ssh-add -l" or "ssh-add -L" to see all keys available through ssh-agent.
Run "ssh-add -s filename.pkcs11" to add a digital certificate to ssh-agent.

"nmap --script ssl-cert localhost" gives me one cert used by port 25 SMTP, called "mint" or "DNS:Mint".

"nmap --script ssl-enum-ciphers localhost" gives me TLS ciphers used by port 25 SMTP, port 631 CUPS.



Security Test / Audit



Lynis
David Mytton's "80+ Linux Monitoring Tools for SysAdmins"
tcpdump:
Daniel Miessler's "A tcpdump Tutorial and Primer with Examples"
"sudo tcpdump -i lo -A | grep Host:"
iptraf
iptop
ntop
netstat: "sudo netstat -atupl"
lsof: "sudo lsof -i" to see established connections.
ss: "sudo ss -lptu".
NixCraft's "ss command: Display Linux TCP / UDP Network/Socket Information"
NixCraft's "Linux: 25 Iptables Netfilter Firewall Examples For New SysAdmins" (see "27. Testing Your Firewall")
nethogs: "sudo nethogs"
ngrep
auditd

CERT's "Intruder Detection Checklist"

See the "Port scanning and router testing" section of my "Computer Security and Privacy" page.

SEI's "Steps for Recovering from a UNIX or NT System Compromise" (PDF)



Miscellaneous



Throttling network bandwidth, for testing purposes:
"sudo tc qdisc add dev enp19s0 root tbf rate 32kbit latency 50ms burst 770"
"sudo tc qdisc delete dev enp19s0 root"

Brendan Gregg's "Linux Performance Tools" diagram