Basics





Derived from Ubuntu Security Podcast episode 83 7/2020:
+/- LSM == Linux Security Module inside the kernel.

Stacking == using multiple security modules in same system.

As of 5.8 kernel, stacking is limited, but this is being changed.

Current (5.8) stacking rules:
Major modules (SELinux, AppArmor, Smack): can't stack with another major module, because they all try to attach their security data blobs to the same hooks inside the kernel.
Minor modules (TOMOYO, Yama, LoadPin): can stack.
Some modules might be allowed to stack but not make sense to stack on each other, they conflict or duplicate.

Wikipedia's "Linux Security Modules"



Network Control and Firewalls



This section is for tools that generally run unattended. For tools used by a person, see the Network Monitoring section.

Some terms:

Ubuntu's "DoINeedAFirewall"
Adrian Grigorof's "Open Source Security Controls"

You can change your MAC address to any value, either for Wi-Fi or for wired Ethernet, via the Network or Network Manager application.



Firewalls

+/-
"Netfilter is the framework in the Linux kernel, which implements the rule and filters provided by the user, through an interface available to user called iptables."

GUFW and UFW (simplified UIs to use instead of iptables)


Coming in KDE: plasma-firewall, a GUI on top of multiple types of back-end firewalls such as ufw and firewalld.

  • /etc/hosts file:
    +/-
    MintGuide's "Hosts - change and manage the /etc/hosts file"


  • iptables:
    +/-
    "modinfo ip_tables"

    +/- There are 5 "tables": filter, nat, mangle, raw, security. We only care about the filter table, which has these built-in "chains" of rules: INPUT, FORWARD, OUTPUT. You can create new chains if you wish. Each rule ends with an action which can be: ACCEPT, DROP, LOG, or name of another chain. (There is more, but that's all we need to know.)

    To see how much traffic is passing through each section of rules in filter table, do "sudo iptables -L -v" (can reset counters via "sudo iptables -Z"). On my system, with Windscribe VPN active, after doing a bunch of downloading, that gives: 
    
+/-
Chain INPUT (policy ACCEPT 2005K packets, 2860M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy DROP 306 packets, 21425 bytes)
 pkts bytes target     prot opt in     out     source               destination
  486 75925 ACCEPT     all  --  any    any     anywhere             192.168.0.0/16
 1104 72461 ACCEPT     all  --  any    any     anywhere             10.0.0.0/8
    0     0 ACCEPT     all  --  any    any     anywhere             172.16.0.0/12
 495K  860M ACCEPT     all  --  any    any     anywhere             localhost
 1339  139K ACCEPT     all  --  any    any     anywhere             104.20.122.38
 1369  143K ACCEPT     all  --  any    any     anywhere             104.20.123.38
 417K   55M ACCEPT     all  --  any    tun+    anywhere             anywhere
    0     0 ACCEPT     all  --  any    any     anywhere             localhost
 423K   78M ACCEPT     all  --  any    any     anywhere             89.238.nnn.nnn
    So my system is doing no routing ("FORWARDing"). And if no rule is matched, the packet gets DROPped if it is output, ACCEPTed if it is input.

    To block SSH connections from any address, do "sudo iptables -A INPUT -p tcp --dport ssh -j DROP". Can do the same with http and https if you're not running a web server.

    There is "ip6tables" which is separate but mostly has the same syntax as "iptables".

    There is a "conntrack" module which lets you do things such as "ctstate" in rules.

    To save changes so they survive across system restart, on Ubuntu-type systems, do "sudo apt install iptables-persistent", then turn off Windscribe VPN and firewall, then do "sudo su", and then "iptables-save >/etc/iptables/rules.v4" and "ip6tables-save >/etc/iptables/rules.v6".

    You can write commands such as "-P INPUT DROP" into a file such as "iptables.txt" and then run "sudo su" then "iptables-restore <iptable.txt"

    How-To Geek's "The Beginner's Guide to iptables, the Linux Firewall"
    Supriyo Biswas's "An In-Depth Guide to iptables, the Linux Firewall"
    Vivek Gite's "Iptables Netfilter Firewall Examples"
    Ravi Saive's "Basic Guide on IPTables (Linux Firewall) Tips / Commands"
    Mitchell Anicas's "Iptables Essentials: Common Firewall Rules and Commands"
    IP sets
    iptables-vis - visualise iptables chains

    I ran a bash file with commands: 
    
iptables -P INPUT DROP
iptables -I INPUT -i lo -j ACCEPT
iptables -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -P FORWARD DROP
    Ran Windscribe VPN client which resulted ("iptables -L") in: 
    
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  anywhere             192.168.0.0/16
ACCEPT     all  --  anywhere             10.0.0.0/8
ACCEPT     all  --  anywhere             172.16.0.0/12
ACCEPT     all  --  anywhere             localhost
ACCEPT     all  --  anywhere             104.20.123.38
ACCEPT     all  --  anywhere             104.20.122.38
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             localhost
ACCEPT     all  --  anywhere             89.238.nnn.nnn
    That anywhere-anywhere rule actually has "out=tun+" attached to it (can see that via "iptables -L -v"), so all that traffic goes through the VPN.

    Put DROP on all IPv6 chains; my ISP doesn't support IPv6.

    Ran a couple of net-testing apps on my phone (which is on my LAN), targeting my PC, and they show all ports blocked, no response to ping, no services offered.

    If you want to log packets that don't match any rule, and end up getting DROPed by the chain policy, add a rule at the END of the chain via a command such as "iptables -A FORWARD -m limit --limit 1/minute -j LOG".

    Default logfile for iptables is /var/log/kern.log;
    "sudo dmesg -T" command shows same log, with some useful coloring added.

    I would like to log/detect all applications that create output connections. But it seems this is very difficult to do in a human-readable way:
    Super User's "With Linux iptables, is it possible to log the process/command name that initiates an outbound connection?"
    Akkana's "Find out what processes are making network connections"

    You can get instantaneous snapshots (not cumulative logs) by running "ss -tp" or "netstat -A inet -p".

    To see IP address and other details about each network interface, run "ip -d address".

    I want to log any incoming attempts on protocols/ports I don't use.

    Found this: "You can add the --syn flag to make the rule match only packets with the SYN flag set, which is set only on new connection attempts."

    Maybe do this:
    "iptables -I INPUT -p tcp --dport ssh --syn -m limit --limit 1/minute -j LOG --log-prefix "Incoming SSH attempt ""
    Do similar for FTP, Telnet, HTTP, HTTPS.

    I didn't bother to make rules for TeamViewer (port 5938), Remote Desktop (port 3389), SMB (port 139), NFS (port 2049), VNC (port 5900), http-alt (port 8080), RTelnet (port 107), TFTP (port 69), Simple FTP (port 115), rsh (port 514), rsync (port 873), Telnet-TLS (port 992), PPTP (port 1723), SSDP (port 1900), CIFS (port 3020), UPnP (port 5000), Socks Proxy (port 1080), Microsoft-DS (port 445) attempts. You can go a little nuts with this stuff. I think I don't have listeners active for any of it, but it would be nice to log and drop the packets.

    But it turns out you can make one rule for multiple ports, so I made:
    "iptables -I INPUT -p tcp --match multiport --dports 5938,3389,139,2049,5900,8080,107,69,115,514,873,992,1723,1900,3020,5000 --syn -m limit --limit 1/minute -j LOG --log-prefix "Incoming suspicious port attempt ""
    Limit of 15 port numbers per rule; had to split it into two.

    If one of your rules logs incoming HTTP attempts, test it by putting address "localhost:80" into browser's address field, then looking in logs. Or test by running "curl localhost" on the CLI.

    If one of your rules logs incoming HTTPS attempts, test it by putting address "https://localhost" into browser's address field, then looking in logs. Or test by running "curl localhost:443" on the CLI.

    If your rules log incoming SSH and/or Telnet and/or FTP attempts, test them by running "ssh localhost" and/or "telnet localhost" and/or "ftp localhost" in CLI, then looking in logs. Or doing "curl localhost:22" and/or "curl localhost:23" and/or "curl localhost:21" in CLI, then looking in logs.

    But I want to drop the packets after logging them. So I did this: 
    
iptables -N I_LOG_DROP
iptables -A I_LOG_DROP -m limit --limit 4/minute -j LOG --log-prefix "IPTABLES-I-LOG-DROP: " --log-level 6
iptables -A I_LOG_DROP -j DROP

iptables -P INPUT DROP
iptables -A INPUT -p tcp --match multiport --dports 20,21,22,23,80,5938,3389,139,2049,5900,8080,107 --syn -j I_LOG_DROP
iptables -A INPUT -p tcp --match multiport --dports 69,115,514,873,992,1723,1900,3020,5000 --syn -j I_LOG_DROP


    At some point, maybe it's easier to list the ports that should be open, instead of those that should be blocked. But maybe some high port numbers get opened dynamically.
    "iptables -I INPUT -p tcp --match multiport ! --dports 80,443 --syn -m limit --limit 1/minute -j LOG --log-prefix "Incoming suspicious port attempt ""

    Did this (in a shell script file): 
    
iptables -F
iptables -Z

iptables -N I_LOG_DROP
iptables -A I_LOG_DROP -m limit --limit 4/minute -j LOG --log-prefix "IPTABLES-I-LOG-DROP: " --log-level 6
iptables -A I_LOG_DROP -j DROP

iptables -N O_LOG_DROP
iptables -A O_LOG_DROP -m limit --limit 4/minute -j LOG --log-prefix "IPTABLES-O-LOG-DROP: " --log-level 6
iptables -A O_LOG_DROP -j DROP

iptables -P INPUT DROP
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p tcp -m tcp --syn -j I_LOG_DROP
# UDP 1194 for ProtonVPN, UDP 5353 is multicast DNS / Avahi
iptables -A INPUT -p udp --match multiport ! --dports 22,67,68,80,443,1194,5353 -j I_LOG_DROP

iptables -P FORWARD DROP

iptables -P OUTPUT ACCEPT
# port 23189 used by SFTP
iptables -A OUTPUT -p tcp --match multiport ! --dports ssh,http,https,23189 --syn -j O_LOG_DROP
# UDP 1194 for ProtonVPN, UDP 5037 when using Wireshark
iptables -A OUTPUT -p udp --match multiport ! --dports 22,53,67,68,80,123,443,1194,1900,5037,5353,30000:64000 -j O_LOG_DROP


    The whole thing gets more complicated because your VPN probably does iptables stuff too. Windscribe VPN adds rules and changes chain policies. I had to have a big shell script to add rules before the VPN starts, then a small script to do a couple of tweaks after VPN has started.

    And the number of ports keeps growing and growing. Apps such as Firefox and torrent-client open lots of high port numbers. Most/all of them may be on localhost, so maybe you have to start wiring addresses into your rules. The whole thing just gets too complicated.

    If you see an iptables LOG line in /var/log/kern.log that doesn't show source and dest ports, but gives "PROTO=n", look up that protocol number in /etc/protocols.

    Do "sudo netstat -tulpn" to see what ports have listeners. (Also "sudo ss -lptu")

    Around this time, I mostly gave up with iptables. I think it was the wrong approach.
    Instead, concentrate on reducing and understanding the number of listeners you have. It doesn't matter if an incoming packet gets through iptables, as long as no process is listening on that port.

    Let the VPN do what it wants with iptables. Maybe do "sudo iptables -L -v" occasionally to see how much traffic is hitting each rule.

    I think it would be different with a server, running only a few services. There you could allow only 10 or so open ports.

    Maybe there are some "listeners" built into the protocol stack ? Turn off protocols you know you aren't using. Maybe do this to see unusual protocols:
    
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
...
# assuming your INPUT chain policy is ACCEPT, put these at the end of the chain
iptables -A INPUT -p tcp -j ACCEPT
iptables -A INPUT -p udp -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -m limit --limit 1/minute -j LOG --log-prefix "Incoming IPv4 protocol "


  • ufw and gufw:
    +/-
    GUFW and UFW (simplified UIs to use instead of iptables)
    Virdo, Boucheron and Juell's "How To Set Up a Firewall with UFW on Debian 10"
    Vivek Gite's "How To Configure Firewall with UFW on Ubuntu 20.04 LTS"
    Jahid Onik's "How To Configure Firewall with UFW on Ubuntu"
    Linuxize's "How to Set Up a Firewall with UFW on Debian 10"
    Daniel Aleksandersen's "How to switch firewalls from FirewallD to UFW"
    "less /etc/default/ufw"
    "less /etc/ufw/sysctl.conf"
    "sudo ufw status verbose"
    Files in /etc/gufw: "sudo less /etc/gufw/gufw.cfg" to find which profile is in use, then "sudo less /etc/gufw/Home.profile" to see that profile.

    UFW at least has the concept of "applications", but I don't know how that appears in GUFW.


  • firewalld:
    +/-
    firewalld
    LinuxTeck's "15 basic useful firewall-cmd commands in Linux"
    Brian Boucheron's "How To Set Up a Firewall Using firewalld on CentOS 8"
    Shashank Nandishwar Hegde's "An introduction to firewalld rules and scenarios"
    GUI for firewalld: firewall-config. Very detailed, a bit overwhelming.

    Avoid duelling firewalls. You can't run firewalld and ufw/gufw at the same time. firewalld and ufw both use iptables.
    "whereis -b -B /usr/bin /usr/sbin -f gufw ufw firewalld"


  • plasma-firewall:
    +/-
    Coming in KDE: plasma-firewall, a GUI on top of multiple types of back-end firewalls such as ufw and firewalld.


  • Nftables:
    +/-
    This is a replacement for iptables, ip6tables, arptables, and ebtables. It is a more procedural language, I think. Should avoid a lot of duplication that you can get in iptables rules, and scale better. Also combines IPv4 and IPv6 in one structure. Not supported by VPNs yet, I think. Still in development 11/2018, I think.

    Debian / wiki / nftables
    Nftables wiki
    Alistair Ross's "Hello nftables, Goodbye iptables"
    "apt show nftables"
    "sudo nft list tables"
    "sudo nft list ruleset | less"

    From someone on reddit:
    Here's how simple it is to get a sensible nftables ruleset in place: 
    
sudo apt install nftables

# Use an example ruleset for a basic workstation
sudo cp /usr/share/doc/nftables/examples/workstation.nft /etc/nftables.conf

# Load that ruleset
sudo nft -f /etc/nftables.conf

sudo systemctl enable nftables
    Have a look at that config file, it's pretty short and all of the rules have comments explaining what they do. There's one rule in there commented out, which you can enable to open any local ports you might want to access from remote systems, e.g. SSH.


  • BPF (Berkeley Packet Filter; AKA cBPF) and eBPF:
    +/-
    BPF programs are run in reaction to events, and run to completion. They're not constantly-running tasks.

    eBPF has been in the Linux kernel since version 3.18.

    Used to make a replacement for iptables and nftables, among many other things.

    Original BPF has a fairly small VM inside the kernel, and byte-code can be placed in it to do network processing.

    eBPF adds attaching code in many more places inside the kernel, adds a Just-In-Time compiler to native code (for some CPU architectures), and adds global state (in arrays, key-value pairs).
    Also has ability to jump out to user-land code, bypassing much of the standard network stack, for example, in cases where reduced features and higher performance make sense. Or go the other way: where there used to be a (costly) jump out to user-land, instead do the operation inside the kernel, in the VM.

    Wikipedia's "Berkeley Packet Filter"
    Filip Nikolovski's "TIL: eBPF is awesome"
    Kevin Dankwardt's "BPF For Observability: Getting Started Quickly"
    Andrew Klotz's "BPF intro: syscall counting"
    Brendan Gregg's "Learn eBPF Tracing: Tutorial and Examples"
    Brendan Gregg's "Extended BPF - A new software type" (video)
    Hongli Lai's "Full-system dynamic tracing on Linux using eBPF and bpftrace"
    eBPF - Introduction, Tutorials & Community Resources
    iovisor / bcc
    iovisor / bcc / docs

    
grep BPF "/boot/config-$(uname -r)"

sudo apt install bpfcc-tools bpftrace

sudo /usr/share/bcc/tools/opensnoop



Mehedi Hasan's "Firewall Software For Protecting Your Linux System"
Eric Geier's "Little Known GUI Firewall Options for Linux"

IDSs and loggers and file-checkers

+/- List of Open Source IDS Tools
Daniel Berman's "6 Open Source SIEM Tools"

Honeypots and tar-pits

+/-
Vinsloev's "Cybersecurity 101 - What is a Honeypot?"
paralax / awesome-honeypots



Nix-Pro Wiki's "Packet journey through Linux kernel"



Application Control and Security



Maybe decide what is important to you.

Isolate applications from each other and the OS:

+/-
Solutions from lightest to heaviest:
  • No isolation: userid gives every app access to all of your files, X gives apps access to each other's event-queues. Native packaging: deb, rpm, etc.

  • Sandbox created by someone other than app dev: AppArmor, Firejail.

  • Container (and permissions) created by app dev: Snap, Flatpak, Docker. All containers run on top of same, shared OS.

  • Micro-VM or unikernel for each app ? A severely stripped-down server image. If you're going to run a single process, no GUI, no SSH, single user, no command-line shell, then strip lots of stuff out of the Linux kernel-plus. Even run kernel and app in same address space.

  • VM for each app. OS is a "full" server distro such as Ubuntu server or a minimal distro such as Alpine.

  • Separate bare metal for each app. OS is a "full" server distro.

Not sure which is better: security set by app dev (e.g. in Snap), or by a third party (e.g. in AppArmor). App dev knows the app best, but OTOH a third party totally focused on security may do a better job.

Latest software:

+/-
Solutions from fastest to slowest:
  • Container / image created by app dev: Snap, Flatpak, Docker, appimage.

  • Distro maintainer does native packaging: deb, rpm, etc.

  • LTS distro / stable repo.

  • VMs created using LTS software.




Low-level kernel stuff and building blocks:

+/-

Lowest-level stuff



  • Namespaces:
    +/-
    Wikipedia's "Linux namespaces"
    Mahmud Ridwan's "Separation Anxiety: A Tutorial for Isolating Your System with Linux Namespaces"
    Ed King's "Linux Namespaces"

    Namespaces are a key feature used to implement isolation for container systems (Docker, flatpak, snap, LXC), applications and daemons (AppArmor, Firejail), and multi-process applications (some browsers). 

    
man namespaces
man lsns

lsns		# list namespaces in use by current user
sudo lsns	# list namespaces in use by all users

unshare -c -n APPNAME    # run APPNAME with no network access

    A program uses the clone() system call instead of fork(), to make a copy of itself running in a separate space. That can mean separate process trees (clone can't see parent process or any pre-existing processes), different views of the network connections, separate mount spaces (/etc/fstab's, I think), pseudo-root privileges, separate IPC spaces, more. There can be sockets set up to communicate between namespaces.

    In the CLI, you can use the "unshare" command to run a process in a separate namespace. The "ps" command has options to display namespace IDs for processes.



  • cgroups (control groups):

    Not a security mechanism. Used to control resource contention among groups of processes. Defines 13 resource categories ("controllers"), and lets you specify that process group A can use amount N of resource category R ?

    So, for example, normally if 2 applications start 5 processes each, those 10 processes all contend for resources on an equal footing. But suppose you want the 5 processes for application A to get higher priority for CPU than the 5 processes for application B ? Define each set of 5 processes as a separate cgroup.

    To run an app using limited amount of memory: 
    
cgcreate -g memory:GROUPNAME
echo 500M >/sys/fs/cgroup/memory/GROUPNAME/memory.limit_in_bytes
echo 5G >/sys/fs/cgroup/memory/GROUPNAME/memory.memsw.limit_in_bytes
cgexec -g memory:GROUPNAME APPNAME

    David Both's "Managing resources with cgroups in systemd"


  • Access Control Lists (ACLs):

    A way to set finer-grained permissions than the base UGO permissions.
    Kuldeep Sharma's "Secure Files/Directories using ACLs (Access Control Lists) in Linux"
    
man acl
man setfacl

grep -i acl /boot/config*"	# see if enabled in kernel
mount | column --table | grep acl	# any FSs mounted with ACL enabled ?

    Works on most Linux filesystems, but not vfat, exFAT, and some other non-Linux filesystems.

  • Capabilities:

    man page
    K3A's "Linux Capabilities in a nutshell"
    Good info in Chapter 39 "Capabilities" of "The Linux Programming Interface" by Michael Kerrisk.

    "The goal of capabilities is to divide the power of superuser into pieces".

    There is a set of defined capabilities: audit-control, audit-write, network admin, network broadcast, etc. 
    
grep 'define CAP_' /usr/src/linux-headers-$(uname -r)/include/uapi/linux/capability.h | grep -v \( | grep -v LAST_CAP
man capabilities
    A set of capabilities can be granted to a file, and then a process that execs that file gets those capabilities. Threads in a process can each have different capabilities. Capabilities can be inherited in various ways by daughter processes.

    Most applications probably aren't written to know or care about capabilities, and don't have any capabilities assigned to their executable file. 
    
getcap /bin/* /usr/bin/* /usr/bin/*
    And anything run by root user gets all capabilities.

Mid-level stuff



  • iptables:

    You could assign an application to belong to a particular group, then make iptables rules about that group using "-m owner --gid-owner" qualifiers in iptables rules.
    discussion


  • seccomp and seccomp-bpf (SECure COMPuting with filters):
    +/-
    A Linux kernel security module that filters syscalls from a process.
    The Linux Kernel's "Seccomp BPF (SECure COMPuting with filters)"
    Wikipedia's "seccomp"


  • SELinux:
    +/-
    A Linux kernel security module. More fine-grained than AppArmor and Firejail, which run at the application level, only at the interface between app and OS. Role-based permissions.

    Steven Vaughan-Nichols' "How to set up SELinux right, the first time"
    Alex Callejas' "A sysadmin's guide to SELinux"
    Gernot Klingler's "SELinux basics"
    Mike Calizo's "Secure your containers with SELinux"
    Wikipedia's "Security-Enhanced Linux"
    RHEL's "SELinux User's and Administrator's Guide"
    SELinux Project
    SELinux Project on GitHub
    Bug-report mailing list: selinux at vger.kernel.org
    Fedora's "Getting started with SELinux"
    Barrow's "Use SELinux Targeted Policy to Secure Your Hosts"
    Tyler Carrigan's "How to read and correct SELinux denial messages"
    Good info in Chapter 24 "Enhancing Linux Security with SELinux" of "Linux Bible" by Christopher Negus.
    /r/selinux

    From Lubos Rendek's "How to disable/enable SELinux on Ubuntu 20.04 Focal Fossa Linux":
    Ubuntu offers AppArmor as an alternative to SELinux. While SELinux is available on Ubuntu, it is rather in an experimental stage and most likely will break your system if set to enforcing mode. In case you must use SELinux, make sure to disable AppArmor first. Also set SELinux first to permissive mode and check your logs for potential issues before you enable enforcing mode.

    I've heard that SELinux is better for servers rather than desktops. My desktop Fedora 34 KDE system does have SELinux enabled and enforcing, by default. But none of the user GUI apps are making use of SELinux; only the system apps and services are using it. 

    
# See if SELinux is installed and/or enabled:
sestatus

# Install it:
sudo apt install selinux-utils selinux-basics policycoreutils setools setroubleshoot setroubleshoot-server
sudo setenforce permissive
sudo selinux-activate

# Allow anything to run:
sudo setenforce permissive
grep CONFIG_SECURITY_SELINUX_DEVELOP /etc/selinux/config	# want "y"

# Possible modes are: disabled permissive enforcing
# In permissive mode, there can be a lot of logging going on
# if lots of operations are (silently) failing security checks.

# Note: changing mode can trigger (upon next reboot) a scan
# of all files to relabel them (check security contexts).

# Possible policy types are: targeted mls minimum
# Edit /etc/selinux/config to set value in line SELINUXTYPE=whatever
# If you set policy type to mls, first install package selinux-policy-mls
# If you set policy type to minimum, first install package selinux-policy-minimum

cat /etc/selinux/config

man -k selinux
man runcon
man sandbox
# runcon is a little dangerous, better to use sandbox

secon -urt      # show security context of current process
id              # show security context of current user
sudo semanage user --list   # show all users/roles
sudo semanage port --list   # show all TCP/IP port labels ?
ls -Z                       # see labels on files
sudo netstat -Ztulp         # see labels on listeners

journalctl -t setroubleshoot --since=14:20

# If you have auditing enabled:
sudo aureport | grep AVC
sudo ausearch -m avc

# If you don't have auditing enabled:
journalctl | grep sealert

    The label (AKA context) format is "user:role:type:level".

    SELinux is enabled / enforcing / targeted by default on RHEL and Fedora ?

    To create a policy set for an app:
    +/-
    Red Hat's "Writing a custom SELinux policy"

    [I tried to write a simple policy for KeePassXC, and gave up. Would have been stuck at zero except for help from people on /r/selinux. Roadblock after roadblock, examples in docs are all about CLI commands instead of policy files, couldn't find any existing policies to copy, all kinds of mysterious things happening, then realized the app's code would have to be changed in concert with applying the policy.]

    1. Install "policycoreutils-devel".
    2. Install the app and its files.
    3. Test the application (unconfined).
    4. Use "sepolicy generate --init APPBINFILE" to generate a package of policy files that captures the current system state for that application.
    5. Run the setup script ("sudo ./APPNAME.sh") from that package to set the system state. This will set context on the app. But labels on any existing files will not be changed. To set app's label on a file, do "sudo semanage fcontext -a -t TYPENAME FILENAME" then "sudo restorecon -v FILENAME". (Seems to be a bug if "semanage FILENAME" uses a relative filename such as "../x") https://wiki.gentoo.org/wiki/SELinux/Tutorials/Controlling_file_contexts_yourself Also creates a manual page you can access via "man -l APPNAME_selinux.8". Have to reboot to make the "restorecon" take effect ? Do "sudo semanage fcontext --list | grep APPNAME" to see contexts labeled on files.
    6. Test the application (confined by new policies). To see that app is confined, "ps -efZ | grep APPNAME". To see that app's files are labeled, "ls -Z FILENAME".
    7. Check those policy files in to the SELinux project.

    A policy is tied to an app binary's filename, so there is a single policy per app. But you can let the user/admin pass values in to the policy (via semanage). So you can have variables such as "network allowed or not" set by the admin.

    Networking can be turned on/off, TCP/IP ports can be controlled, but there is no way to control the IP addresses or domains that can be accessed.
    When the SELinux policy has been loaded, all newly created files will be automatically labelled according to the specifications in the file context (.fc file).

    D-Bus access: better to control via polkit/policykit instead of SELinux.

    There seems to be no way (short of changing app's source code) to say "any time app creates a file anything.EXT anywhere, label it with role APPNAME_EXT_t". You CAN do this (using "filetrans_pattern") if you can confine those files to being created only in a fixed directory.

    Is there a way to apply a label to a shell script, and have it work at run time ? Is there a generic "backupapp_t" type ? I want app K's files to be accessible only to app K and to the backup scripts.

    In an app's C code, include selinux.h and then make calls to libselinux. For example, setfscreatecon() and then open(O_CREATE).




  • Yama:

    A Linux kernel security module. Seems to be very narrowly focused on ptrace ?
    Michael Boelen's "Protect against ptrace of processes: kernel.yama.ptrace_scope"
    Yama


  • Landlock:

    A Linux kernel security module. Coming in kernel 5.13.
    Landlock: unprivileged access control
    Mickael Salaun's "Landlock: unprivileged access control"


  • Lockdown:

    A set of Linux kernel features to optionally close off interfaces that allow root to modify the kernel.
    Matthew Garrett's "Linux kernel lockdown, integrity, and confidentiality"
    Arch Wiki's "Kernel lockdown mode"


  • SMACK:

    A Linux kernel security module.
    Wikipedia's "Smack"
    The Smack Project - Home


  • TOMOYO:

    A Linux kernel security module.
    Wikipedia's "Tomoyo Linux"
    TOMOYO Linux
    eLinux's "TomoyoLinux"


  • Bubblewrap:

    "man bwrap"
    ArchWiki's "Bubblewrap"



# all available kernel modules:
find /lib/modules/$(uname -r)/kernel -name '*.ko' -print | sort | less

lsmod | sort | less		# see loaded modules
modinfo MODNAME			# see info about module, incl params when loaded

cat /proc/cmdline		# see kernel launch command line

sudo sysctl -a | less	# see kernel parameters
# https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html

less /boot/config-$(uname -r)	# see current kernel configuration

less /etc/sysctl.conf	# see system variables
getconf | less



Service and app-level stuff:

+/-
  • Firejail:
    +/-
    An application that restricts the environments of other apps, running them with access only to certain directories, or to fake copies of system directories, or no network access. Application-based permissions.

    From Firejail Security Sandbox:
    Firejail is a SUID program that reduces the risk of security breaches by restricting the running environment of untrusted applications using Linux namespaces and seccomp-bpf. It allows a process and all its descendants to have their own private view of the globally shared kernel resources, such as the network stack, process table, mount table.

    Firejail Security Sandbox
    Easy Linux tips project's "Run your web browser (and other apps) in a secure sandbox"
    Jesse Smith's "The Firejail security sandbox"
    xenopeek's "Using firejail as security sandbox for your programs"
    netblue30 / firejail
    "firejail --version" (shows more than just version number)

    Firejail works on native binaries and on appimages (with --appimage option), but not on Snaps or Flatpaks or Docker containers.

    Does Firejail work on browsers and terminals launched from other apps, such as VSCode ?

    "sudo apt install firejail firejail-profiles firetools".
    Profiles stored in /etc/firejail.

    Poked around a bit, quickest way to see the restrictions is to run "mount | column --table" in CLI before and after running "firejail bash". Also "firejail --audit APPNAME".

    To run an app while denying network access: 
    
firejail --net=none APPNAME
    This is equivalent to editing the ~/.config/firejail/APPNAME.local file to add "net none".

    To run an app using limited amount of memory: 
    
firejail --rlimit-as=NUMBYTES APPNAME

    Local system edits could be done to either ~/.config/firejail/APPNAME.local or /etc/firejail/APPNAME.local.

    Easy way to make any of your big apps run under Firejail: create symlinks for them that link to firejail: 
    
sudo ln -s /usr/bin/firejail /usr/local/bin/firefox
which -a firefox        # check PATH
# then if run "firefox", from either CLI or icon, will run "firejail firefox"


    Easy way to make ALL apps known to Firejail run under Firejail: 
    
sudo firecfg
which -a firefox        # check PATH
# Then if run "firefox", from either CLI or icon, will run "firejail firefox"

# If you install another application, do same again:
sudo firecfg

firecfg --list          # list all links
sudo firecfg --clean    # remove all links

# On Kubuntu 20.10, Firejail version 0.9.62.4:

# I found that I had to remove Dolphin's
# link to fix problems with running sudo in shell scripts:
sudo rm /usr/local/bin/dolphin

# How to put /opt/thunderbird/thunderbird-bin inside Firejail ?

# Firefox wouldn't launch.  Tried various directives in
# firefox-common.local to fix it, no luck.  Removed it from Firejail.

# Liferea wasn't able to send links to Firefox properly;
# snap of KeePassXC was able to.

# VSCode launched, then it didn't.

# Gave up on Firejail on Kubuntu 20.10.


    
# On Fedora 34 KDE using X, Firejail 0.9.64.4:
# Tried to run KeePassXC 2.6.4 under Firejail.
# Clicking on links, and Ctrl+Shift+U to open link, didn't work.
# In debug mode, get message:
# "Unable to detect a web browser to launch 'http...'"
# Bug reports:
#	https://github.com/netblue30/firejail/issues/3329
#	https://github.com/keepassxreboot/keepassxc/issues/6637

# you can apply a profile for one app to another app:
firejail --profile=keepassxc ls ~/.mozilla
    With help from someone, got some basic sanity of Firejail-KeePassXC working. Still fails if browser has to be launched, but if browser is running already, clicking on a link works. Had to make file /etc/firejail/keepassxc.local: 
    
# keepassxc.local

# If you click on an URL in KeePassXC, and default browser is Firefox, FF stub
# gets launched inside the KP firejail, so need this to counteract what is
# done in disable-programs.inc:
noblacklist ${HOME}/.cache/mozilla
noblacklist ${HOME}/.mozilla
# That works if FF is running before you click on an URL in KeePassXC.
# If FF is not running before you click on an URL, a full copy of FF tries
# to launch inside the KP firejail, and crashes.

ignore private-bin
include allow-bin-sh.inc
ignore net none
protocol unix,inet,inet6,netlink
private-etc ca-certificates,crypto-policies,host.conf,hostname,hosts,nsswitch.conf,pki,protocols,resolv.conf,rpc,services,ssl
    Similar does NOT work for Brave or ungoogled-chromium browsers. Only FF works.

    General problem with launching stuff / opening links from inside Firejail, it seems: https://github.com/glitsj16/firejail-handler-http

    You could use Firetools or Firejail Configuration Wizard apps in the GUI. But building a security profile doesn't seem to be persistent, and the Firetools monitor shows a much-too-high number for memory usage. And the UI is really annoying.

    When trying to debug a Firejail profile, launch the app in CLI ("firejail APPNAME"), not via an icon. Apps often put out error messages on stderr. Also, do "sudo ps -ax | grep firejail" to make sure an app isn't hanging; reboot if there is one.

    Firejail project home: netblue30 / firejail

    Interesting debate: Firejail worsen security? #3046



  • green check-mark  AppArmor:
    +/-
    Very similar to Firejail, but implemented as a Linux kernel security module. Application-based permissions. Generally applied to services/daemons, and runs in the background without the user having to know anything about it.

    Do "sudo apparmor_status" to see info.
    To install more: "sudo apt install apparmor-profiles apparmor-profiles-extra apparmor-utils apparmor-easyprof apparmor-notify"
    Maybe add your user to group "adm" to get notifications ? "adduser USERNAME adm"
    "man -k apparmor"

    Wikipedia's "AppArmor"
    AppArmor

    Making profiles:
    Ubuntu Tutorials' "How to create an AppArmor Profile"
    Uzair Shamim's "The Comprehensive Guide To AppArmor: Part 1"
    The Debian Administrator's Handbook's "14.4. Introduction to AppArmor"

    In Mint, Apparmor (user-space parser utility) was installed by default.
    I installed Apparmor-utils through Software Manager.
    Profiles are stored in /etc/apparmor.d directory.
    You could turn on the profile for Firefox by doing "sudo aa-enforce /etc/apparmor.d/usr.bin.firefox".
    To turn it off, do "sudo aa-disable /etc/apparmor.d/usr.bin.firefox".
    If you do a disable when enforce is not on, you'll see a "Profile doesn't exist" error.
    To see the list of active profiles, do "cat /sys/kernel/security/apparmor/profiles".
    To see AppArmor activity, do "grep apparmor /var/log/kern.log".


    When trying to debug an AppArmor profile, launch the app in CLI, not via an icon. Apps often put out error messages on stderr. Also, do "sudo ps -ax | grep APPNAME" to make sure an app isn't hanging; reboot if it is.

    Firefox with AppArmor:
    +/-
    Already had FireFox working in Firejail, so all of the following is with FF running in Firejail.

    Started enforcing AppArmor for Firefox, launched Firefox in Firejail, it showed some "welcome to Mint - file access failed" page, and the kernel log shows a dozen or more AppArmor "denied" messages.

    Quit Firefox, disabled AppArmor for Firefox, launched Firefox, and my Firefox user profile is gone ! Bookmarks, settings, add-ons, everything gone !

    Fortunately, I was able to find it all under a directory in "~/.mozilla/firefox", and was able to restore it by editing "~/.mozilla/firefox/profiles.ini" to point at the old profile instead of the new one. Whew ! Do a backup of that stuff before trying again.

    Looking at "denied" messages in /var/log/kern.log, it looks like access to "/run/firejail/mnt/" is denied, maybe also access to "home/.ecryptfs/USERNAME/...". Edited "/etc/apparmor.d/usr.bin.firefox". Above "# so browsing directories works", I added lines "/run/firejail/mnt/** r," and "/home/.ecryptfs/** rw,".

    Still got "bookmarks and history system will not be functional because one of Firefox's files is in use by another application. Security software can cause this problem." error message.

    Tried AppArmor on, Firejail off, and FF works ! So the problem is somewhere in the interaction of AppArmor and Firejail. Did "sudo xed /etc/apparmor.d/firejail-default /etc/apparmor.d/usr.bin.firefox ~/.config/firejail/firefox.profile". Couldn't fix the problem. Tried adding lines "network unix stream, network netlink stream," and "/run/firejail/mnt/** r, /home/.ecryptfs/** rw," to "/etc/apparmor.d/usr.bin.firefox", didn't help.

    If you caused a bunch of FF crashes while debugging, look in "~/.mozilla/firefox/'Crash Reports'/pending" and delete them.

    KeePassXC with AppArmor:
    +/-
    Mint 19 has no AppArmor profile for KeePassXC.

    Made a profile the stupid way, trial-and-error, not using the profiling tools. Started with no restrictions.
    "sudo aa-enforce /etc/apparmor.d/usr.bin.keepassxc"
    "sudo aa-disable /etc/apparmor.d/usr.bin.keepassxc"
    "sudo apparmor_parser -r /etc/apparmor.d/usr.bin.keepassxc"
    Got it to work, despite showing "qt5ct: D-Bus global menu: no" error (same error when AppArmor is disabled).
    Tried to turn off networking, got "Could not connect to any X display" error. Adding "network unix stream" fixed that. Very unfortunate that I had to do that; my main goal for this profile was to turn off networking.
    Took a lot of fiddling to get the directory-tree permissions okay; I'm sure they're still too generous.
    Ended up with this.




  • green check-mark  OpenSnitch:

    evilsocket / opensnitch
    As of 6/2021: Released and under active development.
    Runs on top of iptables; nftables coming later. Also uses eBPF.
    Handles TCP, UDP, and UDPLITE traffic.
    Only handles outbound connections (inbound may be under development).
    Unclear how it interacts with VPNs and firewalls that write iptables rules; dev says there should be no problems.
    Can block on the basis of applications, as well as protocols and ports.

    "systemctl status opensnitch"

    nixCraft's "OpenSnitch: The Little Snitch application like firewall tool for Linux"
    Step 3 in TokyoNeon's "Using Ubuntu as Your Primary OS, Part 4 (Auditing, Antivirus & Monitoring)"
    Do Son's "opensnitch: GNU/Linux port of the Little Snitch application firewall"
    Linux Uprising's "OpenSnitch Application Firewall 1.4.0"
    Linux Uprising's "OpenSnitch 1.3.0"


  • Douane:

    Douane
    Douane on GitLab

    Tuxdiary's "Douane: easy firewall with app rules"
    Dedoimedo's "Linux per-application firewalls - Doable? Douane"
    Project-insanity's "Application firewall Douane for ArchLinux"


  • eBPFSnitch:

    harporoeder / ebpfsnitch
    Filtering inbound connections not supported as of 3/2021, according to README.
    For any distro except Arch, must be compiled from source, as of 3/2021.


  • Portmaster:

    Portmaster


  • Trickle:

    ArchWiki's "Trickle"
    Reduce bandwidth available to a single process.


  • Docker to isolate a single process ?

    "docker run --rm -it -v $PWD/untrustedprogram:/untrustedprogram:ro ubuntu:latest" ?


  • ulimit

    To run an app using limited amount of memory: 
    
ulimit -Sv NUMKBYTES
APPNAME


  • cpulimit:

    Gregory Bartholomew article
    Limit CPU available to a single process.




My evaluation

+/- The mainstream solutions (at least, in Mint) seem to be Firejail and AppArmor.

  • Firejail better because:
    • Can have different icons that launch an app with/without Firejail.
    • Can have private copies of /dev, /bin, /tmp, /etc.
    • Easy to say nodvd, noshell, noroot.
    • In Mint, there are far more Firejail profiles than AppArmor profiles.


  • AppArmor better because:
    • No way to evade controls when you launch an app.
    • Lots of fine-grained controls on D-Bus.
    • Can set detailed rwxmk permissions on directory trees and files; Firejail just has whitelist/blacklist.
    • Has some kind of controls on "signals".


  • Neither AppArmor nor Firejail supports restricting network access to just X system, or just localhost, or just LAN addresses.

  • Trying to use both AppArmor and Firejail on same app usually fails.

  • In Mint, by default, AppArmor is controlling several daemons, such as dhcp, cups, mysql-akonadi, Clam updater, ippusbxd (USB printer).

  • I think I would try to get my key applications running in Firejail. Leave AppArmor enforcing on the daemons.





Opinion: firewalls / app control / security is a mess:

+/-
Firejail, AppArmor, SELinux, Yama, iptables, ip6tables, nftables, bpf, ufw, gufw, firewalld, netfilter, VPN, IDS, IPS, etc. Why can't we have one integrated solution, with a body of rules that can do filesystem and network access and syscall access and service and app and login control, and take user ID, app ID, MAC address, IP address, IP port, etc as input to the rules ?

Examples:
  • I'd like to control Firefox now. I go to the Security GUI, select Firefox, and I see everything about it: what files/dirs it can access, what syscalls it can do, what devices, what IP ports, domain whitelists and blacklists, what users/groups can access it, everything. All in one place. I don't care if separate kernel modules do the networking and the filesystem control, and some module inside Firefox does the domain whitelist/blacklist.

  • Same for any other app or service or module. I go to the Security GUI and choose my password manager app. In one place, I can control everything about it: filesystem access, network access, syscalls, etc.

  • I don't want my system to do anything with IPv6. I go to the Security GUI and choose IPv6. In one place, I can control everything about it: turn it off for everything, turn off incoming, whatever. Later, maybe I decide only Firefox can do IPv6. I can check a box to allow that.

  • "Networking" should have zones: can/can't access internet, can/can't access router, can/can't access other devices on LAN, maybe also services listening on localhost. And internally X11 does networking, so that has to be allowed.

  • I don't want user "skype" (which owns Skype app and the cron job that updates it) to be able to contact any domains except Microsoft's Skype domain. I go to the Security GUI and choose user "skype" and set the appropriate things.

I think today Firejail and AppArmor already mix some of this. You can set filesystem, syscall and network permissions in a Firejail profile or an AppArmor profile, I think. But Firejail and AppArmor settings are hard to share across many apps, and iptables rules know nothing about applications/processes. Iptables has a module that allows rules based on connection state, but I don't think Firejail and AppArmor have anything like that.



System Hardware Monitoring and Control



Periodically check the health of your drives. Record the stats.


Software Resource Monitoring





Why Did Something Get Installed ?



With deb/apt/dpkg packages: 

ls -lt /var/lib/dpkg/info/*.list | less

zcat -f /var/log/dpkg.log* | grep -i PKGNAME | egrep "\ install\ |\ upgrade\ "
# grab part of datetime from that, and do:
zcat -f /var/log/dpkg.log* | grep -i DATETIME | egrep "\ install\ |\ upgrade\ "

sudo apt remove PKGNAME --simulate

apt-cache rdepends --installed --recurse PKGNAME

aptitude why PKGNAME --show-summary



Network Monitoring



This section is for tools used by a person. For tools that generally run unattended, see the Network Control section.

Some terms:

Traffic monitoring and analysis

+/- Chris Siebenmann's "Internet routing can now vary based on things you wouldn't expect"
Mehedi Hasan's "Linux Monitoring Tools For SysAdmin"
Hayden James' "Linux Networking commands and scripts"
Martin Bruchanov's "Linux Network Administration"

Monitor the traffic in/out of your LAN. Best ways probably are custom software in your router, and a Pi-hole doing DNS filtering. From Security in Five Podcast - Episode 746, investigation of traffic volume exceeding data cap found that iCloud was uploading/downloading the entire collection any time one thing was added, and after that was fixed almost 50% of all traffic was due to blockable scripts (ads, trackers).



Endpoint monitoring

+/-
I want a monitoring app on my Linux system that tells me how much free storage/disk each device on the LAN (Windows laptops, Android phones and tablets, printer) has, last time it was updated, is it on the network right now. That's about it. I don't really care about network traffic monitoring.

It looks like all the managers work by using DNS to name and access devices. You'll have to set static IP assignments in the router, and put names/numbers in /etc/hosts on manager machine, then "sudo systemctl restart systemd-resolved", then "ping NAME".

Some people are telling me that there is a market separation between "network monitoring (SNMP)" products [LibreNMS, Icinga, more] and "mobile device management (MDM)" products [Quest Kace, JAMF, Spiceworks, Intune, more], and you will not find one product which can monitor both servers and smartphones, for example.

System Types

  • Custom agent on each client: install a custom "agent" app or service on each client device, get lots of detailed information and control.

  • SNMP agent on each client: install or enable an SNMP agent app or service on each client device, get varying amount of information and control.

  • Agentless: manager does network access (ping, HTTP, whatever) to each client to see if it is alive and seems to be functioning as intended. Little information, probably no remote control.


Agent Types

  • Passive: agent just collects data and responds to requests/orders from manager. So all clients have to have fixed known IP addresses ?

  • Trap: when some trigger happens (e.g. free disk space goes below 10%), agent sends an event to manager. So only manager has to have fixed known IP address ?

  • Both


Network Managers

+/-
  • EventSentry Light:

    Limited to 5 devices ? Maybe limited only for updating/control ?
    EventSentry
    Server runs on ???
    Agents available for ???


  • Zabbix:

    Zabbix
    Server runs on Linux.
    Agents available for Linux, Mac, Windows.
    There is an Android app "Unofficial Zabbix Agent" by dentier.
    "docker pull zabbix/zabbix-server-mysql" and "docker run -p 8080:8080 zabbix/zabbix-server-mysql"
    "docker pull zabbix/zabbix-web-nginx-mysql" and "docker run -p 8080:8080 zabbix/zabbix-web-nginx-mysql"


  • LibreNMS:

    LibreNMS
    Server runs on Linux and Windows/Docker.
    No agents, devices must have static IP address and SNMP.
    LibreNMS was forked from Observium before Observium became a commercial product.
    Installing LibreNMS

    There is a Docker image, "docker pull librenms/librenms" and "docker run -p 8080:8080 librenms/librenms", but it expects you to have a running MySQL database first.

    Tried native install: 
    
# See https://docs.librenms.org/Installation/Install-LibreNMS/
# I followed the "sudo -s" / Ubuntu / Nginx path.
# but starting mariadb and OpenIPMI services failed.

# Removed everything:
# Do NOT remove mysql-server and mariadb-server !  desktop went with them !
sudo rm -r /opt/librenms
sudo deluser librenms
sudo delgroup librenms
# Ended up in a state where neither MySQL nor mariadb will re-install.



  • Icinga 2:

    Icinga
    Server runs on Linux.
    No agents, devices must have static IP address and SNMP.
    Can use Nagios plug-ins.
    There is an Android app "Probe for OMD and Nagios" by Marco Ribero.
    "docker pull jordan/icinga2" and "docker run -p 8080:8080 jordan/icinga2"
    "sudo apt install icinga2"


  • Observium Community:

    Observium
    Server runs on Linux, more ?
    Agents available for xxx
    Tagged as "simple to use" in reviews.
    It seems that a client device MUST have an agent running on it (responding to SNMP queries) before you can define the device in the manager application; there's no way to define the device and then just get traps from it (at least in CE).

    Tried Docker image (FAILED): 
    
docker pull uberchuckie/observium

# See https://hub.docker.com/r/uberchuckie/observium

# paths have to be full paths !
#docker run -d -v /your-config-location:~/projects/observium/config -e TZ="Madrid" -v /path-to-logs:~/projects/observium/logs -v /path-to-rrds:~/projects/observium/rrd -p 8668:8668 uberchuckie/observium
docker run -d -v /your-config-location:/home/user1/projects/observium/config -e TZ="Madrid" -v /path-to-logs:/home/user1/projects/observium/logs -v /path-to-rrds:/home/user1/projects/observium/rrd -p 8668:8668 uberchuckie/observium
docker container list
# Browse to http://localhost:8668
# Log in as observium / observium

# Fail: it just ran and ran, never created subdirs,
# didn't respond to browser.
# Later, the image owner said there are a whole
# lot of undocumented reqts, such as run inside unRAID,
# assumes that there is a "nobody" user with user id 99 and
# group id 100.  I didn't retry.

# When done:
docker stop CONTAINERID

docker images
docker rmi IMAGEID
# if it says there are stopped containers:
docker rm CONTAINERID

    Tried native install: 
    
# See https://docs.observium.org/install_debian/

wget http://www.observium.org/observium_installscript.sh
chmod +x observium_installscript.sh
sudo ./observium_installscript.sh
# choose Community Edition
# Said "no" to installing snapd and agent.
# set MySQL password
# set Observium admin acct name and password
# Observium CE 20.9.10731
sudo systemctl restart apache2
# Browse to http://localhost:8668 or http://127.0.1.1:8668/
# FAIL
# Browse to http://localhost
# Log in; works okay.

# Decided to remove it.
sudo apt remove apache2 php7.4-mysql mysql-server mysql-client rrdtool
sudo rm -fr /opt/observium*
sudo rm /etc/cron.d/observium
sudo deluser observium
sudo delgroup observium



  • MeshCentral:

    MeshCentral
    /r/MeshCentral
    Seems very beta 11/2020.
    Server runs on Linux or Windows.
    Agents available for Linux, Windows


  • Xymon Monitor:

    The Xymon Monitor
    Server runs on Linux.
    Agents available for Linux, Windows.
    Need to have web server (e.g. Apache) installed, and C compiler and make utilities.
    Seems to expect a dedicated server, where you log in as user "xymon" and run a daemon.
    Latest update 2019-09.


  • Linux snmp CLI utilities:

    There is no constantly-running manager command ?
    http://net-snmp.sourceforge.net/wiki/index.php/TUT:MRTG
    https://www.comparitech.com/net-admin/snmpwalk-examples-windows-linux/

    There IS a trap-handler daemon, which will receive messages from clients and maybe send email to notify. First, set up the Linux agent software (see "agents" section). Then: 
    
sudo systemctl start snmptrapd
# Snmptrapd service says:
#		Warning: no access control information configured.
#		This receiver will *NOT* accept any incoming notifications.
man snmptrapd
man snmptrapd.conf

sudo edit /etc/snmp/snmptrapd.conf
# un-comment line "authCommunity log,execute,net public"
sudo systemctl restart snmptrapd
sudo systemctl status snmptrapd
sudo netstat -tulpn

sudo journalctl --unit='snmp*' --pager-end


  • Simple Android manager-type apps, mainly for testing:

    • "Fing - Network Tools" by Fing Limited.

      Under a machine's "Network Details" section, it will show SNMP machine name, contact, location. But in a port scan, it will not show machine's port 161 as open, I think because SNMP is using UDP.

    • "SNMP MIB Browser" by Zoho Corporation.

      When installed, just has one MIB "RFC1213-MIB", stored on phone as three files in "Internal shared storage / mibs" folder. There is a similar file "/var/lib/snmp/mibs/ietf/RFC1213-MIB" on my Linux system. Copied all MIB files from remote device (e.g. /var/lib/snmp/mibs/ from my Linux laptop) to phone, NOT replacing the RFC1213-MIB that was there already.

      When you add a host to the "Host List", the circle to the left of it will turn green if the SNMP connection to it is successfull.

      When polling for an ObjectId value, you can put in an ObjectId such as ".1.3.6.1.2.1.1.1.0", or a name if you've loaded the MIB where that name is defined. Some names and their MIBs are "sysDescr" (SNMPv2-MIB), "sysLocation.0" (SNMPv2-MIB). "hrSystemProcesses.0" (HOST-RESOURCES-MIB). If you put in a name and start polling and get no response at all, probably you have not loaded the MIB needed to translate that name.

      After you've successfully connected to the client system, you can go into "SNMP MIB Browser" and you will find that all the fields have been filled in with the values obtained from the client.



DNSstuff's "Best Linux Network Monitoring Tools for 2020"

All of the above systems seem to be huge overkill, and few of them support Android phones.

SNMP Agents

+/-
  • Linux snmp CLI utilities:

    To get SNMP agent working locally: 
    
# http://net-snmp.sourceforge.net/

sudo apt install snmp
man snmpget
man snmpcmd
man snmpstatus
man snmptrap
man -k snmp

sudo apt install snmp-mibs-downloader
# https://medium.com/@CameronSparr/downloading-installing-common-snmp-mibs-on-ubuntu-af5d02f85425
sudo sed -i 's/mibs :/# mibs :/g' /etc/snmp/snmp.conf	

sudo apt install snmptrapd	# can log incoming SNMP notifications to syslog etc
sudo systemctl status snmpd
sudo systemctl status snmptrapd
sudo systemctl enable snmptrapd
sudo systemctl start snmptrapd
sudo netstat -tulpn

# https://help.ubuntu.com/community/SNMPAgent
# man snmpd.conf
# sudo edit /etc/snmp/snmpd.conf to add:
rocommunity  public default
includeAllDisks 10%		# or e.g. "disk /home %10"
# Note: default frequency for disk-checking is every 10 mins ?
# But I don't see it logging anything, ever.
# Then:
sudo systemctl restart snmpd
sudo systemctl status snmpd

snmpwalk -c public -v1 localhost | less
# notice that lines start with various "DISMAN-EVENT-MIB",
# "SNMPv2-MIB", "IF-MIB", "SNMPv2-SMI", "IP-MIB",
# "HOST-RESOURCES-MIB", more.

snmpdf -v 2c -CH -c public localhost

snmpps -v 2c -c public localhost | less

watch --interval 5 snmpstatus -v 2c -c public localhost

# https://www.debianadmin.com/linux-snmp-oids-for-cpumemory-and-disk-statistics.html
# every 5 seconds show Total RAM free:
watch --interval 5 snmpget -v 2c -c public localhost .1.3.6.1.4.1.2021.4.11.0

snmpdelta -v 2c -Cp 5 -c public localhost .1.3.6.1.4.1.2021.10.1.3.1
# notice that lines start with "UCD-SNMP-MIB"
    To make SNMP agent accessible from network: 
    
# https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-an-snmp-daemon-and-client-on-ubuntu-18-04

# edit /etc/snmp/snmpd.conf to comment out
# "agentAddress  udp:127.0.0.1:161" line and add:
agentAddress udp:161
# or just comment out all the agentAddress lines.
# add lines:    NO !!!
createUser suser1 MD5 SOMETMPPASSWORD DES
rouser suser1 priv

# A different way to do it:    NO !!!
sudo apt install libsnmp-dev
net-snmp-create-v3-user --help

sudo systemctl restart snmpd
sudo systemctl status snmpd

# Run gufw and add rules to allow IN/OUT on ports 161 and 162
# from/to anywhere, with UDP, with IPv4.

# get your LAN IP address (call it THEIPADDRESS):
ip addr | grep 192.

snmpdf -v 2c -CH -c public localhost
snmpdf -v 2c -CH -c public THEIPADDRESS

snmpget -u suser1 -l authPriv -a MD5 -x DES -A SOMETMPPASSWORD -X SOMETMPPASSWORD THEIPADDRESS 1.3.6.1.2.1.1.1.0
# should get machine's "name", which is output of "uname -a"

# Go to manager app, such as Android apps in previous section,
# and try to query values from this system.

# Got stuck at this point, rebooted my Linux system,
# and things started working !  SNMP agent (snmpd) is working.

sudo journalctl --unit='snmp*' --pager-end


  • Android agent apps:

    I don't see a FOSS SNMP agent app for Android. Is SNMP built into Android by default, or just not available ?
    https://github.com/brnunes/Android-SNMP
    https://sourceforge.net/projects/libandroidsnmpsdk/
    https://gist.github.com/issamux/6709513
    http://www.net-snmp.org/
    https://www.javatpoint.com/simple-network-management-protocol
    https://agentpp.com/
    https://www.programcreek.com/java-api-examples/?api=org.snmp4j.Snmp
    https://apkpure.com/snmp-agent-4a/jp.snmp_agent4a
    https://joyent.github.io/node-snmpjs/agent.html

    Android "trap" app that sends to manager when some event happens:
    • "SNMP Trap" by Maildover LLC.
      http://www.maildover.com/
      http://www.maildover.com/joomla/



SNMP manager and client can be on separate parts (Wi-Fi and ethernet) of the same LAN.

Phone can NOT be using VPN-to-internet (at least, strongSwan the way I have it configured), when acting as a manager.

Linux system CAN be using VPN-to-internet (at least, OpenVPN the way I have it configured), when acting as a client.



Mobile Device Managers (MDMs)

+/-
  • Desktop Central:

    Desktop Central
    Server runs on Windows only, I think.
    Agents available for ???


  • Spiceworks "Inventory":

    Spiceworks
    Server runs on Windows.
    Agents available for ???
    Scans by IP address. No mention of mobile devices.


  • Esper:

    Esper
    Server runs on ???
    Agents available for Android only.
    More of a DevOps thing.


  • Miradore:

    Miradore
    Server runs on ??? Not Linux.
    Agents available for all but Linux.


  • Countly:

    Countly
    Server runs on Linux.
    Your app on mobile has to be compiled with their SDK, to talk to server.


  • ITarian:

    ITarian
    Server ("manager") runs on Mac, Windows, Linux.
    Agents available for Android and iOS mobile devices.


  • Relution:

    Relution
    Can use their cloud server (SaaS) up to 5 devices.
    Agents available for Android and iOS mobile devices.


  • AppTec EMM:

    AppTec EMM
    Server runs on ???
    Agents available for Android and iOS mobile and Windows Mobile devices.


  • ManageEngine:

    ManageEngine
    Server runs on Windows, but can use their cloud server (SaaS) up to 25 devices.
    Agents available for Apple, Android, Windows, Chrome.





At this point, I'm thinking of rolling my own solution: An agent on each client that just does an HTTP GET of the manager's web server every hour, passing in some information.
"MacroDroid - Device Automation" by ArloSoft Tools (easy to send IMEI and Battery%, not much else)

Syslog protocol

+/-
Teknikal's_Domain's "Graylog, and the Syslog Protocol, Explained"
Wikipedia's "Syslog"
RSyslog Documentation
Bozho's "The Syslog Hell"

Port 514 on UDP is simplest way. 

# See if rsyslog service is enabled and listening:
systemctl status rsyslog
sudo netstat -tulp

# If not, enable and check again:
less /etc/rsyslog.conf
# Edit /etc/rsyslog.conf to enable UDP-514.  Then:
sudo systemctl restart rsyslog
# Then check again.

echo "<29>mymachine mycmd[2]: hello" > /dev/udp/127.0.0.1/514
sudo grep hello /var/log/syslog

# Apparently, if the message starts with a date-stamp, that stamp
# will be used in syslog; otherwise rsyslog will add its own stamp.

# Change data to start with different leading value to end up in
# different log files.  Or omit leading "<nn>" entirely.

sudo apt install sendip

sudo sendip -p ipv4 -is 127.0.0.1 -p udp -us 514 -ud 514 -d 'Hello' -v 127.0.0.1
sudo grep Hello /var/log/syslog

# In router admin, set a fixed address for your computer.  Then:
sudo sendip -p ipv4 -is 192.168.1.131 -p udp -us 514 -ud 514 -d 'Hello Again' -v 192.168.1.131
sudo grep Hello /var/log/syslog

# Now go to Android phone and install app and send messages from there.
# This app works for testing purposes but mainly is logging
# its own internal failure msgs:
# https://f-droid.org/en/packages/sk.madzik.android.logcatudp/

sudo tail /var/log/syslog
# If firewall is running on your computer, disable it or add exception for port 514.
# Probably have to disable VPN on phone.

# System log will aggregate repeated identical messages into
# one "repeated N times" message, which is a real pain while testing.
# Edit /etc/rsyslog.conf to set "$RepeatedMsgReduction off".

# Various rules in /etc/rsyslog.d/50-default.conf

# Client for Windows:
# https://www.solarwinds.com/free-tools/event-log-forwarder-for-windows
# https://en.freedownloadmanager.org/Windows-PC/syslog-ng-Agent-for-Windows-FREE.html

"snap install graylog"
"Edit /var/snap/graylog/common/server.conf to set the admin password and mongodb connection string. Connection information for elastic search is also needed."




Security Testing and Penetration Testing





Tightening Security



Really, it seems that 95% of the vulnerabilities are eliminated if you just don't run a web server on your machine. Also don't run SSH or FTP or other login-type services, and keep software updated, and you're above 99%.

From older version of Easy Linux tips project's "Security in Linux Mint: an explanation and some tips":
"Don't install Windows emulators such as Wine, PlayOnLinux and CrossOver, or the Mono infrastructure, in your Linux, because they make your Linux partially vulnerable to Windows malware. Mono is present by default in Linux Mint; run 'sudo apt remove mono-runtime-common' to get rid of Mono."
[First run 'sudo apt --simulate remove mono-runtime-common' to see what else you'd lose.]

Ask Ubuntu's "What are PPAs and how do I use them?"
But: "One thing to keep in mind about using PPAs (Personal Package Archives) is that when you add a PPA to your Software Sources, you're giving Administrative access (root) to everyone that can upload to that PPA. Packages in PPAs have access to your entire system as they get installed (just like a regular package from the main Ubuntu Archive), so always decide if you trust a PPA before you add it to your system."

It's a good idea to get CLI mail working, and check it regularly, since various services and packages will send failure or security notices to root's email. See "Getting Linux local CLI mail working" section.


Easy Linux tips project's "Security in Linux Mint: an explanation and some tips"
The Empire's "An Ubuntu Hardening Guide"
lfit's "Linux workstation security checklist"
blakkheim's "Linux Security Hardening and Other Tweaks"
Madaidan's Insecurities' "Linux Hardening Guide"
Mehedi Hasan's "Best Linux Hardening Security Tips: A Comprehensive Checklist"
ArchWiki's "Security"
Maybe likely to break things:
SK's "How To Password Protect GRUB Bootloader In Linux"
[But that doesn't protect against booting from USB drive.]

See Anti-Virus and Malware Scanners section of my "Using Linux" page.

See Application Control and Security section.



Tightening Privacy





Accounts




sudo bash

# To see accounts that can log in:
grep -v ':\*:' /etc/shadow | grep -v ':\!:' | grep -v ':\!\!:' | grep -v ':\!\*:'

# To see accounts that will be shown on the login page:
grep -v '/usr/sbin/nologin$' /etc/passwd | grep -v '/bin/false$' | grep -v '/bin/sync$' | grep -v '^root'

# To see accounts that can do "sudo":
grep sudo /etc/group

# To see failed login attempts:
sudo lastb



My understanding of accounts:

+/-
  • There are "local" accounts (defined in /etc/passwd), but in a corporate environment there could be "network" accounts that are defined in a server somewhere (e.g. LDAP).

  • Generally there is a "local" home directory for each account (named "/home/USERNAME"), but in a corporate environment the home directory could be on some server, and just mounted onto the local mountpoint.

  • Login can occur on the system desktop GUI (display and keyboard and mouse), through system virtual console (TTY1; Ctrl+Alt+F1 to open, Ctrl+Alt+F7 to close), through Telnet/SSH from a network, through remote-desktop software such as VNC or TeamViewer.

  • The account you specified at installation time is a user that belongs to the "sudoers" group and thus can use the "sudo" command to do super-user things. You can login as that user.

  • There is a "root" account, but it is locked so you can't log in as root. No password can be typed which will match the value in root's password hash field.

    BUT: from Easy Linux tips project's "Security in Linux Mint: an explanation and some tips":
    "In Linux Mint 18.3, the root password is unfortunately no longer set by default. This means that a malicious person with physical access to your computer, can simply boot it into Recovery mode. In the recovery menu he can then select to launch a root shell, without having to enter any password. After which your system is fully his. So, set a password for root (preferably identical to your own password)."

  • There may be a "guest" account, and you can log in as that user. [Disabled by default on Mint 19.] There is no password on that account. Can it login through SSH ?

    Definitely disable guest login, and you have to reboot after doing so. In Mint, run "System Settings", go to "Administration" section, click on "Login Window", click "Users" tab, probably everything should be turned off.

    Abhishek Prakash's "How To Disable Guest Account In Ubuntu"
    Ubuntu's "CustomizeGuestSession"

  • If you create more users, you can choose to add each new user to the "sudoers" group or not.

  • If you are logged in as a user who belongs to the "sudoers" group, you can do any administrative operation, and the password to use when "sudo" asks is the password for that user.
    Sudo make me a sandwich

  • I guess it is okay to routinely log in and do all your daily computing as the user you specified during installation, which belongs to the "sudoers" group. Any malware you run accidentally would have to know your password in order to do "sudo" and then do administrative stuff, and I think sudo accepts password only from keyboard. So it's not quite like routinely running as Administrator under Windows.

  • On a single-user system, the security distinction between root and normal user is not so important. All of the interesting personal files probably are owned by the normal user. So if an attacker can get in as that normal user, they get all the good stuff, no need to escalate to root.

    Escalating to root might let the attacker do a few more things, such as access network hardware at a low level to attack other machines on the LAN.

    Escalating to root on a multi-user system is much more serious/important than on a single-user system.

  • From someone on reddit:
    "Normally, all keyrings should get signed into automatically upon user login to the system. If suddenly you have to log in to a keyring when you never did before, what's gone wrong is that you changed the password with passwd instead of using the account manager built into the GUI. The GUI will automatically change your keyring password too. The command line won't."

Ubuntu's "RootSudo"

Some command-line ways to list all users: "getent passwd", "compgen -u", "cat /etc/passwd".

List users with no password set: "sudo awk -F: '($2 == "") {print}' /etc/shadow"

List users with UID set to 0 (superuser): "sudo awk -F: '($3 == "0") {print}' /etc/passwd"

List info about a user: "id USERNAME"

Set limits on users or groups: /etc/security/limits.conf

Login security can be defeated if attacker has physical access:
+/-
Alarming article about (a hole in) account security:
Abhishek Prakash's "How to Reset Ubuntu Password in 2 Minutes" (boot into Recovery mode)
Maybe there is some way to password-protect GRUB, or maybe this doesn't work if /home is encrypted ?
SK's "How To Password Protect GRUB Bootloader In Linux"

Another way to change passwords if you have physical access: boot the machine from a Live system on USB or CD, do "sudo -i", do chroot to the main system disk, do "passwd $username".

Ask Ubuntu's "How do I reset a lost administrative password?" (boot into Recovery mode)
SK's "How To Reset Root User Password In Linux"

Not sure, but I think these methods work even if user's home is encrypted. Access to the disk encryption passphrase is controlled by the user permissions, so once you login as the user (with any or empty password), software can decrypt the user's home.

PAM (Pluggable Authentication Modules):
+/-
Applications (including the "login manager" and sudo) can be written to be PAM-aware. Then the application doesn't have to invent its own authentication mechanism.

The authentication process can be complex and multi-step. But as far as I can tell, all it does is give a yes/no for access to the application; there's no way to have it do fine-grained control of access to permissions within the application ? To do that, you'd have to define separate users with separate permissions ?

Configuration files are in /etc/pam.d directory. Some are for applications, others for events.
"apt list | grep libpam | less"
PAM can limit resource usage via settings in /etc/security/limits.conf
PAM can enforce password strength rules via libpam-cracklib and /etc/pam.d/system-auth
PAM can lock an account after N failed login attempts via pam_tally2 and /etc/pam.d/system-auth
Mokhtar Ebrahim's "Configure and Use Linux-PAM"
Good info in Chapter 23 "Understanding Advanced Linux Security" of "Linux Bible" by Christopher Negus.
Old but some useful stuff: Debian Reference's "Chapter 4. Authentication"

To enable TOTP on desktop logins:
If you're going to enable this, I would save a copy of "/etc/pam.d/lightdm", then create another user account, login to that account, and enable TOTP on that account, to make sure everything works.

Chris Hoffman's "How to Log In To Your Linux Desktop With Google Authenticator"
Daniel Pellarini's "How To Configure Multi-Factor Authentication on Ubuntu 18.04"
nixCraft's "Secure Your Linux Desktop and SSH Login Using Two Factor Google Authenticator"
Linux Uprising's "How To Login With A USB Flash Drive Instead Of A Password On Linux Using pam_usb"

"sudo apt install libpam-google-authenticator".
"man google-authenticator".


Polkit / PolicyKit:
+/-
"a service running on D-Bus that offers clients a way to perform granular system-wide privilege authentication ... Unlike sudo, that switches the user and grants permission to the whole process, polkit delimits distinct actions, categorizes users by group or name, and decides whether the action is allowed or not. This is all offered system-wide, so that D-Bus services can query polkit to know if clients have privileges or not."

Venam's "D-Bus and Polkit"
ArchWiki's "Polkit"

"systemctl status polkit"
"sudo ps -ax | grep polkit | grep -v grep"

XDG Desktop Portals:
+/-
Flatpak's "Portal Documentation"
mclasen's "Flatpak - a look behind the portal"
Snapcraft's "XDG desktop portals"


Types of keys and certificates:
+/-
What is the difference between a certificate and a key-pair ?
A certificate is like a driver's license: some authority says your identity is X.

A key-pair can be used to do a signature (message M came from someone who knows the private key) or an envelope (message N can be read only by someone who knows the private key).



Tweaking sudoers:

+/-
Edit /etc/sudoers file (maybe use "sudo visudo"): 

# For USERNAME, sudo never ask for password:
USERNAME ALL=(ALL) NOPASSWD: ALL

# For USERNAME, for CMDPATH1 and CMDPATH2, sudo never ask for password:
USERNAME ALL=(ALL) NOPASSWD: CMDPATH2 CMDPATH2




  • RSA / SSH public-private key-pair.

    Usually generated by a client, which authenticates to a server some other way and then sends the public key to that server, for future use in authentication.
    Never expire.
    File id_rsa.pub is the public key and file id_rsa is the private key.
    Can these files hold multiple key-pairs, or only one pair ?
    ~/.ssh directory.

  • Identity certificate.

    X.509, usually generated and signed by a trusted authority.
    Has an expiration date.
    Wikipedia's "X.509"
    Personal cert file types .pem .p12 .pfx.
    Authority cert file types .cer .cert .crt. Also .der ?

  • PGP public-private key-pair.

    Never expire.
    ~/.gnupg directory.



From "man ssh":
"The idea is that each user creates a public/private key pair for authentication purposes. The server knows the public key, and only the user knows the private key. ssh implements public key authentication protocol automatically ..." and "A variation on public key authentication is available in the form of certificate authentication: instead of a set of public/private keys, signed certificates are used. This has the advantage that a single trusted certification authority can be used in place of many public/private keys."
Also relevant "man ssh-keygen".

Steve Cope's "SSL and SSL Certificates Explained For Beginners"
Mike Malone's "Everything you should know about certificates and PKI but are too afraid to ask"

Keyring / GnomeKeyring / ksecretservice:
+/-
setevoy's "What is: Linux keyring, gnome-keyring, Secret Service, and D-Bus" (also Arseny Zinchenko (setevoy)'s "What is: Linux keyring, gnome-keyring, Secret Service, and D-Bus")
GNOME Keyring
Keyrings(7) man page
Arch Wiki's "GNOME/Keyring"
Nurdletech's "Gnome Keyring"

There is a Linux kernel keyring (see "man 7 keyrings"), and a GNOME Keyring (GNOME Keyring).

Is integrated with ssh, sftp, scp, PAM, Chrome, chromium. Can be integrated with Git, GnuPG, Firefox.
swick / mozilla-gnome-keyring (extension for Firefox and Thunderbird)
From Gnome Keyring - Security FAQ:
"Gnome Keyring is integrated with PAM, so that the 'login' keyring can be unlocked when the user logs in.".
LZone's "Using Linux keyring secrets from your scripts"

On CLI, do "cat /proc/keys" to see some of the keys in the Linux kernel keyring.
On CLI, do "man keyctl".

GNOME keyring stored under ~/.local/share/keyrings

Apparently Thunderbird has its own internal keyring.

"Passwords and Keys" application (AKA "Seahorse"):
+/-
Accesses GNOME Keyring.
AKA Seahorse

Under Passwords - Logins, it seems to have a bunch of placeholder entries for web sites, and a couple of things for apps (Chrome, Skype). There's nothing (for me) under Certificates (I do have certs installed in FF, Chrome, Thunderbird, but they don't show up here), and under Secure Shell (OpenSSH = ~/.ssh). But there are several keys under PGP Keys (maybe stored under ~/.gnupg directory ?). Hover mouse over each item to see tooltips.

SSH logins:
+/-
Ubuntu's "SSH / OpenSSH / Installing Configuring Testing"
Chris Hoffman's "How to Secure SSH with Google Authenticator's Two-Factor Authentication"
Linuxaria's "Add security to your ssh daemon with PAM module"
nixCraft's "Top 20 OpenSSH Server Best Security Practices"
Carl Tashian's "SSH Agent Explained"

From Ravi Saive's "How to Setup Two-Factor Authentication (Google Authenticator) for SSH Logins":
"Important: The two-factor authentication works with password based SSH login. If you are using any private/public key SSH session, it will ignore two-factor authentication and log you in directly."

SK's "How To Configure SSH Key-based Authentication In Linux"
Alistair Ross's "How To Set Up SSH Keys"
Carla Schroder's "5 SSH Hardening Tips"

Testing your SSH from outside:
InfoByIp's "SSH server connectivity test"
Rebex SSH Check
But really you need to try to connect from an outside machine and see what happens.

Jesus Vigo's "How to join a Linux computer to an Active Directory domain"

Trusted certificate stores:
+/-
Security certificates can be stored in a number of places ?
  • In some browsers. /usr/share/ca-certificates/mozilla/
  • In other mini-browser-containing apps such as Thunderbird email client ? ~/.thunderbird/PROFILENAME/cert9.db and key4.db ?
  • Electron apps contain the Chromium browser engine; what certificate store is used, or none ?
  • Node.js comes with a built-in store of CA's ?
  • Java has its own store ? Also each Java app could have its own ? /etc/ssl/certs/java/cacerts, jre/lib/security/cacerts ? "man keytool" article
  • LDAP or OpenLDAP ?
  • System store used by openssl (/etc/ssl): 
    
# find the base directory:
openssl version -d
# list the certs:
ls -R `openssl version -d | sed -E 's/OPENSSLDIR: "([^"]*)"/\1/'`/cert*
# or maybe
sudo ls -lAR /etc/ssl
# or
sudo find /etc/ssl -name '*.pem' -print | xargs -I{} openssl x509 -subject -noout -in {}
sudo find /etc/ssl -name '*.crt' -print | xargs -I{} openssl x509 -subject -noout -in {}
sudo find /etc/pki -name '*.pem' -print | xargs -I{} openssl x509 -subject -noout -in {}
# but my personal certs (installed in browsers etc) don't show up in there
man update-ca-certificates
ls -l /etc/ssh
ls -lAR ~/.pki
ls -lAR /etc/pki
# while user is logged in:
sudo ls -lAR /run/user/USERIDNUMBER/keyring

From someone on Stack Exchange:
Most distros put their certificates soft-link in system-wide location at /etc/ssl/certs.
  • Key files go into /etc/ssl/private
  • System-provided actual files are located at /usr/share/ca-certificates
  • Custom certificates go into /usr/local/share/ca-certificates
Whenever you put a certificate in one of the above mentioned paths, run update-ca-certificates to update /etc/ssl/certs lists.

From someone on reddit 11/2019:
Applications that utilize the system cert store: Chrome on macOS/windows [but 11/2020 Chrome is changing to have its own store: article]. Safari on macOS. Edge on windows [old Edge, I assume; don't know about new Edge]. Linux support depends on the distribution. RHEL is probably better than others.

Firefox uses its own key store ...

Java applications will vary in support. It really depends on the implementer.

[Certs can be stored in a hardware device:] A Yubikey with certs provisioned acts as a pkcs#11 device which is an industry standard interface to cryptographic devices. It has good support for all applications that utilize the system cert store. There are plugins to utilize pkcs11 devices for Firefox.

Amit N. Bhagat's "Digital Certificates Explained"
Federal Public Key Infrastructure Guides' "Trust Stores"
Venam's "Key And Trust Store on Unix-like OS"

Places passwords are stored:
+/-
GNOME networking passwords are stored in plaintext in files in /etc/NetworkManager/system-connections

MEGA password discussion
MEGAchat: Technical Security Primer

libsecret-based clients via the Freedesktop.org secret storage DBus API ?
KeePassXC 2.5.x can be used as a vault service by libsecret: https://keepassxc.org/blog/2019-10-26-2.5.0-released/ KeePassXC as "secret service"

KeePassXC password manager can supply SSH keys to an SSH agent: KeePassXC and SSH.

Run "ssh-add -l" or "ssh-add -L" to see all keys available through ssh-agent.
Run "ssh-add -s filename.pkcs11" to add a digital certificate to ssh-agent.

"nmap --script ssl-cert localhost" gives me one cert used by port 25 SMTP, called "mint" or "DNS:Mint".

"nmap --script ssl-enum-ciphers localhost" gives me TLS ciphers used by port 25 SMTP, port 631 CUPS.



Security Test / Audit



Lynis
David Mytton's "80+ Linux Monitoring Tools for SysAdmins"
tcpdump:
Daniel Miessler's "A tcpdump Tutorial and Primer with Examples"
"sudo tcpdump -i lo -A | grep Host:"
iptraf
iptop
ntop
netstat: "sudo netstat -atupl"
lsof: "sudo lsof -i" to see established connections.
ss: "sudo ss -lptu".
NixCraft's "ss command: Display Linux TCP / UDP Network/Socket Information"
NixCraft's "Linux: 25 Iptables Netfilter Firewall Examples For New SysAdmins" (see "27. Testing Your Firewall")
nethogs: "sudo nethogs"
ngrep
auditd


CERT's "Intruder Detection Checklist"

See the "Port scanning and router testing" section of my "Computer Security and Privacy" page.

SEI's "Steps for Recovering from a UNIX or NT System Compromise" (PDF)



Miscellaneous



Throttling network bandwidth, for testing purposes:
"sudo tc qdisc add dev enp19s0 root tbf rate 32kbit latency 50ms burst 770"
"sudo tc qdisc delete dev enp19s0 root"

Brendan Gregg's "Linux Performance Tools" diagram