Processes, Services and System Logs

A computer is never idle. Behind the screen, dozens or hundreds of programs run at the same time, and many of them listen for connections, so they are doors to your system. Knowing what is running, being able to start and stop services, and reading the logs that record everything are daily skills for both administrators and defenders. In this lesson you will learn them on your lab server.

The previous lesson explained who owns what. Now we look at who is doing what, right now, and how to find the trail of what happened earlier.

Processes

A process is a running program. Each has a number, the PID (process identifier), an owner, a parent process, and consumes some CPU and memory. When you type a command, the shell starts a process for it. The first process of the system is the one that starts all the others; every other process descends from it.

Useful commands to see the processes:

ps aux              # all processes, with owner, CPU and memory
ps aux | grep ssh   # look for a particular process
top                 # live view (press q to quit)
pstree              # tree view of parent-child relations
kill 1234           # politely ask process 1234 to stop
kill -9 1234        # force it to stop (last resort)

In the output of ps aux, the first column is the user running the process. Looking at this column is instructive: system programs run as root, and services such as a web server run under their own limited account. If you spot an unexpected program running as root, or a process whose name looks like a system program but runs from /tmp, that deserves a look. Defenders and forensics analysts pay close attention to such oddities.

Processes can run in the foreground (they block the terminal) or the background (add & at the end of the command). Ctrl+C interrupts the foreground process, and jobs shows your background tasks.

What is a service?

A service (also called a daemon) is a process that starts at boot and runs in the background, waiting to do something: a web server waiting for requests, an SSH server waiting for logins, a scheduler running jobs. Services are the network-facing part of a system, which is why the list of active services is part of the "attack surface" of a machine.

On most modern Linux distributions, services are managed by systemd, the system and service manager. It starts services at boot in the correct order, restarts those that fail, and keeps their logs. The main command is systemctl:

systemctl status ssh           # state of a service
sudo systemctl start ssh       # start now
sudo systemctl stop ssh        # stop now
sudo systemctl restart ssh     # stop then start
sudo systemctl enable ssh      # start automatically at boot
sudo systemctl disable ssh     # do not start at boot
systemctl list-units --type=service --state=running

The name of the SSH service varies between distributions (ssh or sshd). The status output shows whether the service is active, its PID, and the last lines of its log. Each service is described by a unit file, a small text file that tells systemd what to run and under which account. You will find them under /etc/systemd/system and /lib/systemd/system. Unit files are also a place where persistence mechanisms hide, and defenders check them.

Services listen on ports, as we saw in chapter 1. To see which service uses which port on your lab server:

sudo ss -tulpn

The options display TCP and UDP sockets that are listening, with numeric ports and the process that owns each. Compare this list with what you expect. A good hardening habit is: if you do not need a service, stop it and disable it. Fewer services mean fewer doors.

Logs: the memory of the system

Every serious event leaves a trace: logins, failed logins, service starts and stops, errors, sudo commands, network changes. Collectively, these traces are the logs. For a defender, they are the first resource when something looks wrong. For a penetration tester, they are a reminder that actions leave marks, and that a good report can show what the client's monitoring did or did not notice.

Logs live in two places that you should know:

1. Text files in /var/log, written by classic logging services. Common ones: syslog or messages (general events), auth.log or secure (authentication and sudo events, depending on the distribution), kern.log (kernel), and directories for individual services such as the web server.
2. The systemd journal, a binary log queried with journalctl. Some recent systems rely mostly on the journal and may not create all the classic text files.

ls /var/log
sudo tail -n 20 /var/log/auth.log
sudo journalctl -n 20
sudo journalctl -u ssh --since "1 hour ago"
sudo journalctl -p err -b

The options of journalctl: -n limits the number of lines, -u filters by service, --since by time, -p err shows only errors and worse, and -b restricts to the current boot. Combine them with grep to look for a word: sudo journalctl -u ssh | grep -i failed.

Each log line typically has a date, a host name, the program that wrote it, and a message. Learn to read one:

Jan  5 10:15:02 lab-server sshd[1042]: Failed password for invalid user admin from 10.10.10.10 port 52114 ssh2

Here the SSH service reports a failed login attempt for an account that does not exist, coming from the address 10.10.10.10. Many such lines in a short time, from the same source, would suggest a password-guessing attempt. A real system on the Internet sees this kind of noise constantly. Reading authentication logs is one of the most practical defensive skills you can build.

Log rotation, protection and the limits of logs

Logs grow, so the system rotates them: old logs are compressed and eventually deleted. For investigations, this means you need to know how long your logs are kept. In companies, logs are usually forwarded to a central server, so that an intruder who erases local logs cannot erase the central copy. Remember also that logs are only as trustworthy as their protection: if an attacker obtains root rights, they can alter local logs. That is why defenders centralise them, a subject we return to in later courses.

Finally, logs can contain personal data (user names, addresses). Treat them as sensitive and respect retention rules.

Your turn: watch a login attempt appear

On your lab server, make sure the SSH service is installed and running (sudo apt install openssh-server if needed, then systemctl status ssh). From the Kali VM, in your own isolated lab, try to connect with ssh student@10.10.10.20 and enter a wrong password once on purpose, then the right one. On the server, follow the authentication log with sudo journalctl -u ssh -f (or sudo tail -f /var/log/auth.log) while you do this. In your notebook, write down the line that shows the failure and the line that shows the success, the source address, and the user name. Finally, run sudo ss -tulpn and say on which port SSH listens.

See a solution

You should see a line similar to "Failed password for student from 10.10.10.10 port ..." and then "Accepted password for student from 10.10.10.10 port ...". The source address is the Kali VM, which makes sense since you connected from it. The ss output shows sshd listening on port 22, usually on all addresses (0.0.0.0:22 and [::]:22). If you do not see any lines in the authentication log, your system may log to the journal only: use journalctl. Once finished, if you do not need SSH, you can stop and disable it: sudo systemctl disable --now ssh. The point of the exercise is to see with your own eyes that every attempt, successful or not, leaves a trace.