Problems faced
- Servers kept unreviewed default settings: unnecessary services running and administrative access that was too permissive.
- Logs stayed on individual hosts and were lost in case of failure, reboot, or tampering.
- Repeated login attempts were neither detected nor limited.
- There was no verifiable baseline: every server was configured by hand, with differences that were hard to spot.
Implemented solution
I defined a hardening baseline applied uniformly to Debian servers: SSH access with keys only, administrative privileges limited to a group, unnecessary services disabled, and automatic security updates. An nftables firewall with a restrictive inbound policy and fail2ban against repeated attempts complete the host-level defense. journald logs are forwarded with rsyslog, over an encrypted channel, to a central collector with defined retention and rotation. Prometheus monitors the state of the services and of the log flow and raises an alert when a host stops sending events.
Results
- Servers share a consistent, verifiable base configuration, with fewer differences from one host to another.
- Security events can be reviewed in one place, even after the host that generated them has rebooted or failed.
- Repeated login attempts are slowed down automatically.
- The absence of logs from a host raises an alert, flagging failures or possible tampering.
Technical description
Hardening reduces the attack surface of individual servers; centralized logs make it possible to notice what is happening and to reconstruct it after an incident. This project combines the two activities into a single baseline, applied uniformly and verifiably. By choice, the security settings are not reported in detail.
Hardening baseline
The baseline covers Debian servers and consists of a few principles:
- administrative access via SSH with keys only, no direct root login, and limited to a dedicated group;
- unnecessary services disabled and listening ports reduced to those needed;
- an nftables firewall with a restrictive default inbound policy and only documented exceptions;
- fail2ban against repeated login attempts, excluding the management segment so the administrator does not get locked out;
- automatic security updates and synchronized clocks, an essential requirement for correlating events.
A checklist for SSH is in the article SSH hardening on Debian: a practical checklist. The baseline is described as code, with Ansible roles, and applied to new servers right when they are created: deviations become visible differences between versions.
Caution
Before restricting SSH or the firewall, keep an already authenticated administrative session open and verify access from a second session: a mistake can lock you out of remote access to the server.
Centralized logs
On every server, journald stores the system logs persistently and makes them available to rsyslog, which forwards them to a central collector over a TLS-encrypted channel. The collector archives them per host, applies rotation and retention, and makes them available even if the originating host is down or compromised.
action(type="omfwd" target="logs.example.com" port="6514" protocol="tcp"
StreamDriver="gtls" StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="logs.example.com"
action.resumeRetryCount="-1"
queue.type="LinkedList" queue.filename="inoltro" queue.saveOnShutdown="on")The on-disk queue prevents losses during collector outages. Prometheus collects metrics on the state of the services and of the log flow: an alert fires when a host stops sending events, a sign of a failure or tampering. A reference is Prometheus and Grafana in LXC.
Logs contain sensitive data: the collector is therefore reachable only from the management segment, and data is kept for the strictly necessary time. Synchronized clocks make it possible to put the events of different hosts on a single timeline.
Testing and acceptance
The baseline is checked with repeatable tests: listing the listening ports, testing access with and without keys, simulating failed logins to verify that fail2ban steps in, sending a test event all the way to the collector, and checking the alert when forwarding is interrupted.
Possible next steps
System auditing with auditd, detection of changes to critical files, automated log analysis, and periodic compliance checks against a reference baseline.