Every security tool covered elsewhere in this series Fail2Ban, UFW, Suricata, Wazuh — exists to prevent a compromise from happening in the first place. But prevention isn’t foolproof, and knowing how to actually respond to a compromised server matters just as much as the tools that try to stop it from happening.
Signs a Server May Be Compromised
- Unexpected outbound network connections – a server suddenly communicating with unfamiliar external IPs, especially at unusual times
- Unexplained high CPU or resource usage – cryptomining malware in particular tends to consume unusually sustained CPU resources
- New, unrecognized user accounts or SSH keys – accounts you didn’t create, or SSH authorized_keys entries you don’t remember adding
- Modified system binaries or unexpected file changes – flagged directly by file integrity monitoring if you’re running Wazuh ()
- Unusual log entries or gaps – suspicious authentication patterns, or conspicuously missing log entries suggesting logs were tampered with to hide activity
- Unexpected scheduled tasks – new cron jobs or systemd timers you didn’t create, a common persistence mechanism for attackers
Immediate First Steps
Isolate, don’t immediately shut down. Powering off a compromised system destroys volatile evidence (running processes, network connections, memory contents) that could help understand what actually happened. Instead, disconnect the server from the network — physically unplugging the network cable or disabling the interface — while keeping it running for investigation.
Preserving Evidence Before Investigating
Before making changes, capture the current state for later analysis:
ps auxf > /tmp/processes.txt
netstat -tulpn > /tmp/connections.txt
last -a > /tmp/login-history.txtCopy these captures to external storage rather than leaving them only on the potentially compromised system itself.
Checking for Common Persistence Mechanisms
Unexpected cron jobs:
crontab -l
cat /etc/cron.d/*Unfamiliar SSH keys:
cat ~/.ssh/authorized_keysUnexpected systemd services:
systemctl list-units --type=service --state=runningNew or modified user accounts:
cat /etc/passwdReviewing Logs for the Initial Entry Point
Understanding how an attacker got in matters as much as removing them, since without knowing the entry point, the same compromise can simply happen again. Review authentication logs, web server access logs (if applicable), and any application-specific logs around the time suspicious activity first appears:
journalctl -u sshd --since "3 days ago"The Case for Rebuilding Rather Than Cleaning
Once you’re confident a server has actually been compromised (not just showing a false alarm), the generally recommended approach is rebuilding from a known-clean backup or fresh installation rather than attempting to manually remove malicious changes. It’s extremely difficult to be fully confident you’ve found and removed every trace of a sophisticated compromise, while restoring from a backup taken before the compromise occurred provides much higher confidence in a genuinely clean result.
Restoring from Backup Safely
If restoring from a Proxmox Backup Server snapshot () or similar backup, verify the specific backup you’re restoring predates the compromise, not just recent enough to seem convenient — restoring an already-compromised backup accomplishes nothing.
After Restoring: Closing the Gap
Before reconnecting the rebuilt or restored server to your network:
- Apply all pending security updates
- Review and, if uncertain, rotate all passwords and SSH keys for that system
- Address whatever the identified entry point actually was (an unpatched vulnerability, weak credentials, an exposed service that shouldn’t have been reachable)
- Consider whether other systems on your network might have been affected through lateral movement from the compromised machine
Learning from the Incident
Once resolved, review what allowed the compromise to happen and address the underlying gap directly — whether that’s SSH hardening you’d been putting off, a firewall rule that was more permissive than necessary, or simply patches that had gone unapplied for too long. The specific tools covered throughout this series (Lynis for auditing, Wazuh for monitoring, UFW and Fail2Ban for prevention) exist specifically to catch the kind of gaps that lead to exactly this scenario.
Final Thoughts
Responding effectively to a compromised server means isolating quickly without destroying evidence, understanding how the compromise actually happened, and generally rebuilding rather than attempting incomplete manual cleanup. While every preventive tool covered throughout this series aims to avoid ever needing this guide, having a clear response plan ready before an incident happens makes an already stressful situation considerably more manageable.
Leave a Reply