How to tell if your server has been compromised

You know a server is likely compromised when there are signs of unexpected access, changes you did not make, or processes and services you did not intend to run. The fastest way to tell is to check recent authentication logs, SSH configuration, scheduled tasks and listening network sockets, then correlate what you find in the system journals. Do not power it off. Preserve what the system can tell you before anything is altered.

This picks up from a server you can already reach over SSH.

Before you start

• Do not turn the server off. CISA’s incident response guidance says you should preserve volatile forensic data. Powering off or rebooting can destroy it.
• Plan to preserve evidence. The UK NCSC’s incident management guidance puts log collection and evidence capture at the centre of the technical response.
• Have out‑of‑band access ready if you might need to touch SSH configuration. With Hostworld VPS you can reach the VNC console through Virtualizor from the client area. That protects you from locking yourself out if a change goes wrong.
• Avoid “cleanup” actions for now. Package upgrades, antivirus scans and removal scripts can overwrite artefacts and alter timelines. Gather evidence first.
• Be aware that attackers often tamper with logs. Missing entries or gaps in authentication logs are a signal in themselves.
• Some steps below read system configuration and logs. They do not modify data. Where a step could change state, it is flagged so you can decide when to do it.

Step 1: Stabilise your approach and preserve what you can

Why this matters: rebooting or hasty changes can erase the very clues you are looking for. You want a clear picture of what happened and when.

• Do not reboot or power off.
• As you run the checks below, keep a record of the outputs somewhere you trust.
• If you suspect you will need to adjust SSH later, confirm you can reach the VNC console in Virtualizor first.

Step 2: Review recent authentication activity

Why this matters: most compromises start with a login. SSH and PAM logs show successful and failed attempts, sudo use and account lockouts.

2.1 Check SSH service logs with journalctl

This filters systemd’s journal for SSH activity so you can search a time window for successful and failed logins.

Ubuntu 24.04 (unit name is ssh.service):

sudo journalctl -u ssh.service --since "YYYY-MM-DD HH:MM"

AlmaLinux 9 (unit name is sshd.service):

sudo journalctl -u sshd.service --since "YYYY-MM-DD HH:MM"

You can also restrict to the current boot to focus your search.

Ubuntu 24.04:

sudo journalctl -u ssh.service -b

AlmaLinux 9:

sudo journalctl -u sshd.service -b

On Ubuntu, authentication events are also written to a file you can review. This is useful if you are correlating with other tools.

Ubuntu 24.04 (auth log file):

sudo cat /var/log/auth.log

On AlmaLinux, authentication messages are available in the journal. Where rsyslog is in use, they may also be persisted here:

AlmaLinux 9 (if present):

sudo cat /var/log/secure

Note: attackers often try to sanitise these logs. Gaps and oddities are worth noting.

2.2 Summarise logins and failures

This reads the system’s login databases to show who logged in, who failed to, and when each account last logged in.

Ubuntu 24.04 and AlmaLinux 9 (same commands on both):

# Show successful and recent sessions
last

# Show failed login attempts
sudo lastb

# Show the most recent login per user
lastlog

Look for unexpected users, unknown IP addresses or odd times. Compare with your known access pattern.

2.3 Check account lockouts and failed attempt tallies

This shows per‑user failure counts and lock states tracked by PAM. It helps you spot brute force attempts or accounts locked for repeated failures.

Ubuntu 24.04 and AlmaLinux 9:

sudo faillock --all

On AlmaLinux 9, pam_tally2 is deprecated. pam_faillock is the supported mechanism. On Ubuntu, faillock is provided by libpam-modules-bin. If you see unexpected tallies or locks, note them.

Step 3: Audit SSH configuration for unexpected changes

Why this matters: attackers often loosen SSH policy or add keys. Changes to root login policy, password authentication or included config files are red flags.

Start by reviewing the main SSH daemon configuration and any drop‑ins. Do not change anything yet. Do not restart SSH unless you have console access in Virtualizor and a recovery plan.

Ubuntu 24.04 (main config and drop‑ins):

sudo cat /etc/ssh/sshd_config
sudo ls -al /etc/ssh/sshd_config.d/
sudo cat /etc/ssh/sshd_config.d/*.conf 2>/dev/null

AlmaLinux 9 (main config and drop‑ins):

sudo cat /etc/ssh/sshd_config
sudo ls -al /etc/ssh/sshd_config.d/
sudo cat /etc/ssh/sshd_config.d/*.conf 2>/dev/null

Review keys such as PermitRootLogin and PasswordAuthentication. On RHEL‑family systems, default hardening in RHEL 9 disallows root password login. A new allowance for root passwords or a widened PermitRootLogin is suspicious. Red Hat ships a 50-redhat.conf drop‑in. Site‑specific overrides must use a lower lexical number to take effect, which matters when you assess the effective policy.

Ubuntu’s OpenSSH uses systemd socket activation in newer releases. If you later change Port or ListenAddress, you may need to restart ssh.socket as well as the service. Do not do that now unless you have console access ready, because it can break your session.

Step 4: Inspect authorised keys for new or altered entries

Why this matters: adding an SSH key is a common way to keep access. The AuthorizedKeysFile defaults include ~/.ssh/authorized_keys. Forced‑command options on a key can give persistence too.

Check root and any privileged user accounts for unexpected keys or options.

Ubuntu 24.04 and AlmaLinux 9:

# Root's authorised keys
sudo ls -al /root/.ssh/
sudo cat /root/.ssh/authorized_keys 2>/dev/null

# A named user's authorised keys (replace <user>)
sudo ls -al /home/<user>/.ssh/
sudo cat /home/<user>/.ssh/authorized_keys 2>/dev/null

Look for keys you do not recognise and options such as command=, from= or environment= that you did not set.

Step 5: Look for persistence in systemd timers and cron

Why this matters: scheduled tasks are a common way to run payloads or exfiltrate data regularly.

5.1 List systemd timers

This shows timers, when they last ran and when they are due next. Unfamiliar entries are worth investigating.

Ubuntu 24.04 and AlmaLinux 9:

sudo systemctl list-timers --all

5.2 Review cron jobs

This lists system cron files and per‑user crontabs so you can spot unexpected jobs.

Ubuntu 24.04 and AlmaLinux 9:

# System-wide cron
sudo cat /etc/crontab
sudo ls -al /etc/cron.d/
sudo ls -al /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/

# Root user's crontab
sudo crontab -l -u root

# A named user's crontab (replace <user>)
sudo crontab -l -u <user>

Record anything new or odd. Do not remove entries yet. Capture first.

Step 6: Enumerate listening services and unexpected network endpoints

Why this matters: a process listening where it should not, or connecting out to an odd address, can reveal a compromise.

This lists TCP and UDP sockets that are listening, with ports and owning processes.

Ubuntu 24.04 and AlmaLinux 9:

sudo ss -tulnp

To focus on TCP connections and see processes, run:

Ubuntu 24.04 and AlmaLinux 9:

sudo ss -tpn

If lsof is installed, you can correlate sockets to files and processes in more detail.

Ubuntu 24.04 and AlmaLinux 9 (where lsof is available):

sudo lsof -i -P -n

Investigate any service you did not expect to be present. Use your change history to confirm what should be running.

Step 7: Verify installed packages for unexpected modifications

Why this matters: replacing a binary on disk is a classic technique. Verifying packages against their metadata highlights altered files.

On RPM‑based systems this compares installed files to the RPM database. It does not repair anything. It reports differences.

AlmaLinux 9:

sudo rpm -Va

On Debian‑based systems there are two approaches. dpkg -V checks files where md5sums exist. debsums can compute and compare MD5 sums of installed files.

Ubuntu 24.04:

# Verify files recorded by dpkg
sudo dpkg -V

# Verify package files using debsums where available
sudo debsums -s

Collect any output lines that show changes. They help you decide which parts of the system you can still trust.

Step 8: Reconstruct activity with journalctl time filters

Why this matters: narrowing to a time window helps you connect events, such as a login followed by a new service starting or a cron job running.

This shows all journal entries for a period. Adjust the date and time to the suspected incident window.

Ubuntu 24.04 and AlmaLinux 9:

sudo journalctl --since "YYYY-MM-DD HH:MM" --until "YYYY-MM-DD HH:MM"

You can also combine with unit filtering to focus on one service. For example, review SSH around a specific time.

Ubuntu 24.04:

sudo journalctl -u ssh.service --since "YYYY-MM-DD HH:MM" --until "YYYY-MM-DD HH:MM"

AlmaLinux 9:

sudo journalctl -u sshd.service --since "YYYY-MM-DD HH:MM" --until "YYYY-MM-DD HH:MM"

Step 9: Check Bash history, but treat it as untrusted

Why this matters: shell history can show recent commands a user ran. It is also easy to edit or disable. Treat it as corroboration, not proof.

Bash writes history to a file controlled by variables such as HISTFILE and HISTFILESIZE. Start with root and any privileged users.

Ubuntu 24.04 and AlmaLinux 9:

# Root's Bash history
sudo cat /root/.bash_history 2>/dev/null

# A named user's Bash history (replace <user>)
sudo cat /home/<user>/.bash_history 2>/dev/null

Note anything that aligns with other evidence, such as package installation, user creation or service changes. If history looks too neat, remember it can be sanitised.

Step 10: Make your logging survive a reboot for the future

Why this matters: by default, the systemd journal can be volatile. If the system reboots during an incident, you want logs to persist.

In the journal’s “auto” storage mode, logs persist only if /var/log/journal exists. Check whether that directory is present. Creating it changes system state. Do this later if you are in evidence‑capture mode and document the change.

Ubuntu 24.04 and AlmaLinux 9:

# Check if persistent journal storage exists
sudo ls -ld /var/log/journal

If you decide to enable persistence for future investigations, you can create the directory.

Ubuntu 24.04 and AlmaLinux 9:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

Document the time you made this change. After that point, logs should be retained across reboots.

What next

If your checks point to a compromise, prepare to contain and eradicate safely. The next step is to lock down SSH, rotate credentials and remove any persistence you identified while keeping a record of what you change.

• If you need help with access or you suspect you will lock yourself out, open a support ticket. Tell us which VPS it is and that you may need Virtualizor console access while you make changes.
• If you are evaluating options, our VPS guides cover common administration tasks on AlmaLinux and Ubuntu.
• If you are planning a fresh start on new capacity, see our UK location on the Linux VPS page.

Next in this playbook: contain the incident by tightening SSH access and rotating keys and passwords, then remove scheduled persistence you found. If you are unsure at any point, open a support ticket so we can help you keep access via Virtualizor while you work.