Reading your server logs when something looks wrong

You start with the systemd journal, then drop into the right files under /var/log. Filter by service, time and severity to find signal. Follow a log live while you reproduce the fault. Learn to ignore background internet noise unless you see success or impact.

This picks up from a server you can already reach over SSH. If SSH is down on a Hostworld VPS, you can use the VNC console in Virtualizor from the client area to run the same commands.

Before you start

• Both AlmaLinux 9 and Ubuntu 24.04 use systemd with modern journalctl features, including unit filtering, time ranges, priority filtering and --grep. The commands below work on both.
• On Ubuntu, you might not have text files like /var/log/syslog if rsyslog is not installed. You can always use journalctl.
• On AlmaLinux 9, the systemd journal is not kept across reboots by default. If you care about post-reboot forensics, enable persistence before the next incident. Details below.
• Do not delete or chmod random files under /var/log to free space. Use journal vacuuming and logrotate. Mass deletion can break logging.
• Run commands as a privileged user. Add sudo if you are not root.

Step 1: Start with the journal for the affected service

Filter the systemd journal by unit, severity and a time window. Replace yourservice.service with the actual unit name, for example nginx.service, apache2.service or sshd.service. This shows warnings and errors from the last hour.

AlmaLinux 9:

journalctl -u yourservice.service --since "1 hour ago" -p warning..err

Ubuntu 24.04:

journalctl -u yourservice.service --since "1 hour ago" -p warning..err

Add -f to follow new events as you troubleshoot or restart the service. This keeps the view open and prints new lines as they arrive.

AlmaLinux 9:

journalctl -fu yourservice.service

Ubuntu 24.04:

journalctl -fu yourservice.service

If you need to search for a keyword, use the journal’s own grep so it can paginate and keep context. This finds lines mentioning a path or error string.

AlmaLinux 9:

journalctl -u yourservice.service -g 'permission denied|EACCES'

Ubuntu 24.04:

journalctl -u yourservice.service -g 'permission denied|EACCES'

If someone told you to run journalctl -xe, know what it does. -e jumps to the end. -x adds any explanatory text the system has. It does not filter for errors. Prefer the -p filter when you need errors and warnings only.

Step 2: Check kernel messages for resource issues

Read kernel messages from this boot. This is where out-of-memory kills, disk I/O resets and NIC problems appear. The priority filter narrows it to likely faults.

AlmaLinux 9:

journalctl -k -b -p warning..err

Ubuntu 24.04:

journalctl -k -b -p warning..err

Look for lines like “Out of memory: Killed process …” which indicate the kernel’s OOM killer stopped a service. You can also search for that wording directly.

AlmaLinux 9:

journalctl -k -b -g 'Out of memory|Killed process'

Ubuntu 24.04:

journalctl -k -b -g 'Out of memory|Killed process'

If you prefer a legacy view of kernel messages, you can read them via dmesg as well, though the journal gives better filtering.

AlmaLinux 9:

dmesg --ctime | grep -E 'Out of memory|error|fail'

Ubuntu 24.04:

dmesg --ctime | grep -E 'Out of memory|error|fail'

Step 3: Compare this boot with the previous one

Check whether your problem correlates with a recent reboot or a previous shutdown. Read the current boot, then the previous one. If the previous boot shows nothing on AlmaLinux 9, you likely do not have persistent journals yet. Enable it in Step 7.

AlmaLinux 9:

journalctl -b
journalctl -b -1

Ubuntu 24.04:

journalctl -b
journalctl -b -1

Step 4: Confirm whether updates or package changes happened

Check if someone or something updated software before the fault appeared. On Ubuntu, APT and dpkg logs record this. On AlmaLinux, DNF and RPM logs do. Use a pager to scroll and press q to quit.

Ubuntu 24.04:

less /var/log/apt/history.log
less /var/log/apt/term.log
less /var/log/dpkg.log

AlmaLinux 9:

less /var/log/dnf.log
less /var/log/dnf.rpm.log

Once you find the approximate time of change, inspect the journal around that window for warnings and errors. Adjust the time strings to your case.

AlmaLinux 9:

journalctl --since "2024-08-15 13:00" --until "2024-08-15 14:00" -p warning..err

Ubuntu 24.04:

journalctl --since "2024-08-15 13:00" --until "2024-08-15 14:00" -p warning..err

Step 5: Read authentication and cron logs

Use the right place for SSH and cron. Background SSH scans are normal on an internet-facing VPS. Treat “Failed password” and “Invalid user” as noise unless followed by a success for an account you did not expect to see.

• Authentication

AlmaLinux 9:

tail -F /var/log/secure

Ubuntu 24.04:

tail -F /var/log/auth.log

• Cron

On AlmaLinux 9, cron has its own file. On Ubuntu, cron activity appears in /var/log/syslog unless you configured a separate log. You can also read the journal for the cron service.

AlmaLinux 9:

tail -F /var/log/cron

Ubuntu 24.04:

grep CRON /var/log/syslog | tail -n 50
journalctl -u cron --since "2 hours ago"

If your cron job runs but you do not see output, capture it yourself by redirecting stdout and stderr in the crontab entry. This writes output to your own file so you can confirm what happened.

# in crontab
*/5 * * * * /usr/local/bin/your-script >> /var/log/your-script.log 2>&1

Step 6: Inspect web server logs, if the problem is web-facing

Check access and error logs side by side. Use the access log to see the pattern of client requests and status codes. Use the error log for backtraces, worker exits or permission messages. Correlate 5xx codes in the access log with errors at the same timestamp.

• Apache on Ubuntu

Ubuntu 24.04:

tail -F /var/log/apache2/access.log
tail -F /var/log/apache2/error.log

• NGINX on Ubuntu

Ubuntu 24.04:

tail -F /var/log/nginx/access.log
tail -F /var/log/nginx/error.log

• NGINX on AlmaLinux

Find the log file locations from your NGINX configuration. The access_log and error_log directives define them.

AlmaLinux 9:

grep -R --line-number -E 'access_log|error_log' /etc/nginx

Then follow the files you find.

AlmaLinux 9:

tail -F /path/to/nginx/access.log
tail -F /path/to/nginx/error.log

Normal web noise includes many 404s and probes of /wp-login.php or /xmlrpc.php. Treat those as background unless they coincide with application errors or a spike in 5xx responses.

Step 7: Follow logs live and survive rotation

Keep a live view while you restart or reproduce the fault. The journal viewer handles rotation for you. For flat files, prefer tail -F so the follow survives log rotation.

AlmaLinux 9:

journalctl -fu yourservice.service
tail -F /var/log/messages

Ubuntu 24.04:

journalctl -fu yourservice.service
tail -F /var/log/syslog

If you want to jump to the newest journal entries once and then scroll back, use -e. Add -x if you want any explanatory annotations the system has.

AlmaLinux 9:

journalctl -u yourservice.service -e -x

Ubuntu 24.04:

journalctl -u yourservice.service -e -x

Step 8: Keep enough history and control log size

Decide how much history you need after a reboot, then enforce size limits so logs do not grow without bound.

Make the journal persistent across reboots

By default, logs are volatile unless /var/log/journal exists or you set persistence in journald.conf. Creating the directory enables persistence for future boots. It does not restore past logs.

AlmaLinux 9:

mkdir -p /var/log/journal

Ubuntu 24.04:

mkdir -p /var/log/journal

Check and trim the journal’s disk usage

See how much space the journal uses, then vacuum archived journals by size or time. The active journal is not removed.

AlmaLinux 9:

journalctl --disk-usage
journalctl --vacuum-size=1G
journalctl --vacuum-time=30d

Ubuntu 24.04:

journalctl --disk-usage
journalctl --vacuum-size=1G
journalctl --vacuum-time=30d

Understand logrotate for flat files

Text logs under /var/log rotate based on the configuration in /etc/logrotate.conf and /etc/logrotate.d/. If you need to verify policies, read those files. Change keep/size/time to suit your retention rules.

AlmaLinux 9:

less /etc/logrotate.conf
ls -1 /etc/logrotate.d/

Ubuntu 24.04:

less /etc/logrotate.conf
ls -1 /etc/logrotate.d/

Bad idea to avoid: do not run rm -rf /var/log/*. It will break services that expect specific files and ownership. Use the journal vacuum options and logrotate instead.

Step 9: Know which log answers which question

Ubuntu 24.04

• System services and boot problems: journalctl with unit filters and boot selectors.
• General system messages: /var/log/syslog if rsyslog is installed.
• Authentication and SSH: /var/log/auth.log.
• Kernel messages: journalctl -k or /var/log/kern.log if present.
• Cron: lines tagged CRON in /var/log/syslog, or journalctl -u cron.
• Package changes: /var/log/apt/history.log, /var/log/apt/term.log, /var/log/dpkg.log.
• Firewall with UFW: /var/log/ufw.log.
• Apache: /var/log/apache2/access.log and /var/log/apache2/error.log.
• NGINX: /var/log/nginx/access.log and /var/log/nginx/error.log.

AlmaLinux 9

• System services and boot problems: journalctl with unit filters and boot selectors.
• General system messages: /var/log/messages.
• Authentication and SSH: /var/log/secure.
• Cron: /var/log/cron.
• Mail: /var/log/maillog.
• Kernel messages: journalctl -k.
• Package changes: /var/log/dnf.log and /var/log/dnf.rpm.log.
• NGINX: paths defined by access_log and error_log in NGINX config. Search under /etc/nginx to find them.

Step 10: Decide noise versus incident

Not every scary-looking line is a breach or outage. Use these rules of thumb.

• SSH bots: many “Failed password” or “Invalid user” attempts are normal scanning. Act when you see a successful login you did not expect.
• Web probes: frequent 404s, or POSTs to /wp-login.php or /xmlrpc.php, are common. Treat them as background unless they line up with application errors or an overload.
• Real trouble: kernel OOM killer messages, repeated crash loops in a service’s journal, consistent 5xx errors correlated with backtraces or “worker process exited” lines in web error logs.
• Reproducibility: if you can reproduce the problem and it writes consistent errors to the journal or error logs, you are looking at a real incident.

Step 11: Export what you found and, if needed, get help

If you want to share logs in a ticket, you can print a focused slice with full timestamps. Adjust the unit, time range and priority to match your case.

AlmaLinux 9:

journalctl -u yourservice.service --since "1 hour ago" -p warning..err --output=short-full > /root/yourservice-last-hour.log

Ubuntu 24.04:

journalctl -u yourservice.service --since "1 hour ago" -p warning..err --output=short-full > /root/yourservice-last-hour.log

If you need hands-on help interpreting a specific incident on your Hostworld VPS, please open a support ticket and include the exact commands you ran and their output.

What next

If your logs point to a single service, restart it while following its journal so you can see the result in real time. If you need a broader view of running services and configuration, work through our VPS guides. If you are planning a fresh start with a clean environment, browse our UK locations at Linux VPS.

If the issue persists and you are on a Hostworld VPS, attach your focused logs and findings and open a support ticket. Tell us what changed, when, and what you have already tried. We will pick it up from there.