Global SSH Logs View with Loki
Do you want to monitor all open or failed SSH connections across your Linux systems in one consolidated view? This simple dashboard provides quick stats for all your machines. It allows you to quickly identify instances with suspicious activity. The setup uses Promtail to collect SSH log lines, which are then sent to Loki. Grafana visualizes this data in an easy-to-read dashboard.
This dashboard provides a quick overview with per-host counts for:
- Number of failed SSH connections
- Number of failed SSH connections by unique IPs
- Number of failed SSH connections by unique users
- Number of open SSH connections
- Number of open SSH connections by unique IPs
- Number of open SSH connections by unique users
Dashboard Preview
Apologies if the images are too small. You may need to zoom in or open them in a new tab.
It’s important to note that this dashboard is not intended to replace the existing SSH Logs - 17514 dashboard. The new dashboard provides a global overview, while the SSH Logs - 17514 dashboard offers detailed information about specific IPs and users for both open and failed connections. If you need to check these details for a specific host, the 17514 dashboard is more suitable. You can also find more details about this host-specific SSH logs dashboard in my other blog post: Parsing SSH Logs with Grafana Loki.
What You Need
To deploy and use this dashboard, you'll need the following:
| What do you need? | Description |
|---|---|
| Grafana | Grafana is an open-source interactive data-visualization platform |
| Grafana Loki | Log aggregation system inspired by Prometheus |
| Grafana Promtail |
| Tested on | Description |
|---|---|
| RedHat Enterprise Linux (RHEL) 8 and 9 | It should work on other RedHat Family systems as well |
| Grafana v11.1.4+ | Tested with Grafana version |
| Loki/Promtail v3+ | Tested with Loki version |
SyslogFacility AUTHPRIV | sshd_config parameter |
LogLevel INFO | sshd_config parameter |
/var/log/secure | Consumed log |
Promtail Configuration
This dashboard requires a single label to be defined consistently across your setup. You need to set the label instance, which should represent your host. While some might prefer using labels like hostname or host, I recommend using instance because it aligns with Prometheus conventions, making it more logical in this context. If you choose a different label, you’ll need to modify all occurrences of instance in the dashboard to match your chosen label.
Here is a sample configuration for Promtail scrape config:
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
instance: intelnuc.local # Label identifier for the instance (hostname -f)
env: dev.voidquark.com
job: secure # Job label
__path__: /var/log/secure
For a detailed explanation of Promtail configuration, please refer to the 📚 Promtail Configuration Documentation.
Dashboard labels
It's important to understand how labels work with this dashboard, as they determine what you see in the results. I’ll use an example to clarify the concept. If you want to view data for a specific environment, like a development environment, simply use the env label. Additionally, my SSH logs are labeled with job="secure" because they originate from the /var/log/secure log file. When I apply these filters, the dashboard displays results for all hosts in the development environment.
This approach allows you to easily filter the global view based on your needs.
Next steps
Now, it's time to import the dashboard. There are multiple ways to do this in Grafana. I prefer the automated provisioning method, but if you prefer manual steps, you can follow this guide on importing dashboard. Once your dashboard is imported and you’ve selected the Loki datasource, everything should be set up.
Parsing logs and configuring panels are not covered here because they are already explained in Parsing SSH Logs with Grafana Loki. I’ve only made slight modifications to the query to group results by instance. If you're curious about how it works, simply refer to that blog post.
Source code available for everyone
Securing SSH is your responsibility. Strongly consider keeping SSH access private and avoid exposing it to the public.
If you like this dashboard, I’d love to hear your feedback! Is there something missing that you’d like to see included? Feel free to contact me.