Rootless Promtail with Ansible Enhancing Security and Efficiency

The Ansible Promtail Role allows you to effortlessly deploy and manage Promtail, agent which ships contents of local logs to private Loki. Role is tailored for operating systems such as RedHat, Rocky Linux, AlmaLinux, Ubuntu, and Debian. Before diving into the details, I highly recommend reading the role's README.md file for a comprehensive understanding.

👀 README.md - promtail Role

This blog post is primarily focused on explaining the rationale behind constructing this role in a specific manner and highlighting the importance of not running Promtail as root.

What is Promtail ?

Promtail is a component of the Loki logging stack, designed to collect log data from applications running on a system. Acting as a log forwarder, tailing log files or accessing system journals, and sends the log entries to a central Loki server for storage and analysis. Promtail is highly efficient, leveraging the power of labels and service discovery from Prometheus, making it easy to configure and integrate with existing monitoring setups.

Avoid Running Promtail as root

Why avoid running Promtail as root? While it may seem easier to configure with elevated privileges, doing so introduces significant security risks. Granting unnecessary root access increases the attack surface, making your system vulnerable to potential exploitation.

On the other hand, opting for a rootless approach ensures better security and adheres to modern best practices. This doesn't imply that running Promtail in root mode will inevitably lead to exploitation. However, it underscores the importance of adopting a mindset focused on configuring your system with caution. Each application should only be granted the permissions it actually needs.

note

Role supports both root and rootless modes, controlled by the promtail_runtime_mode variable. The decision of which mode to use is up to you. It's important to note that by default, ACL (rootless) mode is enabled.

Variable	Supported Mode	Description
promtail_runtime_mode	acl	Rootless mode
promtail_runtime_mode	root	Root mode

How can Promtail access system logs without root ?

Utilizing setcap

Setcap command-line utility is powerful that allow to set specific capabilities for executable files, granting them certain privileges without the need for full root access. For instance, running the command setcap 'cap_dac_read_search=+ep' /usr/bin/promtail adds the capability to read and search directories and files to the /usr/bin/promtail executable. As a result, Promtail gains the ability to perform these actions without relying on full root privileges.

However, there's a significant problem with this approach. Granting such capabilities to Promtail can potentially give it access to sensitive files like /etc/shadow, which are typically restricted to the root user. This introduces unnecessary risks and provides more permissions than needed for Promtail to fulfill its intended purpose.

Due to these security concerns, I have not implemented this setcap approach.

Utilizing ACL

Linux Access Control Lists (ACLs) provide an effective way to manage Promtail's access to specific logs. The Promtail ACL permission needs to be granted. I have simplified the configuration, removing the need for users to provide any additional configurations. Let's consider the following example for the scrape configuration:

promtail_scrape_configs variable example

promtail_scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: secure
          instance: "{{ ansible_fqdn }}"
          __path__: /var/log/secure
      - targets:
          - localhost
        labels:
          job: messages
          instance: "{{ ansible_fqdn }}"
          __path__: /var/log/messages
      - targets:
          - localhost
        labels:
          job: audit
          instance: "{{ ansible_fqdn }}"
          __path__: /var/log/audit/audit.log
      - targets:
          - localhost
        labels:
          job: nginx
          instance: "{{ ansible_fqdn }}"
          __path__: /var/log/nginx/*.log

In the provided configuration, the Promtail role will extract all existing files and directories from __path__ and subsequently utilize the ansible.posix.acl Ansible module to establish correct ACL permissions on all existing directories and files.

Additionally, I cannot predict which logs you want to configure, so I've decided to avoid modifying the system logrotate file located at /etc/logrotate.d/rsyslog. To overcome some limitations of logrotate, this role creates a dummy directory called /var/log/dummy_promtail_acl, along with an empty log file within it.

The following configuration is generated based on the provided variable promtail_scrape_configs:

/etc/logrotate.d/promtail_acl

/var/log/dummy_promtail_acl/dummy_promtail_acl.log
{
    copytruncate
    postrotate
        /usr/bin/setfacl -R -m d:u:promtail:rx /var/log
        /usr/bin/setfacl -m u:promtail:rx /var/log
        /usr/bin/setfacl -R -m d:u:promtail:rx /var/log/audit
        /usr/bin/setfacl -m u:promtail:rx /var/log/audit
        /usr/bin/setfacl -R -m d:u:promtail:rx /var/log/nginx
        /usr/bin/setfacl -m u:promtail:rx /var/log/nginx
        /usr/bin/setfacl -m u:promtail:r /var/log/messages
        /usr/bin/setfacl -m u:promtail:r /var/log/secure
        /usr/bin/setfacl -m u:promtail:r /var/log/audit/audit.log
        /usr/bin/setfacl -m u:promtail:r /var/log/nginx/*.log
    endscript
}

This configuration ensures that the permissions for Promtail are maintained after log rotation.

tip

Role in rootless/root mode supports also glob pattern configuration, such as /var/log/nginx/*.log

---

Utilizing ACLs and carefully managing access to log files enhances system security. This ensures that Promtail has access to only required log files.

👀 GitHub Source Code - Ansible Promtail Role

---

Thanks for reading. I'm entering the void. 🛸 ➡️ 🕳️