m4nifest0-tech

Search the site

Linux and Windows Server

Monitoring infrastructure with Prometheus and Grafana in an LXC container

How to collect metrics from Linux and Proxmox VE nodes with Prometheus, visualize them in Grafana and get alerts, in a dedicated unprivileged LXC.
Published on 8 min readLevel: Intermediate

An unmonitored infrastructure reports problems when they have already become outages: a full disk, a powered-off node, a backup stalled for days. Prometheus and Grafana let you collect metrics, visualize them and get an alert before the failure spreads. This guide describes a lab installation on Debian 12 or 13, in an unprivileged LXC container on Proxmox VE 8 or 9, that observes Linux nodes, Proxmox hosts and, if you want, Windows servers.

Architecture

There are four components, each with a specific job.

  • Prometheus polls targets over HTTP at regular intervals (pull model), stores the time series and evaluates alert rules.
  • Exporters are small services that expose metrics on an HTTP endpoint: node_exporter for Linux, windows_exporter for Windows, prometheus-pve-exporter for Proxmox VE.
  • Grafana collects nothing: it reads from Prometheus and draws the dashboards.
  • Alertmanager receives the alerts generated by Prometheus, groups them and forwards them to the configured channels, such as email, webhooks or chat.
exporter (nodes, Proxmox VE)  ->  Prometheus  ->  Alertmanager  ->  notifications
                                     ^
                                     |
                                  Grafana

The dedicated LXC container

A separate container isolates monitoring from the services it observes and can be backed up, moved and restored as a single unit. The load is modest and no component needs privileges on the host, so an unprivileged LXC is the natural choice: the container's root does not correspond to the host's root, as explained in unprivileged LXC, bind mounts and UID mapping. To keep the metrics history beyond the life of the container, you can set aside a dedicated volume, or a bind mount, for the data directory.

On the Proxmox node
pveam update
pveam available --section system
pveam download local <NOME_TEMPLATE_DEBIAN>
On the Proxmox node
pct create <CTID> local:vztmpl/<NOME_TEMPLATE_DEBIAN> \
  --hostname monitoraggio \
  --unprivileged 1 \
  --cores 2 --memory 2048 \
  --rootfs local-lvm:16 \
  --net0 name=eth0,bridge=vmbr0,ip=192.0.2.20/24,gw=192.0.2.1 \
  --onboot 1
pct start <CTID>

Note

Template and storage names (local, local-lvm) vary from one installation to another: check them with pveam available and pvesm status before running the commands. Addresses and sizes are only examples.

The container belongs in the management network: segmentation with VLANs helps keep only the necessary ports open toward the exporters.

Prometheus and node_exporter from Debian packages

In the container, as root
apt update
apt install prometheus prometheus-node-exporter prometheus-alertmanager

Debian packages are updated along with the rest of the system but follow the version of the stable release. For newer features you can use the official binaries, handling updates by hand.

prometheus-node-exporter must also be installed on every Linux host to be observed, limiting port 9100 to the address of the monitoring container only. Targets are declared in scrape_configs.

/etc/prometheus/prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s
 
rule_files:
  - /etc/prometheus/rules/*.yml
 
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]
 
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
 
  - job_name: "nodi-linux"
    static_configs:
      - targets:
          - "nodo1.example.com:9100"
          - "nodo2.example.com:9100"
          - "nodo3.example.com:9100"
 
  - job_name: "proxmox"
    metrics_path: /pve
    params:
      module: ["default"]
      cluster: ["1"]
      node: ["1"]
    static_configs:
      - targets:
          - "nodo1.example.com"
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: "127.0.0.1:9221"

The proxmox job uses relabeling: the targets are the Proxmox nodes, but requests go to the exporter listening locally, which in turn queries the node through its API. Windows servers, such as the domain controller in the Active Directory lab, are observed with windows_exporter (port 9182 by default) by adding a job similar to nodi-linux.

Check the configuration before applying it:

In the container, as root
promtool check config /etc/prometheus/prometheus.yml
systemctl reload prometheus

Exporter for Proxmox VE and least privilege

The exporter must read the state of the cluster, not modify it. The principle of least privilege translates into three choices: a dedicated user, the predefined PVEAuditor role (read-only) and an API token with separate permissions, revocable without touching anything else.

On the Proxmox node
pveum user add monitoraggio@pve --comment "Read-only for Prometheus"
pveum acl modify / --users monitoraggio@pve --roles PVEAuditor
pveum user token add monitoraggio@pve prometheus --privsep 1
pveum acl modify / --tokens 'monitoraggio@pve!prometheus' --roles PVEAuditor

Important

The token value is shown only once, at creation. Store it in a secrets manager and write it only in the exporter's configuration file, with restrictive permissions and outside any repository.

The prometheus-pve-exporter package is installed with pip. On Debian 12 and later a virtual environment is the way to go, because system-wide installation is blocked:

In the container, as root
apt install python3-venv
python3 -m venv /opt/pve-exporter
/opt/pve-exporter/bin/pip install prometheus-pve-exporter

The exporter is started with pve_exporter --config.file /etc/prometheus/pve.yml, listens on port 9221 and, for stable use, should run as a systemd service under an unprivileged user. The format of the configuration file is described in the project documentation (opens in a new tab) (opens in a new tab).

Grafana from the official APT repository

Grafana is installed from the apt.grafana.com repository, verified with the GPG key published by the project. The commands follow the official documentation at the time of writing and may change: before running them, compare them with the installation guide for Debian (opens in a new tab) (opens in a new tab).

In the container, as root
apt install wget
mkdir -p /etc/apt/keyrings
wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" > /etc/apt/sources.list.d/grafana.list
apt update
apt install grafana
systemctl enable --now grafana-server

Grafana listens on port 3000. At first login, the default administrator password must be replaced. Then add Prometheus as a data source with the address http://localhost:9090 and import community dashboards for node_exporter and for Proxmox, adapting them to your needs.

Alert rules

Rules live in separate files, loaded through rule_files. Two examples: a target unreachable for five minutes and a filesystem above a threshold.

/etc/prometheus/rules/base.yml
groups:
  - name: base
    rules:
      - alert: TargetUnreachable
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is not responding"
          description: "The job {{ $labels.job }} has not responded for over 5 minutes."
 
      - alert: DiskAlmostFull
        expr: |
          1 - node_filesystem_avail_bytes{fstype=~"ext4|xfs|btrfs|zfs"}
            / node_filesystem_size_bytes{fstype=~"ext4|xfs|btrfs|zfs"} > 0.85
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Filesystem {{ $labels.mountpoint }} on {{ $labels.instance }} is over the threshold"
          description: "Used space exceeds 85%. The threshold is indicative."
In the container, as root
mkdir -p /etc/prometheus/rules
promtool check rules /etc/prometheus/rules/base.yml
systemctl reload prometheus

Notification recipients are defined in /etc/prometheus/alertmanager.yml. Channel credentials (SMTP, webhooks, chat) are secrets like any other and stay out of the repository.

What to monitor first

  • Availability. The up metric tells you whether collection succeeds: it is the first alert to enable.
  • Disk space. Full filesystems are a frequent cause of failures. Besides node_filesystem_*, the Proxmox exporter exposes storage usage.
  • SMART. node_exporter does not read SMART data natively: you need a dedicated exporter or a script for the textfile collector, run on the physical host because a container cannot see the disks.
  • Temperatures. hwmon sensors are exposed as node_hwmon_temp_celsius and must be read on the physical node.
  • Backup status. A backup that doesn't start produces no errors: it is best to measure the age of the last successful backup and alert when it exceeds the expected interval. The context is in the 3-2-1 backup strategy with Proxmox Backup Server.

Warning

Prometheus, Alertmanager and the exporters have no authentication by default, and the metrics reveal the structure of the infrastructure. None of these services, Grafana included, should be exposed directly to the Internet: external access goes through a VPN, for example WireGuard, or through a reverse proxy with TLS and authentication. If Prometheus and Grafana share the container, Prometheus can listen only locally with ARGS="--web.listen-address=127.0.0.1:9090" in /etc/default/prometheus.

Related articles