Prometheus and Grafana Installation and Deployment

Overview

I’m familiar with the Prometheus model, so usually some server software and small applications are monitored through Prometheus. Although I often build Prometheus environments, I don’t summarize them, so I need to go to various places to find the configuration (systemd, prometheus, nginx configuration, etc.) every time. Recently, I just had a new environment and needed to rebuild it, so I took the opportunity to document the process of installing and configuring prometheus and grafana.

This build includes installing and deploying prometheus and grafana using Docker, as well as mapping the local disk directory (prometheus data may be quite a bit, so I am using additional mounted disks), and then also includes a reverse proxy for Nginx, and a simple configuration of prometheus, where I will write a simple service discovery service, provided by replacing the configuration file, because my crawl target is an internal container platform, the target ip will often change, so there is a need for a simple dynamic change management.

compose runs prometheus and grafana

install Docker and compose

[root@liqiang.io]# swapoff -a
[root@liqiang.io]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
[root@liqiang.io]# yum install -y docker-ce
[root@liqiang.io]# systemctl start docker
[root@liqiang.io]# systemctl enable docker
[root@liqiang.io]# sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/sbin/docker-compose
[root@liqiang.io]# sudo chmod +x /usr/local/sbin/docker-compose

Environmental Preparation

[root@liqiang.io]# mkdir -p /data/monitoring/{software,configs,grafana,prometheus}
[root@liqiang.io]# mkdir -p /data/monitoring/configs/{grafana,prometheus}
[root@liqiang.io]# cd /data/monitoring/software && tar -xvf prometheus-2.42.0.linux-amd64.tar.gz && ln -s prometheus-2.42.0.linux-amd64 prometheus
[root@liqiang.io]# cp /data/monitoring/software/prometheus/prometheus.yml /data/monitoring/configs/prometheus/prometheus.yml
[root@liqiang.io]# cat > /data/monitoring/configs/grafana/grafana.ini <<EOF
[server]
protocol = http
http_addr = 0.0.0.0
http_port = 3000
domain = grafana.liqiang.io
enforce_domain = true
serve_from_sub_path = true
root_url = http://%(domain)s:%(http_port)s/grafana
EOF

running prometheus and grafana

[root@liqiang.io]# cat <<EOF >/data/monitoring/software/docker-compose.yaml
version: "3.7"
services:
  prometheus:
    image: prom/prometheus:latest
    user: "${UID}:${GID}"
    volumes:
      - /data/monitoring/configs/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - /data/monitoring/prometheus:/prometheus
    command:
      - '--web.enable-lifecycle'
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090
  grafana:
    image: grafana/grafana:latest
    user: "${UID}:${GID}"
    volumes:
      - /data/monitoring/configs/grafana/grafana.ini:/etc/grafana/grafana.ini
      - /data/monitoring/grafana:/var/lib/grafana
    ports:
      - 3000:3000
    links:
      - prometheus
EOF
[root@liqiang.io]# cd /data/monitoring/software/ && docker-compose up -d

At this point, both Prometheus and Grafana have been installed and deployed, and you can access Grafana through http://localhost:3000. However, I used to proxy it with Nginx so that I didn’t have to specify an additional port.

Configuring Nginx

Since I already have Nginx installed, I don’t need to install it again. If you don’t have it installed, the easy option is to run another Nginx with docker-compose, or install one directly with Linux’s package management software. For example, I installed it directly with Yum.

[root@liqiang.io]# yum install -y nginx
[root@liqiang.io]# mkdir -p /etc/nginx/grafana
[root@liqiang.io]# cat> /etc/nginx/grafana/grafana.conf  << EOF
location /grafana {
        proxy_pass              http://127.0.0.1:3000;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        Host $host;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version      1.1;
        proxy_set_header        Upgrade $http_upgrade;
        proxy_set_header        Connection "upgrade";
        proxy_redirect          off;
}
EOF
[root@liqiang.io]# echo "Please add 'include /etc/nginx/grafana/*.conf;' to your nginx.conf"
[root@liqiang.io]# nginx -t
[root@liqiang.io]# nginx -s reload

Configuring Grafana

Open your browser, visit http://grafana.liqiang.io/grafana, and then you will see this screen.

Grafana

The default account password for grafana is

User name: admin
Password: admin

Then you will be asked to change your password, it is best to change it.

Service discovery process

Prometheus supports a very wide variety of service discovery methods, such as the common ones.

kubernetes_sd_config
docker_sd_config
openstack_sd_config

However, these service discovery features are that the URL Path of the Target you need to collect is fixed, and you can only service discovery Endpoint list, which means you can only use to discover the number of containers for a specific service, and you cannot dynamically increase or decrease the type of service.

For example, you have two services, Order and Account, and Prometheus’ service discovery supports you to dynamically change the number of containers for these two services, for example, from 100 to 150. However, if you add a new service: Item, then this service discovery provided by Prometheus is not supported.

So what do we do to dynamically implement service extensions, here I chose to write a program that then dynamically does the service metadata work and then, depending on the actual situation (e.g. the platform where the service is deployed), extend this program to do the dynamic discovery of Endpoint:

Service metadata discovery: this is achieved by replacing the main configuration file of Prometheus and having Prometheus reload the configuration.
Endpiont service discovery: this is implemented through the service discovery interface that comes with Prometheus, since the platform I am using does not support all these API service discoveries, so I choose to implement it in the form of filesd.

Service Metadata Discovery

There are two key points in the implementation of service metadata discovery.

Obtaining service metadata: This step is related to the deployment platform, so it varies from person to person
Constructing the configuration of Prometheus: this step is fixed, once we have determined the service metadata, it can be rendered by the configuration template

Since the code is rather long, I won’t post the code, interested parties can directly open the Github Repo: devops/prometheus/service_discovery to see for themselves~. The practice is to add a service to the end of docker-compose.yaml.

[root@liqiang.io]# cat docker-compose.yaml
... ...
  service-discovery:
    image: liqiangliu/prometheus-service-discovery:0.3.0
    user: "${UID}:${GID}"
    environment:
    - PROMETHEUS_ADDR=http://prometheus:9090
    volumes:
      - /data/monitoring/configs/prometheus:/etc/prometheus
    ports:
      - 5555:5555

Then try to add a service.

[root@liqiang.io]# curl --location --request POST 'http://127.0.0.1:5555/services' --header 'Content-Type: application/json' --data-raw '{
    "name": "prometheus-default",
    "endpoints": [
        {
            "host": "prometheus",
            "port": 9090
        }
    ]
}'

Summary

At this point, our docker compose based prometheus and grafana monitoring system is built, and also supports a simple service discovery feature, the application of this service discovery is very simple, so you can do more extensions yourself is no problem.

Ref

https://liqiang.io/post/install-and-deploy-prometheus-and-grafana

Table of Contents