I’m familiar with the Prometheus model, so usually some server software and small applications are monitored through Prometheus. Although I often build Prometheus environments, I don’t summarize them, so I need to go to various places to find the configuration (systemd, prometheus, nginx configuration, etc.) every time. Recently, I just had a new environment and needed to rebuild it, so I took the opportunity to document the process of installing and configuring prometheus and grafana.
This build includes installing and deploying prometheus and grafana using Docker, as well as mapping the local disk directory (prometheus data may be quite a bit, so I am using additional mounted disks), and then also includes a reverse proxy for Nginx, and a simple configuration of prometheus, where I will write a simple service discovery service, provided by replacing the configuration file, because my crawl target is an internal container platform, the target ip will often change, so there is a need for a simple dynamic change management.
compose runs prometheus and grafana
install Docker and compose
running prometheus and grafana
At this point, both Prometheus and Grafana have been installed and deployed, and you can access Grafana through
http://localhost:3000. However, I used to proxy it with Nginx so that I didn’t have to specify an additional port.
Since I already have Nginx installed, I don’t need to install it again. If you don’t have it installed, the easy option is to run another Nginx with docker-compose, or install one directly with Linux’s package management software. For example, I installed it directly with Yum.
Open your browser, visit
http://grafana.liqiang.io/grafana, and then you will see this screen.
The default account password for grafana is
- User name: admin
- Password: admin
Then you will be asked to change your password, it is best to change it.
Service discovery process
Prometheus supports a very wide variety of service discovery methods, such as the common ones.
However, these service discovery features are that the URL Path of the Target you need to collect is fixed, and you can only service discovery Endpoint list, which means you can only use to discover the number of containers for a specific service, and you cannot dynamically increase or decrease the type of service.
For example, you have two services, Order and Account, and Prometheus’ service discovery supports you to dynamically change the number of containers for these two services, for example, from 100 to 150. However, if you add a new service: Item, then this service discovery provided by Prometheus is not supported.
So what do we do to dynamically implement service extensions, here I chose to write a program that then dynamically does the service metadata work and then, depending on the actual situation (e.g. the platform where the service is deployed), extend this program to do the dynamic discovery of Endpoint:
- Service metadata discovery: this is achieved by replacing the main configuration file of Prometheus and having Prometheus reload the configuration.
- Endpiont service discovery: this is implemented through the service discovery interface that comes with Prometheus, since the platform I am using does not support all these API service discoveries, so I choose to implement it in the form of filesd.
Service Metadata Discovery
There are two key points in the implementation of service metadata discovery.
- Obtaining service metadata: This step is related to the deployment platform, so it varies from person to person
- Constructing the configuration of Prometheus: this step is fixed, once we have determined the service metadata, it can be rendered by the configuration template
Since the code is rather long, I won’t post the code, interested parties can directly open the Github Repo: devops/prometheus/service_discovery to see for themselves~. The practice is to add a service to the end of docker-compose.yaml.
Then try to add a service.
At this point, our docker compose based prometheus and grafana monitoring system is built, and also supports a simple service discovery feature, the application of this service discovery is very simple, so you can do more extensions yourself is no problem.