The Prometheus community is updating so fast that some of the documentation written before is a bit out of date. Recently, I started to focus on observability again, and make up for some knowledge points in operation and maintenance.
1. Explanation of terms
A visualization tool that provides various visualization panels and supports various data sources, including Prometheus, OpenTSDB, MySQL, etc.
A time series database, mainly used to collect, store, and provide query data to the public.
A program used to expose service monitoring metrics, providing API interface to Prometheus to pull monitoring data.
Prometheus’ built-in data query language, which provides support for rich querying, aggregation, and logical computing capabilities for time series data.
2. Install Prometheus
- Add Helm source
- Install Prometheus
If prometheus-node-exporter is not working, the default port 9100 is probably occupied. You can change the default port by editing DaemonSet with the following command.
3. Installing Grafana
- Add the Helm source
It is worth noting that with this installation method, Grafana data is stored under the
/var/lib/grafana path of the Pod, and the associated configuration will be lost if you restart Grafana. For production environments, you need to mount the storage volume. If a StorageClass is available, you can use the following parameters for persistent storage.
- Get the login password for the admin account
- Modify the access method of the Grafana service to NodePort
This is used to access Grafana via the host IP + 31892.
4. Configuring Use
4.1 Adding data sources
In the left navigation bar, find [Data Sources].
Fill in the Prometheus access address http://prometheus-server.monitor.svc and you are done.
4.2 Adding a template
Find the Import button in the left navigation bar [+], I chose the panel with id 10856 to import. You can also go to the Grafana official website and select the appropriate panel to import,
As shown below, here you need to select the data source added above.
4.3 View Panel
4.4 grafana.ini configuration
Some of Grafana’s configurations are implemented by modifying grafana.ini, here are just a few that I use.