Prometheus Routing and Routing Configuration with Nginx Reverse Proxy

Background

The company group an end-to-end containerized test environment, located under different sub-paths, proxied by Nginx: * For example, user A is under /a/prometheus/: /a/prometheus/.

For example, user A is under /a/: /a/prometheus/
For example, user B under /b/: /b/prometheus/

The effect you want to achieve, in addition to the above distinction between different subpaths, requires that there is no such distinction inside the container: i.e., the container can be accessed directly inside prometheus:9090 to request Prometheus.

TLDR: If you also have such a need. If you think the article is too long and don’t want to read it, you can skip directly to the final summary.

Prometheus’ path and route configuration

From the command line parameters of Prometheus, you can see that it provides the following two options to control the path and route configuration.

--web.external-url=<URL>

The URL under which Prometheus is externally reachable (for example, if Prometheus is served via a reverse proxy). Used for generating relative and absolute links back to Prometheus itself. If the URL has a path portion, it will be used to prefix all HTTP endpoints served by Prometheus. If omitted, relevant URL components will be derived automatically.

This is the address (URL) that is externally reachable by Prometheus (after reverse proxy).

Used to generate relative or absolute links (on a page or in a request).
--web.route-prefix=<path>

Prefix for the internal routes of web endpoints. Defaults to path of –web.external-url.

Route prefix for Prometheus internal web requests.

Defaults to the path in --web.external-url.

Direct access test

In order to have direct access to Prometheus’ API inside the container, --web.route-prefix should be configured as /.

And for --web.external-url, since it controls the externally accessible path and relative link generation in the page. So this address should be configured to the path mentioned at the beginning of the article (in the case of user A): /a/prometheus/.

Here’s a real test to see.

`1`	`$ docker run -it --rm -p 9090:9090 prom/prometheus --config.file /etc/prometheus/prometheus.yml --web.route-prefix=/ --web.external-url=/a/prometheus/`

Note: For illustration purposes, nginx and prometheus are used below to represent the two services respectively.

The API is accessible.

1
2

$ curl -s prometheus:9090/api/v1/query | cut -c-50
{"status":"error","errorType":"bad_data","error":"

The home page is redirected to the path with the prefix when you visit it.

$ curl -si prometheus:9090
HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Location: /a/prometheus/graph
Date: Sat, 12 Mar 2022 09:26:58 GMT
Content-Length: 42

<a href="/a/prometheus/graph">Found</a>.

However, if you try to access this redirected address, it will report an error that it cannot be found.

1
2

$ curl -s prometheus:9090/a/prometheus/graph
404 page not found

Isn’t it quite strange? If I directly request the path without the prefix, it is found.

1
2

$ curl -s prometheus:9090/graph | cut -c-50
<!doctype html><html lang="en"><head><meta charset

So that’s what --web.external-url=<URL> means when it says “deeper”: This option is used to control the path before the reflection proxy, which Prometheus itself doesn’t need, and you should rewrite the path in nginx to remove it.

Tested by nginx proxy

Nginx configuration.

1
2
3

location /a/prometheus/ {
  proxy_pass http://prometheus:9090;
}

Tried to access it, and as expected it didn’t work because the path hadn’t been rewritten yet.

1
2

$ curl nginx:8080/a/prometheus/
404 page not found

Try to rewrite the path.

location /a/prometheus/ {
  rewrite ^/a/prometheus(.*)$ $1 break;
  proxy_pass http://prometheus:9090;
}

The rewrite rule above means roughly: the request /a/prometheus/xxx is rewritten to /xxx and then passed to the proxied server.

So, Prometheus receives the request without the prefix. And, it was found to work.

$ curl nginx:8080/a/prometheus/
<a href="/a/prometheus/graph">Found</a>.

$ curl -s nginx:8080/a/prometheus/graph | cut -c-50
<!doctype html><html lang="en"><head><meta charset

After consulting the proxy_pass documentation for nginx, we found that

If proxy_pass is not followed by a URL, the original URL received by nginx is passed directly to the proxied server unchanged.
If proxy_pass is followed by a URL, the prefix in location is removed and appended to the URL before it is proxied.

In this way, the above nginx configuration can be simplified to look like the following.

1
2
3

location /a/prometheus/ {
  proxy_pass http://prometheus:9090/;
}

Note that there is a / at the end.

Summary

The configuration of Prometheus.
- --web.external-url=/你的/前缀/
- --web.route-prefix=/