How to get the client's real IP, starting with a "bug" in Gin

Request IP, as one of the user’s identity attributes, is a very important basic data. In many scenarios, we will do network security attack prevention or access risk control based on the client request IP. Usually, we can get the real IP through the X-Forwarded-For header in the HTTP protocol Request Headers, but is it really reliable to get the real IP through the X-Forwarded-For header?

Concept

The X-Forwarded-For is an HTTP extension header that is not defined in the HTTP/1.1 (RFC 2616) standard and was first introduced by Squid, a caching proxy, to represent the real IP of an HTTP request. It is now a de facto standard and is widely used by major HTTP proxies, load balancing and other forwarding services, and is written into the RFC 7239 (Forwarded HTTP Extension) standard.

We found a “bug” after upgrading the Gin framework to 1.7.2 in one of our HTTP services, where the server could not get the correct client IP after the upgrade, but instead the Nginx Ingress IP in the Kubernetes cluster, so we decided to get the client source code from Gin to investigate.

The business-side service was previously using version v1.6.3, so let’s look at the Context.ClientIP() method implementation.

// ClientIP 方法可以获取到请求客户端的IP
func (c *Context) ClientIP() string {
   // 1. ForwardedByClientIP 默认为 true，此处会优先取 X-Forwarded-For 值，
   // 如果 X-Forwarded-For 为空，则会再尝试取 X-Real-Ip
   if c.engine.ForwardedByClientIP {
      clientIP := c.requestHeader("X-Forwarded-For")
      clientIP = strings.TrimSpace(strings.Split(clientIP, ",")[0])
      if clientIP == "" {
         clientIP = strings.TrimSpace(c.requestHeader("X-Real-Ip"))
      }
      if clientIP != "" {
         return clientIP
      }
   }
   // 2. 如果我们手动配置 ForwardedByClientIP 为 false 且 X-Appengine-Remote-Addr 不为空，则取 X-Appengine-Remote-Addr 作为客户端IP
   if c.engine.AppEngine {
      if addr := c.requestHeader("X-Appengine-Remote-Addr"); addr != "" {
         return addr
      }
   }
   // 3. 最终才考虑取对端 IP 兜底
   if ip, _, err := net.SplitHostPort(strings.TrimSpace(c.Request.RemoteAddr)); err == nil {
      return ip
}
   return ""
}

Looking at the v1.7.2 version, the Contexnt.ClientIP() method implements.

func (c *Context) RemoteIP() (net.IP, bool) {
   ...
   remoteIP := net.ParseIP(ip) // 获取客户端 IP
   ...
   // trustedCIDRs 由 engine 启动时配置的 TrustedProxies 数组解析而来，表示可以信任的前置代理 CIDR 列表。只有配置了 engine.TrustedProxies 才有可能解析出正确的可信任 CIDR 列表。
   // 只有 CIDR 列表不为空，这里才会将 remoteIP 和已配置可信 CIDR 列表进行比对。CIDR 列表中任一 CIDR 包含对端 IP，则将第二个返回值置为 true，表示对端 IP 可信任。
   if c.engine.trustedCIDRs != nil {
      for _, cidr := range c.engine.trustedCIDRs {
         if cidr.Contains(remoteIP) {
            return remoteIP, true
         }
      }
   }
   return remoteIP, false
}
func (c *Context) ClientIP() string {
   // 1. AppEngine 默认为 false，如果应用通过 Google Cloud App Engine 部署，或用户手动设置为 true 且 X-Appengine-Remote-Addr 不为空，则会取 X-Appengine-Remote-Addr 值作为客户端 IP。
   if c.engine.AppEngine {
      if addr := c.requestHeader("X-Appengine-Remote-Addr"); addr != "" {
         return addr
      }
   }
   // 2. 否则通过 RemoteIP() 方法判断对端 IP 是否可信，trusted 为 true 表示可信
   // 详见上文 Context.RemoteIP() 方法内部注释。
   remoteIP, trusted := c.RemoteIP()
   if remoteIP == nil {
      return ""
   }
   // 3. 如对端 IP 可信，且 ForwardedByClientIP 为 true(默认为 true），且
   // RemoteIPHeaders 不为空（默认不为空），则根据 RemoteIPHeaders 中配置的获取 ClientIP 的 Headers 列表中依次获取。默认读取顺序：1. X-Forwarded-For；2. X-Real-IP。
   if trusted && c.engine.ForwardedByClientIP && c.engine.RemoteIPHeaders != nil {
      for _, headerName := range c.engine.RemoteIPHeaders {
         // 对header进行处理，先通过","进行分割，并返回分割后 IP 列表的第一个合法 IP
         ip, valid := validateHeader(c.requestHeader(headerName))
         if valid {
            return ip
         }
      }
   }
   // 3. 最终才考虑取对端 IP 兜底。
   return remoteIP.String()
}
// validateHeader 会对入参header进行校验，先通过","进行分割成 IP 列表后，对每个 IP 进行合法性检查，如果任一 IP 不合法，则此Header不合法；否则返回 IP 列表中第一个 IP。
func validateHeader(header string) (clientIP string, valid bool) {
   if header == "" {
      return "", false
   }
   items := strings.Split(header, ",")
   for i, ipStr := range items {
      ipStr = strings.TrimSpace(ipStr)
      ip := net.ParseIP(ipStr)
      ...
      if i == 0 {
         clientIP = ipStr
         valid = true
      }
   }
   return
}

A detailed discussion of this “bug” can be found at https://github.com/gin-gonic/gin/issues/2697

Analysis

Let’s start by introducing a few concepts/terms that may be covered later.

$remote_addr: This is the real address of the client that Nginx obtains during the TCP connection with the client. Remote Address cannot be forged because it takes three handshakes to establish a TCP connection. If you forge the source IP, you cannot establish a TCP connection, and there will be no subsequent HTTP requests.
X-Client-Real-IP ：This feature is basically supported by most of the cloud vendors (Ali cloud, Huawei cloud, Tencent cloud, etc.). This feature is basically supported by most cloud vendors (Ali cloud, Huawei cloud, Tencent cloud, etc).

Network requests are usually sent by browsers (or other clients), forwarded through layers of network devices, and finally reach the server. Then the $remote_addr in the request received by each link must be the real IP of the upstream link, which cannot be faked. Then from the whole link, if the source of the final request is needed, it is traced by X-Forwarded-For, and the IP ($remote_addr) of each link is added to the X-Forwarded-For field, so that X-Forwarded-For can connect the whole link in series. That is.

`1`	`X-Forwarded-For: client_ip, proxy1_ip, proxy2_ip`

Can X-Forwarded-For be forged?

Whether or not a client can spoof IP depends on how the Edge Node handles the X-Forwarded-For field. The first Proxy node to which a client connects directly is called an Edge Node, whether it is a gateway, CDN, LB, etc. As long as this layer is accessed directly by the client, then it is an Edge Node.

Edge nodes that do not override X-Forwarded-For Edge nodes are insecure if they pass through the X-Forwarded-For header of HTTP, and the client can forge the X-Forwarded-For value in the HTTP request, and the value will be passed backwards.

Thus an edge node that does not override X-Forwarded-For is an insecure edge node and the user can forge X-Forwarded-For.

1
2

# 不安全
X-Forwareded-For：clientX-Forwarded-For(用户请求中的 X-Forwarded-For)，proxy1，proxy2，proxy3...

Edge nodes that override X-Forwarded-For Edge nodes are secure if they override $remote_addr to X-Forwarded-For. The remote_addr obtained by the edge node is the real IP of the client, so an edge node that overrides X-Forwarded-For is a secure edge node and the user cannot forge X-Forwarded-For.

# 边缘节点用 $remote_addr 来覆盖用户请求中的 X-Forwarded-For：
proxy_set_header X-Forwarded-For $remote_addr; 
# 安全
X-Forwareded-For：ClientX-Forwarded-For(边缘节点获取的 remote_addr)，proxy1，proxy2，proxy3...

How can I get the real client IP?

We consider solutions that can obtain real client IPs under common network topologies on public clouds.

Client->WAF->SLB->Ingress->Pod

Using the Nginx real-ip module

To get it using the Nginx real-ip module, configure proxy-real-ip-cidr on Ingress to add both the WAF and SLB (layer 7) addresses. After this operation, the server can fetch real IPs using X-Forwarded-For and fake IPs using X-Original-Forwarded-For.

This option has the following disadvantages.

Since the WAF is maintained by the cloud vendor, the WAF address pool is large and the addresses change, making it extremely difficult to maintain this dynamic configuration, and inaccurate client IPs can be obtained if not updated in a timely manner. -Even with this solution, if the business side wants to use the new version of Gin’s ctx.ClientIP() method, it still needs to change the code to configure all trusted proxies to TrustedProxies, which will lead to coupling of infrastructure and business services, and this solution is obviously unacceptable unless the business side is willing to lock the relying Gin version locked at v1.6.3.

Customizing Header with WAF

Many cloud vendors provide custom Header to get the client real IP ($remote_addr) capability. We can configure custom Header headers in advance in the cloud vendor WAF endpoint, such as X-Appengine-Remote-Addr or X-Client-Real-IP, etc., to get the client real IP.

This option has the following drawbacks.

If you reuse the X-Appengine-Remote-Addr header directly, you need to set engine.AppEngine=true to get the client IP with the ctx.ClientIP() method - if you use other header, such as X -Client-Real-IP, you need to encapsulate the client IP method from X-Client-Real-IP, and you need to do the modification with the business.

The architecture is roughly as follows.

Client->CDN->WAF->SLB->Ingress->Pod

Use real-ip

Using the real-ip module, you need to configure proxy-real-ip-cidr on ingress to add CDN, WAF and SLB (layer 7) addresses, and the server can fetch real IPs using X-Forwarded-For and fake IPs via X-Original-Forwarded-For.

Advantages and disadvantages of this scenario.

This scenario has more CDN layers than 3.2.1, the CDN address pool is larger than WAF, the address pool changes more frequently, and the vendor does not provide a CDN address pool, so it is basically impossible to maintain Ingress configuration. -Even with this solution, if the business side wants to use the new version of Gin’s ctx.ClientIP() method, it still needs to change the code to configure all trusted proxies to TrustedProxies, which will lead to coupling of infrastructure and business services, which is definitely unacceptable unless the business side will Gin version locked at 1.6.3.

Customizing Header with CDN

The advantages and disadvantages of this solution are the same as in 3.1.1. The architecture is approximately as follows.

Client->SLB->Ingress->Pod

You can prevent X-Forwarded-For forgery by setting use-forwarded-headers on Ingress.

use-forwarded-headers=false

For Ingress without a proxy layer in front of it, e.g. directly on a Layer 4 SLB, ingress rewrites X-Forwarded-For to $remote_addr by default, which prevents forging X-Forwarded-For.

use-forwarded-headers=true

For Ingress with a pre-proxy layer, such as a Layer 7 SLB or WAF, CDN, etc., this is equivalent to adding the following configuration to nginx.conf.

1
2
3

real_ip_header      X-Forwarded-For; 
real_ip_recursive   on; 
set_real_ip_from    0.0.0.0/0; // 默认信任所有 IP，无法避免伪造 X-Forwarded-For

The architecture is roughly as follows.

Summary

As we can see from the above, with the complex and changing network topology on the cloud, we have to frequently maintain multiple network configurations such as CDN, WAF, SLB, Ingress, etc. If X-Forwarded-For is not forged, there are only two options for upgrading the Go service of the Gin framework.

Keep trying to get the client’s real IP through X-Forwarded-For. - Try to get the client’s real IP through another Header.

Continue trying to get the client’s real IP via X-Forwarded-For

The business needs to configure all the front agents of the infrastructure into TrustedProxies, including CDN address pools, WAF address pools, and Kunernetest Nginx Ingress address pools, and this solution is basically impossible to implement.

The configuration is too complex and difficult to troubleshoot once the IP is not available.
If the infrastructure makes changes to the CDN, WAF, or Ingress, the business code must be changed simultaneously. -Some of the trusted proxy IPs are not configurable at all, such as CDN address pools.

Try to get the client’s real IP through custom Header

The infrastructure team provides a custom Header to get the client’s real IP, such as X-Client-Real-IP or X-Appengine-Remote-Addr. This scenario requires the infrastructure team to configure the cloud vendor CDN or WAF endpoint accordingly. This solution is.

Simple and reliable configuration, low maintenance cost, only need to configure custom Header in CDN, WAF endpoint. -If you use X-Appengine-Remote-Addr, you don’t need to make any changes to the services using Google Cloud’s App Engine. If you are using a domestic cloud vendor’s service, you need to explicitly configure engine.AppEngine = true and then continue with the ctx.ClientIP() method.
If you use other custom Header, such as X-Client-Real-IP to get the real IP of the client, it is recommended to consider wrapping ClientIP(*gin.Context) string function to get the client IP from X-Client-Real-IP by yourself.

Table of Contents