Request IP, as one of the user’s identity attributes, is a very important basic data. In many scenarios, we will do network security attack prevention or access risk control based on the client request IP. Usually, we can get the real IP through the
X-Forwarded-For header in the HTTP protocol Request Headers, but is it really reliable to get the real IP through the
X-Forwarded-For is an HTTP extension header that is not defined in the HTTP/1.1 (RFC 2616) standard and was first introduced by Squid, a caching proxy, to represent the real IP of an HTTP request. It is now a de facto standard and is widely used by major HTTP proxies, load balancing and other forwarding services, and is written into the RFC 7239 (Forwarded HTTP Extension) standard.
We found a “bug” after upgrading the Gin framework to 1.7.2 in one of our HTTP services, where the server could not get the correct client IP after the upgrade, but instead the Nginx Ingress IP in the Kubernetes cluster, so we decided to get the client source code from Gin to investigate.
The business-side service was previously using version v1.6.3, so let’s look at the
Context.ClientIP() method implementation.
Looking at the v1.7.2 version, the
Contexnt.ClientIP() method implements.
A detailed discussion of this “bug” can be found at https://github.com/gin-gonic/gin/issues/2697
Let’s start by introducing a few concepts/terms that may be covered later.
$remote_addr: This is the real address of the client that Nginx obtains during the TCP connection with the client. Remote Address cannot be forged because it takes three handshakes to establish a TCP connection. If you forge the source IP, you cannot establish a TCP connection, and there will be no subsequent HTTP requests.
X-Client-Real-IP：This feature is basically supported by most of the cloud vendors (Ali cloud, Huawei cloud, Tencent cloud, etc.). This feature is basically supported by most cloud vendors (Ali cloud, Huawei cloud, Tencent cloud, etc).
Network requests are usually sent by browsers (or other clients), forwarded through layers of network devices, and finally reach the server. Then the
$remote_addr in the request received by each link must be the real IP of the upstream link, which cannot be faked. Then from the whole link, if the source of the final request is needed, it is traced by
X-Forwarded-For, and the IP (
$remote_addr) of each link is added to the
X-Forwarded-For field, so that
X-Forwarded-For can connect the whole link in series. That is.
Can X-Forwarded-For be forged?
Whether or not a client can spoof IP depends on how the Edge Node handles the X-Forwarded-For field. The first Proxy node to which a client connects directly is called an Edge Node, whether it is a gateway, CDN, LB, etc. As long as this layer is accessed directly by the client, then it is an Edge Node.
- Edge nodes that do not override X-Forwarded-For Edge nodes are insecure if they pass through the
X-Forwarded-Forheader of HTTP, and the client can forge the
X-Forwarded-Forvalue in the HTTP request, and the value will be passed backwards.
Thus an edge node that does not override
X-Forwarded-For is an insecure edge node and the user can forge
- Edge nodes that override X-Forwarded-For Edge nodes are secure if they override
remote_addrobtained by the edge node is the real IP of the client, so an edge node that overrides
X-Forwarded-Foris a secure edge node and the user cannot forge
How can I get the real client IP?
We consider solutions that can obtain real client IPs under common network topologies on public clouds.
Using the Nginx real-ip module
To get it using the Nginx real-ip module, configure proxy-real-ip-cidr on Ingress to add both the WAF and SLB (layer 7) addresses. After this operation, the server can fetch real IPs using X-Forwarded-For and fake IPs using X-Original-Forwarded-For.
This option has the following disadvantages.
Since the WAF is maintained by the cloud vendor, the WAF address pool is large and the addresses change, making it extremely difficult to maintain this dynamic configuration, and inaccurate client IPs can be obtained if not updated in a timely manner. -Even with this solution, if the business side wants to use the new version of Gin’s ctx.ClientIP() method, it still needs to change the code to configure all trusted proxies to TrustedProxies, which will lead to coupling of infrastructure and business services, and this solution is obviously unacceptable unless the business side is willing to lock the relying Gin version locked at v1.6.3.
Customizing Header with WAF
Many cloud vendors provide custom Header to get the client real IP ($remote_addr) capability. We can configure custom Header headers in advance in the cloud vendor WAF endpoint, such as X-Appengine-Remote-Addr or X-Client-Real-IP, etc., to get the client real IP.
This option has the following drawbacks.
If you reuse the
X-Appengine-Remote-Addr header directly, you need to set
engine.AppEngine=true to get the client IP with the
ctx.ClientIP() method - if you use other header, such as
X -Client-Real-IP, you need to encapsulate the client IP method from
X-Client-Real-IP, and you need to do the modification with the business.
The architecture is roughly as follows.
real-ip module, you need to configure
proxy-real-ip-cidr on ingress to add CDN, WAF and SLB (layer 7) addresses, and the server can fetch real IPs using
X-Forwarded-For and fake IPs via
Advantages and disadvantages of this scenario.
This scenario has more CDN layers than 3.2.1, the CDN address pool is larger than WAF, the address pool changes more frequently, and the vendor does not provide a CDN address pool, so it is basically impossible to maintain Ingress configuration. -Even with this solution, if the business side wants to use the new version of Gin’s
ctx.ClientIP() method, it still needs to change the code to configure all trusted proxies to TrustedProxies, which will lead to coupling of infrastructure and business services, which is definitely unacceptable unless the business side will Gin version locked at 1.6.3.
Customizing Header with CDN
The advantages and disadvantages of this solution are the same as in 3.1.1. The architecture is approximately as follows.
You can prevent
X-Forwarded-For forgery by setting
use-forwarded-headers on Ingress.
For Ingress without a proxy layer in front of it, e.g. directly on a Layer 4 SLB, ingress rewrites
$remote_addr by default, which prevents forging
For Ingress with a pre-proxy layer, such as a Layer 7 SLB or WAF, CDN, etc., this is equivalent to adding the following configuration to nginx.conf.
The architecture is roughly as follows.
As we can see from the above, with the complex and changing network topology on the cloud, we have to frequently maintain multiple network configurations such as CDN, WAF, SLB, Ingress, etc. If
X-Forwarded-For is not forged, there are only two options for upgrading the Go service of the Gin framework.
Keep trying to get the client’s real IP through
X-Forwarded-For. - Try to get the client’s real IP through another Header.
Continue trying to get the client’s real IP via X-Forwarded-For
The business needs to configure all the front agents of the infrastructure into TrustedProxies, including CDN address pools, WAF address pools, and Kunernetest Nginx Ingress address pools, and this solution is basically impossible to implement.
The configuration is too complex and difficult to troubleshoot once the IP is not available.
If the infrastructure makes changes to the CDN, WAF, or Ingress, the business code must be changed simultaneously. -Some of the trusted proxy IPs are not configurable at all, such as CDN address pools.
Try to get the client’s real IP through custom Header
The infrastructure team provides a custom Header to get the client’s real IP, such as
X-Appengine-Remote-Addr. This scenario requires the infrastructure team to configure the cloud vendor CDN or WAF endpoint accordingly. This solution is.
Simple and reliable configuration, low maintenance cost, only need to configure custom Header in CDN, WAF endpoint. -If you use
X-Appengine-Remote-Addr, you don’t need to make any changes to the services using Google Cloud’s App Engine. If you are using a domestic cloud vendor’s service, you need to explicitly configure
engine.AppEngine = trueand then continue with the
If you use other custom Header, such as
X-Client-Real-IPto get the real IP of the client, it is recommended to consider wrapping
ClientIP(*gin.Context) stringfunction to get the client IP from