Tutorials for using resty

resty is an HTTP client library for the Go language. resty is powerful and feature-rich. It supports almost all HTTP methods (GET/POST/PUT/DELETE/OPTION/HEAD/PATCH, etc.) and provides an easy-to-use API.

Quick Use

The code in this article uses Go Modules.

Create the directory and initialize.

1
2

$ mkdir resty && cd resty
$ go mod init github.com/darjun/go-daily-lib/resty

Install the resty library.

`1`	`$ go get -u github.com/go-resty/resty/v2`

Here we get the information on the first page of Baidu.

package main

import (
  "fmt"
  "log"

  "github.com/go-resty/resty/v2"
)

func main() {
  client := resty.New()

  resp, err := client.R().Get("https://baidu.com")

  if err != nil {
    log.Fatal(err)
  }

  fmt.Println("Response Info:")
  fmt.Println("Status Code:", resp.StatusCode())
  fmt.Println("Status:", resp.Status())
  fmt.Println("Proto:", resp.Proto())
  fmt.Println("Time:", resp.Time())
  fmt.Println("Received At:", resp.ReceivedAt())
  fmt.Println("Size:", resp.Size())
  fmt.Println("Headers:")
  for key, value := range resp.Header() {
    fmt.Println(key, "=", value)
  }
  fmt.Println("Cookies:")
  for i, cookie := range resp.Cookies() {
    fmt.Printf("cookie%d: name:%s value:%s\n", i, cookie.Name, cookie.Value)
  }
}

Resty is relatively simple to use.

First, call a resty.New() to create a client object.
call the R() method of the client object to create a request object.
Call the Get()/Post() methods of the request object, pass in the parameter URL, and you can send an HTTP request to the corresponding URL. Returning a response object.
The response object provides a number of methods to check the status of the response, header, cookies, and other information.

In the above program we have obtained.

StatusCode(): status code, e.g. 200
Status(): status code and status information, e.g. 200 OK.
Proto(): protocol, e.g. HTTP/1.1.
Time(): the time from sending the request to receiving the response.
ReceivedAt(): the moment when the response was received.
Size(): the size of the response.
Header(): the response header information, returned as http.Header type, i.e. map[string][]string.
Cookies(): the cookie information set by the server via the Set-Cookie header.

Basic information about the response output from running the program.

Response Info:
Status Code: 200
Status: 200 OK
Proto: HTTP/1.1
Time: 415.774352ms
Received At: 2021-06-26 11:42:45.307157 +0800 CST m=+0.416547795
Size: 302456

header information.

Headers:
Server = [BWS/1.1]
Date = [Sat, 26 Jun 2021 03:42:45 GMT]
Connection = [keep-alive]
Bdpagetype = [1]
Bdqid = [0xf5a61d240003b218]
Vary = [Accept-Encoding Accept-Encoding]
Content-Type = [text/html;charset=utf-8]
Set-Cookie = [BAIDUID=BF2EE47AAAF7A20C6971F1E897ABDD43:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com BIDUPSID=BF2EE47AAAF7A20C6971F1E897ABDD43; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com PSTM=1624678965; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com BAIDUID=BF2EE47AAAF7A20C716E90B86906D6B0:FG=1; max-age=31536000; expires=Sun, 26-Jun-22 03:42:45 GMT; domain=.baidu.com; path=/; version=1; comment=bd BDSVRTM=0; path=/ BD_HOME=1; path=/ H_PS_PSSID=34099_31253_34133_34072_33607_34135_26350; path=/; domain=.baidu.com]
Traceid = [1624678965045126810617700867425882583576]
P3p = [CP=" OTI DSP COR IVA OUR IND COM " CP=" OTI DSP COR IVA OUR IND COM "]
X-Ua-Compatible = [IE=Edge,chrome=1]

Note that there is a Set-Cookie Header, which appears in the Cookie section.

Cookies:
cookie0: name:BAIDUID value:BF2EE47AAAF7A20C6971F1E897ABDD43:FG=1
cookie1: name:BIDUPSID value:BF2EE47AAAF7A20C6971F1E897ABDD43
cookie2: name:PSTM value:1624678965
cookie3: name:BAIDUID value:BF2EE47AAAF7A20C716E90B86906D6B0:FG=1
cookie4: name:BDSVRTM value:0
cookie5: name:BD_HOME value:1
cookie6: name:H_PS_PSSID value:34099_31253_34133_34072_33607_34135_26350

Automatic Unmarshal

Many sites now provide API interfaces that return structured data in JSON/XML format, etc. resty can automatically Unmarshal the response data into the corresponding structured object. To see an example, we know that many js files are hosted on cdn and we can get the basic information of these libraries through api.cdnjs.com/libraries and return a JSON data in the following format.

Next, we define the struct and then use resty to pull information, automatically Unmarshal.

type Library struct {
  Name   string
  Latest string
}

type Libraries struct {
  Results []*Library
}

func main() {
  client := resty.New()

  libraries := &Libraries{}
  client.R().SetResult(libraries).Get("https://api.cdnjs.com/libraries")
  fmt.Printf("%d libraries\n", len(libraries.Results))

  for _, lib := range libraries.Results {
    fmt.Println("first library:")
    fmt.Printf("name:%s latest:%s\n", lib.Name, lib.Latest)
    break
  }
}

As you can see, we just need to create a result type object and call the SetResult() method of the request object, and resty will automatically Unmarshal the response data into the incoming object. Here the request information is set using a chain call approach, i.e., multiple settings are done in one line.

Run

$ go run main.go
4040 libraries
first library:
name:vue latest:https://cdnjs.cloudflare.com/ajax/libs/vue/3.1.2/vue.min.js

There are 4040 libraries in total and the first one is Vue ✌️. We can get the details of Vue by requesting https://api.cdnjs.com/libraries/vue.

Interested in pulling this information yourself with resty.

Normally with a request, resty will infer the data format based on the Content-Type in the response. But sometimes the response has no Content-Type header or is not in the same format as the content, we can force resty to parse the response in a specific format by calling ForceContentType() on the request object.

1
2
3

client.R().
  SetResult(result).
  ForceContentType("application/json")

Request Information

resty provides a rich set of methods for setting request information. We can set the query string in two ways. One is to call SetQueryString() of the request object to set our spliced query string.

1
2
3

client.R().
  SetQueryString("name=dj&age=18").
  Get(...)

The other is to call SetQueryParams() on the request object, pass in map[string]string, and let resty do the splicing for us. Obviously this is more convenient.

client.R().
  SetQueryParams(map[string]string{
    "name": "dj",
    "age": "18",
  }).
  Get(...)

resty also provides a very useful interface for setting path parameters, we call SetPathParams() and pass in the map[string]string parameter, then the keys in this map can be used later in the URL path.

client.R().
  SetPathParams(map[string]string{
    "user": "dj",
  }).
  Get("/v1/users/{user}/details")

Note that the keys in the path need to be wrapped with {}.

Set Header.

1
2
3

client.R().
  SetHeader("Content-Type", "application/json").
  Get(...)

Set the request body.

client.R().
  SetHeader("Content-Type", "application/json").
  SetBody(`{"name": "dj", "age":18}`).
  Get(...)

The message body can be of various types: string, []byte, struct, map[string]interface{}, etc.

Set to carry Content-Length header, resty automatically calculates.

client.R().
  SetBody(User{Name:"dj", Age:18}).
  SetContentLength(true).
  Get(...)

Some sites need to get a token before they can access its API. To set the token.

1
2
3

client.R().
  SetAuthToken("youdontknow").
  Get(...)

Demo

Finally, let’s string together all of the above with a case study. Now we want to get the organization’s repository information through the API provided by GitHub, see the link after the article for the API documentation. The GitHub API request address is https://api.github.com, and the request format for getting repository information is as follows.

`1`	`GET /orgs/{org}/repos`

We can also set these parameters as follows.

accept: Header, this is required and needs to be set to application/vnd.github.v3+json.
org: Organization name, path parameter.
type: repository type, query parameters, e.g. public/private/forks (fork’s repository), etc.
sort: sorting rule of the repository, query parameters, e.g. created/updated/pushed/full_name, etc. default sort by creation time.
direction: ascending asc or descending dsc, query parameters.
per_page: how many entries per page, max 100, default 30, query parameter.
page: current request page number, with per_page for paging management, default 1, query parameter.

The GitHub API must be set up with a token to access it. Log in to your GitHub account, click on your avatar in the top right corner, and select Settings.

Then, select Developer settings.

Select Personal access tokens, then click Generate new token in the upper right corner.

Fill in Note to indicate the purpose of the token, this can be filled in according to your situation. The following checkboxes are used to select which permissions the token has, and are not required here.

Click the Generate token button below to generate a token.

Note that this token can only be seen now, close the page and you won’t be able to see it next time you enter. So save it, and don’t use my token, I will delete the token after testing the program 😭.

The JSON format data in the response is shown below.

There are so many fields that for the sake of convenience I will deal with only a few here.

type Repository struct {
  ID              int        `json:"id"`
  NodeID          string     `json:"node_id"`
  Name            string     `json:"name"`
  FullName        string     `json:"full_name"`
  Owner           *Developer `json:"owner"`
  Private         bool       `json:"private"`
  Description     string     `json:"description"`
  Fork            bool       `json:"fork"`
  Language        string     `json:"language"`
  ForksCount      int        `json:"forks_count"`
  StargazersCount int        `json:"stargazers_count"`
  WatchersCount   int        `json:"watchers_count"`
  OpenIssuesCount int        `json:"open_issues_count"`
}

type Developer struct {
  Login      string `json:"login"`
  ID         int    `json:"id"`
  NodeID     string `json:"node_id"`
  AvatarURL  string `json:"avatar_url"`
  GravatarID string `json:"gravatar_id"`
  Type       string `json:"type"`
  SiteAdmin  bool   `json:"site_admin"`
}

Then use resty to set the path parameters, query parameters, Header, Token, etc., and initiate the request.

func main() {
  client := resty.New()

  var result []*Repository
  client.R().
    SetAuthToken("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").
    SetHeader("Accept", "application/vnd.github.v3+json").
    SetQueryParams(map[string]string{
      "per_page":  "3",
      "page":      "1",
      "sort":      "created",
      "direction": "asc",
    }).
    SetPathParams(map[string]string{
      "org": "golang",
    }).
    SetResult(&result).
    Get("https://api.github.com/orgs/{org}/repos")

  for i, repo := range result {
    fmt.Printf("repo%d: name:%s stars:%d forks:%d\n", i+1, repo.Name, repo.StargazersCount, repo.ForksCount)
  }
}

The above procedure pulls the 3 repositories in ascending order of creation time.

$ go run main.go
repo1: name:gddo stars:1097 forks:289
repo2: name:lint stars:3892 forks:518
repo3: name:glog stars:2738 forks:775

Trace

After introducing the main features of resty, let’s take a look at a secondary feature provided by resty: trace. We enable trace by calling the EnableTrace() method on the request object. Enabling trace records the time taken and other information about each step of the request. resty supports chained calls, which means we can create the request, enable trace, and initiate the request all in one line: trace.

`1`	`client.R().EnableTrace().Get("https://baidu.com")`

After completing the request, we get the information by calling the TraceInfo() method of the request object.

ti := resp.Request.TraceInfo()
fmt.Println("Request Trace Info:")
fmt.Println("DNSLookup:", ti.DNSLookup)
fmt.Println("ConnTime:", ti.ConnTime)
fmt.Println("TCPConnTime:", ti.TCPConnTime)
fmt.Println("TLSHandshake:", ti.TLSHandshake)
fmt.Println("ServerTime:", ti.ServerTime)
fmt.Println("ResponseTime:", ti.ResponseTime)
fmt.Println("TotalTime:", ti.TotalTime)
fmt.Println("IsConnReused:", ti.IsConnReused)
fmt.Println("IsConnWasIdle:", ti.IsConnWasIdle)
fmt.Println("ConnIdleTime:", ti.ConnIdleTime)
fmt.Println("RequestAttempt:", ti.RequestAttempt)
fmt.Println("RemoteAddr:", ti.RemoteAddr.String())

We have access to the following information.

DNSLookup: the DNS lookup time, if a domain name is provided instead of an IP, the corresponding IP needs to be looked up in the DNS system before subsequent operations can be performed.
ConnTime: the elapsed time to obtain a connection, which may be obtained from a connection pool or newly created.
TCPConnTime: the time taken for a TCP connection, from the end of the DNS query to the establishment of the TCP connection.
TLSHandshake: TLS handshake elapsed time.
ServerTime: server processing elapsed time, calculating the time interval from connection establishment to the time the client receives the first byte.
ResponseTime: response elapsed time, the time interval between the reception of the first response byte and the reception of the complete response.
TotalTime: the elapsed time of the whole process.
IsConnReused: whether the TCP connection is multiplexed or not.
IsConnWasIdle: whether the connection was fetched from an idle connection pool.
ConnIdleTime: the connection idle time.
RequestAttempt: the number of requests in the request execution process, including the number of retries.
RemoteAddr: the service address of the remote, in IP:PORT format.

resty makes a very fine distinction between these. In fact resty also uses the functionality provided by the standard library net/http/httptrace. httptrace provides a structure where we can set the callback functions for each stage.

// src/net/http/httptrace.go
type ClientTrace struct {
  GetConn func(hostPort string)
  GotConn func(GotConnInfo)
  PutIdleConn func(err error)
  GotFirstResponseByte func()
  Got100Continue func()
  Got1xxResponse func(code int, header textproto.MIMEHeader) error // Go 1.11
  DNSStart func(DNSStartInfo)
  DNSDone func(DNSDoneInfo)
  ConnectStart func(network, addr string)
  ConnectDone func(network, addr string, err error)
  TLSHandshakeStart func() // Go 1.8
  TLSHandshakeDone func(tls.ConnectionState, error) // Go 1.8
  WroteHeaderField func(key string, value []string) // Go 1.11
  WroteHeaders func()
  Wait100Continue func()
  WroteRequest func(WroteRequestInfo)
}

You can get a brief idea of what the callbacks mean from the field names. resty sets the following callbacks when trace is enabled.

// src/github.com/go-resty/resty/trace.go
func (t *clientTrace) createContext(ctx context.Context) context.Context {
  return httptrace.WithClientTrace(
    ctx,
    &httptrace.ClientTrace{
      DNSStart: func(_ httptrace.DNSStartInfo) {
        t.dnsStart = time.Now()
      },
      DNSDone: func(_ httptrace.DNSDoneInfo) {
        t.dnsDone = time.Now()
      },
      ConnectStart: func(_, _ string) {
        if t.dnsDone.IsZero() {
          t.dnsDone = time.Now()
        }
        if t.dnsStart.IsZero() {
          t.dnsStart = t.dnsDone
        }
      },
      ConnectDone: func(net, addr string, err error) {
        t.connectDone = time.Now()
      },
      GetConn: func(_ string) {
        t.getConn = time.Now()
      },
      GotConn: func(ci httptrace.GotConnInfo) {
        t.gotConn = time.Now()
        t.gotConnInfo = ci
      },
      GotFirstResponseByte: func() {
        t.gotFirstResponseByte = time.Now()
      },
      TLSHandshakeStart: func() {
        t.tlsHandshakeStart = time.Now()
      },
      TLSHandshakeDone: func(_ tls.ConnectionState, _ error) {
        t.tlsHandshakeDone = time.Now()
      },
    },
  )
}

Then, when fetching TraceInfo, the elapsed time is calculated based on each time point.

// src/github.com/go-resty/resty/request.go
func (r *Request) TraceInfo() TraceInfo {
  ct := r.clientTrace

  if ct == nil {
    return TraceInfo{}
  }

  ti := TraceInfo{
    DNSLookup:      ct.dnsDone.Sub(ct.dnsStart),
    TLSHandshake:   ct.tlsHandshakeDone.Sub(ct.tlsHandshakeStart),
    ServerTime:     ct.gotFirstResponseByte.Sub(ct.gotConn),
    IsConnReused:   ct.gotConnInfo.Reused,
    IsConnWasIdle:  ct.gotConnInfo.WasIdle,
    ConnIdleTime:   ct.gotConnInfo.IdleTime,
    RequestAttempt: r.Attempt,
  }

  if ct.gotConnInfo.Reused {
    ti.TotalTime = ct.endTime.Sub(ct.getConn)
  } else {
    ti.TotalTime = ct.endTime.Sub(ct.dnsStart)
  }

  if !ct.connectDone.IsZero() {
    ti.TCPConnTime = ct.connectDone.Sub(ct.dnsDone)
  }

  if !ct.gotConn.IsZero() {
    ti.ConnTime = ct.gotConn.Sub(ct.getConn)
  }

  if !ct.gotFirstResponseByte.IsZero() {
    ti.ResponseTime = ct.endTime.Sub(ct.gotFirstResponseByte)
  }

  if ct.gotConnInfo.Conn != nil {
    ti.RemoteAddr = ct.gotConnInfo.Conn.RemoteAddr()
  }

  return ti
}

Run output.

$ go run main.go
Request Trace Info:
DNSLookup: 2.815171ms
ConnTime: 941.635171ms
TCPConnTime: 269.069692ms
TLSHandshake: 669.276011ms
ServerTime: 274.623991ms
ResponseTime: 112.216µs
TotalTime: 1.216276906s
IsConnReused: false
IsConnWasIdle: false
ConnIdleTime: 0s
RequestAttempt: 1
RemoteAddr: 18.235.124.214:443

We see that TLS consumes almost half of the time.

Summary

In this article I introduce HTTP Client, a very handy and easy to use library for the Go language. resty provides a very useful and rich API. Chain calls, automatic Unmarshal, request parameters/path settings are very convenient and useful, so that we can do our work with less effort. Due to the limitation of space, many advanced features can not be introduced one by one, such as submit form, upload files, etc., etc.. We can only leave it to those who are interested to explore it. 😄

Reference

resty GitHub：github.com/go-resty/resty
GitHub API：https://docs.github.com/en/rest/overview/resources-in-the-rest-api

Reference https://darjun.github.io/2021/06/26/godailylib/resty

Table of Contents