Some Tips for Using Json in Go

This article summarizes the problems and solutions I usually encounter in my projects regarding the interconversion between go language JSON data and structs.

Basic Serialization

First let’s look at the basic usage of json.Marshal() (serialization) and json.Unmarshal (deserialization) in the Go language.

type Person struct {
	Name   string
	Age    int64
	Weight float64
}

func main() {
	p1 := Person{
		Name:   "Qimi",
		Age:    18,
		Weight: 71.5,
	}
	// struct -> json string
	b, err := json.Marshal(p1)
	if err != nil {
		fmt.Printf("json.Marshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("str:%s\n", b)
	// json string -> struct
	var p2 Person
	err = json.Unmarshal(b, &p2)
	if err != nil {
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("p2:%#v\n", p2)
}

Output:

1
2

str:{"Name":"Qimi","Age":18,"Weight":71.5}
p2:main.Person{Name:"Qimi", Age:18, Weight:71.5}

Introduction to the structure tag

The Tag is the meta-information of the structure, which can be read at runtime through the mechanism of reflection. Tag is defined after the structure field, wrapped by a pair of backquotes, in the following format.

`1`	`key1:"value1" key2:"value2"`

The structure tag consists of one or more key-value pairs. The keys are separated from the values using colons and the values are enclosed in double quotes. Multiple key-value pairs tag can be set for the same structure field, with spaces separating the different key-value pairs.

Specify the field name using json tag

Serialization and deserialization use the field name of the structure by default, we can specify the field name generated by json serialization by adding tag to the structure field.

// Use json tag to specify the behavior when serializing and deserializing
type Person struct {
	Name   string `json:"name"` // Specify lowercase name forand deserializing json
	Age    int64
	Weight float64
}

Ignore a field

If you want to ignore a field in the structure when serializing/deserializing json, you can add - to the tag as follows.

// Use json tag to specify the behavior of json when serializing and deserializing
type Person struct {
	Name   string `json:"name"` // Specify lowercase name for json serialization/deserialization
	Age    int64
	Weight float64 `json:"-"` // Specify to ignore this field when serializing/deserializing json
}

Ignore nil fields

Marshal() does not ignore fields in struct when serializing them without a value, but outputs the type zero of the field by default (e.g. int and float type zero is 0, string type zero is “”, object type zero is nil). If you want to ignore these fields that have no value when serializing, you can add omitempty tag to the corresponding field.

As an example.

type User struct {
	Name  string   `json:"name"`
	Email string   `json:"email"`
	Hobby []string `json:"hobby"`
}

func omitemptyDemo() {
	u1 := User{
		Name: "Qimi",
	}
	// struct -> json string
	b, err := json.Marshal(u1)
	if err != nil {
		fmt.Printf("json.Marshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("str:%s\n", b)
}

Output:

`1`	`str:{"name":"Qimi","email":"","hobby":null}`

If you want to remove the null field from the final serialization result, you can define the structure like the following.

// Add omitempty to tag to ignore null values
// Note that here hobby,omitempty together are json tag values, separated by English commas
type User struct {
	Name  string   `json:"name"`
	Email string   `json:"email,omitempty"`
	Hobby []string `json:"hobby,omitempty"`
}

At this point, execute the above omitemptyDemo again and the output will be as follows.

`1`	`str:{"name":"Qimi"} // No email and hobby fields in serialization results`

Ignore nested structure nil fields

First, look at several examples of structure nesting.

type User struct {
	Name  string   `json:"name"`
	Email string   `json:"email,omitempty"`
	Hobby []string `json:"hobby,omitempty"`
	Profile
}

type Profile struct {
	Website string `json:"site"`
	Slogan  string `json:"slogan"`
}

func nestedStructDemo() {
	u1 := User{
		Name:  "Qimi",
		Hobby: []string{"Soccer", "Two Color Ball"},
	}
	b, err := json.Marshal(u1)
	if err != nil {
		fmt.Printf("json.Marshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("str:%s\n", b)
}

The serialized json string when nesting Profile anonymously is a single level of.

`1`	`str:{"name":"Qimi","hobby":["Soccer","Two Color Ball"],"site":"","slogan":""}`

To turn it into a nested json string, you need to change it to a named nest or define the field tag.

type User struct {
	Name    string   `json:"name"`
	Email   string   `json:"email,omitempty"`
	Hobby   []string `json:"hobby,omitempty"`
	Profile `json:"profile"`
}
// str:{"name":"Qimi","hobby":["Soccer","Two Color Ball"],"profile":{"site":"","slogan":""}}

It is not enough to add omitempty if you want to ignore the field when the nested structure has a null value.

type User struct {
	Name     string   `json:"name"`
	Email    string   `json:"email,omitempty"`
	Hobby    []string `json:"hobby,omitempty"`
	Profile `json:"profile,omitempty"`
}
// str:{"name":"Qimi","hobby":["Soccer","Two Color Ball"],"profile":{"site":"","slogan":""}}

It is also necessary to use nested structure pointers.

type User struct {
	Name     string   `json:"name"`
	Email    string   `json:"email,omitempty"`
	Hobby    []string `json:"hobby,omitempty"`
	*Profile `json:"profile,omitempty"`
}
// str:{"name":"Qimi","hobby":["Soccer","Two Color Ball"]}

Ignore nil fields without modifying the original structure

We need json to serialize User, but do not want to serialize the password as well, and do not want to modify the User structure, this time we can use to create another structure PublicUser anonymously nested in the original User, while specifying the Password field as an anonymous structure pointer type, and add omitemptytag, sample code as follows.

type User struct {
	Name     string `json:"name"`
	Password string `json:"password"`
}

type PublicUser struct {
	*User             // Anonymous nesting
	Password *struct{} `json:"password,omitempty"`
}

func omitPasswordDemo() {
	u1 := User{
		Name:     "Qimi",
		Password: "123456",
	}
	b, err := json.Marshal(PublicUser{User: &u1})
	if err != nil {
		fmt.Printf("json.Marshal u1 failed, err:%v\n", err)
		return
	}
	fmt.Printf("str:%s\n", b)  // str:{"name":"Qimi"}
}

Elegant handling of numbers in string format

Sometimes the front-end may use numbers of string type in the passed json data, in this case you can add string to the structure tag to tell the json package to parse the data of the corresponding field from the string.

type Card struct {
	ID    int64   `json:"id,string"`    // add string tag
	Score float64 `json:"score,string"` // add string tag
}

func intAndStringDemo() {
	jsonStr1 := `{"id": "1234567","score": "88.50"}`
	var c1 Card
	if err := json.Unmarshal([]byte(jsonStr1), &c1); err != nil {
		fmt.Printf("json.Unmarsha jsonStr1 failed, err:%v\n", err)
		return
	}
	fmt.Printf("c1:%#v\n", c1) // c1:main.Card{ID:1234567, Score:88.5}
}

Integer to floating

In the JSON protocol there is no distinction between integer and floating point types, they are collectively called number. Numbers in json strings are deserialized by the json package in the Go language and become float64 types. The following code demonstrates this problem.

func jsonDemo() {
	// map[string]interface{} -> json string
	var m = make(map[string]interface{}, 1)
	m["count"] = 1 // int
	b, err := json.Marshal(m)
	if err != nil {
		fmt.Printf("marshal failed, err:%v\n", err)
	}
	fmt.Printf("str:%#v\n", string(b))
	// json string -> map[string]interface{}
	var m2 map[string]interface{}
	err = json.Unmarshal(b, &m2)
	if err != nil {
		fmt.Printf("unmarshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("value:%v\n", m2["count"]) // 1
	fmt.Printf("type:%T\n", m2["count"])  // float64
}

This scenario requires the use of decoder to deserialize the numbers if you want to handle them more rationally, and the sample code is as follows.

func decoderDemo() {
	// map[string]interface{} -> json string
	var m = make(map[string]interface{}, 1)
	m["count"] = 1 // int
	b, err := json.Marshal(m)
	if err != nil {
		fmt.Printf("marshal failed, err:%v\n", err)
	}
	fmt.Printf("str:%#v\n", string(b))
	// json string -> map[string]interface{}
	var m2 map[string]interface{}
	// Deserialize using the `decoder` method, specifying the use of the `number` type
	decoder := json.NewDecoder(bytes.NewReader(b))
	decoder.UseNumber()
	err = decoder.Decode(&m2)
	if err != nil {
		fmt.Printf("unmarshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("value:%v\n", m2["count"]) // 1
	fmt.Printf("type:%T\n", m2["count"])  // json.Number
	// After converting m2["count"] to json.Number, call the Int64() method to get a value of type int64
	count, err := m2["count"].(json.Number).Int64()
	if err != nil {
		fmt.Printf("parse to int64 failed, err:%v\n", err)
		return
	}
	fmt.Printf("type:%T\n", int(count)) // int
}

The source code of json.Number is defined as follows.

// A Number represents a JSON number literal.
type Number string

// String returns the literal text of the number.
func (n Number) String() string { return string(n) }

// Float64 returns the number as a float64.
func (n Number) Float64() (float64, error) {
	return strconv.ParseFloat(string(n), 64)
}

// Int64 returns the number as an int64.
func (n Number) Int64() (int64, error) {
	return strconv.ParseInt(string(n), 10, 64)
}

We need to get the json.Number type first when dealing with number type json.Number fields, and then call Float64() or Int64() depending on the actual type of the field.

Customized parsing time field

The Go language’s built-in json package uses the time format defined in the RFC3339 standard, which has many restrictions on how we can serialize time fields.

type Post struct {
	CreateTime time.Time `json:"create_time"`
}

func timeFieldDemo() {
	p1 := Post{CreateTime: time.Now()}
	b, err := json.Marshal(p1)
	if err != nil {
		fmt.Printf("json.Marshal p1 failed, err:%v\n", err)
		return
	}
	fmt.Printf("str:%s\n", b)
	jsonStr := `{"create_time":"2020-04-05 12:25:42"}`
	var p2 Post
	if err := json.Unmarshal([]byte(jsonStr), &p2); err != nil {
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("p2:%#v\n", p2)
}

The output of the above code is as follows

1
2

str:{"create_time":"2020-04-05T12:28:06.799214+08:00"}
json.Unmarshal failed, err:parsing time ""2020-04-05 12:25:42"" as ""2006-01-02T15:04:05Z07:00"": cannot parse " 12:25:42"" as "T"

That is, the built-in json package does not recognize our common string time format, such as 2020-04-05 12:25:42.

However, we implement custom event format parsing by implementing the json.Marshaler/json.Unmarshaler interface.

type CustomTime struct {
	time.Time
}

const ctLayout = "2006-01-02 15:04:05"

var nilTime = (time.Time{}).UnixNano()

func (ct *CustomTime) UnmarshalJSON(b []byte) (err error) {
	s := strings.Trim(string(b), "\"")
	if s == "null" {
		ct.Time = time.Time{}
		return
	}
	ct.Time, err = time.Parse(ctLayout, s)
	return
}

func (ct *CustomTime) MarshalJSON() ([]byte, error) {
	if ct.Time.UnixNano() == nilTime {
		return []byte("null"), nil
	}
	return []byte(fmt.Sprintf("\"%s\"", ct.Time.Format(ctLayout))), nil
}

func (ct *CustomTime) IsSet() bool {
	return ct.UnixNano() != nilTime
}

type Post struct {
	CreateTime CustomTime `json:"create_time"`
}

func timeFieldDemo() {
	p1 := Post{CreateTime: CustomTime{time.Now()}}
	b, err := json.Marshal(p1)
	if err != nil {
		fmt.Printf("json.Marshal p1 failed, err:%v\n", err)
		return
	}
	fmt.Printf("str:%s\n", b)
	jsonStr := `{"create_time":"2020-04-05 12:25:42"}`
	var p2 Post
	if err := json.Unmarshal([]byte(jsonStr), &p2); err != nil {
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("p2:%#v\n", p2)
}

Custom MarshalJSON and UnmarshalJSON methods

The above method of customizing types is a little bit more verbose, so here is a relatively convenient way to look at it.

The first thing you need to know is that if you can implement the MarshalJSON()([]byte, error) and UnmarshalJSON(b []byte) error methods for a type, then that type will use your custom methods when serializing (MarshalJSON)/deserializing (UnmarshalJSON).

type Order struct {
	ID          int       `json:"id"`
	Title       string    `json:"title"`
	CreatedTime time.Time `json:"created_time"`
}

const layout = "2006-01-02 15:04:05"

// MarshalJSON implements custom MarshalJSON methods for Order types
func (o *Order) MarshalJSON() ([]byte, error) {
	type TempOrder Order // Define a new type consistent with the Order field
	return json.Marshal(struct {
		CreatedTime string `json:"created_time"`
		*TempOrder         // Avoid nesting Order directly into a dead loop
	}{
		CreatedTime: o.CreatedTime.Format(layout),
		TempOrder:   (*TempOrder)(o),
	})
}

// UnmarshalJSON Implement custom UnmarshalJSON methods for Order types
func (o *Order) UnmarshalJSON(data []byte) error {
	type TempOrder Order // Define a new type consistent with the Order field
	ot := struct {
		CreatedTime string `json:"created_time"`
		*TempOrder         // Avoid nesting Order directly into a dead loop
	}{
		TempOrder: (*TempOrder)(o),
	}
	if err := json.Unmarshal(data, &ot); err != nil {
		return err
	}
	var err error
	o.CreatedTime, err = time.Parse(layout, ot.CreatedTime)
	if err != nil {
		return err
	}
	return nil
}

// Custom Serialization Methods
func customMethodDemo() {
	o1 := Order{
		ID:          123456,
		Title:       "《Qimi's Go Learning Notes》",
		CreatedTime: time.Now(),
	}
	// struct -> json string via custom MarshalJSON method
	b, err := json.Marshal(&o1)
	if err != nil {
		fmt.Printf("json.Marshal o1 failed, err:%v\n", err)
		return
	}
	fmt.Printf("str:%s\n", b)
	// json string -> struct via custom UnmarshalJSON method
	jsonStr := `{"created_time":"2020-04-05 10:18:20","id":123456,"title":"《Qimi's Go Learning Notes》"}`
	var o2 Order
	if err := json.Unmarshal([]byte(jsonStr), &o2); err != nil {
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("o2:%#v\n", o2)
}

Output results.

1
2

str:{"created_time":"2020-04-05 10:32:20","id":123456,"title":"《Qimi's Go Learning Notes》"}
o2:main.Order{ID:123456, Title:"《Qimi's Go Learning Notes》", CreatedTime:time.Time{wall:0x0, ext:63721678700, loc:(*time.Location)(nil)}}

Adding fields using anonymous structs

The use of inline structs can extend the fields of a struct, but sometimes it is not necessary to define new structs separately and the operation can be simplified by using anonymous structs.

type UserInfo struct {
	ID   int    `json:"id"`
	Name string `json:"name"`
}

func anonymousStructDemo() {
	u1 := UserInfo{
		ID:   123456,
		Name: "Qimi",
	}
	// Use anonymous structure with embedded User and add additional field Token
	b, err := json.Marshal(struct {
		*UserInfo
		Token string `json:"token"`
	}{
		&u1,
		"91je3a4s72d1da96h",
	})
	if err != nil {
		fmt.Printf("json.Marsha failed, err:%v\n", err)
		return
	}
	fmt.Printf("str:%s\n", b)
	// str:{"id":123456,"name":"Qimi","token":"91je3a4s72d1da96h"}
}

Combining Multiple Structs with Anonymous Structs

Similarly, anonymous structures can be used to combine multiple structures to serialize and deserialize data: the

type Comment struct {
	Content string
}

type Image struct {
	Title string `json:"title"`
	URL   string `json:"url"`
}

func anonymousStructDemo2() {
	c1 := Comment{
		Content: "Never overestimate yourself",
	}
	i1 := Image{
		Title: "QR Code",
		URL:   "https://www.foo.com/qr.png",
	}
	// struct -> json string
	b, err := json.Marshal(struct {
		*Comment
		*Image
	}{&c1, &i1})
	if err != nil {
		fmt.Printf("json.Marshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("str:%s\n", b)
	// json string -> struct
	jsonStr := `{"Content":"Never overestimate yourself","title":"QR Code","url":"https://www.foo.com/qr.png"}`
	var (
		c2 Comment
		i2 Image
	)
	if err := json.Unmarshal([]byte(jsonStr), &struct {
		*Comment
		*Image
	}{&c2, &i2}); err != nil {
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("c2:%#v i2:%#v\n", c2, i2)
}

Output:

1
2

str:{"Content":"Never overestimate yourself","title":"QR Code","url":"https://www.foo.com/qr.png"}
c2:main.Comment{Content:"Never overestimate yourself"} i2:main.Image{Title:"QR Code", URL:"https://www.foo.com/qr.png"}

Handling json with uncertain hierarchy

If the json string does not have a fixed format that makes it difficult to define its corresponding structure, we can use json.RawMessage to save the raw byte data.

type sendMsg struct {
	User string `json:"user"`
	Msg  string `json:"msg"`
}

func rawMessageDemo() {
	jsonStr := `{"sendMsg":{"user":"q1mi","msg":"Never overestimate yourself"},"say":"Hello"}`
	// Define a map with value type json.RawMessage for more flexible processing
	var data map[string]json.RawMessage
	if err := json.Unmarshal([]byte(jsonStr), &data); err != nil {
		fmt.Printf("json.Unmarshal jsonStr failed, err:%v\n", err)
		return
	}
	var msg sendMsg
	if err := json.Unmarshal(data["sendMsg"], &msg); err != nil {
		fmt.Printf("json.Unmarshal failed, err:%v\n", err)
		return
	}
	fmt.Printf("msg:%#v\n", msg)
	// msg:main.sendMsg{User:"q1mi", Msg:"Never overestimate yourself"}
}

Reference https://www.liwenzhou.com/posts/Go/json_tricks_in_go/

Table of Contents

Basic Serialization

Introduction to the structure tag

Specify the field name using json tag

Ignore a field

Ignore nil fields

Ignore nested structure nil fields

Ignore nil fields without modifying the original structure

Elegant handling of numbers in string format

Integer to floating

Customized parsing time field

Custom MarshalJSON and UnmarshalJSON methods

Adding fields using anonymous structs

Combining Multiple Structs with Anonymous Structs

Handling json with uncertain hierarchy