## 1. Introduction to Form

Form, is an important syntax element in HTML markup language. A Form contains not only normal text content, markup, etc., but also special elements called controls. The user usually “completes” the form by modifying the controls (e.g., entering text, selecting menu items, etc.), and then submits the form data to the Web server as an HTTP Get or Post request.

Many beginners always confuse HTML and HTTP. http is usually used as a carrier for html transmission, for example, html is like a passenger and http is like a cab that transports passengers from one place to another. But obviously the cab of http can not only pull html as a passenger, many formats can be used as passengers of the cab of http, such as json (over http), xml (over http).

In an HTML document, the standard format of a form is as follows.

 1 2 3 4 5  


When a Form is loaded into the browser, it will appear as a form, and after entering text in each of the two text boxes (or using the default text as input) and clicking “submit”, the browser will send an HTTP request to http://localhost:8080. The HTTP request will take the input text of the form as the Query String Parameter (in this case ?language=go&since=monthly ) because the method property of the Form is get. After the request is processed on the server side, an HTTP-bearing response is returned, which is received by the browser and rendered in the browser window in a specific style. The above process can be summarized in the following diagram.

The method in Form can also use post, like the following.

 1 2 3 4 5  


What is the difference between the http request sent by a Form form changed to post when it is clicked and submitted and the request when method=get? The difference is that in the case of method=post, the parameters of the form are no longer placed in the request URL as query string parameters, but are written to the HTTP BODY. Let’s summarize this process in a diagram as well.

Since the form parameters are placed in the HTTP Body for transmission (the data in the body is: language=go&since=monthly ), we will find a new header field in the headers of this HTTP request: Content-Type, which in this example has the value application/x-www-form-urlencoded. We can use the enctype attribute in the Form to change the content encoding type of the data transmitted by the Form, the default value of which is application/x-www-form-urlencoded (i.e. key1=value1&key2=value2&… of the form). form). Other optional values for enctype include.

 1 2  text/plain multipart/form-data 

The form parameters of the Form with method=get are put into the http request as query string parameters, which makes its application scenarios relatively limited, for example.

• When there are many parameter values and the parameter values are very long, the URL maximum length limit may be exceeded.
• when passing sensitive data, it is not safe to put parameter values in plaintext in the HTTP request header.
• Not competent in the case of passing binary data (such as a file content).

Therefore, the method=post form is more advantageous in these cases. When the enctype is a different value, the form with method=post transmits data in the http body in the following form.

We see: when enctype=application/x-www-urlencoded, the data in the Body is presented in the form of key1=value1&key2=value2&…, similar to the combined presentation of the query string parameters of the URL; when enctype=text/ plain, this encoding format is also called raw, which means that the data content is transmitted in the Body as it is, keeping the original encoding of the data (usually utf-8); and when enctype=multipart/form-data, the data in the HTTP Body is presented in the form of multipart (part), and the paragraphs are separated by The specified random string is also passed to the server along with the HTTP Post request (placed in the value of Content-Type in the Header, separated from multipart/form-data by a semicolon), e.g.

 1  Content-Type: multipart/form-data; boundary=--------------------------399501358433894470769897 

Let’s look at a diagram of a slightly more complex enctype=multipart/form-data example.

We use Postman to simulate a Post request with five segments (parts), including two text segments (text) and three file segments, and the three files are in different formats, txt, png, and json, respectively. Postman uses the Content-Type in each segment to specify the type of data content for that segment. When the server receives the data, it can parse the segmented data in a targeted manner according to the segment Content-Type indication. The default Content-Type of a file segment is text/plain; for unrecognized file types (e.g., no extension), the Content-Type of a file segment is usually set to application/octet-stream.

Uploading files via Form is a capability given to html by the RFC1867 specification, and it has proven to be very useful and widely used, even we can directly use multipart/form-data as HTTP Post body a data carrying protocol to transfer file data between the two ends.

http.Request provides the ParseMultipartForm method to parse data transferred in multipart/form-data format. Parsing is the process of mapping data into the Request structure’s MultipartForm field.

  1 2 3 4 5 6 7 8 9 10  // $GOROOT/src/net/http/request.go type Request struct { ... ... // MultipartForm is the parsed multipart form, including file uploads. // This field is only available after ParseMultipartForm is called. // The HTTP client ignores MultipartForm and uses Body instead. MultipartForm *multipart.Form ... ... }  Multipart.Form represents a parsed multipart/form-data Body with the following structure:   1 2 3 4 5 6 7 8 9 10 11  //$GOROOT/src/mime/multipart/formdata.go // Form is a parsed multipart form. // Its File parts are stored either in memory or on disk, // and are accessible via the *FileHeader's Open method. // Its Value parts are stored as strings. // Both are keyed by field name. type Form struct { Value map[string][]string File map[string][]*FileHeader } 

We see that this Form structure consists of two maps, one map holds all the value parts (like name, age) and the other map holds all the file parts (like part1.txt, part2.png and part3.json). value part collection There is nothing to say, the key of the map is the “name” of each value part; our focus is on the file part. Each file part corresponds to a set of FileHeader, and the structure of FileHeader is as follows.

 1 2 3 4 5 6 7 8 9  // $GOROOT/src/mime/multipart/formdata.go type FileHeader struct { Filename string Header textproto.MIMEHeader Size int64 content []byte tmpfile string }  The FileHeader for each file part contains five fields. • Filename - the original file name of the uploaded file • Size - the size of the uploaded file (in bytes) • content - the (partial or full) data content of the uploaded file stored in memory • tmpfile - data content of the part of the uploaded file stored in a temporary file local to the server (if the size of the uploaded file is larger than the parameter maxMemory passed to ParseMultipartForm, the remaining part is stored in the temporary file) • Header - the header content of the file part, which is also a map, with the following structure.  1 2 3 4 5  //$GOROOT/src/net/textproto/header.go // A MIMEHeader represents a MIME-style header mapping // keys to sets of values. type MIMEHeader map[string][]string 

We can represent the data mapping process implemented by the ParseMultipartForm method as the following diagram, which looks more intuitive.

With the above explanation of how uploading files in multipart/form-data format works, it is easy to implement a simple Go server that supports uploading files in multipart/form-data format using the Go http package.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48  // github.com/bigwhite/experiments/multipart-formdata/server/file_server1.go package main import ( "fmt" "io" "net/http" "os" ) const uploadPath = "./upload" func handleUploadFile(w http.ResponseWriter, r *http.Request) { r.ParseMultipartForm(100) mForm := r.MultipartForm for k, _ := range mForm.File { // k is the key of file part file, fileHeader, err := r.FormFile(k) if err != nil { fmt.Println("inovke FormFile error:", err) return } defer file.Close() fmt.Printf("the uploaded file: name[%s], size[%d], header[%#v]\n", fileHeader.Filename, fileHeader.Size, fileHeader.Header) // store uploaded file into local path localFileName := uploadPath + "/" + fileHeader.Filename out, err := os.Create(localFileName) if err != nil { fmt.Printf("failed to open the file %s for writing", localFileName) return } defer out.Close() _, err = io.Copy(out, file) if err != nil { fmt.Printf("copy file err:%s\n", err) return } fmt.Printf("file %s uploaded ok\n", fileHeader.Filename) } } func main() { http.HandleFunc("/upload", handleUploadFile) http.ListenAndServe(":8080", nil) } 

We can upload two files part1.txt and part3.json to the above file server at the same time using Postman or the curl command below.

 1 2 3 4 5  curl --location --request POST ':8080/upload' \ --form 'name="tony bai"' \ --form 'age="23"' \ --form 'file1=@"/your_local_path/part1.txt"' \ --form 'file3=@"/your_local_path/part3.json"' 

The output log for running the file upload server is as follows.

 1 2 3 4 5  $go run file_server1.go the uploaded file: name[part3.json], size[130], header[textproto.MIMEHeader{"Content-Disposition":[]string{"form-data; name=\"file3\"; filename=\"part3.json\""}, "Content-Type":[]string{"application/json"}}] file part3.json uploaded ok the uploaded file: name[part1.txt], size[15], header[textproto.MIMEHeader{"Content-Disposition":[]string{"form-data; name=\"file1\"; filename=\"part1.txt\""}, "Content-Type":[]string{"text/plain"}}] file part1.txt uploaded ok  Then we can see: the file upload server successfully stored the received part1.txt and part3.json in the upload directory under the current path! ## 3. Go clients that support uploading files in multipart/form-data format The previous clients for file uploads are either browsers, Postman or curl, but if we want to build our own client that supports uploading files in multipart/form-data format, how should we do it? We need to construct the body of the HTTP request in multipart/form-data format. Fortunately, with the mime/multipart package provided by the Go standard library, we can easily build a body that meets the requirements.   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79  // github.com/bigwhite/experiments/multipart-formdata/client/client1.go ... ... var ( filePath string addr string ) func init() { flag.StringVar(&filePath, "file", "", "the file to upload") flag.StringVar(&addr, "addr", "localhost:8080", "the addr of file server") flag.Parse() } func main() { if filePath == "" { fmt.Println("file must not be empty") return } err := doUpload(addr, filePath) if err != nil { fmt.Printf("upload file [%s] error: %s", filePath, err) return } fmt.Printf("upload file [%s] ok\n", filePath) } func createReqBody(filePath string) (string, io.Reader, error) { var err error buf := new(bytes.Buffer) bw := multipart.NewWriter(buf) // body writer f, err := os.Open(filePath) if err != nil { return "", nil, err } defer f.Close() // text part1 p1w, _ := bw.CreateFormField("name") p1w.Write([]byte("Tony Bai")) // text part2 p2w, _ := bw.CreateFormField("age") p2w.Write([]byte("15")) // file part1 _, fileName := filepath.Split(filePath) fw1, _ := bw.CreateFormFile("file1", fileName) io.Copy(fw1, f) bw.Close() //write the tail boundry return bw.FormDataContentType(), buf, nil } func doUpload(addr, filePath string) error { // create body contType, reader, err := createReqBody(filePath) if err != nil { return err } url := fmt.Sprintf("http://%s/upload", addr) req, err := http.NewRequest("POST", url, reader) // add headers req.Header.Add("Content-Type", contType) client := &http.Client{} resp, err := client.Do(req) if err != nil { fmt.Println("request send error:", err) return err } resp.Body.Close() return nil }  Obviously the core of the above client-side code is the createReqBody function. • The client creates three segments in the body, the first two segments are only intentionally added by me to demonstrate how to create a text part, a real upload file client does not need to create these two segments (parts). • createReqBody uses bytes.Buffer as temporary storage for the http body. • After building the body content, don’t forget to call the Close method of multipart.Writer to write the ending boundary token. We use this client to upload a file to the previous server that supports uploading files in multipart/form-data format.   1 2 3 4 5 6 7 8 9 10  // 客户端$go run client1.go -file hello.txt upload file [hello.txt] ok // 服务端 $go run file_server1.go http request: http.Request{Method:"POST", URL:(*url.URL)(0xc00016e100), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"492"}, "Content-Type":[]string{"multipart/form-data; boundary=b55090594eaa1aaac1abad1d89a77ae689130d79d6f66af82590036bd8ba"}, "User-Agent":[]string{"Go-http-client/1.1"}}, Body:(*http.body)(0xc000146380), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:492, TransferEncoding:[]string(nil), Close:false, Host:"localhost:8080", Form:url.Values{"age":[]string{"15"}, "name":[]string{"Tony Bai"}}, PostForm:url.Values{"age":[]string{"15"}, "name":[]string{"Tony Bai"}}, MultipartForm:(*multipart.Form)(0xc000110d50), Trailer:http.Header(nil), RemoteAddr:"[::1]:58569", RequestURI:"/upload", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc0001463c0)} the uploaded file: name[hello.txt], size[15], header[textproto.MIMEHeader{"Content-Disposition":[]string{"form-data; name=\"file1\"; filename=\"hello.txt\""}, "Content-Type":[]string{"application/octet-stream"}}] file hello.txt uploaded ok  We see that the text file hello.txt was successfully uploaded! ## 4. Customize the header in the file segment From the output of file_server1 above, the Content-Type set by client1 in the file segment (part) is the default application/octet-stream when uploading files. Sometimes, the server side may need to do sorting according to this Content-Type and needs the client to give the exact value. In the client1 implementation above, we use the method multipart.Writer.CreateFormFile to create the file part.  1 2 3 4  // file part1 _, fileName := filepath.Split(filePath) fw1, _ := bw.CreateFormFile("file1", fileName) io.Copy(fw1, f)  The following is the code for the implementation of the CreateFormFile method in the standard library.  1 2 3 4 5 6 7 8 9  //$GOROOT/mime/multipart/writer.go func (w *Writer) CreateFormFile(fieldname, filename string) (io.Writer, error) { h := make(textproto.MIMEHeader) h.Set("Content-Disposition", fmt.Sprintf(form-data; name="%s"; filename="%s", escapeQuotes(fieldname), escapeQuotes(filename))) h.Set("Content-Type", "application/octet-stream") return w.CreatePart(h) } 

We see that CreateFormFile sets the Content-Type to the default value of application/octet-stream, regardless of the type of file to be uploaded. If we want to customize the value of the Content-Type of the Header field in the file part, we cannot use CreateFormFile directly, but we can refer to its implementation.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41  // github.com/bigwhite/experiments/multipart-formdata/client/client2.go var quoteEscaper = strings.NewReplacer("\\", "\\\\", ", "\\\"") func escapeQuotes(s string) string { return quoteEscaper.Replace(s) } func createReqBody(filePath string) (string, io.Reader, error) { var err error buf := new(bytes.Buffer) bw := multipart.NewWriter(buf) // body writer f, err := os.Open(filePath) if err != nil { return "", nil, err } defer f.Close() // text part1 p1w, _ := bw.CreateFormField("name") p1w.Write([]byte("Tony Bai")) // text part2 p2w, _ := bw.CreateFormField("age") p2w.Write([]byte("15")) // file part1 _, fileName := filepath.Split(filePath) h := make(textproto.MIMEHeader) h.Set("Content-Disposition", fmt.Sprintf(form-data; name="%s"; filename="%s", escapeQuotes("file1"), escapeQuotes(fileName))) h.Set("Content-Type", "text/plain") fw1, _ := bw.CreatePart(h) io.Copy(fw1, f) bw.Close() //write the tail boundry return bw.FormDataContentType(), buf, nil } 

We customize the header part of the file part by textproto.MIMEHeader instance, then call CreatePart based on this instance to create the file part, after that we write the file content of hello.txt to the header of this part.

We run client2 to upload the hello.txt file, and on the file_server side, we can see the following log.

 1 2  the uploaded file: name[hello.txt], size[15], header[textproto.MIMEHeader{"Content-Disposition":[]string{"form-data; name=\"file1\"; filename=\"hello.txt\""}, "Content-Type":[]string{"text/plain"}}] file hello.txt uploaded ok 

We see that the value of Content-Type of file part has changed to text/plain which we set.

Buffer to load all the contents of the file to be uploaded, so if the file to be uploaded is large, the memory space consumption is bound to be too large. So how can we limit the memory usage to an appropriate range each time we upload a memory file, or how can we say that the memory space consumed by the uploaded file does not get bigger as the file to be uploaded gets bigger? Let’s look at the following solution.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62  // github.com/bigwhite/experiments/multipart-formdata/client/client3.go ... ... func createReqBody(filePath string) (string, io.Reader, error) { var err error pr, pw := io.Pipe() bw := multipart.NewWriter(pw) // body writer f, err := os.Open(filePath) if err != nil { return "", nil, err } go func() { defer f.Close() // text part1 p1w, _ := bw.CreateFormField("name") p1w.Write([]byte("Tony Bai")) // text part2 p2w, _ := bw.CreateFormField("age") p2w.Write([]byte("15")) // file part1 _, fileName := filepath.Split(filePath) h := make(textproto.MIMEHeader) h.Set("Content-Disposition", fmt.Sprintf(form-data; name="%s"; filename="%s", escapeQuotes("file1"), escapeQuotes(fileName))) h.Set("Content-Type", "application/pdf") fw1, _ := bw.CreatePart(h) cnt, _ := io.Copy(fw1, f) log.Printf("copy %d bytes from file %s in total\n", cnt, fileName) bw.Close() //write the tail boundry pw.Close() }() return bw.FormDataContentType(), pr, nil } func doUpload(addr, filePath string) error { // create body contType, reader, err := createReqBody(filePath) if err != nil { return err } log.Printf("createReqBody ok\n") url := fmt.Sprintf("http://%s/upload", addr) req, err := http.NewRequest("POST", url, reader) //add headers req.Header.Add("Content-Type", contType) client := &http.Client{} log.Printf("upload %s...\n", filePath) resp, err := client.Do(req) if err != nil { fmt.Println("request send error:", err) return err } resp.Body.Close() log.Printf("upload %s ok\n", filePath) return nil } 

In this scenario, we create a read and write pipeline through the io.Pipe function, the write side of which is passed to multipart.NewWriter as an instance of io.Writer, and the read side is returned to the caller for use in building the http request. io.Pipe is based on the channel implementation and does not maintain any internal memory cache.

 1 2 3 4 5 6 7 8 9  // $GOROOT/src/io/pipe.go func Pipe() (*PipeReader, *PipeWriter) { p := &pipe{ wrCh: make(chan []byte), rdCh: make(chan int), done: make(chan struct{}), } return &PipeReader{p}, &PipeWriter{p} }  When the reader returned via Pipe reads the data in the pipe, if no data has been written to the pipe yet, the reader blocks there as if it were reading the channel. Since the http request is sent (client.Do(req)) before it actually reads the Body data based on the reader passed in when constructing the req, the client will block on the read of the pipe. Obviously we can’t put both the read and write operations in one goroutine, that would cause a panic as all goroutines would hang. in the client3.go code above, the function createReqBody creates a new goroutine internally, putting the real work of building the multipart/form- data body in the new goroutine. The new goroutine will eventually write the data of the file to be uploaded to the pipeline via the pipeline write side.  1  cnt, _ := io.Copy(fw1, f)  And this data will also be read by the client and transferred out over the network connection. The implementation of io.Copy is as follows.  1 2 3 4  //$GOROOT/src/io/io.go func Copy(dst Writer, src Reader) (written int64, err error) { return copyBuffer(dst, src, nil) } 

io.copyBuffer internally maintains a small buffer of 32k by default, which tries to read a maximum of 32k of data from src at a time and writes it to dst until it is finished. This way, no matter how big the file to be uploaded is, we actually only have 32k of memory allocated for each upload.

Here is the log of our upload of a file of size 252M using client3.go.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14  $go run client3.go -file /Users/tonybai/Downloads/ICME-2019-Tutorial-final.pdf 2021/01/10 12:56:45 createReqBody ok 2021/01/10 12:56:45 upload /Users/tonybai/Downloads/ICME-2019-Tutorial-final.pdf... 2021/01/10 12:56:46 copy 264517032 bytes from file ICME-2019-Tutorial-final.pdf in total 2021/01/10 12:56:46 upload /Users/tonybai/Downloads/ICME-2019-Tutorial-final.pdf ok upload file [/Users/tonybai/Downloads/ICME-2019-Tutorial-final.pdf] ok$go run file_server1.go http request: http.Request{Method:"POST", URL:(*url.URL)(0xc000078200), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept-Encoding":[]string{"gzip"}, "Content-Type":[]string{"multipart/form-data; boundary=4470ba3867218f1130878713da88b5bd79f33dfbed65566e4fd76a1ae58d"}, "User-Agent":[]string{"Go-http-client/1.1"}}, Body:(*http.body)(0xc000026240), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:-1, TransferEncoding:[]string{"chunked"}, Close:false, Host:"localhost:8080", Form:url.Values{"age":[]string{"15"}, "name":[]string{"Tony Bai"}}, PostForm:url.Values{"age":[]string{"15"}, "name":[]string{"Tony Bai"}}, MultipartForm:(*multipart.Form)(0xc0000122a0), Trailer:http.Header(nil), RemoteAddr:"[::1]:54899", RequestURI:"/upload", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc000026280)} the uploaded file: name[ICME-2019-Tutorial-final.pdf], size[264517032], header[textproto.MIMEHeader{"Content-Disposition":[]string{"form-data; name=\"file1\"; filename=\"ICME-2019-Tutorial-final.pdf\""}, "Content-Type":[]string{"application/pdf"}}] file ICME-2019-Tutorial-final.pdf uploaded ok $ls -l upload -rw-r--r-- 1 tonybai staff 264517032 1 14 12:56 ICME-2019-Tutorial-final.pdf  CopyBuffer instead of io.Copy if you feel that 32k is still very large and you want to use a smaller buffer for each upload.   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37  // github.com/bigwhite/experiments/multipart-formdata/client/client4.go func createReqBody(filePath string) (string, io.Reader, error) { var err error pr, pw := io.Pipe() bw := multipart.NewWriter(pw) // body writer f, err := os.Open(filePath) if err != nil { return "", nil, err } go func() { defer f.Close() // text part1 p1w, _ := bw.CreateFormField("name") p1w.Write([]byte("Tony Bai")) // text part2 p2w, _ := bw.CreateFormField("age") p2w.Write([]byte("15")) // file part1 _, fileName := filepath.Split(filePath) h := make(textproto.MIMEHeader) h.Set("Content-Disposition", fmt.Sprintf(form-data; name="%s"; filename="%s", escapeQuotes("file1"), escapeQuotes(fileName))) h.Set("Content-Type", "application/pdf") fw1, _ := bw.CreatePart(h) var buf = make([]byte, 1024) cnt, _ := io.CopyBuffer(fw1, f, buf) log.Printf("copy %d bytes from file %s in total\n", cnt, fileName) bw.Close() //write the tail boundry pw.Close() }() return bw.FormDataContentType(), pr, nil }  Run this client4.  1 2 3 4 5 6  $go run client4.go -file /Users/tonybai/Downloads/ICME-2019-Tutorial-final.pdf 2021/01/10 13:39:06 createReqBody ok 2021/01/10 13:39:06 upload /Users/tonybai/Downloads/ICME-2019-Tutorial-final.pdf... 2021/01/10 13:39:09 copy 264517032 bytes from file ICME-2019-Tutorial-final.pdf in total 2021/01/10 13:39:09 upload /Users/tonybai/Downloads/ICME-2019-Tutorial-final.pdf ok upload file [/Users/tonybai/Downloads/ICME-2019-Tutorial-final.pdf] ok 

You will see that although the upload is successful, its time consumption for uploading increases a lot for large files because only 1k data can be read per read.