Building a Zero-Copy File Web Server in Golang

This article starts with Golang’s file server, then explores what the sendfile system call is, and finally summarizes the usage scenarios for zero-copy.

Build a file server

How to build a zero-copy file server in Golang, here is the complete code.

package main

import "net/http"

func main() {
        http.Handle("/", http.StripPrefix("/static/", http.FileServer(http.Dir("./output"))))
        http.ListenAndServe(":8000", nil)
}

Well, yes. Two lines of code to implement a file server.

How is the FileServer Handler implemented?

For the Handler that handles file requests, the implementation will be very simple according to our idea: determine the file type: if the request is for a directory, return the directory list; if the request is for a file; then copy the file data to the client via io.Copy.

But the real implementation is slightly more complicated than I thought.

By tracing the code, a simple flowchart was drawn as follows.

Handler handling file requests

In the last step tcp Write, that is, write the data to the tcp stream. serveFile uses io.CopyN(w, sendContent, sendSize) When the code is seen here, I feel very satisfied. Because the implementation does not seem to be too different from what we thought.

Next, take a look at the io.CopyN method.

func CopyN(dst Writer, src Reader, n int64) (written int64, err error) {
    written, err = Copy(dst, LimitReader(src, n)) 
    if written == n {
        return n, nil
    }
    if written < n && err == nil {
        // src stopped early; must have been EOF.
        err = EOF
    }
    return
}

Let’s look at the familiar io.Copy method.

func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
    if wt, ok := src.(WriterTo); ok {
        return wt.WriteTo(dst)
    }
    if rt, ok := dst.(ReaderFrom); ok {
        return rt.ReadFrom(src)
    }
    // ...
    // create buffer
    // for {
    //    read
    //    write
    // }
}

Read from src, write to dst. all together the way we want. No problem.

Wait, I think I’ve seen the ReaderFrom interface somewhere?

1
2
3

type ReaderFrom interface {
    ReadFrom(r Reader) (n int64, err error)
}

func (c *TCPConn) ReadFrom(r io.Reader) (int64, error) {
    if !c.ok() {
        return 0, syscall.EINVAL
    }
    n, err := c.readFrom(r)
    if err != nil && err != io.EOF {
        err = &OpError{Op: "readfrom", Net: c.fd.net, Source: c.fd.laddr, Addr: c.fd.raddr, Err: err}
    }
    return n, err
}

The tcp connection implements the ReadFrom interface. What exactly does this implementation do?


func (c *TCPConn) readFrom(r io.Reader) (int64, error) {
    if n, err, handled := splice(c.fd, r); handled {
        return n, err
    }
    if n, err, handled := sendFile(c.fd, r); handled {
        return n, err
    }
    return genericReadFrom(c, r)
}

If the linux binary is compiled, the system call slice is called in the slice method.

If sendfile is called, the internal implementation calls the system call method Sendfile.

What’s so special about sendfile that read/Write isn’t good?

The answer is found in man.

1
2
3

sendfile()  copies  data between one file descriptor and another.  Because this copying is done within the kernel,
sendfile() is more efficient than the combination of read(2) and write(2), which would require  transferring  data
to and from user space.

sendfile is used to copy data between two file descriptors. Since the data operation is done in the kernel state, it avoids copying data between the kernel buffer and the user buffer, so it is called the zero-copy technique. It is much more efficient than the read/write method, which requires a buffered copy.