Go os/exec Short Tutorial

The Go standard library provides convenient methods to run external commands easily. Generally we use the methods under the os/exec package to execute external commands and interact with external commands. os/exec wraps the os.StartProcess method for easier access to input and output, providing I/O pipe and other features. I will dedicate two articles to Go’s methods for starting new processes and executing external commands, this is the first one, dedicated to the os/exec library.

The os/exec library provides methods similar to those defined in the POSIX standard for the C language, but provides further encapsulation to make it easier to use. I will introduce you to each of them next.

“os/exec”

Run a command

The simplest way to run an external command is to call the Command method, you need to pass in the program to be executed and parameters, it will return a *Cmd data structure, representing an external command to be executed, the main call to its Run, Output, CombinedOutput method after this object can not be reused, and generally we will not reuse this object, but generate a new one when needed.

The following code is to execute the ls command and pass it the parameter -lah.

cmd := exec.Command("ls", "-lah")
err := cmd.Run()
if err != nil {
	log.Fatalf("failed to call cmd.Run(): %v", err)
}

If you execute this command, there is no output in the console, in fact the command has been executed, only our Go program does not capture and process the output, so there is no output in the console.

Run method will execute the external command and wait for the command to finish, if the command is executed normally without errors and the return code is 0, then Run returns err == nil, then it returns an *ExitError, sometimes you need to read cmd.

If you don’t want to wait, then you can call the Start method. If Start succeeds, the Process and ProcessState fields will be set. You can check ProcessState.Exited() to determine if the program has exited. If you want to block again and wait for the program to finish, you can call the Wait method.

In fact, the Run method is implemented using the Start and Wait methods

func (c *Cmd) Run() error {
	if err := c.Start(); err != nil {
		return err
	}
	return c.Wait()
}

Display the output of external commands

The Cmd command contains input and output fields that you can set to enable customization of input and output:

1
2
3

Stdin io.Reader
Stdout io.Writer
Stderr io.Writer

If Stdin is null, then the process will read from null device(os.DevNull).

If Stdin is an *os.File object, then it will read from this file.

If Stdin is os.Stdin, then it will read from standard input like command line.

Stdout and Stderr represent the standard output and error output of the external program process, and if they are null, then they are output to the null device.

If Stdout and Stderr are *os.File objects, then the data is output to a file.

If Stdout and Stderr are set to os.Stdout and os.Stderr respectively, it will output to the command line.

Let’s adapt the previous example to show the command output.

cmd := exec.Command("ls", "-lah")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
	log.Fatalf("failed to call cmd.Run(): %v", err)
}

Work Dir

By default the working path of the process is the folder where the process is called, but you can also specify it manually, for example we specify the working path as the root path.

cmd := exec.Command("ls", "-lah")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Dir = "/"
err := cmd.Run()
if err != nil {
	log.Fatalf("failed to call cmd.Run(): %v", err)
}

External program path

Cmd.Path is the path of the program to be executed, if it is a relative path, then it calculates the relative path based on Cmd.Dir. If the program is already in the system $PATH path, then you can write the program name directly.

cmd := exec.Command("/usr/local/go/bin/go", "env")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
log.Printf("path: %s", cmd.Path)
err := cmd.Run()
if err != nil {
	log.Fatalf("failed to call cmd.Run(): %v", err)
}

Set environment variables

Cmd also has a field called Env, which is used to set the environment variable of the process in the format key=value.

If Env is empty, then the new process will use the environment variables of the caller process.

For example, in the following example, we set the myvar variable, you can comment out the line cmd.Enc = ... that line to compare the results.

cmd := exec.Command("bash", "-c", "echo $myvar")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Env = []string{"myvar=abc"}
err := cmd.Run()
if err != nil {
	log.Fatalf("failed to call cmd.Run(): %v", err)
   }

The underlying Process and ProcessState

As I mentioned above, os/exec is a wrapped convenience library, and at the bottom it is implemented using os.StartProcess, so you can get the underlying Process object and ProcessState object, which represent the process and the process state respectively

cmd := exec.Command("bash", "-c", "sleep 1;echo $myvar")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Start()
if err != nil {
	log.Fatalf("failed to call cmd.Start(): %v", err)
}
log.Printf("pid: %d", cmd.Process.Pid)
cmd.Process.Wait()
log.Printf("exitcode: %d", cmd.ProcessState.ExitCode())

Determine if an external command exists

Sometimes you need to check if an external command exists when you execute it, you can use the LookPath method.

If the incoming argument contains a path separator, then it will look for the program based on the relative or absolute path of Cmd.Dir. If it does not contain a path separator, then it will look for the file from the PATH environment variable

path, err := exec.LookPath("ls")
if err != nil {
	log.Printf("'ls' not found")
} else {
	log.Printf("'ls' is in '%s'\n", path)
}

Get command results

Cmd provides the Output() method to get the bytes of the command execution result if the command is executed correctly:

cmd := exec.Command("ls", "-lah")
data, err := cmd.Output()
if err != nil {
	log.Fatalf("failed to call Output(): %v", err)
}
log.Printf("output: %s", data)

If there is an error with the command, the error message can be obtained with Stderr :

cmd := exec.Command("ls", "-lahxyz")
cmd.Stderr = os.Stderr
data, err := cmd.Output()
if err != nil {
	log.Fatalf("failed to call Output(): %v", err)
}
log.Printf("output: %s", data)

Combining Stdout and Stderr

If you want to have one way to get the output regardless of errors, you can call the CombinedOutput() method, which will return either normal or error output, and a second return err to indicate whether an error was executed

cmd := exec.Command("ls", "-lah")
data, err := cmd.CombinedOutput()
if err != nil {
	log.Fatalf("failed to call CombinedOutput(): %v", err)
}
log.Printf("output: %s", data)

The implementation of the CombinedOutput method is also very simple, in fact it is implemented by sharing the same bytes.Buffer

func (c *Cmd) CombinedOutput() ([]byte, error) {
	if c.Stdout != nil {
		return nil, errors.New("exec: Stdout already set")
	}
	if c.Stderr != nil {
		return nil, errors.New("exec: Stderr already set")
	}
	var b bytes.Buffer
	c.Stdout = &b
	c.Stderr = &b
	err := c.Run()
	return b.Bytes(), err
}

Read Stdout and Stderr separately

Once we understand the implementation of CombinedOutput, we can set bytes.Buffer for Stdout and Stderr respectively, to achieve independent reads.

cmd := exec.Command("ls", "-lah")
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil {
	log.Fatalf("failed to call Run(): %v", err)
}
log.Printf("out:\n%s\nerr:\n%s", stdout.String(), stderr.String())

Show command execution progress

Since we have been able to use our own io.Writer to set Stdout/Stderr, we can do something richer.

For example, if we use the curl command to download a large file, we can display the size of the downloaded data in real time:

cmd := exec.Command("curl", "https://dl.google.com/go/go1.15.6.linux-amd64.tar.gz")
var stdoutProcessStatus bytes.Buffer
cmd.Stdout = io.MultiWriter(ioutil.Discard, &stdoutProcessStatus)
done := make(chan struct{})
go func() {
	tick := time.NewTicker(time.Second)
	defer tick.Stop()
	for {
		select {
		case <-done:
			return
		case <-tick.C:
			log.Printf("downloaded: %d", stdoutProcessStatus.Len())
		}
	}
}()
err := cmd.Run()
if err != nil {
	log.Fatalf("failed to call Run(): %v", err)
}
   close(done)

Set Stdin

While the previous examples demonstrated handling Output, this next example shows how to set up Stdin.

The wc command reads the main.go file and counts how many lines it has in total.

stdin, err := os.Open("main.go")
if err != nil {
	log.Fatalf("failed to open file: %v", err)
}
cmd := exec.Command("wc", "-l")
cmd.Stdin = stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
if err != nil {
	log.Fatalf("failed to call cmd.Run(): %v", err)
   }

Pipe

You can use the output of one command as input to the next command, and so on, to string multiple commands into a pipe.

os/exec provides StderrPipe, StdinPipe, and StdoutPipe methods to get the pipe object.

For example, the following command takes the output of cat main.go as input to the wc -l command:

cmdCat := exec.Command("cat", "main.go")
catout, err := cmdCat.StdoutPipe()
if err != nil {
	log.Fatalf("failed to get StdoutPipe of cat: %v", err)
}
cmdWC := exec.Command("wc", "-l")
cmdWC.Stdin = catout
cmdWC.Stdout = os.Stdout
err = cmdCat.Start()
if err != nil {
	log.Fatalf("failed to call cmdCat.Run(): %v", err)
}
err = cmdWC.Start()
if err != nil {
	log.Fatalf("failed to call cmdWC.Start(): %v", err)
}
cmdCat.Wait()
cmdWC.Wait()

First, the pipeline is created, and then the Start method and Wait method of each command are called in turn.

Generic Pipe method

Here is a more generic way to create a Cmd pipeline.

func main() {
	cmdCat := exec.Command("cat", "main.go")
	cmdWC := exec.Command("wc", "-l")
	data, err := pipeCommands(cmdCat, cmdWC)
	if err != nil {
		log.Fatalf("failed to call pipeCommands(): %v", err)
	}
	log.Printf("output: %s", data)
}
func pipeCommands(commands ...*exec.Cmd) ([]byte, error) {
	for i, command := range commands[:len(commands)-1] {
		out, err := command.StdoutPipe()
		if err != nil {
			return nil, err
		}
		command.Start()
		commands[i+1].Stdin = out
	}
	final, err := commands[len(commands)-1].Output()
	if err != nil {
		return nil, err
	}
	return final, nil
}

bash pipe

If you execute it via the bash command, you can use the bash pipe feature, which is simpler to write.

cmd := exec.Command("bash", "-c", "cat main.go| wc -l")
data, err := cmd.CombinedOutput()
if err != nil {
	log.Fatalf("failed to call pipeCommands(): %v", err)
}
   log.Printf("output: %s", data)

Orphan Process

When the parent process ends before the child process does, then the child process is called an orphan process and the ppid of the child process is set to 1 at this time.

cmd := exec.Command("curl", "-o", "go1.15.6.linux-amd64.tar.gz", "https://dl.google.com/go/go1.15.6.linux-amd64.tar.gz")
err := cmd.Start()
if err != nil {
	log.Fatalf("failed to call Run(): %v", err)
   }

When the main program exits, this curl sub-process will keep downloading and its ppid is set to 1. You can use ps to view the process information, for example in a Mac.

`1`	`ps xao pid,ppid,pgid,sid,comm \| grep curl`

Kill child processes when the program exits

If we want to kill the child process started by the program when it exits, then a stupid way is to get the Process object of the child process and call its Kill method to kill it. But unfortunately it can’t kill the Sun process.

For Linux systems, you can kill the grandchild process as well with the following settings.

cmd := exec.Command("curl", "-o", "go1.15.6.linux-amd64.tar.gz", "https://dl.google.com/go/go1.15.6.linux-amd64.tar.gz")
cmd.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL}
err := cmd.Start()
if err != nil {
	log.Fatalf("failed to call Run(): %v", err)
}

Or more generally, use the following settings:

cmd := exec.Command("curl", "-o", "go1.15.6.linux-amd64.tar.gz", "https://dl.google.com/go/go1.15.6.linux-amd64.tar.gz")
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
err := cmd.Start()
if err != nil {
	log.Fatalf("failed to call Run(): %v", err)
}

see https://groups.google.com/g/golang-nuts/c/XoQ3RhFBJl8

Pass the file opened by the parent process to the child process

In addition to the standard input and output of 0,1,2 files, you can also pass files from the parent process to the child process via the Cmd.ExtraFiles field.

One of the more common scenarios is graceful restart, where the new process inherits the net.Listener that the old process is listening to, so that the network connection does not need to be closed and reopened.

For example, one of the earliest articles introducing the go graceful restart technique, Graceful Restart in Golang, has the following code

file := netListener.File() // this returns a Dup()
path := "/path/to/executable"
args := []string{
    "-graceful"}
cmd := exec.Command(path, args...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.ExtraFiles = []*os.File{file}
err := cmd.Start()
if err != nil {
    log.Fatalf("gracefulRestart: Failed to launch, error: %v", err)
}

Reference https://colobu.com/2020/12/27/go-with-os-exec/

Table of Contents