There are several common ways to run programs in the background of Linux.

  • ctrl+z
  • screen
  • nohup
  • supervisor
  • systemd

The following practice is based on the Ubuntu2204 system, but theoretically other Linux distributions are also applicable.

1. Distinguish usage scenarios

If the command takes a little longer to execute and you don’t need to see the output, use & as a background run flag or use ctrl+z to hang the process in the background after running, combined with jobs and fg to switch the background tasks of the current session.

If you want the command to stay running after the session ends, consider using screen and nohup, screen for short time tasks (e.g. scripts) and nohup for long time tasks (e.g. proxies).

If there is a need for logging/permission control, screen and nohup are not enough, and it is appropriate to use supervisor and systemd and other daemon programs to create background daemons (web applications).

2. Usage

2.1. ctrl-z

Take ping as an example, send 4 icmp packets to 114.114.114.114 and record the result.

1

ping -c 4 114.114.114.114 > 114-ping.txt &

The effect of the & symbol is similar to ctrl-z after running, so if you don’t want to record the results and want to switch back to view them easily at any time, consider using ctrl+z to suspend the process to the background.

1

ping 114.114.114.114 # Press ctrl+z to hang

After hanging, use jobs -l to view the hung background processes.

1
2

➜  jobs -l
[1]  - 241975 suspended  ping 114.114.114.114

Use the command fg %num to bring the background task process back up and you can see that the process is executing again.

1
2
3

➜  fg %1
64 bytes from 114.114.114.114: icmp_seq=3 ttl=117 time=6.19 ms
64 bytes from 114.114.114.114: icmp_seq=4 ttl=117 time=6.37 ms

The above methods are only valid for the current session, if the session ends, the program will end its run.

2.2. screen

If you want the program to continue running after the session ends, consider using screen, which some Linux distributions need to install themselves.

1

sudo apt install screen

Use screen to create a ping session.

1

screen -R ping

After exiting the session, recreate the session and use the create ping session command again.

1

screen -R ping

You can see that the command is still running.

2.3. nohup

screen is more suitable for executing some scripts than nohup, while nohup is suitable for running long-time tasks. nohup is called no hang up, which means running the program without hanging up, ignoring the system’s Hang Up signal.

Searching the web for ways to use nohup, the more common ways to run it are as follows.

1

nohup command > command.log 2>&1 &

What is more puzzling here is the meaning of 2>&1.

There are three types of inputs and outputs defined in Linux, which are

  • 0: standard input
  • 1: standard output
  • 2: Error output

The standard output 1 can be left unwritten at runtime, so executing command > command.log is equivalent to command 1 > command.log, and after understanding this, the meaning of 2>&1 becomes clearer, indicating that the error output 2 is redirected to the standard output variable &1, and then combined with > to be output to the log in the log.

Writing the full command is equivalent to the following.

1

nohup command 1> command.log 2>&1 &

Note: nohup command 2>&1 > command.log & is incorrectly written, the error output (2) is redirected to the standard output (1) and subsequently the standard output (1) is redirected to the file, so the error output is disabled.

As an example, take the nc program to scan the port and execute it with nuhup as follows.

1

nohup nc -ztv -w 2 114.114.114.114 1-100 > /tmp/nc.log 2>&1 &

Check the /tmp/nc.log file and you can see that it is executing.

1
2
3
4

➜  ~ tail -f /tmp/nc.log 
nc: connect to 114.114.114.114 port 1 (tcp) timed out: Operation now in progress
nc: connect to 114.114.114.114 port 2 (tcp) timed out: Operation now in progress
...

nohup is still very good for performing simple background tasks, but if you have requirements for output log cutting, running users, booting, etc., you need to consider daemon type programs such as supervisor and systemd.

2.4. supervisor

supervisor is a lightweight process control system similar to systemd, used to run programs for long periods of time, often used for web background programs, proxy programs, etc. On most Linux distributions you need to install it yourself, e.g. install and start the service in ubuntu2204.

1
2
3

sudo apt install supervisor
sudo systemctl enable supervisor
sudo systemctl start supervisor

supervisor supports very rich parameter settings, environment settings and daemons compared to nohup. After installing supervisor, take the synchronization software syncthing as an example.

Create the /etc/supervisor/conf.d/syncthing.conf file, as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[program:syncthing]
directory=/home/apps/syncthing/syncthing-linux-amd64-v1.19.0
command=/home/apps/syncthing/syncthing-linux-amd64-v1.19.0/syncthing
autostart=true
autorestart=true
startsecs=10
stdout_logfile=/var/log/supervisor/%(program_name)s-stdout.log
stdout_logfile_maxbytes=5MB
stdout_capture_maxbytes=5MB
stdout_logfile_backups=5
stderr_logfile=/var/log/supervisor/%(program_name)s-stderr.log
stderr_logfile_maxbytes=5MB
stderr_capture_maxbytes=5MB
stderr_logfile_backups=5
user = apps
environment = HOME="/home/apps/syncthing/syncthing-linux-amd64-v1.19.0"

Most of the parameters are very simple and easy to understand, where

  • autorestart: boot self-start
  • stdout_*: standard output log
  • strerr_*: exception output log
  • user: the user who runs the program
  • environment: set the environment parameters for running

Makes the newly created service effective.

1

sudo supervisorctl update

supervisorctl supports multiple operations, such as

  • update: means update the list of services and re-run the new/modified services
  • status: check the running status of all services
  • start/stop/restart: start/stop/restart the service

2.5. systemd

After 2015, systemd is the system suite that comes with most distributions to replace SysV init, providing reliable parallelism during the boot process and centralized management of processes, daemons, services and mount points.

If you are familiar with modern Linux operating systems, you will most likely not be unfamiliar with systemd. systemd is a relatively heavyweight system suite, and unless there is a clear need to use it, it is generally not considered as a way to run small applications in the background. supervisor is sufficient for most application scenarios.

The systemd common user unit configuration directory is as follows (as distinct from the system unit configuration)

  • $HOME/.config/systemd/user
  • /usr/lib/systemd/user
  • $HOME/.local/share/systemd/user
  • /etc/systemd/user

Take the synchronization software syncthing as an example, add a configuration file $HOME/.config/systemd/user/syncthing.service with the following content.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

[Unit]
Description=Sync Files
Documentation=https://syncthing.net
After=network.target

[Service]
Type=simple
Environment=HOME="/home/apps/syncthing/syncthing-linux-amd64-v1.19.0"
ExecStart=/home/apps/syncthing/syncthing-linux-amd64-v1.19.0/syncthing
# Logging requires systemd version > 240
StandardOutput=append:/home/apps/syncthing/syncthing-linux-amd64-v1.19.0/logs/standard-output.log
StandardError=append:/home/apps/syncthing/syncthing-linux-amd64-v1.19.0/logs/standard-error.log

[Install]
WantedBy=multi-user.target

Start the service.

1

systemctl start syncthing --user

If you modify the configuration sheet, you need to run a refresh of systemd.

1

systemctl daemon-reload --user

The following is a description of the key parts of the configuration file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

[Unit]
Description= # Description of the program
Documentation= # Reference Documents
After= # Service start order, e.g. network.target means start the service after the network service starts
Before= # Same as above, indicating that before a service is started
Wants= # Service weak dependency, make sure the service referred to by wants is started properly before starting the service
Requires= # Strong dependency, if the service Requires refers to is no longer running, this service will also end

[Service]
Type= # Service type, value type can be simple/fork/oneshot/dbus/notify/idle
Environment= # Environment variables are added here
EnvironmentFile= # Environment variable files (e.g. .bashrc)
ExecStart= # Command to be executed when starting the process
ExecReload= # Commands to execute when restarting the service
ExecStop= # Commands to execute when stopping the service
ExeStartPre= # Commands to execute before starting the process
ExeStartPost= # Commands executed after starting the process
ExeStopPost= # Commands executed after stopping the process
Restart= # Defines whether to restart the service after exit, values can be no/on-success/on-failure/on-abnormal/on-abort/onwatchdog/always
RestartSec= # Restart interval after process exit

[Install]
WantedBy= # Indicates the attributed target, this part is more complicated, commonly used is multi-user.target and graphical.target