Sometimes we need to check the growth rate of a file, such as a log file, to get a feel for the load on the system, because in general, the faster the logs are written, the heavier the load on the system.

In this article, we will introduce several ways to view the growth rate of logs in Linux, as follows.

Using dd

The first thing to introduce is dd, as the dd command comes with almost all major distributions and requires no additional installation, as follows.

1
2

$ tail -F app.log | dd of=/dev/null status=progress
3875840 bytes (3.9 MB) copied, 8.228797 s, 471 kB/s

As above, use tail -F to get the newly written data, then pipe the data to dd, which copies it to /dev/null, where status=progress is used to show the copy speed, as you can see, the write speed of our log is 471 kB/s.

Using pv

The pv command can be thought of as cp with progress, as follows.

1
2
3
4

$ yum install -y pv

$ tail -F app.log | pv >/dev/null
2.05MiB 0:00:03 [ 330kiB/s] [       <=>                                ]

The principle is similar to dd, but the command is changed to pv.

Using the cv command

Since log data are written by programs (such as java), and in the following pseudo-file, which holds information about files opened by programs, as follows.

  • /proc/<pid>/fd : holds the open file descriptor
  • /proc/<pid>/fdinfo : holds the offset to which the file descriptor is written

So reading this offset periodically will tell you how fast the file is being written, which is how the cv command is implemented, as follows.

1
2
3
4
5
6
7
8

$ yum install -y cv

$ cv -mc java
[    1] java app.log 100.0% (6.1 GiB / 6.1 GiB) 390.2 KiB/s

# In fact, since most commands on Linux (such as cp) don't come with a progress viewer, cv is a good supplement
# For example, to view the progress of a cp copy file
$ cp app.log app.log.bak & cv -mc cp

The new version of the cv command has been renamed to progress. When you can’t install the cv package, you can try to install the progress package.

Write a small script

By writing a small script to periodically observe the file size and also view the file write speed, as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Get the file size per second, and calculate the speed by subtracting the size of the previous second
while sleep 1; do  \
  sz=`stat -c %s app.log`; \
  numfmt --from=auto --to=iec $((sz-psz)); \
  psz=$sz; \
done

492K
750K
370K

A further look through the watch command also gives a rough idea of the speed, as follows.

1

watch -d -t -n1 du app.log

watch file

Think of it this way

In fact, if we output the log in a function, and then filter out this log by grep, then we just count the number of lines of log output per second, is this not the QPS of the function execution!

1
2
3
4
5
6
7

# Use grep to filter out logs, tr to remove non-line breaks, so dd shows the number of line breaks!
$ tail -F app.log \
    | grep --line-buffered '/order/get' \
    | stdbuf -oL tr -dc '\n' \                     
    | dd of=/dev/null bs=1 status=progress

151 bytes (151 B) copied, 15.523018 s, 0.0 kB/s

Here the QPS can be calculated from 151/15 to be 10, and since it does not exceed 1000, it is seen as 0.0kB/s. It would be simpler to use the pv command as follows.

1
2
3
4
5

$ tail -F app.log \
    | grep --line-buffered '/order/get' \
    | pv -l >/dev/null
    
144  0:00:03 [11.5 /s] [       <=>                         ]