Prometheus Gauge

1. What is a Gauge?

Those of you who are familiar with Prometheus will know that Prometheus offers four main metric types.

  • Counter
  • Gauge
  • Histogram
  • Summary

Histogram and Summary are in the same category, but are a little more complex to understand, so we’ll leave that aside for now; Counter only provides an Add method, which is an increasing value, while Gauge, which is also a value, but unlike Counter, provides not only an Add method but also a Sub method. If you have a metric that can be incremented or decremented or need to support negative numbers, then Gauge is clearly a more suitable metric type than Counter.

2. The rationale for the Gauge Add/Sub operation

In the Prometheus Go client package, we see that Gauge is an interface type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// github.com/prometheus/client_golang/prometheus/gauge.go
type Gauge interface {
    Metric
    Collector

    // Set sets the Gauge to an arbitrary value.
    Set(float64)
    // Inc increments the Gauge by 1. Use Add to increment it by arbitrary
    // values.
    Inc()
    // Dec decrements the Gauge by 1. Use Sub to decrement it by arbitrary
    // values.
    Dec()
    // Add adds the given value to the Gauge. (The value can be negative,
    // resulting in a decrease of the Gauge.)
    Add(float64)
    // Sub subtracts the given value from the Gauge. (The value can be
    // negative, resulting in an increase of the Gauge.)
    Sub(float64)

    // SetToCurrentTime sets the Gauge to the current Unix time in seconds.
    SetToCurrentTime()
}

The client package also provides the default implementation type of the interface, gauge.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// github.com/prometheus/client_golang/prometheus/gauge.go
type gauge struct {
    // valBits contains the bits of the represented float64 value. It has
    // to go first in the struct to guarantee alignment for atomic
    // operations.  http://golang.org/pkg/sync/atomic/#pkg-note-BUG
    valBits uint64

    selfCollector

    desc       *Desc
    labelPairs []*dto.LabelPair
}

Looking at the definition of the gauge type, the core field of gauge, which is the instantaneous value of a gauge, is the uint64 type valBits, which stores the instantaneous value represented by the gauge indicator.

However, we see that the arguments to the Add and Sub methods of the Gauge interface type are of type float64. There is no excuse for using float64 as an argument to the methods of the Gauge interface type, because Gauge has to support floating point numbers and decimals, and floating point numbers can be converted to integers, but integers cannot be converted to floating point numbers with a decimal part .

So why does the gauge type use a field of type uint64 rather than a field of type float64 to store the instantaneous value represented by the gauge? This starts with a feature of the Prometheus go client, which is that modifications to the instantaneous value of gauge are goroutine-safe. specifically, gauge uses atomic operations provided by the atomic package to ensure that this concurrent access is safe. However, the atomic package of the standard library supports atomic operations of type uint64, but not of type float64, and the size of both float64 and uint64 is 8 bytes. The Prometheus go client takes advantage of the fact that uint64 supports atomic operations and that both uint64 and float64 are 64bits long to implement the Add and Sub methods of the gauge type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

// github.com/prometheus/client_golang/prometheus/gauge.go

func (g *gauge) Add(val float64) {
    for {
        oldBits := atomic.LoadUint64(&g.valBits)
        newBits := math.Float64bits(math.Float64frombits(oldBits) + val)
        if atomic.CompareAndSwapUint64(&g.valBits, oldBits, newBits) {
            return
        }
    }
}

func (g *gauge) Sub(val float64) {
    g.Add(val * -1)
}

We see that the Sub method actually calls the Add method, but multiplies the val value by -1 as an argument to the Add method. Let’s focus on the gauge’s Add method.

The implementation of the gauge Add method is a typical use of the CAS (CompareAndSwap) atomic operation, i.e. in an infinite loop, the current instantaneous value is read atomically, then it is summed with the incoming incremental value to get the new value, and finally the new value is set to the current instantaneous value by the CAS operation. If the CAS operation fails, the loop is repeated.

However, it is worth looking at the respective functions of the float64 and uint64 types in the Add method and their conversion to each other.

The Add method first reads the value of valBits using the atomic.LoadUint64 atom. It then converts it to float64 using math.Float64frombits, and then adds the instantaneous value of float64 to val to get the new value we want.

The next step is to re-store it in valBits. float64 does not support atomic operations, so the Add method needs to convert the new value back to uint64 before calling CAS, which is why the above code calls math.Float64bits. The newBits is then written to valBits using the atomic.CompareAndSwapUint64 method.

You must be wondering how math.Float64frombits and math.Float64bits do the conversion between uint64 and float64, let’s take a look at their implementation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// $GOROOT/src/math/unsafe.go

// Float64bits returns the IEEE 754 binary representation of f,
// with the sign bit of f and the result in the same bit position,
// and Float64bits(Float64frombits(x)) == x.
func Float64bits(f float64) uint64 { return *(*uint64)(unsafe.Pointer(&f)) }

// Float64frombits returns the floating-point number corresponding
// to the IEEE 754 binary representation b, with the sign bit of b
// and the result in the same bit position.
// Float64frombits(Float64bits(x)) == x.
func Float64frombits(b uint64) float64 { return *(*float64)(unsafe.Pointer(&b)) }

We see that these two functions only do type conversions using the unsafe package and do not do any arithmetic operations.

To summarise.

  • the valBits of type uint64 in the gauge structure are essentially only used as “carriers” for float64 values, and are updated in real time with the support of atomic operations on their type; they do not themselves participate in any integer or floating-point calculations.
  • The operations in the Add method are performed between floating-point types. The Add method reduces the IEEE 754-compliant floating-point representation carried in uint64 to a floating-point type via math.Float64frombits, then sums the input parameters, also of type float64, and the result of the calculation is then converted to a uint64 type via the math.Float64bits function to uint64, without any change in the bit pattern of the 8-byte field, and finally the result value (the new bit pattern) is written to valBits via a CAS operation.

3. Summary

This model of implementing float64 atomic operations via bit-mode conversion, as used by the gauge structure and its Add method, is worth learning from.