May 1, 2025 (May 5, 2025)

This blog post is technically incorrect. It only aims to give you an in practice correct enough understanding of using OpenTelemetry in your Go projects and how to set it up. Lets dive in.

Update 2024-05-02: Code has been updated to use the autoexport package to simplify setup a lot.

Concepts

There are a few concepts we need to be aware of:

  • Providers.
  • Meters and Tracers.
  • Metric instruments and Spans.
  • Exporters.
  • Semantic conventions.

Providers

We start with Providers. They are factory methods that construct Meters and Tracers. That’s all you need to care about right now. You can have multiple of these, but you don’t have to. Moving on.

Meters and Tracers

The Tracer and the Meter create Spans and Metric Instruments, respectively. The Meter and the Tracer represent an “instrumentation scope”, usually a library or a service.

They can be requested from your Trace Provider or Meter Provider. Other libraries may create their own. Tracers and Meters attach some metadata like the fact that they were the one to create a span or a metric instrument.

Metric instruments and Spans

A Metric Instrument lets you measure a thing. They can be counters, gauges, histograms and so on. They’re what you typicall refer to as a metric.

A Span captures one part of the execution of your program. You typically start it at some root, like the HTTP Handler. As your program executes you create child spans for meaningful works of computation that you want to independently track in the graph. Spans can have additional attributes, events and other things associated with them. They also have a Status.

You’ll usually include a middleware at the top of your HTTP handler stack which will emit metrics for the request and automatically create the root span for you.

Exporters

Finally, exporters. This is how you send traces and metrics elsewhere. For traces we’ll use the OTLP protocol using gRPC or HTTP. For metrics you can additionally expose them using the Prometheus exposition format so you can scrape them instead. There are also stdout exporters that you can use during development.

Semantic Conventions

For traces, you can think of them as a very wide structured logging line. It has attributes which can be nested and so on. Semantic convetions provides a number of predefined keys and namespaces for commonly used attributes so various systems can emit things under the same name with the same semantics. It makes dashboarding and correlating things across services easier.

If there is a semantic convention attribute, always prefer it over a custom attribute.

Setting up traces and meters

We’ll need to start with a bunch of boiler plate to create and set our trace and meter providers. We will set this as our default trace and meter providers, so we can easily access them for elsewhere in our code.

We will use the autoexport package to help with automatically configuring the span exporters and the metric readers using standard OTEL environment variables. You can find the documented here.

Resource

First, we need a Resource. A resource represents an “entity” that’s creating the metrics and traces. Our code, if you will. This provides a bunch of metadata, but otherwise doesn’t do much. We’ll create a schemaless one as it makes life a bit easier when mix-matching libraries that may use different versions of the semantic convetions.

import (
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.30.0"
)

func defaultResource() *resource.Resource {
	r, _ := resource.Merge(
		resource.Default(),
		resource.NewSchemaless(
			semconv.ServiceName("our-backend"),
			semconv.ServiceVersion("our-backend-version"),
		),
	)

	return r
}

Trace provider

Next, a trace provider.

First up, automatically configure a span exporter:

import (
    "context"
    "log"

	"go.opentelemetry.io/contrib/exporters/autoexport"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func InitTracer(ctx context.Context) sdktrace.SpanExporter {
   	se, err := autoexport.NewSpanExporter(ctx)
	if err != nil {
		log.Fatalf("Failed to create OTEL span exporter: %s", err)
	}
	return se
}

Now, lets hoop it up to the trace provider:

import ( 
	"context"
	"log"
	"time"

	"go.opentelemetry.io/contrib/exporters/autoexport"
	"go.opentelemetry.io/otel"
	metricnoop "go.opentelemetry.io/otel/metric/noop"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.30.0"
	tracenoop "go.opentelemetry.io/otel/trace/noop"
)

func InitOtel(ctx context.Context,
	metrics, traces bool,
) func() {
	shutdownFuncs := []func(){}
	if !traces {
		otel.SetTracerProvider(tracenoop.NewTracerProvider())
		shutdownFuncs = append(shutdownFuncs, func() {})
	}

	resource := defaultResource()

	if traces {
		tp := sdktrace.NewTracerProvider(
			sdktrace.WithResource(resource),
			sdktrace.WithBatcher(initTracer(ctx)),
		)

		otel.SetTracerProvider(tp)
		otel.SetTextMapPropagator(
			propagation.NewCompositeTextMapPropagator(
				propagation.TraceContext{},
				propagation.Baggage{},
			),
		)

		shutdownFuncs = append(shutdownFuncs, func() {
			tpCtx, tpCancel := context.WithTimeout(context.Background(), 10*time.Second)
			defer tpCancel()
			if err := tp.Shutdown(tpCtx); err != nil {
				log.Printf("Failed to gracefully shut down traces exporter: %s", err)
			}
		})
	}

	return func() {
		for _, f := range shutdownFuncs {
			f()
		}
	}
}

Our InitOtel() returns a func() that the caller can defer. This will try to gracefully shutdown a trace provider.

Meter provider

The meter provider is the same.

import (
    "context"
    "log"

    "go.opentelemetry.io/otel/sdk/metric"
)

func InitMeter(ctx context.Context) metric.Reader {
	mr, err := autoexport.NewMetricReader(ctx)
	if err != nil {
		log.Fatalf("Failed to create OTEL metric reader: %s", err)
	}
	return mr
}

And plug it into our InitOtel() function:

import ( 
	"context"
	"log"
	"time"

	"go.opentelemetry.io/contrib/exporters/autoexport"
	"go.opentelemetry.io/otel"
	metricnoop "go.opentelemetry.io/otel/metric/noop"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.30.0"
	tracenoop "go.opentelemetry.io/otel/trace/noop"
)

func InitOtel(ctx context.Context,
	metrics, traces bool,
) func() {
	shutdownFuncs := []func(){}
	if !traces {
		otel.SetTracerProvider(tracenoop.NewTracerProvider())
		shutdownFuncs = append(shutdownFuncs, func() {})
	}

	if !metrics {
		otel.SetMeterProvider(metricnoop.NewMeterProvider())
		shutdownFuncs = append(shutdownFuncs, func() {})
	}

	resource := defaultResource()

	if traces {
		tp := sdktrace.NewTracerProvider(
			sdktrace.WithResource(resource),
			sdktrace.WithBatcher(initTracer(ctx)),
		)

		otel.SetTracerProvider(tp)
		otel.SetTextMapPropagator(
			propagation.NewCompositeTextMapPropagator(
				propagation.TraceContext{},
				propagation.Baggage{},
			),
		)

		shutdownFuncs = append(shutdownFuncs, func() {
			tpCtx, tpCancel := context.WithTimeout(context.Background(), 10*time.Second)
			defer tpCancel()
			if err := tp.Shutdown(tpCtx); err != nil {
				log.Printf("Failed to gracefully shut down traces exporter: %s", err)
			}
		})
	}

	if metrics {
		mp := metric.NewMeterProvider(
			metric.WithResource(resource),
			metric.WithReader(initMeter(ctx)),
		)

		otel.SetMeterProvider(mp)
		shutdownFuncs = append(shutdownFuncs, func() {
			mpCtx, mpCancel := context.WithTimeout(context.Background(), 10*time.Second)
			defer mpCancel()
			if err := mp.Shutdown(mpCtx); err != nil {
				log.Printf("Failed to gracefully shut down metrics exporter: %s", err)
			}
		})
	}

	return func() {
		for _, f := range shutdownFuncs {
			f()
		}
	}
}

The autoexport automatically creates a separate Prometheus registry for its metrics. It automatically starts a second HTTP server with just the /metrics endpoint. See the autoexport documentation for how to configure the host and port this binds to.

Getting a tracer or a meter

Since we’ve installed our providers as the defaults, we can simply call otel.Meter("name") or otel.Tracer("name"). If a meter or tracer by that name doesn’t yet exist, they’ll be created on the fly. Otherwise it’ll return the meter or tracer matching that name.

The name should be a package name, like my.awesome.tld/backend/internal/metrics. Other libraries you use may also create their own tracers and meters from the default providers. This lets you distinguish where they came from.

You can call these functions whenever. If no Meter or Tracer providers have been installed, they’ll be no-op tracers and meters. They’ll be recreated automatically once a meter or trace provider is created. This means it’s not strictly necessary to set the noop Meter and Trace providers in our Init functions. But we do it to be explicit.

It can be helpful to create a meter and tracer once, and then access them through package globals, dependency injection or by explicitly passing them to functions or methods. But you can also call the otel.Meter() or otel.Tracer() functions anywhere.

Creating spans and metrics

Tracing and metrics work a bit differently at this point, so lets look at those independently.

Spans

There are two ways to create child spans, in the context of an HTTP handler.

You can either do so directly, if you have access to the tracer:

newCtx, childSpan := tracer.Start(r.Context(), "child-operation", trace.WithSpanKind(trace.SpanKindServer))
defer childSpan.End()

Alternatively, you can get the tracer from an existing span:

tracer := trace.SpanFromContext(r.Context()).TracerProvider().Tracer("name")

Adding additional attributes to a span can be done with SetAttributes on a span. There are additional methods for various things.

Metrics

Here we’ll need to create a meter first, and store it somewhere. A package global, through dependency injection, by passing it into a function etc. Whatever suits your application. Then we’ll need to do the same for our meter instrument:

meter = otel.Meter("name")
reqC, _ := meter.Int64Counter(
		"http.server.requests",
		metric.WithDescription("Total number of HTTP requests"),
		metric.WithUnit("{request}"),
)

Make sure to check the documentation for metric.WithUnit to understand what values you can set.

In any part of the code, like an HTTP handler, you can:

attrs := []attribute.KeyValue{
    attribute.String("http.method", r.Method),
    attribute.String("http.route", r.URL.Path),
}

reqC.Add(ctx, 1, metric.WithAttributes(attrs...))

You should ensure you set the same attributes for the same metric. Take a look at all the examples in the metric package.

Conclusion

Congratulations, you now know enough OpenTelemetry to be dangerous. Go forth and instrument!

Remember:

  • As a library, do not set the default trace and meter providers. Your consumers should do that. Do create your own Tracer and Meter.
  • Initialise the Trace and Meter providers as early as possible in package main. Ideally right after flag parsing and the initial context handling.
  • Have a way to gracefully shut down the providers.
  • Look at OpenTelemetry Go Contrib for middleware for various Go HTTP frameworks and a http.RoundTripper for http.Client.
  • Documentation for the OTEL environment variables can be found here.
tech  golang  opentelemetry  observability