Potential Resource Leak from Periodic Reader Goroutines After Reporter Eviction

[jcswiftie13]

Description

Summary

A potential resource leak has been identified in the metrics reporting pipeline where periodic reader goroutines continue running after their associated reporters are evicted from the LRU cache, preventing garbage collection and causing the OTEL endpoint to receive empty metric exports.

Root Cause

1. Per-Service Metric Provider Creation
   When the MetricsReporter processes spans in the onSpan function, it creates a per-service Metrics instance through the ReporterPool. Each Metrics instance contains a MeterProvider with a PeriodicReader that exports metrics at configured intervals.
   

   opentelemetry-ebpf-instrumentation/pkg/export/otel/metrics.go

   Lines 1125 to 1151 in 1807454

   	func (mr *MetricsReporter) onSpan(spans []request.Span) {
   	for i := range spans {
   	s := &spans[i]
   	if s.InternalSignal() {
   	continue
   	}
   	if !s.Service.ExportModes.CanExportMetrics() {
   	continue
   	}
   	// If we are ignoring this span because of route patterns, don't do anything
   	if request.IgnoreMetrics(s) {
   	continue
   	}
   	reporter, err := mr.reporters.For(&s.Service)
   	if err != nil {
   	mlog().Error("unexpected error creating OTEL resource. Ignoring metric",
   	"error", err, "service", s.Service)
   	continue
   	}
   	reporter.record(s, mr)
   	if mr.commonCfg.Features.AppHost() {
   	hostInfo, attrs := mr.hostInfo.ForRecord(s)
   	hostInfo.Record(mr.ctx, 1, instrument.WithAttributeSet(attrs))
   	}
   	}
   	}

   
   Each service gets its own MeterProvider with a PeriodicReader:
   

   opentelemetry-ebpf-instrumentation/pkg/export/otel/metrics.go

   Lines 651 to 666 in 1807454

   	opts := []metric.Option{
   	metric.WithResource(resources),
   	metric.WithReader(metric.NewPeriodicReader(mr.exporter,
   	metric.WithInterval(mr.cfg.Interval))),
   	}
   	opts = append(opts, mr.otelMetricOptions(mlog)...)
   	opts = append(opts, mr.spanMetricOptions(mlog)...)
   	return Metrics{
   	ctx: mr.ctx,
   	service: service,
   	provider: metric.NewMeterProvider(
   	opts...,
   	),
   	}

2. Background Goroutine in PeriodicReader
   The PeriodicReader starts a background goroutine that continuously collects and exports metrics on a ticker interval
   

   opentelemetry-ebpf-instrumentation/pkg/export/otel/metric/periodic_reader.go

   Lines 126 to 129 in 1807454

   	go func() {
   	defer func() { close(r.done) }()
   	r.run(ctx, conf.interval)
   	}()

   
   

   opentelemetry-ebpf-instrumentation/pkg/export/otel/metric/periodic_reader.go

   Lines 163 to 181 in 1807454

   	func (r *PeriodicReader) run(ctx context.Context, interval time.Duration) {
   	ticker := newTicker(interval)
   	defer ticker.Stop()
   	for {
   	select {
   	case <-ticker.C:
   	err := r.collectAndExport(ctx)
   	if err != nil {
   	otel.Handle(err)
   	}
   	case errCh := <-r.flushCh:
   	errCh <- r.collectAndExport(ctx)
   	ticker.Reset(interval)
   	case <-ctx.Done():
   	return
   	}
   	}
   	}

3. Reporter Eviction Without Goroutine Cleanup
   After the TTL expires and expireOldReporters removes a reporter from the cache, the eviction callback only calls ForceFlush asynchronously, but does not call Shutdown
   

   opentelemetry-ebpf-instrumentation/pkg/export/otel/otelcfg/common.go

   Lines 293 to 306 in 1807454

   	func (rp *ReporterPool[K, T]) expireOldReporters() {
   	now := rp.clock()
   	if now.Sub(rp.lastExpiration) < rp.ttl {
   	return
   	}
   	rp.lastExpiration = now
   	for {
   	_, v, ok := rp.pool.GetOldest()
   	if !ok || now.Sub(v.lastAccess) < rp.ttl {
   	return
   	}
   	rp.pool.RemoveOldest()
   	}
   	}

   
   

   opentelemetry-ebpf-instrumentation/pkg/export/otel/metrics.go

   Lines 284 to 299 in 1807454

   	mr.reporters = otelcfg.NewReporterPool[*svc.Attrs, *Metrics](cfg.ReportersCacheLen, cfg.TTL, timeNow,
   	func(id svc.UID, v *Metrics) {
   	llog := log.With("service", id)
   	llog.Debug("evicting metrics reporter from cache")
   	v.cleanupAllMetricsInstances()
   	if !mr.pidTracker.ServiceLive(id) {
   	mr.deleteTargetMetrics(&id)
   	}
   	go func() {
   	if err := v.provider.ForceFlush(ctx); err != nil {
   	llog.Warn("error flushing evicted metrics provider", "error", err)
   	}
   	}()
   	}, mr.newMetricSet)

Impact

The periodic reader goroutine continues running indefinitely because:

1. The goroutine only stops when ctx.Done() is signaled (line 177 in periodic_reader.go)
2. The context is only canceled in the Shutdown method (line 320 in periodic_reader.go)
3. The eviction callback only calls ForceFlush, not Shutdown

This causes:

• Memory leak: The evicted Metrics instance and its MeterProvider cannot be garbage collected
• Unnecessary network traffic: The OTEL endpoint continues receiving metric exports with no actual data
• Resource waste: Background goroutines accumulate over time for services that are no longer active

The Dilemma

Calling MeterProvider.Shutdown() in the eviction callback would solve the goroutine leak, but creates another problem

opentelemetry-ebpf-instrumentation/pkg/export/otel/metric/provider.go

Lines 129 to 143 in 1807454

	func (mp *MeterProvider) Shutdown(ctx context.Context) error {
	// Even though it may seem like there is a synchronization issue between the
	// call to `Store` and checking `shutdown`, the Go concurrency model ensures
	// that is not the case, as all the atomic operations executed in a program
	// behave as though executed in some sequentially consistent order. This
	// definition provides the same semantics as C++'s sequentially consistent
	// atomics and Java's volatile variables.
	// See https://go.dev/ref/mem#atomic and https://pkg.go.dev/sync/atomic.
	mp.stopped.Store(true)
	if mp.shutdown != nil {
	return mp.shutdown(ctx)
	}
	return nil
	}

The MeterProvider.Shutdown() calls the unified shutdown function for all readers

opentelemetry-ebpf-instrumentation/pkg/export/otel/metric/config.go

Lines 25 to 35 in 1807454

	func (c config) readerSignals() (forceFlush, shutdown func(context.Context) error) {
	var fFuncs, sFuncs []func(context.Context) error
	for _, r := range c.readers {
	sFuncs = append(sFuncs, r.Shutdown)
	if f, ok := r.(interface{ ForceFlush(context.Context) error }); ok {
	fFuncs = append(fFuncs, f.ForceFlush)
	}
	}
	return unify(fFuncs), unifyShutdown(sFuncs)
	}

Which in turn calls the PeriodicReader.Shutdown()
The critical issue: Line 338 in periodic_reader.go calls r.exporter.Shutdown(ctx), which would shut down the shared exporter instance used by all services' metric providers, breaking metrics export for all other active services.

opentelemetry-ebpf-instrumentation/pkg/export/otel/metric/periodic_reader.go

Lines 309 to 350 in 1807454

	func (r *PeriodicReader) Shutdown(ctx context.Context) error {
	err := ErrReaderShutdown
	r.shutdownOnce.Do(func() {
	// Prioritize the ctx timeout if it is set.
	if _, ok := ctx.Deadline(); !ok {
	var cancel context.CancelFunc
	ctx, cancel = context.WithTimeout(ctx, r.timeout)
	defer cancel()
	}
	// Stop the run loop.
	r.cancel()
	<-r.done
	// Any future call to Collect will now return ErrReaderShutdown.
	ph := r.sdkProducer.Swap(produceHolder{
	produce: shutdownProducer{}.produce,
	})
	if ph != nil { // Reader was registered.
	// Flush pending telemetry.
	m := r.rmPool.Get().(*sdkmetricdata.ResourceMetrics)
	err = r.collect(ctx, ph, m)
	if err == nil {
	err = r.export(ctx, m)
	}
	r.rmPool.Put(m)
	}
	sErr := r.exporter.Shutdown(ctx)
	if err == nil || errors.Is(err, ErrReaderShutdown) {
	err = sErr
	}
	r.mu.Lock()
	defer r.mu.Unlock()
	r.isShutdown = true
	// release references to Producer(s)
	r.externalProducers.Store([]Producer{})
	})
	return err
	}

Steps to Reproduce

Environment:
Run opentelemetry-ebpf-instrumentation like the docker compose example on opentelemetry docs

• opentelemetry-ebpf-instrumentation version: v0.3.0
• Setup: Docker Compose with PostgreSQL database

version: "3.7"

services:
  postgres:
    image: postgres:16.10
    ports:
      - 5432:5432
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
  obi:
    ​image: otel/ebpf-instrument:v0.3.0
    environment:
    	- OTEL_EBPF_OPEN_PORT=5432
    	- OTEL_EBPF_CONFIG_PATH=/etc/obi/config.yaml
    volumes:
    	- ./obi-config.yaml:/etc/obi/config.yaml
    privileged: true
    pid: "service:postgres"
    depends_on:
      - postgres

# obi-config.yaml
  
log_level: DEBUG

otel_metrics_export:
  ttl: 5m
  endpoint: http://otel-collector:4317
  features:
    - network

Steps:

1. Deploy the above configuration in the Docker Compose environment
2. Connect to PostgreSQL database and issue a create table command
3. Observe that the metrics reporter is instantiated
4. After the TTL has passed, connect to PostgreSQL database and issue a create table command to trigger expiration
5. Observe that the eviction callback is being called
6. Note that there are now two metrics reporter sending messages, the old one did not dissapear

Expected Behavior

When a metrics reporter is evicted from the cache, the following shutdown sequence should occur:

1. Complete MeterProvider Shutdown: The MeterProvider associated with the evicted reporter must be fully shut down by calling its Shutdown() method, not just ForceFlush(). This ensures that the PeriodicReader properly terminates its background goroutine, preventing resource leaks.
2. Exporter Independence: The PeriodicReader.Shutdown() implementation calls r.exporter.Shutdown(ctx), but since multiple MeterProvider instances share the same exporter instance, the shutdown of one reader should not shut down the shared exporter and must not affect other active readers.

Questions for Maintainers

1. Is the current behavior intentional? Is it expected that periodic reader goroutines continue running after reporter eviction, or is this an oversight?
2. What is the intended lifecycle management strategy for per-service MeterProvider instances in the context of TTL-based cache eviction?
3. Why does PeriodicReader.Shutdown() call exporter.Shutdown()? Given that multiple readers may share the same exporter instance, shouldn't the exporter lifecycle be managed separately from individual readers?
4. What is the recommended approach to properly clean up evicted reporters without affecting other active services?

[grcevski]

Hi, is it possible to get a pprof diff showing the goroutine leak?

[jcswiftie13]

Sure. I have collected the requested pprof diff data.

To expedite the reproduction and ensure precise cache eviction, I configured the metrics expiration TTL to 1 minute (down from the default 5 minutes).

I also wrote a script to simulate the workload with deliberate pauses to rule out race conditions.

Reproduction Methodology:

1. Configuration: Set the instrumentation metrics TTL to 1m.

2. Base Profile: Captured base.pprof (system idle).

3. Workload Script: Ran a script that executed exactly 15 iterations.

   • Each iteration followed this sequence:
     • Connect to Postgres.
     • Sleep 10s (wait for instrumentation attach).
     • Create Table.
     • Sleep 10s.
     • Drop Table.
     • Sleep 10s.
     • Close Connection.

4. Wait: After the script finished, I waited for >1 minute (to strictly exceed the configured TTL) to ensure all inactive sessions were evicted.

5. Target Profile: Captured target.pprof.

Findings: The attached pprof diff (go tool pprof -base base.pprof target.pprof) clearly shows a net increase of +15 goroutines executing (*PeriodicReader).run (and related internal structures).

Since the loop ran 15 times and the connection was explicitly closed each time, the expected diff should be zero. The fact that the leaked count (+15) matches the iteration count exactly—even after the TTL period has passed—confirms a deterministic leak where the monitoring goroutine fails to exit.

[grcevski]

Out of curiosity, which LLM am I talking to?

[jcswiftie13]

When drafting the initial issue, I used DeepWiki. For the subsequent comment, Gemini. I use them only to articulate the problem more clearly and comprehensively, thereby aiming to reduce potential miscommunication.

[jcswiftie13]

I can understand the use of LLM might cause you concern on whether or not this is an actual issue and I'm sorry for that, as I'm not a native English speaker and not that experienced in Golang, I thought that it would help me check if my methodology is rigorous enough and if the information conveyed is clear enough. This is a real problem I encountered and I hope we can work together to improve Open-telemetry's ecosystem

[grcevski]

OK, thanks for explaining @jcswiftie13. How about we start from the beginning, are you running into memory leaks issues while using OpenTelemetry eBPF Instrumentation? How long are you running for and how many services are you instrumenting?

The reason that I'm asking is that the configuration you provided doesn't make any sense with the problem you are describing. Your configuration specifies only "network" as export features, however, the code you are pointing out is for application observability. The affected code will never run with "network" only metrics, where we don't even keep per service information.

[jcswiftie13]

I apologize if my previous explanation of the reproduction steps was confusing. I understand your concerns regarding the configuration.

My initial setup of only the "network" feature enabled was primarily due to an observation I made in the v0.3.0 tag of the OpenTelemetry eBPF Instrumentation codebase.

Specifically, in my docker-compose setup, the environment variable OTEL_EBPF_OPEN_PORT was set. This led OBI to consider the application feature as enabled, based on the first condition on line 441 of config.go. You can see this here:

opentelemetry-ebpf-instrumentation/pkg/obi/config.go

Lines 436 to 445 in 6acd646

	func (c *Config) Enabled(feature Feature) bool {
	switch feature {
	case FeatureNetO11y:
	return c.NetworkFlows.Enable || c.promNetO11yEnabled() || c.otelNetO11yEnabled()
	case FeatureAppO11y:
	return c.Port.Len() > 0 || c.AutoTargetExe.IsSet() || len(c.Discovery.Instrument) > 0 ||
	c.Exec.IsSet() || len(c.Discovery.Services) > 0
	}
	return false
	}

Subsequently, within the ReportMetrics function, newMetricsReporter would be executed because cfg.Enabled() evaluated to true due to the network feature being configured. Relevant code sections:  

opentelemetry-ebpf-instrumentation/pkg/export/otel/metrics.go

Lines 155 to 184 in 6acd646

	func ReportMetrics(
	ctxInfo *global.ContextInfo,
	cfg *otelcfg.MetricsConfig,
	selectorCfg *attributes.SelectorConfig,
	unresolved request.UnresolvedNames,
	input *msg.Queue[[]request.Span],
	processEventCh *msg.Queue[exec.ProcessEvent],
	) swarm.InstanceFunc {
	return func(ctx context.Context) (swarm.RunFunc, error) {
	if !cfg.Enabled() {
	return swarm.EmptyRunFunc()
	}
	otelcfg.SetupInternalOTELSDKLogger(cfg.SDKLogLevel)
	mr, err := newMetricsReporter(
	ctx,
	ctxInfo,
	cfg,
	selectorCfg,
	unresolved,
	input,
	processEventCh,
	)
	if err != nil {
	return nil, fmt.Errorf("instantiating OTEL metrics reporter: %w", err)
	}
	return mr.reportMetrics, nil
	}
	}

opentelemetry-ebpf-instrumentation/pkg/export/otel/otelcfg/config_metrics.go

Lines 171 to 173 in 6acd646

	func (m *MetricsConfig) Enabled() bool {
	return m.EndpointEnabled() && (m.OTelMetricsEnabled() || m.AnySpanMetricsEnabled() || m.NetworkMetricsEnabled())
	}

I was actually planning to submit this as a separate issue, but I noticed that this specific behavior was addressed and modified in your recent PR, where the check before creating the Metrics Reporter was adjusted.

opentelemetry-ebpf-instrumentation/pkg/export/otel/metrics.go

Lines 156 to 187 in e25ddf5

	func ReportMetrics(
	ctxInfo *global.ContextInfo,
	cfg *otelcfg.MetricsConfig,
	mcfg *perapp.MetricsConfig,
	selectorCfg *attributes.SelectorConfig,
	unresolved request.UnresolvedNames,
	input *msg.Queue[[]request.Span],
	processEventCh *msg.Queue[exec.ProcessEvent],
	) swarm.InstanceFunc {
	return func(ctx context.Context) (swarm.RunFunc, error) {
	if !(cfg.EndpointEnabled() && mcfg.Features.AppOrSpan()) {
	return swarm.EmptyRunFunc()
	}
	otelcfg.SetupInternalOTELSDKLogger(cfg.SDKLogLevel)
	mr, err := newMetricsReporter(
	ctx,
	ctxInfo,
	cfg,
	mcfg,
	selectorCfg,
	unresolved,
	input,
	processEventCh,
	)
	if err != nil {
	return nil, fmt.Errorf("instantiating OTEL metrics reporter: %w", err)
	}
	return mr.reportMetrics, nil
	}
	}

I now notice that my "Steps to Reproduce" in the original issue might have inadvertently conflated these two distinct observations.

To provide a clearer and more accurate reproduction of this issue, I can confirm that the problem still occurs when using the main version of the OpenTelemetry eBPF Instrumentation image with the application feature enabled in otel_metrics_export.  
Here's the modified docker-compose.yaml and obi-config.yaml to reproduce the issue:  

# docker-compose.yaml

version: "3.7"

services:
  postgres:
    image: postgres:16.10
    ports:
      - 5432:5432
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
  obi:
    ​image: otel/ebpf-instrument:main
    environment:
    	- OTEL_EBPF_OPEN_PORT=5432
    	- OTEL_EBPF_CONFIG_PATH=/etc/obi/config.yaml
    volumes:
    	- ./obi-config.yaml:/etc/obi/config.yaml
    privileged: true
    pid: "service:postgres"
    depends_on:
      - postgres

# obi-config.yaml
  
log_level: DEBUG

otel_metrics_export:
  ttl: 5m
  endpoint: http://otel-collector:4317
  features:
    - application