POC Probe implementation and machinery integration with opentelemetry-go-instrumentation ("Steel Thread")

[damemi]

To further the design of a Custom Probe API, we first need to understand the needs of projects that intend to adopt Probes.

Current Overlap

The main overlap between OBI and Go-Auto is:

• OBI Tracer = Go-Auto Probe
  The Go-Auto Probe (the "Probe API") is conceptually equivalent to OBI’s Tracer.
  Currently, the Probe API is responsible for defining instrumentation libraries (e.g., uprobes, function symbols, offsets, and config options), managing their lifecycle (e.g., loading eBPF programs, maps), and handling emitted events. In Go-Auto, this is defined as a Go interface.

• OBI Instrumenter = Go-Auto Manager
  The Manager in Go-Auto plays a role similar to the OBI Instrumenter. It is the core orchestration component, instantiated via auto.NewInstrumentation(), and manages probe lifecycles and runtime behavior.

tl;dr
obi.Tracer = auto.Probe
obi.Instrumenter = auto.Manager

Our aim is to unify these overlapping concepts and remove redundant implementations across OBI and Go-Auto.

---

Proposal

I propose refactoring the Go-Auto architecture to shift most lifecycle responsibilities from Probe to Manager. Under this model:

• Probes become declarative:
  The Probe API should focus on describing instrumentation (what to trace, where, and how) without dealing with lifecycle management.

• Managers control execution:
  The Manager becomes the sole runtime controller, responsible for initializing, loading, and managing instrumentations.

This change will simplify and stabilize the Probe API, allowing developers to define portable, reusable Probes. Any eBPF tool built with OpenTelemetry standards could import and use these Probes without needing to fork or customize core logic, provided they conform to the Probe API.

Benefits of This Approach

1. Reduced API Surface Area: By stripping lifecycle logic from Probes, we make the API easier to maintain and less prone to breaking changes.

2. Extensibility in Go-Auto: The Manager can evolve and gain new features without impacting stable Probe definitions.

3. Vendor Enablement: Projects like Odigos and Beyla can use the Go-Auto framework as a foundation, without necessarily pulling in OBI as a dependency.

4. Clear Project Roles:

   • OBI becomes the standard implementation and community end-user tool for OTel eBPF auto-instrumentation.
   • Go-Auto acts as the reusable library layer for vendors to build upon.
   • OBI also serves as a “vendor” in this model to validate Go-Auto’s usability and promote adoption and community participation.

Implementation for OBI

In this unified model, OBI would:

• Import and use Probes defined via Go-Auto
• Rely on the Go-Auto Manager to load and coordinate instrumentations

This allows OBI to focus on experience and usability, while contributing improvements upstream to Go-Auto.

Challenges

OBI currently provides functionality not yet available in Go-Auto. Examples include:

• Cross-process Tracer sharing: OBI supports shared eBPF maps across Tracers in different processes.
• Custom Go offsets map: OBI uses a specialized map to handle Go runtime offsets.
• Optimized eBPF span structures: OBI improves span efficiency for event-heavy scenarios at the C level.

To enable this proposal, we would contribute these capabilities to the Go-Auto Manager, and encourage future contributions like this from OBI and any other vendors (given that they align with OpenTelemetry's commitment to vendor neutrality).

Bonus challenge: C-level eBPF structures

While Probes interact with the Manager, developers are still responsible for writing eBPF code that aligns with the assumptions of the Go-Auto framework. Critical functionality like context propagation, goroutine tracking, and OpenTelemetry-compatible span structures cannot be fully enforced or abstracted through the Go-level Probe API alone. To support this, Go-Auto provides a shared C library that handles these runtime behaviors and is used across existing Probe implementations. To improve reusability and consistency, this C library should be standardized and made easily accessible to Probe developers. This will ensure that custom Probes can be reliably integrated into any OTel-based tool without requiring deep coupling or tool-specific adaptations.

What to do now

We want to build a proof of concept that demonstrates a Go-Auto Probe being successfully imported and used by OBI. This will serve as a “steel thread” between Go-Auto and OBI to validate that the Probe API is functional, minimal, and practical for third-party consumption.

As @grcevski showed in today’s SIG call, the current implementation is already close to supporting this integration, minus the features listed above. The key decision now is how and where to introduce those missing capabilities. Instead of expanding the Probe interface to handle this additional functionality, the proposal is to move these responsibilities into the Go-Auto Manager where possible.

This essentially comes down to the removal of probe.Load() and replacing OBI’s Instrumenter with the Go-Auto Manager. The next step is to determine what functionality OBI currently requires and how to generalize and integrate that into the Manager. Any remaining gaps point to necessary features of the Probe API, and our design of that v1 API becomes much clearer.

---

x-ref: open-telemetry/opentelemetry-go-instrumentation#2029 -- shifting some probe lifecycle functionality to the Manager

[grcevski]

and replacing OBI’s Instrumenter with the Go-Auto Manager.

I think it might be hard to fully replace the OBI instrumenter, because it handles many other program types that are not necessarily related to uprobe based Go instrumentation, like kprobes, tracepoints, socket filters, etc. I think we should be able to use the Go-Auto Manager for Go programs.

[damemi]

@grcevski it's optimistic for sure, but I wanted to put that as the "goal" to make the overall intent clear. Do you think OBI would be able to wrap the Manager? Should the Manager be extendable on its own to enable this? Just some thoughts

If the starting point is just replacing it for Go programs, I think that's a reasonable scope for now too. But wdyt?

[grcevski]

@grcevski it's optimistic for sure, but I wanted to put that as the "goal" to make the overall intent clear. Do you think OBI would be able to wrap the Manager? Should the Manager be extendable on its own to enable this? Just some thoughts

You bring up a good point here. If we don't use just one, we'll be duplicating some code and it's debatable how little or much that is. We definitely will have duplication in loading, constants, reading the events from userspace etc. Maybe the proper way to do this is to start a new repo which serves both projects as a library? This new repo can handle what Go auto needs, but also handle all the other kinds of probes and program types OBI uses. Something like eBPF tools, where we can also put the code we have around parsing binaries etc.

I don't mind either way, as long as we can do what we need to :). If we move all this code for handling Go unrelated eBPF programs in Go Auto it may look like unused functionality and could cause inadvertently breakages, because the project itself isn't using say something like loading Linux Traffic Control programs. To keep things solid, we'd have to move/write new tests for this unused functionality, so tests will exist in Go Auto for functionality the project doesn't actually use.