Supports External Control of Agent Behavior Dynamically

[pepeshore]

Is your feature request related to a problem? Please describe.

When a sudden surge in business traffic causes an application to reach its performance limits, we often need to downgrade some functions of the mounted agent to ensure business availability. Currently, the configuration of the agent requires a restart to take effect, and some behaviors of the probes cannot be dynamically managed. This poses significant risks in real-world enterprise scenarios.

Describe the solution you'd like

We hope to dynamically control the following behaviors of the agent from the external:

• Control whether data is reported
• Control whether the instrumenter executes
• Adjust the sampling rate
• And so on

Describe alternatives you've considered

No response

Additional context

No response

[jackshirazi]

I've been looking at these. It will gradually become more dynamic but never fully. In terms of your top three there

• dynamic sampling rate has this implementation available
• dynamically reporting data currently needs a custom exporter, which just does nothing really other than decide whether to export or not. I think it's reasonable to add a dynamic config to the default exporter, but I haven't tried this yet
• dynamic instrumentation has two aspects
  • Firstly the actual bytecode transformations. I don't think that's going to be dynamic anytime within the next couple of years. (it's feasible but the agent isn't really designed to handle this nicely)
  • The second aspect is whether the instrumentation generates spans. This is in progress to enable/disable dynamically

[pepeshore]

I noticed that the community has been discussing OpAMPs recently. Is it possible to use them to achieve this functionality?

[pepeshore]

https://docs.google.com/document/d/1WK9h4p55p8ZjPkxO75-ApI9m0wfea6ENZmMoFRvXSCw/edit

[jackshirazi]

The short answer is OpAMP doesn't help.

The long answer is that OpAMP and dynamic capability are not tied. You can have the above dynamic capability without OpAMP. Conversely adding in OpAMP does not require dynamic capabilities in the agent or SDK. Current thinking is that with OpAMP you have a few possibilities:

• the agent waits until it receives the OpAMP configuration then initializes based on that
• a changed configuration obtained via OpAMP after agent initialization re-sets the entire SDK & agent (this is feasible for the SDK but unlikely to be achieved for the agent any time soon)
• a changed configuration obtained via OpAMP identifies a "diff" in the configuration which alters only those configuration options that are changed

Using OpAMP to get configuration changes is probably going to be done in a proprietary way for now because the OpAMP specification doesn't require any particular config structure or mechanism. It may evolve to recommend the use of declarative configuration, but that doesn't make it any more supportive of dynamic behaviour

[jackshirazi]

We have the "Control whether data is reported & whether instrumenter executes" working in the Elastic distribution, and we'll use it specifically with central config (using OpAMP). So before I look at contributing this to the contrib repo, I think it's worth asking how to expose the capabilities?

Our current implementation has a programmatic API and, for testing, the ability to check for a system property change. Maybe a programmatic API is all that should be exposed? If not, what mechanism?

The generic OpAMP implementation is already being contributed, but the hookup between OpAMP and dynamic API would be proprietary atm since there is no clear definition of what OpAMP sends/receives in terms of structures (we'll use JSON and only send the specific config changed)

[trask]

since there is no clear definition of what OpAMP sends/receives in terms of structures

I'd ❤ to see something standardized so that you can point the Java agent at any compliant opamp server and get dynamic updates

cc @andykellr @evan-bradley @tigrannajaryan

[jackshirazi]

Well let's be clear because I've inadvertently conflated separate things.

1. I'll make a contribution that provides dynamic update capability. It will have a code API. That's separate - we can discuss the actual implementation, API and capability when I generate the PR
2. Do we want to have any mechanism that executes that API from config, and if so what would that be?
3. If we provide a bridge or hook between the API and OpAMP, exactly what should that be - I think that's what @trask is suggesting we standardize

[trask]

1. I'll make a contribution that provides dynamic update capability. It will have a code API. That's separate - we can discuss the actual implementation, API and capability when I generate the PR

👍

2. Do we want to have any mechanism that executes that API from config, and if so what would that be?

I didn't follow what you meant here by "from config"

3. If we provide a bridge or hook between the API and OpAMP, exactly what should that be - I think that's what @trask is suggesting we standardize

👍

[jackshirazi]

I didn't follow what you meant here by "from config"

The OpAMP solution applies changes from remote/centralized config. We can also support local config eg existing or new config which can be changed locally. For example, a new otel.javaagent.dynamic-configuration-file which is monitored for changes, or directly support the existing otel.javaagent.configuration-file and monitor that for changes in the supported (and new) config