Avoid race conditions on feature discovery during Writer creation

[daniel-mohedano]

What Does This Do

• Avoids a race condition in datadog.trace.common.writer.WriterFactory where a call to datadog.communication.ddagent.SharedCommunicationObjects#featuresDiscovery scheduled the discovery of features, but these were accessed immediately after without checking if they had been initialized or not.
• Avoids an additional race condition by changing the scheduled task to discoverIfOutdated instead of discover, to ensure the features are not reset while reading them in other parts of the code.
• The unit test has been updated to simulate this race condition with a delaying HttpRequest on feature discovery.

Motivation

• This race condition incorrectly configured the writer to AgentWriter, even when IntakeWriter was supposed to be used, with info logs like CI Visibility functionality is limited. Please upgrade to Agent v6.40+ or v7.40+ or enable Agentless mode. incorrectly appearing. This had other impacts on the Test Optimization product like session and module data not being reported.

Additional Notes

• The upgraded test correctly identified the failures of the original code:

Contributor Checklist

• Format the title according the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any usefull labels
• Don't use close, fix or any linking keywords when referencing an issue.
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, move, or deletion
• Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

[mcculls]

Approving because the writer/tracer changes are limited to CI-Visibility mode

(we don't want to call discoverIfOutdated in these places for other modes because that would re-introduce a blocking I/O call on the application thread.)

[nikita-tkachenko-datadog]

It looks like with the way things are the return value of datadog.communication.ddagent.SharedCommunicationObjects#featuresDiscovery is unusable without a subsequent discoverIfOutdated call, as the caller has no guarantee that the returned instance is initialised.
Perhaps it'd be cleaner/more robust if featuresDiscovery returned a Future? Then those who need to use the result would be forced to wait on it, reducing the chance of error, and those who don't need to use it would just pass the future along. @mcculls what do you think?

[pr-commenter]

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/evp-proxy-writer
git_commit_date	1752571825	1752577505
git_commit_sha	51532a8	4e880c7
release_version	1.52.0-SNAPSHOT~51532a82ba	1.52.0-SNAPSHOT~4e880c76b9

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752579323	1752579323
ci_job_id	1028839261	1028839261
ci_pipeline_id	70561734	70561734
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-akr3jkbk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-akr3jkbk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 8 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.52.0-SNAPSHOT~4e880c76b9, baseline=1.52.0-SNAPSHOT~51532a82ba

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (998.255 ms) : 0, 998255
Total [baseline] (8.577 s) : 0, 8576849
Agent [candidate] (1.0 s) : 0, 1000232
Total [candidate] (8.555 s) : 0, 8555030
section iast
Agent [baseline] (1.129 s) : 0, 1129120
Total [baseline] (9.271 s) : 0, 9271118
Agent [candidate] (1.131 s) : 0, 1131091
Total [candidate] (9.293 s) : 0, 9292619

Loading

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	998.255 ms	-
Agent	iast	1.129 s	130.865 ms (13.1%)
Total	tracing	8.577 s	-
Total	iast	9.271 s	694.269 ms (8.1%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.0 s	-
Agent	iast	1.131 s	130.859 ms (13.1%)
Total	tracing	8.555 s	-
Total	iast	9.293 s	737.589 ms (8.6%)

gantt
    title insecure-bank - break down per module: candidate=1.52.0-SNAPSHOT~4e880c76b9, baseline=1.52.0-SNAPSHOT~51532a82ba

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (689.479 ms) : 0, 689479
BytebuddyAgent [candidate] (690.362 ms) : 0, 690362
GlobalTracer [baseline] (242.288 ms) : 0, 242288
GlobalTracer [candidate] (243.589 ms) : 0, 243589
AppSec [baseline] (30.742 ms) : 0, 30742
AppSec [candidate] (30.508 ms) : 0, 30508
Debugger [baseline] (6.028 ms) : 0, 6028
Debugger [candidate] (6.011 ms) : 0, 6011
Remote Config [baseline] (685.123 µs) : 0, 685
Remote Config [candidate] (686.559 µs) : 0, 687
Telemetry [baseline] (8.251 ms) : 0, 8251
Telemetry [candidate] (8.274 ms) : 0, 8274
section iast
BytebuddyAgent [baseline] (803.765 ms) : 0, 803765
BytebuddyAgent [candidate] (805.663 ms) : 0, 805663
GlobalTracer [baseline] (232.074 ms) : 0, 232074
GlobalTracer [candidate] (232.3 ms) : 0, 232300
AppSec [baseline] (30.827 ms) : 0, 30827
AppSec [candidate] (32.19 ms) : 0, 32190
Debugger [baseline] (5.814 ms) : 0, 5814
Debugger [candidate] (5.725 ms) : 0, 5725
Remote Config [baseline] (607.191 µs) : 0, 607
Remote Config [candidate] (594.541 µs) : 0, 595
Telemetry [baseline] (7.982 ms) : 0, 7982
Telemetry [candidate] (7.926 ms) : 0, 7926
IAST [baseline] (27.314 ms) : 0, 27314
IAST [candidate] (25.949 ms) : 0, 25949

Loading

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.52.0-SNAPSHOT~4e880c76b9, baseline=1.52.0-SNAPSHOT~51532a82ba

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (992.676 ms) : 0, 992676
Total [baseline] (10.58 s) : 0, 10579755
Agent [candidate] (1.004 s) : 0, 1004050
Total [candidate] (10.656 s) : 0, 10656152
section appsec
Agent [baseline] (1.177 s) : 0, 1176695
Total [baseline] (10.701 s) : 0, 10700911
Agent [candidate] (1.177 s) : 0, 1176793
Total [candidate] (10.753 s) : 0, 10753043
section iast
Agent [baseline] (1.138 s) : 0, 1137637
Total [baseline] (10.806 s) : 0, 10805723
Agent [candidate] (1.131 s) : 0, 1130830
Total [candidate] (10.8 s) : 0, 10800286
section profiling
Agent [baseline] (1.242 s) : 0, 1241904
Total [baseline] (10.997 s) : 0, 10997117
Agent [candidate] (1.24 s) : 0, 1239583
Total [candidate] (10.948 s) : 0, 10948005

Loading

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	992.676 ms	-
Agent	appsec	1.177 s	184.019 ms (18.5%)
Agent	iast	1.138 s	144.961 ms (14.6%)
Agent	profiling	1.242 s	249.228 ms (25.1%)
Total	tracing	10.58 s	-
Total	appsec	10.701 s	121.155 ms (1.1%)
Total	iast	10.806 s	225.968 ms (2.1%)
Total	profiling	10.997 s	417.362 ms (3.9%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.004 s	-
Agent	appsec	1.177 s	172.743 ms (17.2%)
Agent	iast	1.131 s	126.779 ms (12.6%)
Agent	profiling	1.24 s	235.533 ms (23.5%)
Total	tracing	10.656 s	-
Total	appsec	10.753 s	96.891 ms (0.9%)
Total	iast	10.8 s	144.134 ms (1.4%)
Total	profiling	10.948 s	291.853 ms (2.7%)

gantt
    title petclinic - break down per module: candidate=1.52.0-SNAPSHOT~4e880c76b9, baseline=1.52.0-SNAPSHOT~51532a82ba

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.54 ms) : 0, 684540
BytebuddyAgent [candidate] (694.011 ms) : 0, 694011
GlobalTracer [baseline] (241.886 ms) : 0, 241886
GlobalTracer [candidate] (243.5 ms) : 0, 243500
AppSec [baseline] (30.507 ms) : 0, 30507
AppSec [candidate] (30.566 ms) : 0, 30566
Debugger [baseline] (6.029 ms) : 0, 6029
Debugger [candidate] (6.044 ms) : 0, 6044
Remote Config [baseline] (675.782 µs) : 0, 676
Remote Config [candidate] (688.727 µs) : 0, 689
Telemetry [baseline] (8.139 ms) : 0, 8139
Telemetry [candidate] (8.289 ms) : 0, 8289
section appsec
BytebuddyAgent [baseline] (709.141 ms) : 0, 709141
BytebuddyAgent [candidate] (712.375 ms) : 0, 712375
GlobalTracer [baseline] (236.594 ms) : 0, 236594
GlobalTracer [candidate] (234.852 ms) : 0, 234852
AppSec [baseline] (171.782 ms) : 0, 171782
AppSec [candidate] (170.806 ms) : 0, 170806
Debugger [baseline] (5.777 ms) : 0, 5777
Debugger [candidate] (5.705 ms) : 0, 5705
Remote Config [baseline] (608.98 µs) : 0, 609
Remote Config [candidate] (607.113 µs) : 0, 607
Telemetry [baseline] (8.092 ms) : 0, 8092
Telemetry [candidate] (8.031 ms) : 0, 8031
IAST [baseline] (23.73 ms) : 0, 23730
IAST [candidate] (23.497 ms) : 0, 23497
section iast
BytebuddyAgent [baseline] (810.699 ms) : 0, 810699
BytebuddyAgent [candidate] (805.401 ms) : 0, 805401
GlobalTracer [baseline] (234.059 ms) : 0, 234059
GlobalTracer [candidate] (232.579 ms) : 0, 232579
AppSec [baseline] (28.119 ms) : 0, 28119
AppSec [candidate] (28.797 ms) : 0, 28797
Debugger [baseline] (7.458 ms) : 0, 7458
Debugger [candidate] (5.75 ms) : 0, 5750
Remote Config [baseline] (585.461 µs) : 0, 585
Remote Config [candidate] (580.965 µs) : 0, 581
Telemetry [baseline] (7.846 ms) : 0, 7846
Telemetry [candidate] (7.907 ms) : 0, 7907
IAST [baseline] (28.001 ms) : 0, 28001
IAST [candidate] (29.024 ms) : 0, 29024
section profiling
BytebuddyAgent [baseline] (674.984 ms) : 0, 674984
BytebuddyAgent [candidate] (674.651 ms) : 0, 674651
GlobalTracer [baseline] (361.454 ms) : 0, 361454
GlobalTracer [candidate] (360.339 ms) : 0, 360339
AppSec [baseline] (32.65 ms) : 0, 32650
AppSec [candidate] (32.899 ms) : 0, 32899
Debugger [baseline] (9.764 ms) : 0, 9764
Debugger [candidate] (9.891 ms) : 0, 9891
Remote Config [baseline] (677.083 µs) : 0, 677
Remote Config [candidate] (663.496 µs) : 0, 663
Telemetry [baseline] (10.329 ms) : 0, 10329
Telemetry [candidate] (8.758 ms) : 0, 8758
ProfilingAgent [baseline] (103.419 ms) : 0, 103419
ProfilingAgent [candidate] (103.81 ms) : 0, 103810
Profiling [baseline] (103.444 ms) : 0, 103444
Profiling [candidate] (103.834 ms) : 0, 103834

Loading

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/evp-proxy-writer
git_commit_date	1752571825	1752577505
git_commit_sha	51532a8	4e880c7
release_version	1.52.0-SNAPSHOT~51532a82ba	1.52.0-SNAPSHOT~4e880c76b9

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752578996	1752578996
ci_job_id	1028839263	1028839263
ci_pipeline_id	70561734	70561734
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-jecfvr8p 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-jecfvr8p 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 4 performance improvements and 1 performance regressions! Performance is the same for 7 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	worse [+206.968µs; +508.036µs] or [+2.481%; +6.090%]	unstable [-95.563op/s; +50.188op/s] or [-17.192%; +9.029%]	8.699ms	533.156op/s	8.341ms	555.844op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load	better [-599.040µs; -211.601µs] or [-5.738%; -2.027%]	unstable [-37.212op/s; +73.024op/s] or [-8.360%; +16.405%]	10.035ms	463.031op/s	10.441ms	445.125op/s
scenario:load:petclinic:iast:high_load	better [-3.439ms; -2.625ms] or [-7.677%; -5.860%]	unstable [-0.332op/s; +15.532op/s] or [-0.318%; +14.872%]	41.761ms	112.037op/s	44.792ms	104.438op/s
scenario:load:petclinic:no_agent:high_load	better [-1.705ms; -1.065ms] or [-4.568%; -2.854%]	unstable [-2.054op/s; +14.949op/s] or [-1.639%; +11.928%]	35.937ms	131.772op/s	37.322ms	125.325op/s
scenario:load:petclinic:profiling:high_load	better [-3.418ms; -2.433ms] or [-6.799%; -4.839%]	unstable [-1.356op/s; +12.856op/s] or [-1.456%; +13.805%]	47.347ms	98.875op/s	50.273ms	93.125op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~4e880c76b9, baseline=1.52.0-SNAPSHOT~51532a82ba
    dateFormat X
    axisFormat %s
section baseline
no_agent (37.322 ms) : 37022, 37623
.   : milestone, 37322,
appsec (46.727 ms) : 46302, 47151
.   : milestone, 46727,
code_origins (45.373 ms) : 44968, 45779
.   : milestone, 45373,
iast (44.792 ms) : 44403, 45181
.   : milestone, 44792,
profiling (50.273 ms) : 49788, 50758
.   : milestone, 50273,
tracing (43.222 ms) : 42855, 43588
.   : milestone, 43222,
section candidate
no_agent (35.937 ms) : 35644, 36231
.   : milestone, 35937,
appsec (46.858 ms) : 46438, 47277
.   : milestone, 46858,
code_origins (45.522 ms) : 45125, 45919
.   : milestone, 45522,
iast (41.761 ms) : 41394, 42127
.   : milestone, 41761,
profiling (47.347 ms) : 46918, 47776
.   : milestone, 47347,
tracing (42.593 ms) : 42260, 42927
.   : milestone, 42593,

Loading

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.322 ms [37.022 ms, 37.623 ms]	-
appsec	46.727 ms [46.302 ms, 47.151 ms]	9.404 ms (25.2%)
code_origins	45.373 ms [44.968 ms, 45.779 ms]	8.051 ms (21.6%)
iast	44.792 ms [44.403 ms, 45.181 ms]	7.47 ms (20.0%)
profiling	50.273 ms [49.788 ms, 50.758 ms]	12.95 ms (34.7%)
tracing	43.222 ms [42.855 ms, 43.588 ms]	5.899 ms (15.8%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	35.937 ms [35.644 ms, 36.231 ms]	-
appsec	46.858 ms [46.438 ms, 47.277 ms]	10.92 ms (30.4%)
code_origins	45.522 ms [45.125 ms, 45.919 ms]	9.584 ms (26.7%)
iast	41.761 ms [41.394 ms, 42.127 ms]	5.823 ms (16.2%)
profiling	47.347 ms [46.918 ms, 47.776 ms]	11.41 ms (31.7%)
tracing	42.593 ms [42.26 ms, 42.927 ms]	6.656 ms (18.5%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~4e880c76b9, baseline=1.52.0-SNAPSHOT~51532a82ba
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.449 ms) : 4398, 4500
.   : milestone, 4449,
iast (9.25 ms) : 9094, 9406
.   : milestone, 9250,
iast_FULL (14.147 ms) : 13862, 14431
.   : milestone, 14147,
iast_GLOBAL (10.441 ms) : 10257, 10624
.   : milestone, 10441,
profiling (8.341 ms) : 8206, 8477
.   : milestone, 8341,
tracing (7.529 ms) : 7422, 7636
.   : milestone, 7529,
section candidate
no_agent (4.36 ms) : 4309, 4411
.   : milestone, 4360,
iast (9.184 ms) : 9034, 9334
.   : milestone, 9184,
iast_FULL (14.411 ms) : 14121, 14700
.   : milestone, 14411,
iast_GLOBAL (10.035 ms) : 9859, 10212
.   : milestone, 10035,
profiling (8.699 ms) : 8555, 8843
.   : milestone, 8699,
tracing (7.568 ms) : 7455, 7681
.   : milestone, 7568,

Loading

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.449 ms [4.398 ms, 4.5 ms]	-
iast	9.25 ms [9.094 ms, 9.406 ms]	4.801 ms (107.9%)
iast_FULL	14.147 ms [13.862 ms, 14.431 ms]	9.698 ms (218.0%)
iast_GLOBAL	10.441 ms [10.257 ms, 10.624 ms]	5.992 ms (134.7%)
profiling	8.341 ms [8.206 ms, 8.477 ms]	3.893 ms (87.5%)
tracing	7.529 ms [7.422 ms, 7.636 ms]	3.08 ms (69.2%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.36 ms [4.309 ms, 4.411 ms]	-
iast	9.184 ms [9.034 ms, 9.334 ms]	4.824 ms (110.6%)
iast_FULL	14.411 ms [14.121 ms, 14.7 ms]	10.05 ms (230.5%)
iast_GLOBAL	10.035 ms [9.859 ms, 10.212 ms]	5.675 ms (130.2%)
profiling	8.699 ms [8.555 ms, 8.843 ms]	4.339 ms (99.5%)
tracing	7.568 ms [7.455 ms, 7.681 ms]	3.208 ms (73.6%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/evp-proxy-writer
git_commit_date	1752571825	1752577505
git_commit_sha	51532a8	4e880c7
release_version	1.52.0-SNAPSHOT~51532a82ba	1.52.0-SNAPSHOT~4e880c76b9

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1752579554	1752579554
ci_job_id	1028839265	1028839265
ci_pipeline_id	70561734	70561734
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-f5mp0txi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-f5mp0txi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~4e880c76b9, baseline=1.52.0-SNAPSHOT~51532a82ba
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.486 ms) : 1474, 1498
.   : milestone, 1486,
appsec (2.423 ms) : 2373, 2473
.   : milestone, 2423,
iast (2.213 ms) : 2150, 2276
.   : milestone, 2213,
iast_GLOBAL (2.263 ms) : 2199, 2326
.   : milestone, 2263,
profiling (2.072 ms) : 2020, 2124
.   : milestone, 2072,
tracing (2.019 ms) : 1970, 2067
.   : milestone, 2019,
section candidate
no_agent (1.483 ms) : 1472, 1495
.   : milestone, 1483,
appsec (2.42 ms) : 2371, 2470
.   : milestone, 2420,
iast (2.203 ms) : 2141, 2266
.   : milestone, 2203,
iast_GLOBAL (2.262 ms) : 2199, 2325
.   : milestone, 2262,
profiling (2.057 ms) : 2007, 2108
.   : milestone, 2057,
tracing (2.021 ms) : 1972, 2070
.   : milestone, 2021,

Loading

• baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.486 ms [1.474 ms, 1.498 ms]	-
appsec	2.423 ms [2.373 ms, 2.473 ms]	936.859 µs (63.0%)
iast	2.213 ms [2.15 ms, 2.276 ms]	726.816 µs (48.9%)
iast_GLOBAL	2.263 ms [2.199 ms, 2.326 ms]	776.534 µs (52.3%)
profiling	2.072 ms [2.02 ms, 2.124 ms]	586.023 µs (39.4%)
tracing	2.019 ms [1.97 ms, 2.067 ms]	532.457 µs (35.8%)

• candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.483 ms [1.472 ms, 1.495 ms]	-
appsec	2.42 ms [2.371 ms, 2.47 ms]	937.202 µs (63.2%)
iast	2.203 ms [2.141 ms, 2.266 ms]	720.068 µs (48.5%)
iast_GLOBAL	2.262 ms [2.199 ms, 2.325 ms]	778.446 µs (52.5%)
profiling	2.057 ms [2.007 ms, 2.108 ms]	573.975 µs (38.7%)
tracing	2.021 ms [1.972 ms, 2.07 ms]	537.714 µs (36.3%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~4e880c76b9, baseline=1.52.0-SNAPSHOT~51532a82ba
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.949 s) : 14949000, 14949000
.   : milestone, 14949000,
appsec (14.668 s) : 14668000, 14668000
.   : milestone, 14668000,
iast (18.789 s) : 18789000, 18789000
.   : milestone, 18789000,
iast_GLOBAL (17.965 s) : 17965000, 17965000
.   : milestone, 17965000,
profiling (14.839 s) : 14839000, 14839000
.   : milestone, 14839000,
tracing (14.778 s) : 14778000, 14778000
.   : milestone, 14778000,
section candidate
no_agent (15.511 s) : 15511000, 15511000
.   : milestone, 15511000,
appsec (14.842 s) : 14842000, 14842000
.   : milestone, 14842000,
iast (18.325 s) : 18325000, 18325000
.   : milestone, 18325000,
iast_GLOBAL (17.769 s) : 17769000, 17769000
.   : milestone, 17769000,
profiling (15.149 s) : 15149000, 15149000
.   : milestone, 15149000,
tracing (15.232 s) : 15232000, 15232000
.   : milestone, 15232000,

Loading

• baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.949 s [14.949 s, 14.949 s]	-
appsec	14.668 s [14.668 s, 14.668 s]	-281.0 ms (-1.9%)
iast	18.789 s [18.789 s, 18.789 s]	3.84 s (25.7%)
iast_GLOBAL	17.965 s [17.965 s, 17.965 s]	3.016 s (20.2%)
profiling	14.839 s [14.839 s, 14.839 s]	-110.0 ms (-0.7%)
tracing	14.778 s [14.778 s, 14.778 s]	-171.0 ms (-1.1%)

• candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.511 s [15.511 s, 15.511 s]	-
appsec	14.842 s [14.842 s, 14.842 s]	-669.0 ms (-4.3%)
iast	18.325 s [18.325 s, 18.325 s]	2.814 s (18.1%)
iast_GLOBAL	17.769 s [17.769 s, 17.769 s]	2.258 s (14.6%)
profiling	15.149 s [15.149 s, 15.149 s]	-362.0 ms (-2.3%)
tracing	15.232 s [15.232 s, 15.232 s]	-279.0 ms (-1.8%)

[mcculls]

Perhaps it'd be cleaner/more robust if featuresDiscovery returned a Future? Then those who need to use the result would be forced to wait on it, reducing the chance of error, and those who don't need to use it would just pass the future along. @mcculls what do you think?

Firstly a warning to be careful of using certain Future related types and other concurrency primitives during premain because if they're loaded before we get to do field-injection then we lose our chance to inject the field and have to fall back to a weak map which is much less performant (it's why we have FieldInjectionSmokeTest to watch out for this)

Secondly we already have a way for products/services to wait for the initial result: https://github.com/DataDog/dd-trace-java/blob/v1.51.0/communication/src/main/java/datadog/communication/ddagent/SharedCommunicationObjects.java#L63

This is used to defer part of the Writer setup: https://github.com/DataDog/dd-trace-java/blob/v1.51.0/dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java#L690

Unfortunately it's hard to do the same for CI-Vis because of how it's currently integrated into the tracer, so I'm fine with these simpler changes in the short-term.

In the medium to long-term I'd prefer to rework startup to use run-levels, so it's clear what a product/feature can do at various points during startup and it's easier to order things. It would also make clearer the dependencies between products/services and help to untangle things in Agent , SharedCommunicationObjects, etc.

EDIT:

those who need to use the result would be forced to wait on it

This is where we need to be careful in the general case, because often we're setting things up on the application thread - and waiting here blocks the application. Ideally we would only do enough work on the application thread to capture observations/traces, etc. from it (since that can be buffered.) Once we get discovery back from the agent then we know how to send that data and can offload the buffer.

This is a simple example, depending on the feature it may not be possible to do much until we complete the agent discovery - it gets harder when you have dependencies between products and their startup (hence the need for run-levels or something similar)