Application Insights SDK sending data to unexpected endpoint, why?

[xiaomi7732]

We have some legacy code that parse and set the endpoint manually, for example:

            _telemetryConfig = TelemetryConfiguration.CreateDefault();
            var telemetryChannel = new ServerTelemetryChannel();
            // Set to: `https://westus2-2.in.applicationinsights.azure.com/v2/track/`, the app insights is in west us 2.
            telemetryChannel.EndpointAddress = Endpoints.AdaptIngestionEndpoint(endpointAddress).AbsoluteUri;

            _telemetryConfig.TelemetryChannel = telemetryChannel;
            // We understand that ikey is obsoleted. Should use connection string instead.
            _telemetryConfig.InstrumentationKey = instrumentationKey;

            var identityTelemetryModule = new ApplicationInsightsIdentityTelemetryModule();
            identityTelemetryModule.Initialize(_telemetryConfig); // Setup the Credential Envlope

            telemetryChannel.Initialize(_telemetryConfig);

            _client = new TelemetryClient(_telemetryConfig);
...

• AAD auth is enabled.

The behavior that we have been seeing, is that at the beginning of the app running, telemetries are ingested successfully to the correct region, the effect endpoint is https://westus2-2.in.applicationinsights.azure.com/v2.1/track.

However, after a couple of hours running, the payloads will be sent to east us 2, and got dropped on the breeze side:

What could cause this behavior? Will using the connection string instead fixing the issue for us?

[xiaomi7732]

I have tried to use connection string instead of manipulating the endpoint manually, the issue is still there. Maybe it is a DNS issue or something?

[xiaomi7732]

Keep digging. Get a network tracing. It shows up, the initial TCP handshakes are going to the right region (westus2), and after a while, it goes to the wrong region (eastus). And the handshakes aren't especially close to each other.

One thing I noticed is that in our process, there's another telemetry server channel, that uses the global endpoint (a different iKey) that sends telemetries to east us (the app insights component is in east US).

Both server channels don't specify StorageFolder. Is it possible that they are interfering?

I checked the ApplicationFolderProvider,

            string appIdentity = this.identityProvider.GetName() + "@" + Path.Combine(baseDirectory, Process.GetCurrentProcess().ProcessName);
            string subdirectoryName = GetSHA256Hash(appIdentity);
            string subdirectoryPath = Path.Combine("Microsoft", "ApplicationInsights", subdirectoryName);
            DirectoryInfo subdirectory = root.CreateSubdirectory(subdirectoryPath);

It looks like the identityProvider (WindowsIdentityProvider) would be the same for the same instance, it's the same process, and thus, the same process name, does that mean they are going to use the same file storage?

I am trying to connect the dots because I see code about endpoint being serialized, is it possible that the transmission picked up the endpoint from the serialized data (and the wrong one)?

Tagging @TimothyMothra for expertise.

[JJong-nl]

Are you running in azure? Is it a webapp, with connected application insight?
Than azure platform hooks into runtime to inject the application insight on platform level.
Are there spesific enviroment variables for applicationinsight connectionstring?

[xiaomi7732]

It is a bit complex. Our application runs as Azure WebJob. We have 2 telemetry configurations sending data to 2 application insights components.

It looks like these are the conditions for the issue to repo:

1. At the beginning, there's no problem - the issue starts to show up after a couple of hours or longer.
2. AAD Auth enabled for one of the application insights, disabled for the other.
3. Those 2 application insights components are deployed to 2 different regions.
4. One of the telemetry configuration uses connection string, the other uses iKey.
5. The one that uese iKey has large quantity of data and is sampling. (can't repro with another app insights that does not sampling.)

We currently worked-around the issue by using connection string (regional breeze endpoint) strictly. It is just not very clear why the issue happens.