knative 1.19 links against interim / broken release of otel-go SDK

[ywwg]

The otel-go SDK is in a bit of an interim state (sorry!) and is producing different metric names than it did previously. This breaks continuity with previous metrics, and the new metrics are UTF-8 which is not supported in some situations. We haven't yet finished adding the code to make the metric name translation mode configurable, so this leaves users in an unfixable state without downgrading knative. Downgrading the otel-go sdk to 0.58 should fix this issue, and I'd recommend pinning to that version until we finish our work (open-telemetry/opentelemetry-go#7083)

( @dashpole @ArthurSens )

[dashpole]

Downgrading to v0.58.0 won't have the intended effect. See open-telemetry/opentelemetry-go#7102 (comment). I suggest you do:

import "github.com/prometheus/common/model"

func init() {
    model.NameValidationScheme = model.LegacyValidation
}

[dprotaso]

@ywwg @dashpole is there a patch release we should upgrade to?

[dprotaso]

Also one thing to note - we just switched to OTel and changed our metric names. So I'd prefer to pick a solution that doesn't require future changes.

[dashpole]

@dprotaso We are in the process of migrating our configuration for metric names, but you should be able to choose the names you want regardless, and keep those across versions.

There are two dimensions of config:

• Type/Unit suffixes
• "Escaping" dots (and other historically disallowed characters) with underscores.

Today, you configure type and unit suffixes using the WithoutCounterSuffixes and WithoutUnits options, and can configure whether dots are escaped by doing #3224 (comment).

In the future, this will be replaced with a single enum option: WithTranslationStrategy, with UnderscoreEscapingWithSuffixes , UnderscoreEscapingWithoutSuffixes, NoUTF8EscapingWithSuffixes, and NoTranslation as values. Currently, the default is effectively NoUTF8EscapingWithSuffixes, but we will likely change to UnderscoreEscapingWithSuffixes in a coming release, and we don't expect to change the default after that.

I think (correct me if i'm wrong) that @ywwg has since found a bug where the go client was producing metrics with dots even when the scraper indicated (using content negotiation headers) that did not support it. I believe that ended up being the root cause of the issue, but i'll let him confirm.

[dprotaso]

but we will likely change to UnderscoreEscapingWithoutSuffixes in a coming release,

Is there a pre-release/SHA version we can bump to where this is already done? We just had our release a week or two ago and I figure it would be good to get a point release out where we switch to the new default.

[dprotaso]

@dashpole @ywwg is there an ETA on a new patch version with fix?

[ywwg]

Sorry, we are still in the review process for the fix PR, we hope to have it in soon

[dprotaso]

@ywwg which PR should I follow - you only linked an issue in otel-go but I don't a referenced PR.

[ywwg]

open-telemetry/opentelemetry-go#7111