How to reduce cross-implementation incompatibility when new IDs are introduced?

[justingrant]

While reviewing bug reports on existing ECMAScript canonicalization behavior, I ran across a problematic case: when a new TZDB identifier is introduced, inevitably some environments will get it before others, and the slower-to-update environments will throw an exception when provided with that new ID. @Yqwed in #5 also raised this case.

When environments are evergreeen browsers, then the delay to get new IDs (and their accompanying Zone rules) is relatively short. But other environments could take months or years to update. For example:

• Server apps, e.g. that may be running on a 2-year old Node version
• Libraries that embed TZDB data, which may require a PR to update. typo in zone name Europe/Kyiv ? iamkun/dayjs#2059 is one example.
• IoT devices, which may require a firmware update to change their TZDB
• Android devices, which are dependent on the carrier to roll out updates
• Enterprise environments where IT delays updates for a long time.
• My family, all of whom seem to wait to update their devices until they stop working without applying updates. :-)
• Or other slow-to-update cases

How can we help make this problem less bad?

When the new ID is present because of a name change like Kiev=>Kyiv, then there is something we could recommend in the spec: that implementations first introduce the identifier without making it canonical (so that it will be recognized if others send it, but because non-canonical it won't be sent to others), wait a bit (maybe a few months?), and then make the new name canonical?

This would have a weird side effect though, because currently (at least once we fix the 13 outdated canonical identifiers in V8 and WebKit) non-canonical identifiers never become canonical; instead, the transitions only go one way. Is it OK to break that regularly? And even if yes, is it good to delay canonicalizing new renames?

On the other hand, when it's a brand-new identifier representing a new Zone, I'm not sure there's much the spec can do, other than perhaps adding additional metadata to time zones, e.g. an addedDate property to Temporal.TimeZone which could allow applications to decide to hide newly-introduced identifiers from dropdown time zone choosers in UIs. This would add a maintenance burden because this added-date data is not available in TZDB. And it'd complicate the Temporal API. So I'm on the fence about whether the value of such an additional metadata is worth it.

I have also been thinking about adding a Temporal.TimeZone.p.metadata getter which initially could return an object, e.g. { version: '2022g' } and could be extended with additional metadata later like an added date if there's enough customer demand. See #7 for discussion.

Another thing we could do would be to encourage implementations to dynamically update to the latest TZDB more quickly (e.g. dynamically on install instead of being bundled into the distribution) but this would vary behavior for the same version of a browser, so I'm not sure that this would be good.

Regardless, one thing we should do is to encourage (e.g. in MDN docs) ECMAScript developers to anticipate this "unknown but not invalid identifier" case and to defend against it, using clearer error messages when an identifier is not recognized.

Feedback encouraged on this issue!

[justingrant]

something we could recommend in the spec: that implementations first introduce the identifier without making it canonical (so that it will be recognized if others send it, but because non-canonical it won't be sent to others), wait a bit (maybe a few months?), and then make the new name canonical?

After seeing other bug reports about this topic, I now think this idea is a good one. The time when a new identifier is released will be a sensitive one, so delaying making a brand-new identifier canonical seems like a good idea so that when the change does happen, most of the Web will already know about the new identifier.

Here's a few example bugs where this approach might help.

Firefox 1796393: Javascript returns problematic timezone, breaking sites
Firefox 1825512: Europe/Kyiv is not a valid IANA timezone identifier
typo in zone name Europe/Kyiv ? - iamkun/dayjs - #2059

[Yqwed]

I don't think using brand new timezone is reasonable (ignoring emotional part). You never practically need to expose raw Olson ID and who is likely to suffer are your users. I agree with IANA's position that Olson ID is just ID and it could be anything - random string, UUID, a number and so on.

implementations first introduce the identifier without making it canonical (so that it will be recognized if others send it, but because non-canonical it won't be sent to others), wait a bit (maybe a few months?), and then make the new name canonical?

What is the use case of canonicalisation? When an user of the a timezone library will need it?

wait a bit (maybe a few months?)

Items you mentioned are unlikely to be updated in that time frame.

Android devices, which are dependent on the carrier to roll out updates

nit: it is up to OEMs, not carriers. Though Android 10+ phones (roughly speaking, it is more complicated) can get updates from Mainline.

[justingrant]

Hi @Yqwed, thanks so much for your thoughtful replies. A few follow-up notes and questions are below.

What is the use case of canonicalisation? When an user of the a timezone library will need it?

I agree that the use cases for canonicalization are dubious! The current ECMAScript spec uses canonicalization everywhere a time zone identifier is used. There's no way for ECMAScript to return a non-canonical ID back to the caller from any ECMAScript API, including Temporal Stage 3.

A goal of this proposal is to change this to reduce the scope of user-observable canonicalization in ECMAScript. The basic idea is that identifiers should never be canonicalized before being returned back to callers via APIs like Temporal.ZonedDateTime.prototype.toString() or Intl.DateTimeFormat.prototype.resolvedOptions().timeZone.

So if a caller provides Asia/Calcutta as input, they'll get Asia/Calcutta in their output. If a caller provides Asia/Kolkata as input, they'll get Asia/Kolkata in their output. This would help make most ECMAScript programs less vulnerable to IANA changing the canonical ID for a time zone.

One thing I'm not sure about is how we should handle cases where the time zone ID comes not from an ECMAScript caller, but from the OS itself. Assume that an OS's IANA ID is, for example, Europe/Kiev but ECMAScript's canonical ID is Europe/Kyiv. If an ECMASCript program calls Temporal.Now.timeZoneId(), which ID do you think should be returned? The OS's ID or the canonical one according to IANA?

The one place where I think that we must expose canonical values to developers is Intl.supportedValuesOf('timeZone'), because the use case for this API is to return one value for each group of aliases, instead of returning both , for example, Asia/Calcutta or Asia/Kolkata. Picking the currently canonical ID seems like the easiest way to choose which ID to return.

But other than that one API, I want to consider removing user-observable canonicalization everywhere else. See https://docs.google.com/presentation/d/1oapwqvqAtauGV5gqpMqWlSfsFE4A38Ylh7aFiMOSIBI/edit#slide=id.g23a7465b127_0_5 for more discussion about this.

I don't think using brand new timezone is reasonable (ignoring emotional part).

Could you explain more what you mean by this? It's IANA, not ECMAScript, that's adding new identifiers in response to geopolitical changes like Kiev being renamed to Kyiv. If IANA introduces a new identifier, we'd have to start using it at some point, right?

The context is that there are many years' worth of complaints about outdated city names being exposed by ECMAScript. So continuing to expose outdated names and angering developers isn't good either. Sadly there's no perfect answer here, only tradeoffs.

Android devices, which are dependent on the carrier to roll out updates

nit: it is up to OEMs, not carriers. Though Android 10+ phones (roughly speaking, it is more complicated) can get updates from Mainline.

This is really helpful info to understand. Thanks for sharing!

wait a bit (maybe a few months?)

Items you mentioned are unlikely to be updated in that time frame.

Makes sense. Thanks for clarifying.

[ljharb]

How could I use Temporal to get from Asia/Calcutta to Asia/Kolkata? Is there some kind of .canonicalID() method or something?

[justingrant]

My current thinking is that offering a "canonicalize this ID" API is a bad idea, because it would encourage users to take a dependency on a potentially moving target, and make code fragile if the canonical ID changes (like Kiev=>Kyiv did in 2022).

There are essentially three choices:

1. Always canonicalize, but never change the canonical ID. This is V8's and Safari's current behavior that developers complain about.
2. Always canoncalize, and update the canonical ID when it changes (like Firefox does), which risks breaking working code when the engine updates.
3. Stop canonicalizing user inputs (and maybe the OS ID too, per above), so that developers stop caring about which ID is the canonical one because it no longer matters. When developers need to compare two IDs for equality, we'd add (in this proposal) a new TimeZone.p.equals method that would (non-observably) canonicalize the IDs and return true if they match, false if not.

I'm currently thinking that (3) is the least-bad choice. The model I have in mind is like a case-insensitive SQL database colunm, where "Jordan", "jordan", and "JORDAN" are considered the same when it comes to comparison but the table still stores whatever string the user provided.

There are tradeoffs to this approach, notably that === becomes unreliable to compare IDs, but honestly it's already unreliable (at least in Firefox) because canonical IDs can change. If you compare an ID you stored last year to an ID you store today, they may be different. No matter what, there needs to be some ECMAScript code run to tell you if two IDs represent the same time zone.

It's possible that there may be use cases where a "canonicalize this ID" API is desperately needed, but so far I haven't heard any. Also, such an API could always be added later if needed, so I think it's probably good to ship without it and see how developers adjust to a world where canonicalization matters less before assuming we need that API.

[ljharb]

I like 3 in that I then don't have to care about which is canonical; it would indeed make comparing the strings by === a bad practice (also, since strings are allowed and not just TimeZone instances, i assume a static method that takes two strings would be available, with or without the prototype method)

[justingrant]

I like 3 in that I then don't have to care about which is canonical; it would indeed make comparing the strings by === a bad practice

Yes, exactly! It's already a bad practice, but most developers probably don't realize that it's bad yet. The changes we're proposing will just make it more explicit that === should be avoided by providing a discoverable alternative.

since strings are allowed and not just TimeZone instances, i assume a static method that takes two strings would be available, with or without the prototype method

We considered a static method here, for exactly the reason you noted, but the existing convention in Temporal is that anything you do with a time zone (like getPossibleInstantsFor, getNextTransition, or getPreviousTransition) requires instantiating a TimeZone instance. So my assumption was that we'd follow the same convention with equals too.

If that becomes a problem, a later proposal could always add a batch of static methods to TimeZone. Sound OK?

[ljharb]

The entire motivation of allowing strings, as i recall, was to avoid the performance penalty of creating an instance, so that seems like a very unfortunate thing to force.

[justingrant]

I think it's helpful to differentiate two cases:

1. Cases where requiring objects made it harder for implementations to optimize common operations, for example the month getter or add method of a ZonedDateTime, where a calendar and time zone is used behind the scenes but userland callers don't call TimeZone or Calendar functionality directly.
2. Cases where developers want to call TimeZone or Calendar functionality directly. In Temporal, almost all functionality lives on the prototype of each type, with the only exceptions being from factory methods and the compare methods used for sorting. So there'd have to be a really strong case to break from that pattern.

The problem with (1) was that implementers couldn't optimize unless they knew that no observable time zone or calendar code had to be run while performing those operations. With object-only calendars and time zones, knowing "is this an unmodified built-in calendar and/or time zone?" was hard and/or brittle. Now, it's easy: if the slot has a string, then it's safe to optimize.

For (2), I'd be hesitant to break Temporal's prototype-focused API for this relatively uncommon case. Especially since every other Temporal type's equals method lives in the prototype.

I guess we could also offer a TimeZone.compare static method that sorted alphabetically by canonical ID, matching the sort order of Intl.suportedValuesOf('timeZone'). That might be slightly faster than a prototype equals. But I'd be inclined to wait until a future proposal to see if that's really needed.

[Yqwed]

which ID do you think should be returned? The OS's ID or the canonical one according to IANA?

Isn't it up to factory method? What I mean is that if timezone is X according to the OS and Temporal.Now.timeZoneId() result is different to TimeZone.of("X") that might be surprising and non-obvious.

nit: there is no "IANA canonical" thing.

If IANA introduces a new identifier, we'd have to start using it at some point, right?

Unfortunately I don't have a good answer here. Currently we are planning to use Europe/Kyiv in Android V only (to be released in 2024).

What I've meant is that one needs to be careful with new IDs: you really don't need to expose Olson ID to users and the use of new ID might affect user experience as it will take time for all the server you interact with to update.

So continuing to expose outdated names and angering developers isn't good either

I see it boiling down to "happy developers" vs "happy end users".

Android devices, which are dependent on the carrier to roll out updates

I forgot to mention that carriers are responsible for NITZ signal. They can't ignore timezone changes. So we depend on carriers - it's not phones they need to update, but signals their cell towers send.

[justingrant]

which ID do you think should be returned? The OS's ID or the canonical one according to IANA?

Isn't it up to factory method? What I mean is that if timezone is X according to the OS and Temporal.Now.timeZoneId() result is different to TimeZone.of("X") that might be surprising and non-obvious.

There are three sources of time zone IDs are output by ECMAScript:

1. IDs that originally came from programmer inputs to ECMAScript APIs like Temporal.TimeZone.from (the ECMAScript factory method)
2. The ID of the OS
3. An enumeration of IDs (i.e. what you'd use as the value of a UI time-zone picker, where the user-visible text of that picker would be the localized name of the time zone) that comes from the version of the IANA TZDB baked into the ECMAScript implementation. (I know that other platforms like Java offer an out-of-band way to update the IANA TZDB without updating Java, but currently ECMAScript doesn't do this. Instead, ES implementations package the TZDB inside the implementation and it currently cannot be overridden.)

The fundamental change in this proposal is in case (1): when a developer provides an Olson ID as input to an ECMAScript API. Currently, that ID is always canonicalized to ECMAScript's current canonical ID for that time zone. We're proposing to stop doing that. The proposed behavior is to retain the ID that the caller provided: if I pass Asia/Calcutta as input, then I get Asia/Calcutta as output. If I pass Asia/Kolkata as input, then I get Asia/Kolkata as output. The goal is to reduce the impact to programs when canonical IDs change for a time zone.

Case (3) seems like we can't avoid returning the canonical ID back to the caller, because it would be bad to have a UI time zone picker with two identical entries with the same localized name, but one has a value of Asia/Calcutta and the other has a value of Asia/Kolkata. We have to pick a single ID for each set of IDs that correspond to the same time zone, and I'm assuming that the canonical ID would be a better choice to return than some other way to choose the winning ID, e.g. alphabetic sort.

For case (2), we're unsure. Should (2) align with (1) and always return the ID that the OS gives us without changing it? Or should (2) align with (3) and always canonicalize what the OS provides before returning it to the caller?

It sounds like your opinion is that aligning (1) and (2) is more important than aligning (2) and (3)? Is that correct?

So continuing to expose outdated names and angering developers isn't good either

I see it boiling down to "happy developers" vs "happy end users".

Yep, agree. Trying to resolve this tradeoff is one of the main problems that this proposal is trying to solve.

One way we're trying to resolve it is by reducing the importance of canonicalization to how ECMAScript programs behave. If ECMAScript doesn't change the input it gets from developers (and from the OS, per above?), then even if canonical IDs change then there's less of a chance that it will break existing code. At least that's the theory. What do you think about this plan?

If IANA introduces a new identifier, we'd have to start using it at some point, right?

Unfortunately I don't have a good answer here. Currently we are planning to use Europe/Kyiv in Android V only (to be released in 2024).

What I've meant is that one needs to be careful with new IDs: you really don't need to expose Olson ID to users and the use of new ID might affect user experience as it will take time for all the server you interact with to update.

Yeah, this is a hard problem without an obvious easy answer. The best idea I've had is similar to what you're doing in Android: wait for some period of time before showing that ID to users.

Specifically, what I'm thinking is that ECMAScript would do this:
a) when new IDs are added, start recognizing them ASAP to accommodate remote systems who send that ID
b) but for renamed IDs (like Kiev=>Kyiv), we'd recognize the ID but not make the new name canonical (meaning not return it in the enumeration case (3) above) until some period of time has elapsed, so that other servers have time to catch up. (It sounds like Android is using 2 years, which seems reasonable to me.)

I forgot to mention that carriers are responsible for NITZ signal. They can't ignore timezone changes. So we depend on carriers - it's not phones they need to update, but signals their cell towers send.

This is good info, thanks! Does the NITZ signal provide the Olson time zone name? Or just the UTC offset? The specification isn't clear and there are no examples in it. I found this document which suggests that it's just the offset, but I wasn't sure if that varied between carriers.

If it's just an offset, then does Android automatically translate between the NITZ timezone and Olson ID using geo-location? If not, then how is the OS's Olson ID determined after NITZ tells the phone that the time zone has changed?

nit: there is no "IANA canonical" thing.

Yep, what I mean when I say "IANA canonical" is "The canonical ID according to the time zone database that the ECMAScript implementation embeds, which may be different from the IANA TZDB used by the OS."

Note that some ECMAScript implementations like V8 don't use TZDB directly, but instead use CLDR data which roughly follows TZDB if it were built with PACKRATDATA=backzone PACKRATLIST=zone.tab, except that the canonical ID is never changed after it's introduced, so CLDR still lists Asia/Calcutta, Europe/Kiev, and Asia/Saigon as canonical.

So maybe a better way of saying what I was trying to ask would be this: "which ID do you think should be returned? The OS's ID or the canonical one according to ECMAScript's current time zone data?"

[Yqwed]

UI time-zone picker

I think timezone picker and enumeration of all the supported timezones are related, but different problems.
(I might unknowingly push the way it is done in Android. We keep per-region list of timezones. It is not yet exposed via APIs though, result file is read by the Settings timezone picker only).

It sounds like your opinion is that aligning (1) and (2) is more important than aligning (2) and (3)? Is that correct?

Ahhh, now I see what you mean. I thought that Intl.supportedValuesOf('timeZone') returns equivalent IDs, like ICU's getEquivalentID.

Before I answer the question: Do I understand correctly that supportedValuesOf('timeZone') will return only one of Asia/Kolkata and Asia/Calcutta?

At least that's the theory. What do you think about this plan?

That sounds reasonable to me.

Does the NITZ signal provide the Olson time zone name

Unfortunately, they don't. NITZ provides offset and DST flag only. Then we try to find matching timezone for that country. Please see our documentation.

Geo-location based timezone detection was recently introduced, and is also documented.

PACKRATDATA=backzone PACKRATLIST=zone.tab

Be careful with backzone file [1]. TBH I thought ICU does not use it at all. Or maybe that's the way we build ICU dat files in Android.

So maybe a better way of saying what I was trying to ask would be this: "which ID do you think should be returned? The OS's ID or the canonical one according to ECMAScript's current time zone data?"

It's tough. Will that ID be communicated back to the OS? Or is that browser's implementation detail?

[1] https://mm.icann.org/pipermail/tz/2023-May/032948.html

[justingrant]

We keep per-region list of timezones.

This data is well-structured. Very helpful to see. Thanks so much for sharing it. This file seems to have a lot of overlap in its intent and usage as CLDR's timezone.xml. Except the Android one has more useful metadata about how to choose the right alias. I wonder if it might make sense at some point to merge these into a single CLDR-maintained data set? Or to otherwise more closely align CLDR's data with what you're doing here so that there'd be more consistency among various CLDR-using software that Android devices communicate with?

I really like how this file already has Europe/Kyiv added as an alias although it's not canonical yet. This is the same least-worst option I was considering for ECMAScript, where we'd quickly add new IDs to the "recognized" list, but wait some period of time (1-2 years?) before making them canonical if they're renames of an existing ID. It's encouraging that you adopted the same approach too.

What do you do in Android about brand-new IDs that are not renames, like America/Ciudad_Juarez that was recently introduced? Do you add them as separate zones as soon as they're added to TZDB, or do you also wait a few years for other systems to catch up? Or do you add them as aliases of an existing zone, and a few years later split them into their own zone?

Ahhh, now I see what you mean. I thought that Intl.supportedValuesOf('timeZone') returns equivalent IDs, like ICU's getEquivalentID.

Nope. Currently in V8, it returns the results of an ICU enumeration API:

TimeZone::createTimeZoneIDEnumeration(UCAL_ZONE_TYPE_CANONICAL_LOCATION, NULL, NULL, errcode);

In Safari, it returns the same thing but adds UTC. In Firefox, it returns canonical IDs according to FF's build of IANA data, including Etc/* zones that the other engines omit. See tc39/ecma402#778 for my concerns about this variance.

But regardless, only one ID per set of aliases are returned by Intl.supportedValuesOf('timeZone'). The core use case is supplying IDs for the value of a time zone picker UI, and that use case essentially requires having only one ID per set of aliases.

Before I answer the question: Do I understand correctly that supportedValuesOf('timeZone') will return only one of Asia/Kolkata and Asia/Calcutta?

Correct. It will return only one of Asia/Kolkata and Asia/Calcutta. Currently, Firefox (which gets its canonicalization directly from IANA) returns Asia/Kolkata. V8 and Safari return what ICU returns from TimeZone::GetCanonicalID, which is Asia/Calcutta. Fixing this divergence is one of the reasons why this proposal exists.

ICU and CLDR are currently investigating whether they can expose an API (likely a new one) that would enable returning Asia/Kolkata in that case.

Be careful with backzone file [1].

Thanks for the pointer. My assumption (may be wrong?) was that we needed to use backzone, otherwise not all IDs in zone.tab would have time zone rules in the build output of TZDB. And also that zones in backzone that do have valid data before 1970 (e.g. Europe/Copenhagen, Europe/Oslo, Atlantic/Reykjavik) would be replaced with other zones' data because TZDB's 2022 releases merged those zones into different Zones like Europe/Berlin or Africa/Abidjan. Maybe these were incorrect assumptions?

In general I think we agree most closely to the philosophy of the global-tz fork maintained by the Joda Time maintainer, and my understanding was that PACKRATDATA=backzone PACKRATLIST=zone.tab should generate the same data as global-tz. But that assumption also might be mistaken?

TBH I thought ICU does not use it at all. Or maybe that's the way we build ICU dat files in Android.

I know that ICU imports data from TZDB via some scripts they run when new TZDB releases come out. I'm not sure exactly how their import process works. I suspect you're correct here, although I am not sure.

One of the work items in this proposal is to come up with normative guidelines for how ECMAScript engines should build TZDB in order to ensure more consistency across engines. Part of these guidelines will probably be recommending the build options of TZDB and/or a particular way of accessing CLDR, if not as a requirement then as an example of one way to obtain time zone data in the recommended way.

So maybe a better way of saying what I was trying to ask would be this: "which ID do you think should be returned? The OS's ID or the canonical one according to ECMAScript's current time zone data?"

It's tough. Will that ID be communicated back to the OS? Or is that browser's implementation detail?

If "communicated back to the OS" means what I think it does, then no. The data flow for Temporal.Now.timeZoneId() is one-way: OS => ECMAScript engine => ECMAScript program calling that API. An ECMAScript program will almost never set the OS's time zone ID, with the only really-rare exception might be a native app or script that is written in ECMAScript and that changes OS settings.

That said, I could also see cases where an ECMAScript program (like a React Native app) might call Temporal.Now.timeZoneId() in some places, but also use native libraries that call native code to fetch the OS ID. So even if ECMASCript is (almost) never used to set the OS's time zone, there's still an opportunity for the variation between the OS and ECMAScript to be noticeable and to cause inconsistency inside of ECMAScript programs.

Also, another case I was thinking of was sysadmins who wanted to lock the OS on a particular older ID (e.g. Europe/Kiev) for backwards-compatibility or political reasons. My assumption is that those sysadmins would not like ECMAScript to ignore their overrides.

I'm not sure about the priority of those cases. Both seem realistic but probably uncommon.

Do you think that it's more important for Temporal.Now.timeZoneId() to align with the behavior of the factory method ("keep the input ID unchanged") or the enumeration method ("always return the current canonical ID")?