Inconsistent results in DifferenceISODate?

[anba]

The current DifferenceISODate definition returns the following results. Notice that it's always "P1M".

js> var end = new Temporal.PlainDate(1970, 2, 28)
js> var start = new Temporal.PlainDate(1970, 1, 28)
js> start.calendar.dateUntil(start, end, {largestUnit:"months"}).toString()
"P1M"
js> var start = new Temporal.PlainDate(1970, 1, 29)
js> start.calendar.dateUntil(start, end, {largestUnit:"months"}).toString()
"P1M"
js> var start = new Temporal.PlainDate(1970, 1, 30)
js> start.calendar.dateUntil(start, end, {largestUnit:"months"}).toString()
"P1M"
js> var start = new Temporal.PlainDate(1970, 1, 31)
js> start.calendar.dateUntil(start, end, {largestUnit:"months"}).toString()
"P1M"

When the end date is 1970-02-27, the results are more like what I'd expect:

js> var end = new Temporal.PlainDate(1970, 2, 27)
js> var start = new Temporal.PlainDate(1970, 1, 27)
js> start.calendar.dateUntil(start, end, {largestUnit:"months"}).toString()
"P1M"
js> var start = new Temporal.PlainDate(1970, 1, 28)
js> start.calendar.dateUntil(start, end, {largestUnit:"months"}).toString()
"P30D"
js> var start = new Temporal.PlainDate(1970, 1, 29)
js> start.calendar.dateUntil(start, end, {largestUnit:"months"}).toString()
"P29D"
js> var start = new Temporal.PlainDate(1970, 1, 30) 
js> start.calendar.dateUntil(start, end, {largestUnit:"months"}).toString()
"P28D"
js> var start = new Temporal.PlainDate(1970, 1, 31)                         
js> start.calendar.dateUntil(start, end, {largestUnit:"months"}).toString()
"P27D"

Also compare to java.time.temporal, which doesn't always return "P1M":

jshell> import java.time.LocalDate
jshell> var end = LocalDate.of(1970, 2, 28)
end ==> 1970-02-28
jshell> var start = LocalDate.of(1970, 1, 28)
start ==> 1970-01-28
jshell> start.until(end)
$4 ==> P1M
jshell> var start = LocalDate.of(1970, 1, 29)
start ==> 1970-01-29
jshell> start.until(end)
$6 ==> P30D
jshell> var start = LocalDate.of(1970, 1, 30)
start ==> 1970-01-30
jshell> start.until(end)
$8 ==> P29D
jshell> var start = LocalDate.of(1970, 1, 31)
start ==> 1970-01-31
jshell> start.until(end)
$10 ==> P28D

[justingrant]

You raise an interesting point. There are essentially two options for balancing a duration:

a. Only add a larger unit if the larger unit can be added without exceeding the starting date, after clamping the values of smaller units according to the default overflow: 'constrain' behavior of addition and subtraction.
b. Only add a larger unit to the result if smaller units match or exceed their value in the starting date, otherwise add smaller units instead. Per the example above, it looks like Java is using this algorithm.

@ptomato I suspect that we probably discussed this tradeoff many years ago. Do you remember the reasoning for choosing one over the other?

It looks like currently we're doing (a) to calculate from 1970-01-31 to 1970-02-28:
a. Start with 1970-01-31.
b. Try adding one month to get 1970-02-31.
c. Constrain that intermediate result to get 1970-01-28.
d. The value in (c) does not exceed the starting date, so add one month to the result.

Using the alternative algorithm (b) would look like this:
a. Start with 1970-01-31.
b. Try adding one month to to get 1970-02-31.
c. This value exceeds the starting date, so don't add one month and try smaller units instead.
d. Add 28 days to the result.

It'd be useful to understand how these two algorithmic choices would affect two other similar cases:

• ZonedDateTime.p.since with largestUnit: 'day' and where the initial result is during the hour skipped by a forward DST transition. This case also seems to be using (a):

// DST skipped the hour 2:00-3:00 on 2020-03-08 in this time zone
Temporal.ZonedDateTime.from('2020-03-09T02:30-07:00[America/Los_Angeles]')
  .since('2020-03-08T03:30-07:00[America/Los_Angeles]', { largestUnit: 'day' });
// => P1D

• largestUnit: 'year' with lunisolar calendar leap years (where years have different numbers of months) appears to be using (b), at least in the polyfill. Although "exceeding the starting date" is a bit fuzzy here. Should we consider the corresponding non-leap-month "equal" in a non-leap year?

// 2001 is a leap year with leap month M04L. 2000 is not a leap year.
Temporal.PlainDate.from({ year: 2001, monthCode: 'M04L', day: 1, calendar: 'chinese' })
  .until({ year: 2002, monthCode: 'M04', day: 1, calendar: 'chinese' }, { largestUnit: 'year' })
// => P12M

My initial opinion before digging into this too deeply is that it'd be good for all three of these cases to behave consistently. I don't yet have an opinion about whether (a) or (b) is the "more correct" behavior.

[justingrant]

Meeting 2023-04-13: Let's research why we originally picked the current behavior, and once we know that history then we can figure out if the existing behavior should be changed, or if the original reason is compelling enough to deviate from Java behavior.

[ptomato]

Related issues:

• difference should always return a "top-heavy balanced" duration with largest-first order of operations #993
• Proposed Temporal changes required to adopt RFC 5545 (iCalendar) meaning of durations #759
• Negative durations vs. order of operations #913
• Add test coverage for Temporal.ZonedDateTime.since/until #1140

We can probably pull up discussions in more detail in the meeting notes, but I haven't looked for those yet.

What I read in those various issue threads, consistently, is that we've considered this behaviour correct, and this is particularly important to @pipobscure:

const posDuration = earlier.until(later);
const negDuration = later.until(earlier);
earlier.add(posDuration).equals(later);
later.add(negDuration).equals(earlier);
earlier.subtract(negDuration).equals(later);
later.subtract(posDuration).equals(earlier);

[ptomato]

Summary of discussion on 2023-04-27:

@pipobscure's statement of commutativity is concise: A + D = B ⇔ B - D = A

In other words, for two dates A and B, and a duration D, if A.add(D) is B then B.subtract(D) is A, and vice versa. Additionally A.until(B) and B.since(A) are D, although that doesn't necessarily have to follow, depending on your definition.

The example in the OP is one where commutativity does not hold, because of the default overflow: "constrain" behaviour. This is why we have the option for overflow: "reject", so you can weed out those cases if you don't want them.

However, the inconsistent behaviour for the 27th versus the 28th of the month looks suspicious. It might just be a bug, in the first place. We still need to research this.

[gibson042]

Unless I'm misinterpreting, commutativity would hold if all the cases where start day-of-month is greater than end day-of-month would return a duration in which the largest unit is days, as seems to be the case for Java. I think the culprit in Temporal's divergence is the inherent privileging of last-day-of-month by the embedded "constrain" in step 1.k of DifferenceISODate (and presumably also in step 1.n.iii for other similar cases).

[justingrant]

I'm finally done researching this. Sorry for delays. TL;DR - I think the current spec is correct, given the criteria used when we designed the algorithm about 3 years ago.

Some relevant meeting discussion is here: https://github.com/tc39/proposal-temporal/blob/main/meetings/agenda-minutes-2020-10-15.md#anything-left-to-discuss-on-order-of-operations-993--913

The background for decisions in this issue is best captured here: #993 (comment). That comment is still a bit out of date (it refers to an older operation difference, instead of the modern until and since names that we ended up using) but the design thinking in that comment is good context to understand my comments below. You may want to read the subsequent discussion in that issue after the comment, because it goes into some corner cases, including difference in behavior from Java.

In other words, for two dates A and B, and a duration D, if A.add(D) is B then B.subtract(D) is A

Regardless of how this issue is resolved, this invariant cannot hold for all values of A, B, and D. This is because add and subtract will constrain the result. For example,

a = Temporal.PlainDate.from('2023-01-31')
d = Temporal.Duration.from({months: 1});
b = a.add(d); // => 2020-02-28 (constrained!)
b.subtract(d); // => 2020-01-28 
a.equals(b.subtract(d)); // => false

However, different invariants should hold in the current design:

• if A.until(B, {largestUnit: 'month'}) is D, then B.subtract(D) is A
• if A.since(B, , {largestUnit: 'month'}) is D, then B.add(D) is A

Assuming that my memory is correct, making these invariants work and enabling reversibility was the reason why we wrote the DifferenceISODate algorithm like we did. There may have been other reasons too.

So I don't think that a spec change is needed, unless there's some new information that would lead us to de-prioritize reversibility in favor of some other reason (e.g. Java compatibility) or unless the reversible invariants above don't hold with the current spec.

I'm not saying that the current spec yields perfect results. The results shown in the OP are indeed confusing. But I think the high-level point is that all results are potentially confusing, depending on what the user's expectations are and what use cases are being targeted. There is no perfect solution here, only less-bad solutions for some users and some use cases.

So the approach we took in Temporal was to: a) optimize the algorithm for reversibility, and b) set the default largestUnit to days or smaller for all Temporal types (except PlainYearMonth, where this problem doesn't apply). This default ensures that the confusion inherent in arithmetic with variable-length months doesn't show up unless the user opts into a non-default largestUnit.

Feel free to push back if I got anything wrong above!

[ptomato]

That's great news. I read through the discussion and what you described (optimizing for reversibility and defaulting largestUnit to days) matches my recollection as well. Thanks for looking into it!

[anba]

However, different invariants should hold in the current design:

* if `A.until(B, {largestUnit: 'month'})` is `D`, then `B.subtract(D)` is `A`

* if `A.since(B, , {largestUnit: 'month'})` is `D`, then `B.add(D)` is `A`

Assuming that my memory is correct, making these invariants work and enabling reversibility was the reason why we wrote the DifferenceISODate algorithm like we did. There may have been other reasons too.

So I don't think that a spec change is needed, unless there's some new information that would lead us to de-prioritize reversibility in favor of some other reason (e.g. Java compatibility) or unless the reversible invariants above don't hold with the current spec.

These invariants don't hold with the current spec. For example:

A = 1972-01-02
B = 1972-03-01

Then D = P1M28D, but B - D = 1972-01-04

Script to test the invariants:

function* dates() {
  for (let y = 1972; y <= 1973; ++y) {
    for (let m = 1; m <= 12; ++m) {
      for (let d = 1; d <= 31; ++d) {
        try {
          yield new Temporal.PlainDate(y, m, d);
        } catch {}
      }
    }
  }
}

function assertEq(a, e, m) {
  if (a !== e) console.log(`${m}: ${a} != ${e}`);
}

for (let a of dates()) {
  for (let b of dates()) {
    let d = a.until(b, {largestUnit: "month"});
    let e = b.subtract(d);

    assertEq(e.toString(), a.toString(), `a=${a}, b=${b}, d=${d}, e=${e}`);
  }
}

[gibson042]

The issue description clearly demonstrates a case where currently specified behavior is less useful than a particular alternative, with the alternative actually being more reversible. In what cases is there a compensating benefit—and if none can be identified, then what would discourage switching to what seems like a strictly superior approach? This is advertised as a fundamental reason to introduce Temporal, so getting it wrong would undermine quite a bit and approach catastrophe.

[gibson042]

During discussion today, I came to the conclusion that since and until should both behave such that varying one input while keeping the other constant should always produce distinguishable non-colliding output, regardless of largestUnit (but in the absence of rounding, which by definition produces collisions).

[justingrant]

Meeting 2023-10-12:

• Summary of current state: Temporal behavior is different from Java, the behavior is unintuitive for at least one person who has dug into it, and the behavior in the spec doesn't seem to match what we thought one of the intended invariants was. So it's worth more investigation and, depending on outcome, maybe a normative change in the November TC39 meeting.
• I'll investigate why the current spec doesn't match the invariants that we thought it should. Was it a bug in writing down the algorithm text, or was it a more fundamental issue in the design of the algorithm? I'll try to dig up more of the original prose to express the algorithm (I think this was part of my work on ZonedDateTime design) and report back what I find out.
• @ptomato will investigate how other platforms' date libraries behave. If everybody matches Java except us, then that's a strong hint that we got something wrong. And if we want to follow Java, then we'll need to either reverse-engineer Java's algorithm or (better) find it in the JSR-310 spec.
• We'll discuss findings in the next champions meeting in 2 weeks, and if a change is needed then we can propose the change in the November TC39 meeting.