doctest-0.23 fails with GHC 8.6 and 8.8 when it succeeds with 8.0-4 and 9.0-10

[andreasabel]

Weirdly, doctest-0.23 fails on my project with not-in-scope identifiers for just GHC 8.6 and 8.8, but succeeds for all other GHCs in the range 8.0 - 9.10:

E.g. https://github.com/BNFC/bnfc/actions/runs/11596085398/job/32286287814?pr=488#step:21:30

Preprocessing library for BNFC-2.9.6...
Error: bytecode compiler can't handle unboxed tuples and sums.
  Possibly due to foreign import/export decls in source.
  Workaround: use -fobject-code, or compile this module to .o separately.
src/BNFC/Backend/Agda.hs:389: failure in expression `importPragmas ByteStringToken False ["qualified Foo.Abs", "Foo.Print (printTree)", "qualified Foo.Layout"]'
expected: ...
 but got: 
          ^
          <interactive>:27:1: error:
              Variable not in scope: importPragmas :: t0 -> Bool -> [[Char]] -> t

I can also reproduce this locally on my machine.

[andreasabel]

This seems to be a regression in 0.23, because doctest-0.22.9 still succeeds: https://github.com/BNFC/bnfc/actions/runs/10579618933/job/29312530584#step:20:26

[sol]

@andreasabel please provide exact and minimal steps on how to reproduce.

[andreasabel]

I am assuming ghc points to ghc-84.4. E.g you have ghcup and installed GHC 8.8.4 with it.

ghcup set ghc 8.8.4

Then, in a temp directory:

git clone https://github.com/BNFC/bnfc
cd bnfc
git checkout v2.9.5
cd source
make doctest

[sol]

Thanks @andreasabel. Can you please try to produce a minimal example?

[andreasabel]

Frankly, no.
I often have to shrink big Agda development by others to hunt down a but and get a minimal testcase for our regression test suite. I am grateful if users provide me with an already minimal example, but I can not always expect it. I care for Agda so I am willing to invest the time (usually 2 hours shrinking per such report).
I can't both do the shrinking of bugs reported to my tools and shrink bugs for the tools I depend on.
For GHC I have done this a couple of times, shrinking the whole 400+ module Agda code base to isolate a regression in GHC, but they didn't even care to fix the bug.
So my policy is now to not do the shrinking for other tools than my own.
I supplied a reproducer; if you care about not introducing regressions in doctest, you have some reproducible evidence you can start investigating with.

[sol]

For GHC I have done this a couple of times, shrinking the whole 400+ module Agda code base to isolate a regression in GHC

I hear your frustration.

I, myself, have filled over 50 GHC bugs over the last 15 years, many of them related to regression in ghci. I always provided concrete and minimal steps on how to reproduce them. I know that this takes considerable time and effort.

they didn't even care to fix the bug

Filling a bug is no guarantee for a fix.

That said if doctest-0.23.0 introduced a regression then I'm more than eager to fix it.

However, doctest-0.22.9 fails for me with the same error:

$ git clone https://github.com/BNFC/bnfc
$ cd bnfc/source
$ git checkout v2.9.5
$ cabal install doctest-0.22.9 -z --overwrite-policy=always
$ cabal build && cabal repl -w doctest --repl-options='-w'
...
Examples: 305  Tried: 147  Errors: 0  Failures: 98

Frankly, I don't know why it has been working for you in the past, and now it stopped working.

I'm not saying there is no regression or bug, at this point I simply don't know.

Scenarios:

Scenario 1: Some version of doctest introduced a regression

If that is the case, then I'm eager to fix it. What I'll need from you:

1. The version of doctest that introduced the regression.
2. Some easy way to verify that this is indeed the case.

Scenario 2: You somehow managed to trigger some bug in GHC 8.8.4

This is less clearcut. There may be a way to work around it in doctest. How much effort I am going to spend on it depends on the impact. What I'll need from you:

1. A minimal example on how to reproduce.

The reason I need a minimal example is to both:

• Assess the impact.
• Make sure that it is not a user error (e.g. using :set instead of :seti).

[sol]

Without a repro, this is not a valid bug report.

[andreasabel]

The evidence is still fresh and out there: https://github.com/BNFC/bnfc/actions/runs/16466475271/job/46545019708#step:24:42
But I will now go for the painkiller (preferred method of the Swedish healthcare to address complex problems) and simply switch off doctest for 8.8.4 and 8.6.5 there.