Mojibake in indexing document

[wayland]

Problem or new feature

In https://docs.raku.org/language/pod#Indexing_terms , there are some extraneous characters that are probably caused by encoding problems. Specifically, I see the following:

ÿÿÿÿXÿÿÿÿ

The Y-umlauts should not be there.

[coke]

This is probably a result of the following rakudoc:

=begin code :allow<B> :lang<rakudoc>
An B<X<array>> is an ordered list of scalars indexed by number,
starting with 0. A B<X<hash>> is an unordered collection of scalar
values indexed by their associated string key.
=end code

Guessing it’s the allow: @finanalyst?

[finanalyst]

@wayland @coke Sorry for the long delay. I just saw this thread when @coke closed a duplicate

The bad characters are a result of an attempt to handle a difficult escaping problem.

As @coke mentioned the offending option is :allow.

=code is specified to treat its contents verbatim. Except that sometimes we want to highlight things. So :allow allows for RakuDoc markup to be 'allowed' inside =code contents.

Then the fashion for coloured code syntax highlighting became fashionable, which some people dont like.

So the problem comes when :allow and syntax highlighting are both operating. Whichever comes second over-rides the styling that comes first.

The approach first tried is to extract the string styled with the allow markup, putting markers into the the string (the bad chars listed above), then after the second styling (syntax highlighting) is finished, replace the markers with the allow styled string.

Except that approach does not always work, as can be seen in the texts above.

Subsequently, RakuDoc v2 was developed. The issue was discussed at length and it was decided that either syntax highlighting OR allow-styling would be obeyed, but not necessarily both.

The new Rakuast::RakuDoc::Render which implements all of RakuDoc v2 handles this. The new version of the docs website (https://new-raku.finananlyst.org) should not have this problem.

I have not had the time to implement this in Raku::Pod::Render, which the renderer used for the production site.