🐛 Colored output from standard grep is not parsed correctly

[dandavison]

Delta officially supports two ways of highlighting grep output:

1. git grep with delta configured as git's pager either via core.pager = delta or pager.grep = delta
2. rg --json ... | delta

However, delta does try to highlight other grep output that is piped to it. This issue shows it failing to handle output from standard grep (grep (BSD grep) 2.5.1-FreeBSD)

cf

Similarly

[dandavison]

It looks like parse_style_sections is failing to parse the ANSI escape sequences emitted by BSD grep and GNU grep (and ag, pt; whereas ack, rg and sift seem to work by default). For example,

delta(794-ansi-parse-bug *) grep --color=always -Hn s src/**/*.rs | head | bat -A   
src/align.rs:1:u␛[01;31m␛[Ks␛[m␛[Ke·␛[01;31m␛[Ks␛[m␛[Ktd::cmp::max;␊
src/align.rs:2:u␛[01;31m␛[Ks␛[m␛[Ke·␛[01;31m␛[Ks␛[m␛[Ktd::collection␛[01;31m␛[Ks␛[m␛[K::VecDeque;␊
src/align.rs:4:con␛[01;31m␛[Ks␛[m␛[Kt·SUBSTITUTION_COST:·u␛[01;31m␛[Ks␛[m␛[Kize·=·1;␊
src/align.rs:5:con␛[01;31m␛[Ks␛[m␛[Kt·DELETION_COST:·u␛[01;31m␛[Ks␛[m␛[Kize·=·1;␊
src/align.rs:6:con␛[01;31m␛[Ks␛[m␛[Kt·INSERTION_COST:·u␛[01;31m␛[Ks␛[m␛[Kize·=·1;␊
src/align.rs:11:····Sub␛[01;31m␛[Ks␛[m␛[Ktitution,␊
src/align.rs:13:····In␛[01;31m␛[Ks␛[m␛[Kertion,␊
src/align.rs:16:u␛[01;31m␛[Ks␛[m␛[Ke·Operation::*;␊
src/align.rs:18:///·Needleman-Wun␛[01;31m␛[Ks␛[m␛[Kch·/·Wagner-Fi␛[01;31m␛[Ks␛[m␛[Kcher·table·for·computation·of·edit·di␛[01;31m␛[Ks␛[m␛[Ktance·and·a␛[01;31m␛[Ks␛[m␛[K␛[01;31m␛[Ks␛[m␛[Kociated␊
src/align.rs:21:␛[01;31m␛[Ks␛[m␛[Ktruct·Cell·{␊

delta(794-ansi-parse-bug *) grep --color=always -Hn s src/**/*.rs | delta        
[src/ansi/mod.rs:76] &s[start..end] = "u"
[src/ansi/mod.rs:76] &s[start..end] = "\u{1b}"
[src/ansi/mod.rs:76] &s[start..end] = "\u{1b}["
[src/ansi/mod.rs:76] &s[start..end] = "\u{1b}"
[src/ansi/mod.rs:76] &s[start..end] = "\u{1b}[Ktd::cmp::max;"
thread 'main' panicked at 'String mismatch encountered while superimposing style sections: 's 't'', src/paint.rs:921:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
src/align.rs:1:

The rust dbg! output shows that some ANSI escape sequences are being parsed as text.

[alerque]

I just started getting this panic when used as git's diff filter.

$ git -c pager.diff=delta diff --staged

removed: ibraniler/20211114-en.md

────────────────────────────┐
ibraniler/20211114-en.md:0: │
────────────────────────────┘
thread 'main' panicked at 'String mismatch encountered while superimposing style sections: '…' vs ' '', src/paint.rs:921:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
$ delta --version
delta 0.10.1

Running git -c pager.diff=cat diff --staged gives me the expected output with nothing egregious in the content, just a bunch of markdown input lines added in one file and the file deletion.

[dandavison]

Hi @alerque, thanks for this. Could you post the raw diff that's causing this? I.e. the file produced bygit diff (your args) > diff.txt.

[alerque]

I can't post a dump of the full content of the diff as it has some private unpublished stuff, but I have at least two commits in this project now that are giving the same panic results so when I get a second I'll try to comb through them and figure out what bit is delta is tripping on.

[alerque]

It isn't just the raw content.

$ git show 57e5082
snip... thread 'main' panicked at 'String mismatch encountered while superimposing style sections: '↵' vs '↴'', src/paint.rs:921:17
$ git show 57e5082 > diff.txt
$ cat diff.txt | delta
snip... correctly rendered diff of 57e5082

[alerque]

The difference above is in the delta arguments used. In my Git config I have:

[delta]
	features = side-by-side line-numbers decorations line-wrap
	hunk-header-style = file line-number syntax
    whitespace-error-style = 22 reverse
	syntax-theme = Monokai Extended
	max-line-length = 0

If I drop the side-by-side feature it works fine as a git pager. Only if I add that as an argument and have it render the diff.txt as per my example that works fine too.

[dandavison]

Do you have git configured to display moved chunks with color-moved? Is the problem unicode characters in moved chunks?

If you could try to make the diff smaller so that it still gives the problem but isn't too private to share that would help a lot!

[alerque]

If I could get the output diff content to make delta panic it would be easy to break the diff up into chunks and find offending content and sent a version that doesn't publicize anything private. The issue is I haven't gotten it to fail except when used directly as a git's diff viewer.

Yes I do have git -c diff.colorMoved=zebra set and yes it does work (including in side-by-side mode) if I disable that. So this is some combination of specific content + delta.side-by-side + git.diff.colorMoved.

[dandavison]

The issue is I haven't gotten it to fail except when used directly as a git's diff viewer.

Oh I'm sorry, I should have pointed out -- you need to use git show --color=always to force git to emit ANSI color escape sequences when its output is not going to the screen.

[alerque]

Okay I have something for you. I exctracted Git's output from the offending commit like this:

$ git -c pager.show=nvim show 57e5082

In Neovim I ran :w diff-codes.txt to get a file with all the original escape codes intact.

Then tried to render it like this:

$ < diff-codes.txt delta --side-by-side --pager cat

Seeing that that produced the same panic, I then went through and started deleting lines until I found the offending one. Here is a file to replicate the problem: diff-codes.txt

Delta appears to have a beef with a very much not-private previously published line of Bible text taken from Hebrews 4:5 (ESV):

And again in this passage he said, "They shall not enter my rest."

This (and a few hundred surrounding lines) were deleted from one file and moved to another in this commit. In messing around with the content, the problem went away if I changed the part before the comma to be "foo".

Lastly in preparing this post I had a moment of panic because I couldn't replicate it on a second pass when I was all done with the writeup. The difference turned out to be terminal width: I get a panic on a narrow terminal (98 cols) but not a wide one (197 cols).

[dandavison]

Great, thanks! I can reproduce that (as you say, dependent on terminal width).

I also found an example while you were doing that. Going backwards through nginx commits with side-by-side active causes the error below, with a wide terminal and side-by-side.

nginx(master) git log --reverse -p | delta --side-by-side > /dev/null
[src/paint.rs:925] s1 = "ex_module ��� ������ ������� �\u{363}� "
[src/paint.rs:926] s2 = "ex_module ��� ������ ������� �\u{363}�↵"
thread 'main' panicked at 'String mismatch encountered while superimposing style sections: ' ' vs '↵'', src/paint.rs:927:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace