bat shows BOM at the start of the line

[rasjani]

when the source code has BOM at the begining of the line, its shown as "<U+FEFF>". I have now huge problem with this per se but it would be more user friendly to strip that piece away when showing the code and putting that info as interpreted value into same line as "File:" is, like additional piece "Encoding: UTF-16 Big Endian" ..

Happens atleast with bat 0.5.0

See https://en.wikipedia.org/wiki/Byte_order_mark for details

[sharkdp]

Thank you for your feedback.

I have recently written a small library (https://crates.io/crates/content_inspector) that I plan to use within bat. It will be able to detect BOMs, which should make this feature request possible. I love the idea of showing the encoding within the header. I was always planning to put more information there.

[sharkdp]

Just curious: I have never encountered a UTF-16BE file "in the wild". Where do they appear? Windows?

[rasjani]

Yeah, this came up in a c# code that only targets windows. But to be honest, i didnt check if that FEFF bom was BE or LE :)

[lzybkr]

There was a commonly used Windows tool that generated UTF-16BE files by default (for no good reason really), but the default was thankfully changed to UTF8.

When adding BOM detection, it would be great to convert UTF16 to UTF8. Right now, UTF16-LE renders ^@ between each character, presumably because the text is being treated as UTF8, e.g.:

[sharkdp]

When adding BOM detection, it would be great to convert UTF16 to UTF8.

That would be the plan, yes.

There was a commonly used Windows tool that generated UTF-16BE files by default

Notepad? :-)

[lzybkr]

Notepad? :-)

That'd be funny, but no, it was powershell_ise.exe.

Thanks for the quick fix.

[sharkdp]

Released in v0.8.0. @rasjani Could you please check if this works for you?