Code Helper attempts to index MPEG-TS files in project and eats 100% cpu

[mjbvz]

TS Template added by @mjbvz

TypeScript Version: 4.3.5

Search Terms

• video
• mpeg
• mp4
• binary

---

From @jon-valliere on January 10, 2018 22:54

• VSCode Version: 1.19.2
• OS Version: 10.13.2

Steps to Reproduce:

1. Add a MPEG-TS file (e.g. somefile.ts) outside of your normal code folder but somewhere in the project so it shows up in "Explorer" in VSC
2. Look at your processes -- intellicode totally stops working and the process eats 100%

The TypeScript processor should be smart enough to be able to detect if the file is text or not.

Copied from original issue: microsoft/vscode#41439

[mjbvz]

From @usernamehw on January 10, 2018 23:34

I wonder if you can exclude media folder(or files) from tsconfig.json:

{
  // ...
  "exclude": [
    "media"
  ]
}

[mjbvz]

There are workarounds we can tell users about on the VSCode side, but moving over to TS to see if the server should be smarter detecting if a file is binary when indexing

[mhegazy]

not sure what we can do here.. one option is to bailout on parsing if we see too many parse errors..

[jon-valliere]

I have a pretty novel idea: check for data[0] == 0x47 and data[188] == 0x47 // assume this is a mpeg-ts file. You could go through the whole file if you wanted to be "extra" sure.

[shirakaba]

I'm not sure what the convention is about where to continue discussion when an issue is a duplicate, but:

To explain @jon-valliere 's suggestion of reading date[0] == 0x47 (quoted text from issue #21453 ):

The Wikipedia page indicates that an MPEG-TS file is merely a "sequence of packets, without any global header". Thus, a valid MPEG-TS file will start with a header packet (the format is again described on the Wikipedia page), beginning with a sync byte, invariably with the value 0x47 (The character 'G').

The data[188] === 0x47 part checks the header at the start of the next packet. It adds some safety, but also wouldn't identify an (admittedly rare) single-packet MPEG-TS file. Unfortunately, the remaining three bytes of the header can all effectively take any value, so they can't be used to extend the fingerprint.

@mhegazy As an alternative to bailing out on high numbers of parse errors, one could bail out if the file size of the .ts file is above a default (yet configurable) threshold, say, 1MB. Java has a similar restriction, wherein the bytecode generated for any method needs to be under 64kB, although I believe that's a hard limit. We are already doing something similar for large JS projects with --disableSizeLimit (albeit without allowing one to configure an explicit size limit): #7444.

[shirakaba]

@mjbvz Yes, I've tested just now, and that exclude option does indeed fix the problem. However, it is ultimately no more than a workaround, as it requires the developer to be fully aware of this issue beforehand; the very nature of this problem is that it is very hard to track down due to the cryptic compiler errors. I could just as easily rename my MPEG-TS files to .tsv or .tsa, but I've lost a whole day of work on more than one occasion by not being able to identify what the actual cause of the error is.