Explore using `ArrayBuffer` as backend for editor text contents

[bpasero]

Explore to not use string but ArrayBuffer for the contents of a text editor which brings higher memory limits. See #167719 (comment)

Original Issue (now outdated)

I see how https://github.com//issues/42839 introduced a way to avoid opening large files by telling the user to quit and restart with a larger `--max-memory`. First of all, I think asking the user to confirm opening a very large file is good because it may result in freezes or slowdown. However, I am no longer able to reproduce the endless freeze, even when opening a very large file (see comment [1] how to create a large file). Maybe given our work on sandboxing this issue is now different or not existent anymore?

At the same time, with the Electron 22 update, we get https://www.electronjs.org/blog/v8-memory-cage so I wonder if --max-memory is even still supported (maybe Deepak could clarify).

This issue is to:

• validate that our original assumptions for adding this still apply
• see if --max-memory still works with Electron 22
• and if not, maybe we change the UX to still block opening the file at first by asking the user for confirmation (similar to how we ask for confirmation before opening a binary file)

[1] How to generate a large file
Inspired by this page, the following script that you can run from a macOS terminal produces a 4.23 GB random file for me after ~2min:

ruby -e 'a=STDIN.readlines;100000000.times do;b=[];4.times do; b << a[rand(a.size)].chomp end; puts b.join(" "); end' < /usr/share/dict/words > file.txt

//cc @deepak1556

[bpasero]

Some interesting discoveries:

• I can open 2 4gb each random files without problems from the E22 branch
• ArrayBuffer is granted a larger space than 4 GB because (as far as I understand) they are not allocated on the heap
• we have maxFileSize and maxHeapSize driving the decision if such a large file can open and I had to disable that [1]

[1]

vscode/src/vs/platform/files/common/files.ts

Lines 1321 to 1322 in 9718242

	maxFileSize: arch === Arch.IA32 ? 300 * ByteSize.MB : 16 * ByteSize.GB, // https://github.com/microsoft/vscode/issues/30180
	maxHeapSize: arch === Arch.IA32 ? 700 * ByteSize.MB : 2 * 700 * ByteSize.MB, // https://github.com/v8/v8/blob/5918a23a3d571b9625e5cce246bdd5b46ff7cd8b/src/heap/heap.cc#L149

[rebornix]

We discussed offline about this task and one of the idea is using ArrayBuffer directly as the backing store for the text buffer instead of using string so we won't be affected by the heap size limit. I experimented a bit (in https://github.com/microsoft/vscode/tree/rebornix/anonymous-donkey) and found moving from string to array buffer for PieceTree is pretty easy.

The real challenge is currently most our editor internal parts eagerly read line contents (through getLineContent). If we cache lineContent, it defeats the purpose of using ArrayBuffer. If we don't, then a component attempts to read every line content of the text model, might block the whole UI if the file is large. I have already moved getLineContent to getLineCharCode if possible in https://github.com/microsoft/vscode/tree/rebornix/anonymous-donkey but following components are not easy to move off line content:

• Folding range compute

  vscode/src/vs/editor/contrib/folding/browser/indentRangeProvider.ts

  Line 155 in 4786a49

  const lineContent = model.getLineContent(line);

• Line breaks computer

  vscode/src/vs/editor/common/viewModel/viewModelLines.ts

  Line 133 in 4786a49

  lineBreaksComputer.addRequest(this.model.getLineContent(i + 1), lineInjectedText, previousLineBreaks ? previousLineBreaks[i] : null);

• Model syncing to web worker (this can be no-op though once we have sab)

[bpasero]

👏 for pushing this forward, would be great if we can look into the 2 remaining ones, I am happy to help if possible.

[tahmidbintaslim]

Hi Everyone, I still see this issue happening to me whenever I try to open big files (Example: Larger than 10 mb) will all installed plugin and settings. Any update on this?

[bpasero]

I still see this issue happening to me whenever I try to open big files

What issue?

[bpasero]

Fyi I have removed support for --max-memory and related editor integrations from our product via 6d5b854. Turns out that max_old_space_size is unsupported since a very long time (electron/electron#31330).