Unable to allocate 370. GiB for an array with shape (294791, 336473) and data type int32

[HiromuHota]

Description of the bug

When parsing a large HTML file (32MB), the following error occurred:

/home/user/.venv/lib/python3.7/site-packages/fonduer/parser/parser.py:286: UserWarning: Document XXX not added to database, because of parse error: 
Unable to allocate 370. GiB for an array with shape (294791, 336473) and data type int32

Note that "XXX" above error message is masked for privacy.

To Reproduce

Steps to reproduce the behavior:

1. Enable lingual=True
2. Parse a large HTML file

The above error message is not always shown.
Most of the times, Fonduer silently fails.

Expected behavior

A machine with 24GB can parse a HTML file of 32MB.

Error Logs/Screenshots

See above.

Environment (please complete the following information)

• OS: Docker (based on HiromuHota/fonduer-tutorials:0.8.2 but updated to the latest commit on the master branch)

• PostgreSQL Version: 12.1

• Poppler Utils Version: N/A

• spaCy Version: 2.1.9

• Fonduer Version: master (9a33ada)

• Docker Host

  • OS: Ubuntu (20.04)
  • MEM: 24GB
  • CPU: 8 cores

Additional context

The original PDF has 389 pages and I converted this into a HTML using pdftotext -bbox-layout.
The generated HTML file is 32MB.
When I disable all the four modals like structural=False, tabular=False, lingual=False, visual=False, this issue did not happen.
When I enabled only lingual (ie structural=False, tabular=False, lingual=True, visual=False, this issue happened.

I think this issue is directly related to #439 (comment), where we were discussing

Parser itself is very memory-hungry.

[HiromuHota]

As I'm almost certain this issue came from spaCy. I searched issues on https://github.com/explosion/spaCy with a query "Unable to allocate" takes me to explosion/spaCy#4847.

[HiromuHota]

A debugging revealed that the memory allocation issue happens during "parser" component in the spaCy pipeline.

https://github.com/HazyResearch/fonduer/blame/9a33adaac989c204a45b352662853ad9d7c16a8b/src/fonduer/parser/lingual_parser/spacy_parser.py#L178-L179

Fact checks:

• self.model.max_length is 1,000,000 (by default).
• sentence_batches has two inner list of sentences, and each has 9714 and 7007 sentences.

[HiromuHota]

A workaround is to lower max_length.
Currently, max_length is not configurable but hopefully becomes configurable.

max_length actually depends on PARALLEL.
E.g.,
if PARALLEL=1, max_length could be 100,000, but
if PARALLEL=2, max_length could be 50,000.

[HiromuHota]

I found that the error message originally came from NumPy.

$ python3
Python 3.8.2 (default, Apr 27 2020, 15:53:34) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> t = np.zeros((294791, 336473),dtype=np.int32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: Unable to allocate 370. GiB for an array with shape (294791, 336473) and data type int32

[HiromuHota]

I think the implication of this issue is two-hold:

1. It'd be good to make max_length configurable.
2. It'd be good to print a stack trace of an error to show where the error originates from as an error message alone is hard to debug.

[HiromuHota]

Regarding the 2nd point above,
with #479 being merged, a stack trace will be logged. This makes makes debugging easier.