Alternative pdf parsers

[yasirdemircan]

Is your feature request related to a problem? Please describe.
No , it is an enhancement

Describe the solution you'd like
Using docling as the pdf parser instead of pdf2text if the performance requirements would not outweight the benefits.

Performance of the library and issues about it

Approximately 30 seconds for parsing a FIPS-140 document and 1 minutes for a CC artifact.
I have tested the library with 72 FIPS certs and 1 CC certificate so far.

The library is better at parsing tables than pdf2text and lets you output Pandas dataframes from PDF tables.

Page headings labeling feature can be used to distinguish cert id clearly from references.

The library is bad at detecting sentence boundaries , additional sentence tokenization is needed for semantic analysis work.

We can discard unnecessary parts of the certificates during analysis like captions or footnotes from detected labels.

[adamjanovsky]

Better parser would be a welcomed additional. I'm proposing some next steps:

• Identify number of documents to run the qualitative evaluation against
• Run docling on those documents, measure the performance (on GPU).
• Specify requirements for incremental updates (in order to avoiding unnecessary conversions in periodic pipelines)
• Evalute the output quality, compare to pdftotext.

I propose waiting a bit with this. As @mukrop said, we'll host a meeting soon to prioritize goals for next year. Maybe there'll be more pressing tasks.

[J08nY]

• Evalute the output quality, compare to pdftotext.

I will use this issue to make some notes on this topic as I have some thoughs to save. Evaluating output quality is hard, as we essentially have two approaches:

• Use some tool to output raw text (pdftotext/OCR/olmOCR/some VLM...) and then process that text using our pipeline. This is done currently and swapping out the conversion tool here is easy, though sometimes at little gain, because what we get is still just (better) raw text and the rest of the pipeline will use it as is and will not (without some effort) use any new features/structure.
• Use some tool to parse the PDF and get structured output (docling/some LLM...) and then use the structure for better heuristics and further analysis. This requires much more work but should also bring more gain. On this front I believe we could gain by using the structure in:
  • Header and footer for better certificate ID heuristic.
  • Tables for FIPS algorithm table handling.
  • Bibliography for better reference annotation (in case of indirect reference through the bibliography)
  • Table of contents maybe?

We can evaluate the performance on some properties:

• Certificate ID assignment (Was the cert ID correctly assigned to the certificate?)
• Certificate reference assignment (Were the cert IDs of other certificates mentioned in it correctly assigned as references?)
• Keyword matches (Are we getting less or more keyword matches? Are they clearly wrong?)
• Extracted SARs (Are we getting less or more?)
• Does the conversion ever hallucinate? This is not a problem with something simple as pdftotext but for example olmOCR can actually hallucinate text.
• Is the method deterministic? While this is not necessary, it would be a nice to have.

Finally, we should take usability and ease of deployment into account. We want users to be able to run the tool and also get usable results.

[J08nY]

Listing our options so far:

• pdftotext (and its -raw and -layout options, so 4 in total)
• docling (and a bunch of its options, including which OCR to use [3 options] and other OCR options [depends], and the PDF backend [3 options]).
• olmOCR
• tesseract (and a bunch of its options, this is somewhat included in docling, but we could also use it directly)

This easily leads to tens of different options to try.