Add logic to track rendering area of various PDF ops #19043

nicolo-ribaudo · 2024-11-14T12:32:47Z

I started working towards #6419. This PR introduces the logic to track where different elements of the PDF are rendered, and hooks it up to the debugger since @calixteman mentioned that it would be useful.

I'm marking this as draft because there are a few changes I need to make:

change the various methods in canvas.js to receive the index as a param, rather than returning a function that takes the index
clean up the "dependencies tracking", since currently it's all over the place. Ideally most of this logic should be self-contained in CanvasRecorder, so that when not recording it doesn't have a performance impact.
improve the dependency tracking (so far I'm only tracking some of them)
do not track extra dependencies (for example, a stroke path doesn't depend on the fill color)
track object dependencies
fix image dependencies tracking for transform (currently there is a .setTransform that makes it get lost)

However, I'd love to receive feedback on the direction.

Commit 1:

Add logic to track rendering area of various PDF ops

This commit is a first step towards #6419, and it can also help with
#13287. To support rendering part of a page, we will need to
first compute which ops can affect what is visible in that part of
the page.

This commit adds logic to track "group of ops" with their respective
bounding boxes. Each group eather corresponds to a single op or
to a range, and it can have dependencies earlier in the ops list that
are not contiguous to the range.

Consider the following example:
0. setFillRGBColor
1. beginText
2. showText "Hello"
3. endText
4. constructPath [...]
5. eoFill
here we have two groups: the text (range 1-3) and the path (range 4-5).
Each of them has a corresponding bounding box, and a dependency
on the op at index 0.

This tracking happens when first rendering a PDF: we wrap the canvas
with a "canvas recorder" that has the same API, but with additional
methods to mark the start/end of a group.

Commit 2:

Hook up the ops bbox logic to the pdf debugger

When using the pdf debugger, when hovering over a step now:

it highlights the steps in the same groups

it highlights the steps that they depend on

it highlights on the PDF itself the bounding box

This is an example of what the debugger integration looks like (note: I couldn't figure out how to make my cursor show up in the recording 😅 I'm moving it over the steps list):

Screen.Recording.2024-11-14.at.16.35.58.mov

By default it doesn't show all the bounding boxes because on some PDFs it's too much noise, but if you click on the checkbox then it shows the boxes and you can click on a box to scroll into view the corresponding ops.

src/display/canvas.js

src/display/api.js

src/display/canvas.js

nicolo-ribaudo · 2024-12-17T17:20:28Z

master...nicolo-ribaudo:pdf.js:draw-page-portion-optimized is a branch merging this PR together with #19128. In the video below you can see that it first renders in the background a low-resolution image "the old way" taking 12 seconds, and then it renders the "detail view" on top taking only 1.4 seconds and only running one fifth of the PDF operations :)

Screen.Recording.2024-12-17.at.18.10.30.mp4

Still keeping this as draft because there are significant bugs (in the PDF I'm using for testing, it often skips rendering some pieces of text even if they are visible on screen, or it renders some paths with the wrong color), but it's nice to see some progress.

bobsingor · 2025-01-01T11:14:51Z

Very good progress on this! This is a feature that the community is waiting a long time for. Can't wait to see more progress on this.

src/display/canvas.js

nicolo-ribaudo · 2025-06-02T14:05:57Z

Update!

I've reworked the dependency tracking to be based on PDF operations rather than on canvas operations. Doing it on canvas operations originally seemed cleaner, but it introduces a lot of complexity because each PDF op calls many canvas ops, and they read state from the canvas in a way that caused the tracking logic to loose information of where that state was originally coming from.
I've now hooked it up to the "detail view" logic, so that we record dependencies/bboxes while rendering the background page and then use that information when rendering the detail view.

This video shows how we are skipping some ops while rendering the detail view as we scroll around the page :)

Screen.Recording.2025-06-02.at.16.00.49.mov

The main missing task is that I have to properly hook this logic up to the reftests, maybe rendering a fraction of the page with the logic and checking that it matches the same fraction of the page with the unoptimized rendering. Once this is done, I can go through the failing tests one by one and add the missing tracking.

src/display/canvas_dependency_tracker.js

moz-tools-bot · 2025-08-04T09:11:25Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @nicolo-ribaudo received. Current queue size: 0

Live output at: http://54.241.84.105:8877/65835d880b1e3fa/output.txt

moz-tools-bot · 2025-08-04T09:11:25Z

From: Bot.io (Windows)

Received

Command cmd_test from @nicolo-ribaudo received. Current queue size: 0

Live output at: http://54.193.163.58:8877/3d1a5e199c07639/output.txt

moz-tools-bot · 2025-08-04T10:11:26Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/65835d880b1e3fa/output.txt

Total script time: 60.00 mins

nicolo-ribaudo · 2025-08-04T10:12:54Z

/botio test

moz-tools-bot · 2025-08-04T10:12:56Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @nicolo-ribaudo received. Current queue size: 0

Live output at: http://54.241.84.105:8877/5c0824b19d5019f/output.txt

moz-tools-bot · 2025-08-04T10:12:56Z

From: Bot.io (Windows)

Received

Command cmd_test from @nicolo-ribaudo received. Current queue size: 1

Live output at: http://54.193.163.58:8877/9087a0a216afcf8/output.txt

nicolo-ribaudo · 2025-08-04T10:26:52Z

There are two failures in the new tests:

issue8078-partial
intelisa-84-partial

They only happen in headless Firefox, and not in "full" Firefox or in Chrome, and the diff is that black lines are very slightly thicker. Any idea of what it could be?

moz-tools-bot · 2025-08-04T10:46:56Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/5c0824b19d5019f/output.txt

Total script time: 33.99 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

  errors: 44

Image differences available at: http://54.241.84.105:8877/5c0824b19d5019f/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2025-08-04T11:42:11Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/3d1a5e199c07639/output.txt

Total script time: 150.75 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: Passed

moz-tools-bot · 2025-08-04T12:44:59Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/9087a0a216afcf8/output.txt

Total script time: 62.82 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: Passed

This commit is a first step towards mozilla#6419, and it can also help with first compute which ops can affect what is visible in that part of the page. This commit adds logic to track "group of ops" with their respective bounding boxes. Each group eather corresponds to a single op or to a range, and it can have dependencies earlier in the ops list that are not contiguous to the range. Consider the following example: ``` 0. setFillRGBColor 1. beginText 2. showText "Hello" 3. endText 4. constructPath [...] 5. eoFill ``` here we have two groups: the text (range 1-3) and the path (range 4-5). Each of them has a corresponding bounding box, and a dependency on the op at index 0. This tracking happens when first rendering a PDF: we wrap the canvas with a "canvas recorder" that has the same API, but with additional methods to mark the start/end of a group.

When using the pdf debugger, when hovering over a step now: - it highlights the steps in the same groups - it highlights the steps that they depend on - it highlights on the PDF itself the bounding box

Account for line width when stroking Workaround for paintImageMaskXObject Fix transform tracking when using a temporary canvas Track more text dependencies Track text movement properly Forward GState dep for transparency groups showText affects positioning of next text on the same line Mark `bug1734802-partial` as known mismatch Track text font/color across `beginText` calls Fix tracking of transitive dependencies Reset sameLineText in beginText Mark `bug1443140-partial` as a known mismatch Minor cleanup Do not allocate throwaway arrays Fix tracking of leading for moveText Mark issue13130-partial as known partial mismatch Fix tracking of some text drawn by paintChar Account for unbalanced save/restore Mark artofwar-partial as known mismatch Track bbox of type 3 glyphs Track bbox of paintSolidColorImageMask Temporarily skip issue8078-partial Fix .transform call in TilingPattern helper Fix tracking of dependencies of TilingPattern Account for PDFs with no drawings PDFs with morre endText than beginText Track marked content blocks Add missing recordFullPageBBox to CanvasNestedDependencyTracker Mark pr8808-partial as known partial mismatch Fix _createMaskCanvas tracking Mark issue12295-partial as known partial mismatch Use full page bbox for type 3 fonts with no bbox

Track bbox of paintImageMaskXObjectGroup Mark bug1365930-partial as known partial mismatch Use CanvasNestedDependencyTracker for type3 fonts Cache variables in hot loops Track paintInlineImageXObjectGroup bbox before restoring ctx Mark `issue1905-partial` as a known mismatch (it's not visible, at the edge) Track bbox in paintChar when patternFill/patternStroke Todo Fix tracking of smask group transform Mark issue1466-partial as known partial mismatch Avoid multiple CanvasNestedDependencyTracker Mark bug1898802-partial as known mismatch Fix bbox computing with rotations Fix bbox computing of text drawn by clipping Mark bug887152-partial as known partial mismatch Mark issue4926-partial as known mismatch Track text-based clip as dependencies Handle unbalanced save/restore in type3 fonts Ensure that there is a bbox in type3 font operations Fallback for fonts without bbox Track fill dependencies for shadingFill Track filters Mark issue17779-partial as known mismatch Mark a couple more known partial mismatches Fix bbox tracking for invalidPDFjsFont Avoid double fontMatrix transform this.ctx -> ctx Use a float32array for pendingBBox Use existing axialAlignedBoundingBoxfor bbox computation Use bbox stored in font when possible Add test case for untrustworthy font bbox Cache ctx stack transform multiplication Fix recordBBox in CanvasNestedDependencyTracker Remove unnecessary sorting Track rectangular clip boxes Intersect bbox with bbox of clip path data.idx -> idx Fix TS types Fix knownPartialMismatch markings

nicolo-ribaudo · 2025-08-04T13:49:05Z

/botio test

moz-tools-bot · 2025-08-04T13:49:08Z

From: Bot.io (Windows)

Received

Command cmd_test from @nicolo-ribaudo received. Current queue size: 0

Live output at: http://54.193.163.58:8877/e977ca91524b114/output.txt

moz-tools-bot · 2025-08-04T13:49:08Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @nicolo-ribaudo received. Current queue size: 0

Live output at: http://54.241.84.105:8877/44ba5a9fb922d6c/output.txt

moz-tools-bot · 2025-08-04T14:06:49Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/44ba5a9fb922d6c/output.txt

Total script time: 17.67 mins

Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

  errors: 1211

Image differences available at: http://54.241.84.105:8877/44ba5a9fb922d6c/reftest-analyzer.html#web=eq.log

Fix rebasing mistake

moz-tools-bot · 2025-08-04T14:26:47Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/e977ca91524b114/output.txt

Total script time: 37.64 mins

Unit tests: FAILED
Integration Tests: FAILED
Regression tests: FAILED

  errors: 1211

Image differences available at: http://54.193.163.58:8877/e977ca91524b114/reftest-analyzer.html#web=eq.log

nicolo-ribaudo · 2025-08-04T19:59:45Z

/botio test

moz-tools-bot · 2025-08-04T19:59:47Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @nicolo-ribaudo received. Current queue size: 0

Live output at: http://54.241.84.105:8877/3edb10cbbfc3cf2/output.txt

moz-tools-bot · 2025-08-04T19:59:47Z

From: Bot.io (Windows)

Received

Command cmd_test from @nicolo-ribaudo received. Current queue size: 0

Live output at: http://54.193.163.58:8877/54120fbb86bd19d/output.txt

moz-tools-bot · 2025-08-04T20:17:16Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/3edb10cbbfc3cf2/output.txt

Total script time: 17.47 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

  errors: 1211

Image differences available at: http://54.241.84.105:8877/3edb10cbbfc3cf2/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2025-08-04T20:33:46Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/54120fbb86bd19d/output.txt

Total script time: 33.97 mins

Unit tests: FAILED
Integration Tests: FAILED
Regression tests: FAILED

  errors: 1211

Image differences available at: http://54.193.163.58:8877/54120fbb86bd19d/reftest-analyzer.html#web=eq.log

nicolo-ribaudo commented Nov 14, 2024

View reviewed changes

src/display/canvas.js Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Nov 14, 2024

View reviewed changes

src/display/api.js Fixed Show fixed Hide fixed

src/display/api.js Fixed Show fixed Hide fixed

nicolo-ribaudo force-pushed the compute-bounding-boxes branch 4 times, most recently from 2475b16 to 5a6a877 Compare November 14, 2024 15:54

timvandermeij added core viewer performance labels Nov 17, 2024

github-advanced-security bot found potential problems Nov 18, 2024

View reviewed changes

src/display/canvas.js Fixed Show fixed Hide fixed

nicolo-ribaudo mentioned this pull request Nov 29, 2024

[api-minor] Render high-res partial page views when falling back to CSS zoom (bug 1492303) #19128

Merged

nicolo-ribaudo force-pushed the compute-bounding-boxes branch from 49c4689 to eac70e4 Compare December 16, 2024 15:34

timvandermeij mentioned this pull request Jan 26, 2025

Mobile Chrome crash on Pinch and Zoom > 200% ( Android 13,built in Web View but latest Chrome as well) stephanrauh/ngx-extended-pdf-viewer#2568

Closed

marco-c mentioned this pull request Apr 3, 2025

Add a pref in order to cap the canvas area to a factor of the window one (bug 1958015) #19755

Merged

marco-c added this to PDF.js quality May 17, 2025

github-project-automation bot moved this to In progress in PDF.js quality May 17, 2025

marco-c moved this from In progress to High priority in PDF.js quality May 17, 2025

nicolo-ribaudo force-pushed the compute-bounding-boxes branch 2 times, most recently from 4cd3d42 to e24c57c Compare May 19, 2025 13:33

nicolo-ribaudo mentioned this pull request May 19, 2025

Show the op dispatched by constructPath in the debugger #19961

Merged

nicolo-ribaudo force-pushed the compute-bounding-boxes branch from e24c57c to 8184a06 Compare May 29, 2025 14:00

github-advanced-security bot found potential problems May 29, 2025

View reviewed changes

src/display/canvas.js Fixed Show fixed Hide fixed

nicolo-ribaudo force-pushed the compute-bounding-boxes branch from 8184a06 to cad8d31 Compare June 1, 2025 16:43

nicolo-ribaudo commented Jun 3, 2025

View reviewed changes

src/display/canvas_dependency_tracker.js Outdated Show resolved Hide resolved

nicolo-ribaudo force-pushed the compute-bounding-boxes branch 3 times, most recently from c294316 to 740b221 Compare June 9, 2025 17:36

nicolo-ribaudo force-pushed the compute-bounding-boxes branch from d676b96 to 9b0b5b9 Compare August 4, 2025 09:38

nicolo-ribaudo added 7 commits August 4, 2025 15:21

Hook up the ops bbox logic to the pdf debugger

c3dd63b

When using the pdf debugger, when hovering over a step now: - it highlights the steps in the same groups - it highlights the steps that they depend on - it highlights on the PDF itself the bounding box

Support reftests for partial rendering

466922f

Crop test canvas at integer coordinates

dbd7a21

Add enableOptimizedPartialRendering pref, false by default

1d5754d

nicolo-ribaudo force-pushed the compute-bounding-boxes branch from c19bd43 to 1d5754d Compare August 4, 2025 13:23

fixup! Add logic to track rendering area of various PDF ops

cd9beb4

Fix rebasing mistake

Add logic to track rendering area of various PDF ops #19043

Are you sure you want to change the base?

Add logic to track rendering area of various PDF ops #19043

Uh oh!

Conversation

nicolo-ribaudo commented Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nicolo-ribaudo commented Dec 17, 2024

Uh oh!

bobsingor commented Jan 1, 2025

Uh oh!

Uh oh!

nicolo-ribaudo commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Linux m4)

Received

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Windows)

Received

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Linux m4)

Failed

Uh oh!

nicolo-ribaudo commented Aug 4, 2025

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Linux m4)

Received

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Windows)

Received

Uh oh!

nicolo-ribaudo commented Aug 4, 2025

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Linux m4)

Failed

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Windows)

Failed

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Windows)

Failed

Uh oh!

nicolo-ribaudo commented Aug 4, 2025

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Windows)

Received

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Linux m4)

Received

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Linux m4)

Failed

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Windows)

Failed

Uh oh!

nicolo-ribaudo commented Aug 4, 2025

Uh oh!

moz-tools-bot commented Aug 4, 2025

From: Bot.io (Linux m4)

Received

nicolo-ribaudo commented Nov 14, 2024 •

edited

Loading

nicolo-ribaudo commented Jun 2, 2025 •

edited

Loading