🦣🦣🦣 Table virtualization #14743

lucasbordeau · 2025-09-26T14:05:04Z

This big PR implements table virtualization with an offset paging, allowing a way more fluid UX.

It is a v1 that should be improved in the future with partial data loading and optimization of the browser display performance of a row.

But with this PR we have the solid enough technical foundation, both frontend and backend, to get to a smooth table UX.

Fixes and improvements after first successful round of development (needed to have main clean) :

Fixes twentyhq/core-team-issues#1613 that contains other bugs to be fixed before merge

greptile-apps

Greptile Overview

Summary

This PR implements table virtualization for the record table to improve performance when handling large datasets. The implementation introduces a virtualized row system that renders only visible rows plus a buffer, dynamically fetches data as users scroll, and manages state through a treadmill pattern that reuses DOM elements.

Key Changes

Virtualization Core: RecordTableRowVirtualized component that renders rows based on virtual/real index mapping
Data Loading: RecordTableVirtualizedDataLoaderEffect handles progressive data fetching with batched UI updates
Scroll Management: RecordTableVirtualizedRowTreadmillEffect manages the scroll-to-virtualization mapping with proper debouncing
State Management: Multiple new state atoms track virtualization indices, fetch status, and scroll positions
Utility Function: New getRange utility added to twenty-shared for array operations

Issues Found

Debug output present: The virtualized row component contains debug output that should be removed before production
Missing error handling: Async data loading operations lack proper error handling which could cause silent failures
Hardcoded values: Magic numbers used instead of established constants for consistency
API design: getRange function has confusing parameter semantics that could lead to off-by-one errors

Confidence Score: 3/5

This PR introduces significant performance improvements but contains debug output and missing error handling that need attention
Score reflects the architectural complexity and solid implementation approach, but several issues must be addressed: debug output needs removal, async operations need error handling, and hardcoded values should use constants. The virtualization logic itself is well-structured with proper debouncing and state management.
Pay close attention to RecordTableRowVirtualized.tsx (debug output removal) and RecordTableVirtualizedDataLoaderEffect.tsx (error handling)

Important Files Changed

File Analysis

Filename	Score	Overview
packages/twenty-front/src/modules/object-record/record-table/virtualization/components/RecordTableRowVirtualized.tsx	2/5	Core virtualized row component with debug output that needs removal and hardcoded values that should use constants
packages/twenty-front/src/modules/object-record/record-table/virtualization/components/RecordTableVirtualizedDataLoaderEffect.tsx	3/5	Data loading logic with missing error handling for async operations that could cause silent failures
packages/twenty-shared/src/utils/array/getRange.ts	3/5	Utility function with confusing parameter naming that could lead to off-by-one errors

Sequence Diagram

sequenceDiagram
    participant User
    participant RecordTable
    participant VirtualBody
    participant DataLoader
    participant TreadmillEffect
    participant API
    
    User->>RecordTable: Initial load
    RecordTable->>VirtualBody: Mount virtualized body
    VirtualBody->>DataLoader: Initialize data loading
    DataLoader->>API: fetchRecordsLazy()
    API-->>DataLoader: records + totalCount
    DataLoader->>VirtualBody: resetVirtualizedRowTreadmill()
    DataLoader->>VirtualBody: setRecordTableData(records)
    
    User->>RecordTable: Scroll down
    RecordTable->>TreadmillEffect: handleScroll event
    TreadmillEffect->>TreadmillEffect: Calculate new visible range
    TreadmillEffect->>VirtualBody: Update realIndexByVirtualIndex mapping
    
    Note over TreadmillEffect: Checks if more data needed
    alt lastRealIndexSet > hasAlreadyFetchedUpToRealIndex
        TreadmillEffect->>DataLoader: Trigger fetch more
        DataLoader->>API: fetchMoreRecords(150)
        API-->>DataLoader: more records
        DataLoader->>DataLoader: Batch process records (10 at a time)
        loop For each batch
            DataLoader->>VirtualBody: Update recordIdByRealIndex state
            DataLoader->>DataLoader: Wait 50ms between batches
        end
    end
    
    VirtualBody->>RecordTable: Render virtualized rows
    RecordTable-->>User: Display updated table

_{46 files reviewed, 7 comments}

_{Edit Code Review Agent Settings | Greptile}

...c/modules/object-record/record-table/virtualization/components/RecordTableRowVirtualized.tsx

...ect-record/record-table/virtualization/components/RecordTableVirtualizedDataLoaderEffect.tsx

greptile-apps · 2025-09-26T14:10:19Z

packages/twenty-shared/src/utils/array/getRange.ts

+export const getRange = (start: number, end: number) => {
+  return Array.from({ length: end }, (_, index) => index + start)
+}


logic: Function behavior is confusing - getRange(0, 300) generates 0-299, not 0-300 as might be expected

Suggested change

export const getRange = (start: number, end: number) => {

return Array.from({ length: end }, (_, index) => index + start)

}

export const getRange = (start: number, length: number) => {

return Array.from({ length }, (_, index) => index + start);

};

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/twenty-shared/src/utils/array/getRange.ts Line: 1:3 Comment: logic: Function behavior is confusing - `getRange(0, 300)` generates 0-299, not 0-300 as might be expected ```suggestion export const getRange = (start: number, length: number) => { return Array.from({ length }, (_, index) => index + start); }; ``` How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2025-09-26T14:10:20Z

...ord/record-table/virtualization/components/RecordTableNoRecordGroupVirtualizedBodyEffect.tsx

+    totalNumberOfRecordsToVirtualize,
+  ]);
+
+  return <></>;


style: Use null instead of empty fragment when returning nothing

Suggested change

return <></>;

return null;

Context Used: Context - Avoid using fragments when there is only one child component in a return statement. (link)

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/twenty-front/src/modules/object-record/record-table/virtualization/components/RecordTableNoRecordGroupVirtualizedBodyEffect.tsx Line: 108:108 Comment: style: Use `null` instead of empty fragment when returning nothing ```suggestion return null; ``` **Context Used:** **Context -** Avoid using fragments when there is only one child component in a return statement. ([link](https://app.greptile.com/review/custom-context?memory=51414064-3127-4b1d-ad7e-62ce2c3739e9)) How can I resolve this? If you propose a fix, please make it concise.

...ect-record/record-table/virtualization/components/RecordTableVirtualizedDataLoaderEffect.tsx

github-actions · 2025-09-26T14:17:19Z

🚀 Preview Environment Ready!

Your preview environment is available at: http://bore.pub:43433

This environment will automatically shut down when the PR is closed or after 5 hours.

seer-by-sentry · 2025-10-10T16:41:22Z

...rc/modules/object-record/record-drag/table/hooks/useRecordTableWithoutGroupDragOperations.ts

        });

        if (dragOperationType === 'single') {
+          const targetRecordId = allSparseRecordIds.at(
+            result.destination.index,
+          );
+
+          if (!isDefined(targetRecordId)) {
+            throw new Error(
+              `Target record id cannot be found, this should not happen`,
+            );
+          }
+
          const singleDragResult = processSingleDrag({
-            result,
-            recordPositionData,
-            recordIds: allRecordIds,
+            sourceRecordId: draggedRecordId,
+            targetRecordId: targetRecordId ?? '',
+            recordsWithPosition: contiguousRecordsWithPosition,
          });


Potential bug: A record's position can be null, but it's passed to a function expecting a number, causing a runtime error during drag-and-drop.

Description: During a drag-and-drop operation, records are mapped to an array of objects with an id and position. The position is derived using getSnapshotValue(...)?.position ?? null, which can result in null if a record's data is not fully loaded in the store. This array is then passed to sortByProperty, which does not support null values and will throw an error: "Property type not supported in sortByProperty, only string and number are supported". This will cause the drag-and-drop feature to fail.

Suggested fix: Modify the position retrieval logic to provide a default numeric value instead of null. For example, change ?.position ?? null to ?.position ?? 0, aligning with the position field's default value in the entity definition.
_{severity: 0.8, confidence: 0.95}

_{Did we get this right? 👍 / 👎 to inform future reviews.}

charlesBochet · 2025-10-10T17:14:17Z

packages/twenty-front/src/modules/activities/utils/getActivityTargetObjectRecords.ts

  activityRecord: Note | Task;
  objectMetadataItems: ObjectMetadataItem[];
-  activityTargets?: NoteTarget[] | TaskTarget[];
+  activityTargets?: Nullable<NoteTarget[] | TaskTarget[]>;


charlesBochet · 2025-10-10T17:14:36Z

...dules/command-menu/pages/message-thread/hooks/__tests__/useEmailThreadInCommandMenu.test.tsx

          $orderBy: [MessageOrderByInput]
          $lastCursor: String
          $limit: Int
+          $offset: Int


why this change? looks unrelated to the PR

charlesBochet · 2025-10-11T07:54:38Z

packages/twenty-front/src/modules/object-record/hooks/useLazyFindManyRecordsWithOffset.ts

+  'objectNameSingular'
+>;
+
+export const useLazyFindManyRecordsWithOffset = ({


fileName <> componentName

charlesBochet · 2025-10-11T08:02:54Z

...ject-record/record-table/record-table-cell/components/RecordTableCellCheckboxPlaceholder.tsx

+  border-left: 1px solid transparent;
+`;
+
+export const RecordTableCellCheckboxPlaceholder = () => {


would be nice to uniformize naming Skeleton <> Placeholder

charlesBochet · 2025-10-11T08:05:16Z

...ject-record/record-table/virtualization/components/RecordTableRowVirtualizedRouterLevel1.tsx

+  realIndex: number;
+};
+
+export const RecordTableRowVirtualizedRouterLevel1 = ({


Router naming is confusing!

charlesBochet · 2025-10-11T08:06:32Z

...ct-record/record-table/virtualization/components/RecordTableVirtualizedDataChangedEffect.tsx

+      lastObjectOperation?.id !==
+      lastObjectOperationThatResettedVirtualization?.id
+    ) {
+      setLastObjectOperationThatResettedVirtualization(lastObjectOperation);


I think we should discuss this, this seems hard to maintain

charlesBochet · 2025-10-11T08:07:38Z

...ecord/record-table/virtualization/components/RecordTableVirtualizedInitialDataLoadEffect.tsx

+import { useEffect } from 'react';
+
+// TODO: see if we can merge the initial and load more processes, to have only one load at scroll index effect
+export const RecordTableVirtualizedInitialDataLoadEffect = () => {


that's a lot of effects would be great to see if we can refactor. DevXP is a bit complex regarding data loading, we might want to take a tradeoff to simplify

charlesBochet · 2025-10-11T08:08:20Z

...front/src/modules/object-record/record-table/virtualization/hooks/useAssignRecordsToStore.ts

+            .getLoadable(recordStoreFamilyState(record.id))
+            .getValue();
+
+          if (JSON.stringify(currentRecord) !== JSON.stringify(record)) {


isDeeplyEqual

lucasbordeau marked this pull request as draft September 26, 2025 14:05

greptile-apps bot reviewed Sep 26, 2025

View reviewed changes

seer-by-sentry bot reviewed Sep 26, 2025

View reviewed changes

...ect-record/record-table/virtualization/components/RecordTableVirtualizedDataLoaderEffect.tsx Outdated Show resolved Hide resolved

lucasbordeau added the -PR: wip label Sep 26, 2025

lucasbordeau force-pushed the refactor/table-infinite-scroll branch 3 times, most recently from bbb5d12 to 8b625bf Compare October 1, 2025 08:57

lucasbordeau changed the title ~~Table virtualization~~ 🦣 Table virtualization Oct 1, 2025

lucasbordeau force-pushed the refactor/table-infinite-scroll branch 8 times, most recently from 2bcf465 to 4e97a8f Compare October 6, 2025 11:49

lucasbordeau mentioned this pull request Oct 7, 2025

RecordBoards show white screen if scrolled fast #8550

Closed

lucasbordeau force-pushed the refactor/table-infinite-scroll branch 3 times, most recently from 2e7760d to c04d362 Compare October 8, 2025 17:02

lucasbordeau changed the title ~~🦣 Table virtualization~~ 🦣🦣🦣 Table virtualization Oct 9, 2025

lucasbordeau force-pushed the refactor/table-infinite-scroll branch from c04d362 to a1c0971 Compare October 9, 2025 16:08

Ok

1ea2596

lucasbordeau force-pushed the refactor/table-infinite-scroll branch from ac6eaae to 1ea2596 Compare October 10, 2025 16:29

lucasbordeau marked this pull request as ready for review October 10, 2025 16:34

Fixed tests

b4c1e98

seer-by-sentry bot reviewed Oct 10, 2025

View reviewed changes

Merge branch 'main' into refactor/table-infinite-scroll

3639012

charlesBochet reviewed Oct 10, 2025

View reviewed changes

lucasbordeau added the preview-app label Oct 10, 2025

lucasbordeau and others added 3 commits October 10, 2025 19:38

Fix build

013ea0d

Fixed loading

337ee3c

Fix not logged in

a7c7e1b

charlesBochet reviewed Oct 11, 2025

View reviewed changes

Fix

fdb4d37

charlesBochet reviewed Oct 11, 2025

View reviewed changes

🦣🦣🦣 Table virtualization #14743

Are you sure you want to change the base?

🦣🦣🦣 Table virtualization #14743

Uh oh!

Conversation

lucasbordeau commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Greptile Overview

Summary

Key Changes

Issues Found

Confidence Score: 3/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps bot Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seer-by-sentry bot Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lucasbordeau commented Sep 26, 2025 •

edited

Loading

github-actions bot commented Sep 26, 2025 •

edited

Loading