feat(core): Add Data Store Backend API (no-changelog) #17824

CharlieKolb · 2025-07-30T09:19:19Z

Summary

Initial draft for the data store backend.

Remaining work for initial deliverable:

Still misses clearStore, deleteRow and updateRow operations.
Should start storing sizing data if feasible - currently the approach of tracking size of inserted/updated/deleted data manually seems favorable

I'm new to this, so there's probably a few issues, particularly I'm unsure about the following:

When to use dtos vs just taking one or two parameters in-place.
Whether it's okay to rely on these dtos in the repository typings - I'm using some where it makes sense and am adding to api-types where I need modification

Related Linear tickets, Github issues, and Community forum posts

Linear project with links to notion phases https://linear.app/n8n/project/project-data-store-d3559ac715b1
Linear ticket for this PR https://linear.app/n8n/issue/ADO-3851/feature-implement-controller-and-service

Review / Merge checklist

PR title and summary are descriptive. (conventions)
Docs updated or follow-up ticket created.
Tests included.
PR Labeled with release/backport (if the PR is an urgent fix that needs to be backported)

CharlieKolb · 2025-07-30T09:44:10Z

packages/@n8n/permissions/src/roles/scopes/global-scopes.ee.ts

@@ -79,6 +79,13 @@ export const GLOBAL_OWNER_SCOPES: Scope[] = [
 	'insights:list',
 	'folder:move',
 	'oidc:manage',
+	'dataStore:create',


these are for personal projects also, right?

It should be for all projects I think

packages/@n8n/permissions/src/roles/scopes/project-scopes.ee.ts

dariacodes · 2025-07-31T12:33:53Z

packages/@n8n/backend-common/src/modules/module-registry.ts

@@ -23,7 +23,7 @@ export class ModuleRegistry {
 		private readonly modulesConfig: ModulesConfig,
 	) {}

-	private readonly defaultModules: ModuleName[] = ['insights', 'external-secrets'];
+	private readonly defaultModules: ModuleName[] = ['insights', 'external-secrets', 'data-store'];


[nit] Shall we make it plural for consistency?

packages/@n8n/db/src/migrations/common/1747814180618-CreateDataStoreTables.ts

packages/cli/src/modules/data-store/data-store-rows.repository.ts

packages/cli/src/modules/data-store/data-store.controller.ts

packages/cli/src/modules/data-store/data-store.entity.ts

packages/cli/src/modules/data-store/data-store.types.ts

packages/cli/src/modules/data-store/data-store-column.repository.ts

…efault

Cadiac

Lets ship this at this stage like this 🚀

I've verified that all the endpoints work and most of the code changes are within the module, which isn't enabled by default.

packages/cli/src/modules/data-store/__tests__/data-store.service.test.ts

currents-bot · 2025-08-12T12:49:38Z

✅

E2E Tests: n8n tests passed after 4m 39.1s

🟢 514 · 🔴 0 · ⚪️ 0

View Run Details

Run Details

Project: n8n
Groups: 3
Framework: Playwright
Run Status: Passed
Commit: 79af2d2
Spec files: 107
Overall tests: 514
Duration: 4m 39.1s
Parallelization: 3

Groups

GroupId	Results	Spec Files Progress
ui	🟢 39 · 🔴 0 · ⚪️ 0	13 / 13
ui:isolated	🟢 6 · 🔴 0 · ⚪️ 0	2 / 2
No name	🟢 469 · 🔴 0 · ⚪️ 0	92 / 92

This message was posted automatically by currents.dev | Integration Settings

CharlieKolb added 14 commits July 22, 2025 18:25

initial module commit

046b88e

Add functional backend skeleton

454db2f

untested wip code

ada6537

add working tests

cc24e98

Working list API

973c1c9

expose endpoints

cf6df74

add test for joining columns

507bcd0

test pagination

309e5d8

add sort by tests

3e11e93

Add row-based operations

9318e6e

expose apis

8b469ff

also expose getColumns

0e7d5ac

fix file casing

9fcbb9f

Rename API, adjust dtos and include project scopes

0695ed0

CharlieKolb changed the title ~~feature(core): Add Data Store Backend API (no-changelog)~~ feat(core): Add Data Store Backend API (no-changelog) Jul 30, 2025

n8n-assistant bot added core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team labels Jul 30, 2025

CharlieKolb commented Jul 30, 2025

View reviewed changes

packages/@n8n/permissions/src/roles/scopes/project-scopes.ee.ts Outdated Show resolved Hide resolved

CharlieKolb added 4 commits July 30, 2025 12:25

self review

01900d5

wip commit

331fc9b

fix build and add tests for shiftColumns

85a6230

implement column moving

0f8e2b1

dariacodes reviewed Jul 31, 2025

View reviewed changes

CharlieKolb and others added 6 commits July 31, 2025 17:41

throw errors instead of returning strings

cb0455a

rename endpoints, modify update dto, fix flaky test

b4a10a3

fix sql utils test

0ff50cf

apply interface and more types

33f7afe

fix tests

1e8a78f

fix import for sql utils test

461d23c

refactor(core): Avoid type casts

91069c1

scdekov reviewed Aug 11, 2025

View reviewed changes

packages/cli/src/modules/data-store/data-store-column.repository.ts Outdated Show resolved Hide resolved

scdekov reviewed Aug 11, 2025

View reviewed changes

packages/cli/src/modules/data-store/data-store-column.repository.ts Show resolved Hide resolved

dariacodes and others added 21 commits August 11, 2025 17:31

add a project scope to data store aggregate

4551e64

remove migration

205a612

Merge branch 'master' into ADO-3851

bc9ff11

rename columnIndex to index

8afdfce

remove entity from data store table names

44a8c93

rename entities

933a47d

include data-store as module

62a5f37

discard migration changes

7270296

add createdAt and updatedAt to column entity

e0cae29

test(core): Expect data-store to be missing from eligibleModules by d…

a05ecf7

…efault

move create user table logic to rows repository

e9e51b4

Merge branch 'master' into ADO-3851

c647a90

rename data store entity for new files

3c5e814

move drop column from table to rows repository

2ef3816

clean up

526a676

remove getPrimaryKeyAutoIncrement

fe8935d

fix(core): Fix data store read rows endpoint

f4ee35b

populate data store create with project

81004be

fix(core): Parse row read filters from query params correctly

a43e4d2

fix ts errors in test

69842f5

fix(editor): change the delete column endpoint to not take request body

79af2d2

dariacodes self-assigned this Aug 12, 2025

Cadiac approved these changes Aug 12, 2025

View reviewed changes

packages/cli/src/modules/data-store/__tests__/data-store.service.test.ts Outdated Show resolved Hide resolved

dariacodes merged commit 98dc71e into master Aug 12, 2025
49 of 50 checks passed

dariacodes deleted the ADO-3851 branch August 12, 2025 12:54

idirouhab pushed a commit that referenced this pull request Aug 13, 2025

feat(core): Add Data Store Backend API (no-changelog) (#17824)

27acde0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(core): Add Data Store Backend API (no-changelog) #17824

feat(core): Add Data Store Backend API (no-changelog) #17824

Uh oh!

CharlieKolb commented Jul 30, 2025 •

edited

Loading

Uh oh!

CharlieKolb Jul 30, 2025

Uh oh!

guillaumejacquart Aug 5, 2025

Uh oh!

Uh oh!

dariacodes Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Cadiac left a comment

Uh oh!

Uh oh!

currents-bot bot commented Aug 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

feat(core): Add Data Store Backend API (no-changelog) #17824

feat(core): Add Data Store Backend API (no-changelog) #17824

Uh oh!

Conversation

CharlieKolb commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Linear tickets, Github issues, and Community forum posts

Review / Merge checklist

Uh oh!

CharlieKolb Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

guillaumejacquart Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dariacodes Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Cadiac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

currents-bot bot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CharlieKolb commented Jul 30, 2025 •

edited

Loading

currents-bot bot commented Aug 12, 2025 •

edited

Loading