feat(monorepo): split indexer into apps and packages #68

tk-o · 2025-01-22T20:51:34Z

Related to #52

Goals

easy exporting for ponder schema to support the upcoming @ponder/client feature
- ~~ponder plugins to be split into packages~~
  - cannot be done as ponder app requires all handlers to be defined within ponder's rootDir
  - rootDir is a single location, hence, having plugins in separate directories won't work
- ponder-shema package to host ponder.schema.ts
ensnode-common package to host all primitives that can be used outside of ponder app context, for example, on the client side
opt-in to using the Subgraph-compatible middleware
- the Subgraph middleware will have its own package
single directory apps to host all runnable modules
- ensnode ponder app will be the first module here
- ~~ensnode app will import ponder schema & config from the ponder-bootstrap package~~
  - the ponder app expects these files to be loaded from a single rootDir — having ponder plugins in play, also place in the rootDir, prevents extracting ponder.config.ts & ponder.schema.ts into a separate package

Use abstraction to detach the subgraphGraphQL factory function from ponder

Use abstraction to detach the helpers.ts functions from ponder

Use abstraction to detach the ids.ts functions from ponder

Move the indexer source code into its dedicated place in the monorepo workspace

… package

lightwalker-eth

@tk-o Exciting to see this taking shape 👍 A few very small comments.

apps/ensnode/README.md

package.json

Simplify common packages version management

lightwalker-eth

@tk-o Hey nice updates 👍 Shared a few more small ideas.

package.json

README.md

packages/ensnode-common/src/ids.spec.ts

apps/ensnode/package.json

packages/ensnode-common/src/plugin-helpers.ts

packages/ensnode-common/package.json

packages/ensnode-subgraph-graphql/package.json

packages/ensnode-common/package.json

tk-o · 2025-01-23T19:08:37Z

@shrugs, do you think, for this PR, that we should also work towards:

easy exporting for ponder schema to support the upcoming @ponder/client feature

?

I wanted to export ponder.schema.ts from one package and import it in another, but it gives me some drizzle type resolution errors, for example:

DTS Build start
../../apps/ensnode/ponder.schema.ts(4,14): error TS2742: The inferred type of 'domain' cannot be named without a reference to '.pnpm/[email protected]_@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/drizzle-orm/pg-core'. This is likely not portable. A type annotation is necessary.

I've found an explanation to that drizzle problem, but didn't yet figure how to solve it. Not sure how to progress about that.

What do you think?

shrugs

great work on this with the catalog definitions and the readmes and whatnot. some minor stuff i've noted from afar, but i'll check this out and start running it now

packages/ensnode-utils/tsconfig.json

apps/ensnode/README.md

shrugs · 2025-01-23T20:38:45Z

packages/ensnode-utils/src/helpers.ts

@@ -0,0 +1,5 @@
+export const uniq = <T>(arr: T[]): T[] => [...new Set(arr)];


this file can/should live in the indexer, it doesn't need to be shared. and the file name helpers really only makes sense in that context.

shrugs · 2025-01-23T20:40:06Z

packages/ensnode-utils/src/ids.ts

 // produces `blocknumber-logIndex` or `blocknumber-logindex-transferindex`
-export const makeEventId = (event: Event, transferIndex?: number) =>
-  [event.block.number.toString(), event.log.logIndex.toString(), transferIndex?.toString()]
+export const makeEventId = (blockNumber: bigint, logIndex: number, transferIndex?: number) =>


i would prefer this file live in the indexer as well — i'm not sure we'll need to generate any of these ids outside of that context, and these ids are indexer-specific. there was a mention of maybe resolverId being used outside, but let's move all of these back to indexer for now, and only extract that one iff we actually need to do so.

packages/ensnode-subgraph-graphql/package.json

pnpm-workspace.yaml

packages/ensnode-utils/tsconfig.json

lightwalker-eth

@tk-o Nice updates 👍 Shared a few more ideas with feedback 😄

packages/ensnode-utils/tsconfig.json

packages/ensnode-utils/README.md

packages/ensnode-subgraph-graphql/package.json

README.md

apps/ensnode/README.md

shrugs

may not need to include vite resolution in root package.json?

package.json

shrugs · 2025-01-23T21:37:17Z

just noting for myself: i'd really like for env variable management to be exclusively at the root — can likely configure this with dotenv executing the script before running ponder if necessary

shrugs · 2025-01-23T21:38:09Z

also note to self: i worry this is too package-happy but the plugin framework can be extracted into its own package, initially private but publishable in the future

tk-o · 2025-01-24T07:41:37Z

Thanks you so much for your support here, @shrugs 🚀

…/monorepo

lightwalker-eth

@tk-o Hey super updates! 😄 Shared a few more small ideas with feedback. I think we're very close now!

package.json

README.md

lightwalker-eth · 2025-01-24T09:47:38Z

apps/ensnode/README.md

  - (possible) continued backwards compatibility with subgraph
  - support indexing subset of data, i.e. only domains under parent node

-## next up


@tk-o Hey if I remember correctly I saw an earlier request from Matt to remove the ideas in this "next up" section and transfer them into GitHub issues. I think it's best we action that outside of this PR and have Matt take the lead on that transition of ideas, since he's the original author of these and he'll understand them best.

Let's revert this change so that we retain these ideas as they are for now.

Sorry for the back and forth on this.

Reverted. FYI @shrugs

apps/ensnode/package.json

packages/ponder-schema/package.json

packages/shared-configs/package.json

lightwalker-eth

@tk-o Great updates! Just one more set of open questions based on your proposed CI updates.

.github/workflows/ci.yml

lightwalker-eth

@tk-o Super updates. Replied to your question and small comments 👍

.github/workflows/ci.yml

lightwalker-eth

@tk-o Awesome! PR now looks good to me 🚀 😄

@shrugs Passing PR review over to you 🙌

shrugs · 2025-01-24T17:16:26Z

if ensnode-utils is going to be internal, let's add private: true to the package.json and we can remove the extra stuff like the repo and homepage sections.

do we need the typecheck script in any of these packages?

we should standardize on whether we're going to org-prefix package names or not. i've done so for shared-configs (it's package name is @namehash/shared-configs, as an example. the benefit is that for internal packages where the name might be relatively generic ('shared-configs') when we import it we import it like @namehash/shared-configs which communicates that it's a namehash-specific module. if it's just named shared-configs a reader might assume that it's some npm package called shared-configs. up to you if we want to do this or not for the other packages, or whether we want to remove the prefix for shared-configs and go un-prefixed only.

shrugs

ready to merge, up to you on package name stuff, or whether to tackle separately. well done!

shrugs · 2025-01-24T17:17:54Z

.github/workflows/ci.yml

+          RPC_URL_59144: https://linea.drpc.org
+          RUNTIME_CHECK_TIMEOUT_SECONDS: 10
+        run: |
+          pnpm dev -vv &


start command here, yeah?

lightwalker-eth · 2025-01-24T17:50:58Z

@shrugs Agreed on your feedback about org prefixing all our package names here 👍

Let's keep all our packages private for now, including ensnode-utils, but I like the idea of keeping fields there that will be relevant once we change that to public and setup everything for automatically publishing to NPM, etc.. But let's handle that in some future separate PR. This one is already big enough!

shrugs · 2025-01-24T18:04:56Z

if ensnode-utils is intended to be public in the future, that's reasonable and i agree - i had understood it was meant to stay private forever

tk-o · 2025-01-24T18:31:49Z

I'm going to update the Custom Start Command on Railway for the indexing service.

From:

pnpm start --schema $RAILWAY_DEPLOYMENT_ID

to:

pnpm --filter ensnode start --schema $RAILWAY_DEPLOYMENT_ID

lightwalker-eth · 2025-01-24T18:34:59Z

@tk-o Sounds good! Thanks!

tk-o added 9 commits January 22, 2025 21:21

refactor(api): abstract ponder schema

f45a792

Use abstraction to detach the subgraphGraphQL factory function from ponder

refactor(api): abstract ponder config

d347040

Use abstraction to detach the helpers.ts functions from ponder

refactor(api): abstract ponder events

18b6c8b

Use abstraction to detach the ids.ts functions from ponder

refactor: create apps/ensnode dir

a79f9e0

Move the indexer source code into its dedicated place in the monorepo workspace

refactor: extract helpers to ensnode-core package

686ffff

fix: pnpm definition

6085dbe

refactor: use single ci workflow with multiple jobs

d9d1ba9

refactor: extract ens subgraph middleware to ensnode-subgraph-graphql…

55a3ce9

… package

refactor: link from ensnode app to ensnode-common package

2d2beea

tk-o changed the title ~~Feat/monorepo~~ feat(monorepo): split indexer into apps and packages Jan 23, 2025

lightwalker-eth reviewed Jan 23, 2025

View reviewed changes

apps/ensnode/README.md Show resolved Hide resolved

package.json Outdated Show resolved Hide resolved

package.json Show resolved Hide resolved

tk-o added 3 commits January 23, 2025 12:59

fix: apply PR feedback

890a961

feat(monorepo): use dependency catalog

5c223f0

Simplify common packages version management

fix: use hono as peer dep

5d8e794

lightwalker-eth reviewed Jan 23, 2025

View reviewed changes

lightwalker-eth mentioned this pull request Jan 23, 2025

v1 Label Healing Strategy #55

Closed

tk-o added 5 commits January 23, 2025 15:23

fix: apply PR feedback

586edbb

refactor: move plugin-helpers into ensnode app

305a089

refactor: rename ensnode-common to ensnode-utils

8cd50c2

docs: update metadata on all modules

c7019b8

docs: add LICENSE files

d75ac5f

tk-o marked this pull request as ready for review January 23, 2025 18:54

tk-o requested a review from a team as a code owner January 23, 2025 18:54

Merge remote-tracking branch 'origin/main' into feat/monorepo

8095d83

shrugs requested changes Jan 23, 2025

View reviewed changes

lightwalker-eth requested changes Jan 23, 2025

View reviewed changes

shrugs reviewed Jan 23, 2025

View reviewed changes

package.json Show resolved Hide resolved

tk-o added 2 commits January 24, 2025 08:29

fix: apply pr feedback

51c6b10

fix: apply pr feedback

71b443c

Merge branch 'feat/monorepo' of github.com:namehash/ensnode into feat…

35f0b58

…/monorepo

lightwalker-eth requested changes Jan 24, 2025

View reviewed changes

tk-o added 3 commits January 24, 2025 12:14

fix: apply pr feedback

ac6c2bb

fix(ci): ponder runtime integrity check

1ab968f

fix: apply pr feedback

4b2355f

lightwalker-eth requested changes Jan 24, 2025

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

fix(ci): ponder runtime integrity check

915f706

tk-o force-pushed the feat/monorepo branch from 56962f8 to 915f706 Compare January 24, 2025 12:16

lightwalker-eth requested changes Jan 24, 2025

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

.github/workflows/ci.yml Outdated Show resolved Hide resolved

.github/workflows/ci.yml Outdated Show resolved Hide resolved

tk-o added 2 commits January 24, 2025 14:21

feat(ci): use working directory to run ponder runtime checks

cd7eb18

feat(ci): update runtime checks exit script

4dfb7c1

tk-o force-pushed the feat/monorepo branch from ae00d6a to 4dfb7c1 Compare January 24, 2025 13:31

tk-o added 4 commits January 24, 2025 14:32

fix(ci): extend runtime checks timeout

819bcd3

fix(ci): include debug logs

7d591b0

fix(ci): contract runtime checks timeout

d54da92

fix: apply pr feedback

39ee809

lightwalker-eth approved these changes Jan 24, 2025

View reviewed changes

shrugs approved these changes Jan 24, 2025

View reviewed changes

tk-o merged commit 0ea1fd2 into main Jan 24, 2025
2 checks passed

tk-o deleted the feat/monorepo branch January 24, 2025 18:32

This was referenced Jan 24, 2025

Fix Static Analysis in CI #59

Closed

Move www.ensnode.io to monorepo #52

Closed

		@@ -0,0 +1,5 @@
		export const uniq = <T>(arr: T[]): T[] => [...new Set(arr)];

feat(monorepo): split indexer into apps and packages #68

feat(monorepo): split indexer into apps and packages #68

Uh oh!

Conversation

tk-o commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goals

Uh oh!

lightwalker-eth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lightwalker-eth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tk-o commented Jan 23, 2025

Uh oh!

shrugs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shrugs Jan 23, 2025

Choose a reason for hiding this comment

Uh oh!

shrugs Jan 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lightwalker-eth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shrugs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shrugs commented Jan 23, 2025

Uh oh!

shrugs commented Jan 23, 2025

Uh oh!

tk-o commented Jan 24, 2025

Uh oh!

lightwalker-eth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lightwalker-eth Jan 24, 2025

Choose a reason for hiding this comment

Uh oh!

tk-o commented Jan 22, 2025 •

edited

Loading