Add initial architeture documentation #10077

Viicos · 2024-08-07T16:56:29Z

Change Summary

Related issue number

Checklist

The pull request title is a good summary of the changes - it will be used in the changelog
Unit tests for the changes exist
Tests pass on CI
Documentation reflects the changes where applicable
My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

cloudflare-workers-and-pages · 2024-08-07T16:59:46Z

Deploying pydantic-docs with Cloudflare Pages

Latest commit:	`b4e61ee`
Status:	✅ Deploy successful!
Preview URL:	https://ee7a3294.pydantic-docs.pages.dev
Branch Preview URL:	https://architecture-docs.pydantic-docs.pages.dev

View logs

codspeed-hq · 2024-08-07T17:03:23Z

CodSpeed Performance Report

Merging #10077 will not alter performance

_{Comparing architecture-docs (b4e61ee) with main (4253e14)}

Summary

✅ 17 untouched benchmarks

Viicos · 2024-08-08T14:45:28Z

docs/architecture.md

+The JSON Schema generation of a model is handled by the [`GenerateJsonSchema`][pydantic.json_schema.GenerateJsonSchema] class.
+The [`generate`][pydantic.json_schema.GenerateJsonSchema.generate] method is the main entry point and is given the core schema of that model.
+
+### Customizing the core schema and JSON schema


This section goes a bit too much into some details imo. The goal was to document that __get_pydantic_core_schema__ is a general pattern used throughout the code base, but it doesn't necessarily apply to the Annotated pattern.

Viicos · 2024-08-08T14:46:43Z

docs/architecture.md

+the `source` will be the type being annotated. When [defining a custom type](concepts/types.md#as-a-method-on-a-custom-type),
+the `source` will be the actual class where `__get_pydantic_core_schema__` is defined.
+
+## Model validation and serialization


This section should definitely be expanded to document how this validation and serialization is performed (and not only state that it happens in pydantic-core). But I'll need to dive deeper in the Rust code base first.

Viicos · 2024-08-09T10:16:54Z

Having a schema could be nice, probably once the section about pydantic-core is more complete

github-actions · 2024-08-09T10:20:28Z

Coverage report

This PR does not seem to contain any modification to coverable code.

sydney-runkle

@Viicos,

Wonderful start here. I think this will be a super helpful tool for contributors looking to get some more context about internal patterns, as well as advanced users looking to customize schemas.

I think there's a bit of restructuring to do to make this a bit more intuitive, and I've left some line by line comments. In general, I think that you have a good pattern going regarding the mixing of high level concept explanation mixed with examples with context.

sydney-runkle · 2024-08-10T02:44:27Z

docs/architecture.md

@@ -0,0 +1,185 @@
+Starting with Pydantic V2, part of the codebase is written in Rust in a separate package called `pydantic-core`.
+This architecture documentation will partly focus on how the two `pydantic` and `pydantic-core` packages interacts together.


Maybe something that touches on a few of the other things covered, like:

This architecture documentation will first cover how pydantic and pydantic-core interact, then will delve into architecture specifics for various patterns, such as model definition, validation, and serialization.

sydney-runkle · 2024-08-10T02:44:54Z

docs/architecture.md

@@ -0,0 +1,185 @@
+Starting with Pydantic V2, part of the codebase is written in Rust in a separate package called `pydantic-core`.


Maybe a bit more of a "why" here - primarily, performance of course.

docs/architecture.md

sydney-runkle · 2024-08-10T02:47:48Z

docs/architecture.md

+Once the model class is successfully built, a core schema for the model will be constructed
+and set as the [`__pydantic_core_schema__`][pydantic.main.BaseModel.__pydantic_core_schema__]
+attribute (this is done by the `GenerateSchema` class). A core schema is a structured Python
+dictionary (represented using [`TypedDict`][typing.TypedDict] definitions) representing a
+specific validation and serialization logic. It is the core data structure used to communicate
+between the `pydantic` and `pydantic-core` packages. Every core schema has a required `type` key,
+and extra properties depending on this `type`.


This is a great start here. Maybe we could title this subsection "pydantic-core schema or something similar to give a bit more context.

I restructured this section to first talk about what exactly is a core schema, and then give the example of our model class core schema and the corresponding __pydantic_core_schema__ (which is specific to models, not necessarily core schemas).

And also edited the title of the section.

sydney-runkle · 2024-08-10T02:48:55Z

docs/architecture.md

+!!! note
+    It is not possible to define a custom core schema. A core schema needs to be understood by the
+    `pydantic-core` package, and as such we only support a fixed number of core schema types.
+    This is also part of the reason why the `GenerateSchema` is private and not documented.


Unfortunately (imo) we do sort of document it, though we've sort of shied away from further documentation bc we want to reserve some rights to modify how we generate core schema. So we might want to reword here.

sydney-runkle · 2024-08-10T02:51:59Z

docs/architecture.md

+You may have noticed that the previous serialization core schema has a `return_schema` key.
+This is because the core schema is also used to generate the corresponding JSON Schema.


Gah, an interesting pattern. Not my favorite, but definitely helpful (and efficient in a lot of contexts) 😅

Well in this case from what I saw the return_schema is also used by pydantic-core but I'm not sure in which context. But I felt having this return_schema here made a nice transition to JSON Schema generation. After all, GenerateJsonSchema.generate is the main entry point and is given the core schema so it makes sense.

docs/architecture.md

sydney-runkle

Initial review of your changes looks good. I'm going to do another pass tomorrow morning with a fresh mind.

sydney-runkle · 2024-08-13T00:59:19Z

docs/architecture.md

@@ -1,5 +1,10 @@
 Starting with Pydantic V2, part of the codebase is written in Rust in a separate package called `pydantic-core`.
-This architecture documentation will partly focus on how the two `pydantic` and `pydantic-core` packages interacts together.
+This was done partly in order to improve validation and serialization performance (with the cost of limited


Hmm not sure if we want to advertise this as a cost which has a pretty negative connotation, but I get your point.

sydney-runkle · 2024-08-13T00:59:31Z

docs/architecture.md

+This architecture documentation will first cover how the two `pydantic` and `pydantic-core` packages interact
+together, then will go through the architecture specifics for various patterns (model definition, validation,
+serialization, JSON Schema).


Nice 🚀 !

sydney-runkle · 2024-08-13T01:00:45Z

docs/architecture.md

+```
+
+Note that this is also a core schema definition, just that it is only relevant for `pydantic-core` during serialization.
+Core schemas cover a broad scope, and are used whenever we want to communicate between the Python and Rust side.


sydney-runkle

Oops, meant to just comment (not approve, yet)

sydney-runkle · 2024-08-15T00:42:59Z

Other than linting issues, I'm happy to merge this. We've marked the page as a draft, and I'd rather go ahead with a first draft to make things easier for contributors 🚀.

Great work @Viicos. I'll take a pass over this before we release v2.9 to further develop the docs.

github-actions bot added the documentation label Aug 7, 2024

Initial stab at architeture docs

639b4ca

Viicos force-pushed the architecture-docs branch from 174f6d6 to 6e9048a Compare August 8, 2024 14:44

Viicos commented Aug 8, 2024

View reviewed changes

Viicos force-pushed the architecture-docs branch from 6e9048a to a0e58cd Compare August 8, 2024 15:01

Small enhancements

d96c8ec

Viicos force-pushed the architecture-docs branch from a0e58cd to d96c8ec Compare August 8, 2024 15:05

Viicos marked this pull request as ready for review August 8, 2024 15:05

Viicos changed the title ~~Initial stab at architeture docs~~ Add initial architeture documentation Aug 8, 2024

Weird lint error happening only now

f039a18

sydney-runkle reviewed Aug 10, 2024

View reviewed changes

PR feedback

251fd45

sydney-runkle mentioned this pull request Aug 12, 2024

pydantic weekly project plan (Sydney Runkle) - August 12th, 2024 #10109

Closed

25 tasks

sydney-runkle approved these changes Aug 13, 2024

View reviewed changes

sydney-runkle reviewed Aug 13, 2024

View reviewed changes

link, add wip

b4e61ee

Viicos enabled auto-merge (squash) August 15, 2024 14:13

Viicos merged commit f5d6acf into main Aug 15, 2024

Viicos deleted the architecture-docs branch August 15, 2024 14:35

		@@ -0,0 +1,185 @@
		Starting with Pydantic V2, part of the codebase is written in Rust in a separate package called `pydantic-core`.
		This architecture documentation will partly focus on how the two `pydantic` and `pydantic-core` packages interacts together.

		You may have noticed that the previous serialization core schema has a `return_schema` key.
		This is because the core schema is also used to generate the corresponding JSON Schema.

Uh oh!

Add initial architeture documentation #10077

Add initial architeture documentation #10077

Uh oh!

Conversation

Viicos commented Aug 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

Related issue number

Checklist

Uh oh!

cloudflare-workers-and-pages bot commented Aug 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying pydantic-docs with Cloudflare Pages

Uh oh!

codspeed-hq bot commented Aug 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #10077 will not alter performance

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Viicos commented Aug 9, 2024

Uh oh!

github-actions bot commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

sydney-runkle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sydney-runkle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sydney-runkle left a comment

Choose a reason for hiding this comment

Uh oh!

sydney-runkle commented Aug 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Viicos commented Aug 7, 2024 •

edited

Loading

cloudflare-workers-and-pages bot commented Aug 7, 2024 •

edited

Loading

codspeed-hq bot commented Aug 7, 2024 •

edited

Loading

github-actions bot commented Aug 9, 2024 •

edited

Loading