Guided tour to the CPython source code

[gvanrossum]

I propose that we collaborate on creating a guided tour to the CPython source code. I think such a thing is vastly overdue, and I am not aware of existing resources (though I'll gladly take pointers that prove me wrong).

I've more or less promised @emilymorehouse to help her get started with something like this, and I'm happy to put time in. @willingc I hope you think this would be a useful document to maintain in this repo? I suppose it would go under the cpython tree.

The proposed document differs from the existing guides like the devguide -- it doesn't tell you anything about the patch submission flow, nor even how to get things to build -- it should focus on how to read the CPython source code (using your editor of choice).

Some topics could include:

• structure of the source code (what goes into Include, Objects, Python etc.)
• bytecode (could start with @akaptur's videos)
• reference counting
• how to find the definition of something
• how to find where a given error comes from

[brettcannon]

Some of this is lightly touched in the devguide (e.g. http://cpython-devguide.readthedocs.io/en/latest/setup.html#directory-structure), but in no way thoroughly.

[gvanrossum]

Ah, thanks. I would like to go a step deeper for the directories containing most of the fundamental C code:

• Include
• Objects
• Python
• Modules
• Parser

Other topics should probably include

• how the parser works
• how to define an object type (or is this in the C/API docs already?

[lorenanicole]

@gvanrossum are you soliciting assistance on this from others that aren't core devs too?

[brettcannon]

For documentation about defining a type, it depends. There's the C API tutorial and then there's xxsubtype.c.

@lorenanicole Guido can say if I'm wrong, but there's absolutely no reason to restrict this to core devs. Honestly, it would be best to have someone who isn't a core dev involved so no bad assumptions of pre-existing knowledge muddles the tour.

[gvanrossum]

@lorenanicole: What Brett said. The core devs already know this and probably have huge blind spots. My own blind spots are even bigger. (I wrote some of the stuff Brett pointed to and I had forgotten about it. :-)

I also believe that a lot of the existing docs in this area assume lots of other skills that aren't so easy to come by. While I don't think we should include a C tutorial, there are a lot of C patterns that aren't unique to Python but are still worth explaining in some detail, either because Python's version has certain important details, or just because they aren't that widely known.

Some more random topics:

• the GIL (Larry's Gilectomy notwithstanding)
• open hashing
• low-level memory allocation
• memory locality

[brettcannon]

• How to handle errors (e.g. when to use a goto, making sure you deref appropriately)

[willingc]

@gvanrossum Love this idea! It's similar to what I want to do with notebooks and Phillip Guo's grad school course. It also captures well the spirit of our PyCon conversation.

All,

The next paragraph that I'm writing is done with gentle kindness, respect, and thoughtfulness:

I do want the contributors to the documentation developed here to primarily or at least 50% written by those that are not core developers. A few reasons:

• 1. to reinforce truth that technical talent already exists in the PyLadies community;
• 2. to best communicate technical information in a manner that resonates with the PyLadies interested in being mentored (of course, all information will be openly available on GitHub so no one is being excluded.
• 3. to provide an opportunity for PyLadies to own some of the maintainer responsibility for the document (i.e. merge privileges)
• 4. to simplify the contribution process, review, and merging of changes.

Just open a PR with the folder name and outline that we would like to iterate on over time.

Also, please let me know if you would like to be added as a maintainer on the repo. I'll be on vacation with limited internet access beginning Friday to the 2nd week of August.

[jackiekazil]

IMHO, I think that it would be great to frame out by core devs and then have non-core devs tear it apart, ask questions, restructure and edit. However, I do not do work like this every day. But @Bradamant3 does! :-)

@Bradamant3, do you have thoughts on ways to approach this?

Background on @Bradamant3
Recent talks she has given...

• Tightly coupling your (REST) API docs
• What can the open source software of today learn from the history of software documentation?

[Bradamant3]

@jackiekazil and all -- I am honored to be mentioned here, and I'd be thrilled to be involved. I'm anything but a core dev -- I'm still pretty new to Python, still working my way through Learn Python the Hard Way, although I've taken a couple of handfuls of programming classes (no C, sadly) and write sample code for part of my living. (I'm a technical writer by trade, on the more technical side of things.) I'll need to spend some time over the next week looking through what's already in the repo and the other resources y'all have mentioned to see where I think I can best help out. I have much to learn, and if I can help others learn by documenting my own learning as I help out with this project, that would be the biggest thrill of all.

[willingc]

Hi @Bradamant3, We would love for you to be part of this. There's a special place in my heart for technical writing since it's key to breaking down barriers for users to use your software and critical for onboarding developers 🌻

Please feel free to ask questions (large or small). Looking forward to you helping out.

[akaptur]

My bytecode chapter for the Architecture of Open Source Applications book might be a better example than my bytecode talk (although the content substantially overlaps): http://aosabook.org/en/500L/a-python-interpreter-written-in-python.html

One of the hardest parts of writing the chapter was getting clarity about the intended audience and the intended goal. I eventually settled on an audience of Python programmers who want to learn more about the language for its own sake. (I personally don't think knowing about the guts of the interpreter helps you write better Python, but I do think it helps you deepen your knowledge of computer science and is darn fun.)

Given that, for this project, who's the intended audience, and what's the goal? What is the reader hoping to get out of the guides?

Some possible goals I can imagine:

• Onboarding people who want to be core developers
• Providing shortcuts, pointers, and hints for people who want to contribute to CPython but haven't worked with a large codebase before
• Satisfying the curiosity of people who want to know how stuff works (I heard someone describe this recently as "Be the Julia Evans you want to see in the world")
• Teaching people fundamental programming & debugging skills (e.g. Guido's last two bullets above)

We don't need perfect agreement on the goals, but I think some clarity early on will pay off significantly later on!

With more clarity on the goals, I'd be happy to contribute to this. I think the object model and garbage collection would be particularly rich topics.

[Bradamant3]

@akaptur and all:
Just some preliminary thoughts as I look superficially through SO MANY AMAZING RESOURCES and read Allison's brilliant questions about audience:
It seems to me that Allison's goals are fundamentally all compatible. Folks who want to be core developers may not have worked with a large codebase, they may have found the Python community to be as amazingly welcome as I have and therefore find it the right place to satisfy their curiosity -- and addressing all the above can also help teach those crucial fundamental programming and debugging skills. (These are all personal goals of mine, and I have zero aspirations toward becoming a professional developers. Just sayin'. There's a huge contribution to make to the world by making it easier for as many people as possible to understand these things.)
+10K to object model and garbage collection (as someone who has far more to learn about both than she currently understands)
also +10K to @jackiekazil's suggestion about framing out by core devs and then letting the rest of us adjacently-obsessed have at it.

[lorenanicole]

+10k as well to all the above :-) once a more definitive outline is settled I would love to step in and help. @jackiekazil + @willingc I'll defer to you two for directions.

[willingc]

Nice to see @akaptur here. Hope to see you at PyBay.

One of the things that I have wanted to do for over a year is to edit down @pgbovine's great 10 hour course http://www.pgbovine.net/cpython-internals.htm that walked through the Python2 codebase. I wanted to create a series of Jupyter notebooks that would correspond to each of his videos and link to source code. Here's a link to one of the notebooks that I started last year: https://github.com/willingc/pyladies-cpython/blob/master/Notes%20on%20Lecture%201.ipynb

I'm wondering (if @pgbovine is cool with us referring to the content and @gvanrossum and @brettcannon see value) if we could use the @pgbovine course as a starting point and outline linking out to other resources and docs as appropriate. Perhaps creating a study group to do two lectures a month or something.

This is everyone's group so no need to defer to me. If you have a good idea, please run with it. Personally, I want to see something that digs into the guts and I see some value in the video lecture combined with written word.

Overall, I'm better on code guts than usability so @Bradamant3's talent and insights are very valued. Thoughts?

P.S. @akaptur I think your bullet list covers the audience.

[willingc]

Also leaving for Japan for a long overdue vacation. @jackiekazil, would you mind working with @estherbester and @audreyr to get yourself and others push access to this repo? I tried to add folks last night but needed org access not just repo admin access.