A Fast, spec compliant Python 3.13+ tokenizer that runs on older Pythons.

pip install pytokens

python -m pytokens path/to/file.py

• Create and activate a virtual environment
• Run pip install -r requirements-dev.txt to do an editable install
• Run pytest to run tests

Run mypy .

Make sure to bump the version in setup.cfg.

Then run the following commands:

rm -rf dist
python -m build

Then upload it to PyPI using twine:

twine upload dist/*