Added vizdoom python interface file to aid IDE for hinting #620

Trenza1ore · 2025-07-24T11:38:14Z

Problem to solve

Currently ViZDoom (python binding) lacks any form of hinting with modern IDEs like VS Code due to the actual code being in a compiled binary (vizdoom, vizdoom.exe, etc.)

Proposed solution

A vizdoom.pyi file written according to official documentation at:

Possible side effects

Not to my knowledge aside from being easier to work with :-)

Changed files

[Edit] setup.py: Added vizdoom.pyi and py.typed to package_data
[Edit] src/lib_python/CMakeLists.txt: Added copy command for vizdoom.pyi and py.typed
[New] src/lib_python/vizdoom.pyi: New python interface file
[New] src/lib_python/py.typed: New py.typed file for partial type checking
[Edit] examples/python/*: Correction for type checking
[Edit] tests/*: Correction for type checking

Before

After

…ck errors but that's for others to fix)

…rapper) pass pre-commit

mwydmuch · 2025-07-25T22:48:33Z

Hi @Trenza1ore, thank you very much for this contribution. This indeed seems to be helpful. Can I ask how you specifically generated the pyi file? It would be nice to have a nice way to update it in case of changes to docstrings/new methods.

Trenza1ore · 2025-07-26T01:59:38Z

Hi @Trenza1ore, thank you very much for this contribution. This indeed seems to be helpful. Can I ask how you specifically generated the pyi file? It would be nice to have a nice way to update it in case of changes to docstrings/new methods.

Hi @mwydmuch it should be me thanking you guys for maintaining this awesome repo and make RL with Doom so much easier. My under/postgrad thesis wouldn't have been nearly as cool without the amazing library of ViZDoom.

For the creation of the pyi file, here are the steps I've taken:

I decomposed the stub-writing job into several steps
In Cursor (an LLM-powered IDE), I aided the Claude-4-Sonnet LLM to create stubs for each section according to documentation for Python API
Still using Cursor to refine, I now feed it the C++ API documentation which covers both the C++ & Python datatypes (Python documentation does not contain types for class properties and these can only be found in C++ documentation, maybe we can update the documentation as well?)
I manually fixed the obvious errors LLM had made
I check through all python examples and test files to find issues with typing
I check documentation to find issues, try to inspect vizdoom.__dict__ to find anything missing, make sure mypy's stub test passes, currently still at this step

I am looking for a way to automate the workflow (especially manual checking part), will share any tool that aided me here (although for future incremental update that might be less useful).

Also I expect to need one more day to finish validating the vizdoom.pyi

…related, good to go!

Trenza1ore · 2025-07-26T05:31:43Z

I believe that's it, the vizdoom.pyi has finally passed stubtest, going from 900+ errors to only 14. 👍
The rest of the errors to my knowledge cannot be passed, since they are all complaining about metaclass being pybind11_builtins.pybind11_type:

error: vizdoom.vizdoom.DoomGame is inconsistent, metaclass differs
Stub: in file /Users/Sushi/Desktop/ViZDoom/.conda/lib/python3.11/site-packages/vizdoom/vizdoom.pyi:631
N/A
Runtime:
<class 'pybind11_builtins.pybind11_type'>

Here's my script for validating correctness via stubtest while filtering out different categories of errors to not get overwhelmed, not sure if it would be useful:

import re
import os
import platform
import subprocess

os.system("cls" if platform.system() == "Windows" else "clear")
os.system("sh ./update.sh")  # copies latest vizdoom.pyi to installed ViZDoom 1.2.4

filter_placeholder = re.compile(r"error: (vizdoom\..*) variable differs from runtime type (vizdoom\..*)"
                                r"\nStub:.*\ntypes.EllipsisType\nRuntime:\n.*[0-9]+>\n*")

filter_builtin_methods = re.compile(r"error: (vizdoom\..*__.*__) is not present in stub\nStub:."
                                    r"*\nMISSING\nRuntime:\n.*\n*")

filter_pybind11_metaclass = re.compile(r"error: (vizdoom\..*) is inconsistent, metaclass differs\nStub:."
                                       r"*\nN/A\nRuntime:\n.*\n*")

filter_missing_attribute = re.compile(r"error: (vizdoom\.(?:vizdoom\.)?(.*)) is not present in "
                                      r"stub\nStub:.*\nMISSING\nRuntime:.*\n(.*)\n*")

extract_function_info = re.compile(r"error: (vizdoom\.(?:vizdoom\.)?(DoomGame)\.(.*)) is not present at " 
                                   r"runtime\nStub:.*\ndef (\((.*?)\)( -> .*?\.(.*))?)\nRuntime:\nMISSING\n*")

def filter_all_error_messages(error_messages: str) -> str:
    # error_messages = filter_placeholder.sub("", error_messages)
    # error_messages = filter_builtin_methods.sub("", error_messages)
    error_messages = filter_pybind11_metaclass.sub("", error_messages)
    # error_messages = filter_function_false_negatives(error_messages)
    # error_messages = filter_missing_enum_val_false_negatives(error_messages)
    return error_messages.strip()


def filter_function_false_negatives(error_message: str) -> str:
    with open("src/lib_python/vizdoom.pyi", "r", encoding="utf-8") as f:
        stub_code = f.read().splitlines()
    for match in extract_function_info.finditer(error_message):
        class_name = match.group(2)
        function_name = match.group(3)
        function_signature = match.group(4).replace("vizdoom.", "").replace(class_name, "").strip().replace("self: ", "self").replace("builtins.", "")
        function_return = match.group(4)
        if any(function_signature in line for line in stub_code):
            error_message = error_message.replace(match.group(0), "")
    return error_message


def filter_missing_enum_val_false_negatives(error_message: str) -> str:
    with open("src/lib_python/vizdoom.pyi", "r", encoding="utf-8") as f:
        stub_code = f.read().splitlines()
    for match in extract_function_info.finditer(error_message):
        attribute = match.group(2)
        if any(attribute in line for line in stub_code):
            error_message = error_message.replace(match.group(0), "")
    return error_message


stub_test_result = subprocess.run(["stubtest", "vizdoom"], capture_output=True, text=True)
std_out = filter_all_error_messages(stub_test_result.stdout)

# Save output to file for later analysis
with open("stubtest_output.txt", "w") as f:
    f.write("=== STUBTEST OUTPUT ===\n")
    f.write(f"Return code: {stub_test_result.returncode}\n\n")
    f.write("=== STDOUT ===\n")
    f.write(std_out)
    f.write("\n=== STDERR ===\n")
    f.write(stub_test_result.stderr)

Trenza1ore · 2025-07-26T05:47:41Z

Also I noticed that there seems to be an undocumented SignalException, not sure what it's for.

…iZDoom Enums

Trenza1ore · 2025-07-26T13:23:02Z

Additionally add a enum_typing example to demonstrate expected duck-typing behaviour of ViZDoom Enums (which should be represented as subclass of Python's enum.IntEnum in stub files but is actually:

not a subclass of enum.IntEnum
inherits from some pybind11 internal class

Also in retrospect the workflow could have been much better and highly automated.
@mwydmuch Here's the automated workflow I've tested and should be working:

pip install mypy, I used 1.17.0
Use stubgen from mypy to generate the initial stub file (for incremental update this can be used as a reference or "doner file" to copy from)
Use stubtest from mypy to test whether the current generated stub is correct (inconsistent, metaclass differs errors can be safely ignored)
Add docstrings and additional comments if needed (for this part, can still use an LLM with browser tool or read from documentation programatically, etc.)
Format the vizdoom.pyi file, make it easier to read and locate things

mwydmuch · 2025-07-26T21:45:40Z

Thank you for describing the process! It seems that building mypy stubgen fails to recognize some signatures, like docstrings from the pybind11 extension. Adding them using LLM is not reliable and cumbersome to automate. There is basically this package https://github.com/sizmailov/pybind11-stubgen to generate stubs for pybind11 modules. By running pybind11-stubgen vizdoom.vizdoom I got very similar vizdoom.pyi file to the one you propose in this PR. Do you think you could use this instead?

mwydmuch · 2025-07-26T21:50:26Z

Also I noticed that there seems to be an undocumented SignalException, not sure what it's for.

This one is no longer used, and if something like this is still in the codebase, then it's a leftover from the old signal handling system that should be removed.

Trenza1ore · 2025-07-26T23:39:28Z

Thank you for describing the process! It seems that building mypy stubgen fails to recognize some signatures, like docstrings from the pybind11 extension. Adding them using LLM is not reliable and cumbersome to automate. There is basically this package https://github.com/sizmailov/pybind11-stubgen to generate stubs for pybind11 modules. By running pybind11-stubgen vizdoom.vizdoom I got very similar vizdoom.pyi file to the one you propose in this PR. Do you think you could use this instead?

I just realized that there are already scripts to generate python docstrings from C++ files. (Previously I used LLM to extract these from documentation htmls, which seems to be totally unnecessary after reading the code base more.) The type stub file doesn't need to be pretty, so I'll work on an automated generation solution that generates the stub file then insert relevant Python docstrings to make it useful for the user (if not already inserted by pybind11-stubgen which I haven't tried yet).

🙂 Will take a look at the pybind11-stubgen module and introduce stub generation as part of the building process.

Trenza1ore · 2025-07-26T23:43:30Z

Also I noticed that there seems to be an undocumented SignalException, not sure what it's for.

This one is no longer used, and if something like this is still in the codebase, then it's a leftover from the old signal handling system that should be removed.

Oh! Thanks for clarification. Found it to still be in vizdoom.__dict__ and was surprised (can still be raised normally).

Trenza1ore · 2025-07-27T18:33:45Z

I've successfully integrated the type stubs generation into the build process of ViZDoom! @mwydmuch

A custom target generate_stubs was added to src/lib_python/CMakeLists.txt (when option CREATE_PYTHON_STUBS is on) to run ${VIZDOOM_PYTHON_SRC_DIR}/generate_vizdoom_stubs.py which creates the stub file by:

using pybind11-stubgen for generation
removing the __all__ = ... line
adding import numpy as np after import typing
optionally, black and isort for simple formatting
adding a header comment
replacing the -> typing.Any properties with actual return behaviour (np.ndarray or typing.Optional[np.ndarray])

vizdoom.pyi and py.typed would be copied to the package directory but generate_vizdoom_stubs.py has finished its purpose so it'll only be in source distribution and not in installed package.

To test the build process, I have tested on an Apple Silicon Macbook Air via:

python -m build
pip install dist/vizdoom-1.3.0.dev2-cp311-cp311-macosx_15_0_arm64.whl

I have modified the documentation in docs/introduction/building.md to add instructions regarding the building of Python type stubs (vizdoom.pyi), additionally I added a few lines to the building on MacOS section to help fellow Apple Silicon MacOS users.

…option

Trenza1ore · 2025-07-27T19:32:18Z

Should be everything, got stub generation work as a standalone script as well

mwydmuch

Thank you for your changes, I like most of them. However, I think I would prefer that generated_vizdoom_stubs.py be more of a utility tool/script (by default OFF in CMake) than a required part of the main building step.

Since stubs do not depend on the building process, IMO they should be committed to the repo, with the possibility to regenerate and to diff changes.

mwydmuch · 2025-07-27T21:38:00Z

src/lib_python/CMakeLists.txt

@@ -1,102 +1,114 @@
-set(BUILD_PYTHON_VERSION "" CACHE STRING "Version of Python to build bindings for")


Why is all of this in diff? Could you modify only the necessary parts?

This is quite strange, let me check whether it's line endings

Original file had \r\n (Windows) line endings, while my uploaded file does not, I have added the \r\n endings back

mwydmuch · 2025-07-27T21:43:29Z

docs/introduction/building.md

+To enable Python typing support, the creation of a Python Interface file `vizdoom.pyi` is now part of the build process.
+
+To ensure `vizdoom.pyi` is properly created, the following dependencies are required:
+* pybind11-stubgen


If pybind11-stubgen is needed as a part of the building process, then it should be added as build deps to pyproject.toml

Added it to pyproject.toml

mwydmuch · 2025-07-27T21:46:00Z

src/lib_python/generate_vizdoom_stubs.py

+- adding `import numpy as np` after `import typing`
+- optionally, `black` and `isort` for simple formatting
+- adding a header comment
+- replacing the `-> typing.Any` properties with actual return behaviour (`np.ndarray` or `typing.Optional[np.ndarray]`)


I think the correct type annotation is actually numpy.typing.NDArrady

Changed to NDArray

mwydmuch · 2025-07-27T22:01:16Z

examples/python/enum_typing.py

What is the purpose of having it as an example?

It was a leftover from when I used IntEnum as base-class for the Enum classes, I'll remove it

Trenza1ore · 2025-07-27T23:30:34Z

👍 I've made the requested changes:

pyproject.toml
- Added "pybind11-stubgen" into build requires
CMakeLists.txt
- Set option CREATE_PYTHON_STUBS to be OFF by default
docs/introduction/building.md
- Updated part about CREATE_PYTHON_STUBS
src/lib_python/CMakeLists.txt
- Added back the \r\n Windows line endings in src/lib_python/CMakeLists.txt to resolve the large diff.
- Changed behavior to only run generate_vizdoom_stubs.py if CREATE_PYTHON_STUBS specified to be ON while vizdoom.pyi and py.typed are always copied.
src/lib_python/vizdoom.pyi
- Add back generated vizdoom.pyi since it's no longer generated every build by default
src/lib_python/generate_vizdoom_stubs.py
- Updated to also be a useful utility script, python src/lib_python/generate_vizdoom_stubs.py -p can directly add typing to installed vizdoom package
- Removed all the Unicode special characters since non-ascii characters would introduce issues in some systems
scripts/add_typing_to_vizdoom.sh
- Demonstrates how to add typing to currently installed vizdoom library directly
examples/python/enum_typing.py
- Removed since it serves no purpose now that Enums are properly type stubbed

mwydmuch

Thank you for all the changes <3 LGTM!

mwydmuch · 2025-07-28T06:49:25Z

Ok, merged, thank you @Trenza1ore. Great PR with very helpful stuff!

Trenza1ore added 2 commits July 24, 2025 11:11

Added vizdoom python interface

c3bd0e0

Added to CMakeLists

a3bbf9e

Trenza1ore mentioned this pull request Jul 24, 2025

[Info] How to Add Python Hinting Support to ViZDoom #621

Open

Trenza1ore added 11 commits July 24, 2025 13:27

Update vizdoom.pyi to fix a few oversights

ad4ee72

Update vizdoom.pyi (mostly formatting)

78df416

Update vizdoom.pyi to add comment

540cbf3

Create py.typed

e67b908

Update CMakeLists.txt

47c4bb8

Update setup.py

e4ea837

Fixed vizdoom.pyi issues

2a5b8a2

Updated setup.py for pre-commit formatting

c544e5d

Updated outdated examples & tests (gymnasium still has a few type che…

af97222

…ck errors but that's for others to fix)

Forgot to update learning_pytorch.py

bf258f8

Formatted vizdoom.pyi better, make sure all files (except gymnasium_w…

fd2ffe8

…rapper) pass pre-commit

mwydmuch added the enhancement label Jul 25, 2025

Trenza1ore added 3 commits July 26, 2025 05:24

Made sure stub is as close to mypy requirements as possible

7099437

Format fix for gymnasium_vect_bench.py

6bd9610

Finally resolved all stubtest errors except those pybind11 metaclass …

936fc46

…related, good to go!

Additional format fix for pre-commit check

322f412

Added a enum_typing example to demonstrate the expected behavior of V…

0331263

…iZDoom Enums

Integrated type stub generation into build, updated docs

429ea1b

Trenza1ore added 2 commits July 27, 2025 19:26

Updated comments

cee1ca6

Updated docs

070c7ba

Trenza1ore added 2 commits July 27, 2025 19:56

Make CREATE_PYTHON_STUBS an option

001a163

Updated docs with building instructions for new DCREATE_PYTHON_STUBS …

6a422a5

…option

Make generate_vizdoom_stubs.py work outside of build process

ea88f27

mwydmuch reviewed Jul 27, 2025

View reviewed changes

Trenza1ore added 3 commits July 27, 2025 23:58

Made the requested changes

cf40878

Added \r\n (Windows line end to make the Github diff go away)

58b838c

Also write py.typed when patching

3a15bd6

Removed the UTF-8 emojis, should have known they won't work

019c881

mwydmuch approved these changes Jul 28, 2025

View reviewed changes

mwydmuch merged commit 8b59a68 into Farama-Foundation:master Jul 28, 2025
28 checks passed

		@@ -1,102 +1,114 @@
		set(BUILD_PYTHON_VERSION "" CACHE STRING "Version of Python to build bindings for")

Uh oh!

Added vizdoom python interface file to aid IDE for hinting #620

Added vizdoom python interface file to aid IDE for hinting #620

Uh oh!

Conversation

Trenza1ore commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem to solve

Proposed solution

Possible side effects

Changed files

Before

After

Uh oh!

mwydmuch commented Jul 25, 2025

Uh oh!

Trenza1ore commented Jul 26, 2025

Uh oh!

Trenza1ore commented Jul 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Trenza1ore commented Jul 26, 2025

Uh oh!

Trenza1ore commented Jul 26, 2025

Uh oh!

mwydmuch commented Jul 26, 2025

Uh oh!

mwydmuch commented Jul 26, 2025

Uh oh!

Trenza1ore commented Jul 26, 2025

Uh oh!

Trenza1ore commented Jul 26, 2025

Uh oh!

Trenza1ore commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Trenza1ore commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mwydmuch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Trenza1ore commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mwydmuch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mwydmuch commented Jul 28, 2025

Uh oh!

Uh oh!

Trenza1ore commented Jul 24, 2025 •

edited

Loading

Trenza1ore commented Jul 26, 2025 •

edited

Loading

Trenza1ore commented Jul 27, 2025 •

edited

Loading

Trenza1ore commented Jul 27, 2025 •

edited

Loading

Trenza1ore commented Jul 27, 2025 •

edited

Loading