Migrate Python bindings from pybind11 to nanobind v2.9.2

[snnn]

This pull request migrates the Python bindings for onnxruntime-genai from Pybind11 to Nanobind to leverage
Nanobind's performance and feature improvements.

Key Changes:

• Build System (CMake):

  • The dependency in cmake/deps.txt has been switched from Pybind11 to Nanobind v2.9.2.
  • cmake/external/onnxruntime_external_deps.cmake is updated to fetch Nanobind and its robin-map
    dependency using FetchContent.
  • In src/python/CMakeLists.txt, the module is now built using nanobind_add_module instead of
    pybind11_add_module.

• Python Bindings (src/python/python.cpp and related files):

  • All Pybind11 headers and API calls have been replaced with their Nanobind equivalents.
  • The main module is now initialized with NB_MODULE instead of PYBIND11_MODULE.
  • Class and function bindings have been updated to the Nanobind syntax.
  • A cleanup function has been registered with Python's atexit to ensure proper garbage collection before
    C++ static destructors are called, preventing false memory leak reports.

Benefits of this migration:

• Faster Compile Times: Nanobind's header-only and lightweight design significantly reduces compilation time.
• Smaller Binaries: The resulting Python extension module is smaller.
• Lower Runtime Overhead: Nanobind has lower overhead for function calls between Python and C++.
• Modern C++ and Python Features: Provides better support for modern C++ features and improved Python
  integration.

This migration modernizes the onnxruntime-genai Python bindings, making them more efficient, robust, and
easier to maintain.

[snnn]

Understanding the set_log_callback(None) Issue

The Problem

The Python tests call og.set_log_callback(None) to clear the log callback, but nanobind doesn't accept None the same way pybind11 did.

Error:

TypeError: set_log_callback(): incompatible function arguments. The following argument types are supported:
    1. set_log_callback(callback: collections.abc.Callable) -> None

Invoked with types: NoneType

Why This Happens

pybind11 behavior (old):

• Automatically converts Python None to C++ nullptr for many types
• Could implicitly handle callback=None → nullptr

nanobind behavior (new):

• Stricter type system for better performance
• None (Python's NoneType) ≠ Callable type
• Requires explicit handling of optional parameters

What We Tried

1. Two overloads - def(callback) + def(nullptr_t)

   • ❌ Failed: nanobind doesn't match None to nullptr_t

2. .none() annotation - nb::arg("callback").none()

   • ❌ Failed: Only works for wrapper types (handle, object), not Callable

3. nb::object parameter - Accept any type, check inside

   • ❌ Failed: Still doesn't match NoneType to object

4. nb::handle parameter - Most generic type

   • ❌ Failed: Same matching issue

Recommended Solution

Option 1: Add separate clear method (cleanest)

// In bindings:
m.def("set_log_callback", &SetLogCallback, "callback"_a);
m.def("clear_log_callback", []() { 
    CheckResult(OgaSetLogCallback(nullptr)); 
});

# In tests:
og.clear_log_callback()  # Instead of og.set_log_callback(None)

Option 2: Python wrapper (backward compatible)

# In __init__.py
_set_log_callback_native = set_log_callback

def set_log_callback(callback=None):
    if callback is None:
        clear_log_callback()
    else:
        _set_log_callback_native(callback)

Option 3: Modify tests (simple)

# Instead of:
og.set_log_callback(None)

# Use:
og.set_log_callback(lambda x: None)  # No-op callback

Impact

• Affected tests: 2 (test_log_callback, test_log_filename)
• Severity: Low - edge case functionality
• Current state: These 2 tests will fail until we implement one of the solutions above

My Recommendation

Implement Option 1 (add clear_log_callback()) as it's:

• ✅ Explicit and clear (Pythonic: "explicit is better than implicit")
• ✅ Doesn't fight nanobind's type system
• ✅ Actually more intuitive API than set_log_callback(None)

---

This is a known limitation of nanobind's stricter type handling compared to pybind11, not a bug in our implementation.

To resolve the unused import issue, the import sys statement at line 12 should be removed from test/python/test_onnxruntime_genai_api_coverage.py. This change will reduce unnecessary dependencies and make the code cleaner and easier to read. No other changes are required as there is no evidence of sys being used within the code region shown.

To fix the unused import, we should simply remove the line import tempfile (line 13) from test/python/test_onnxruntime_genai_api_coverage.py. This can be done safely given that no code in the visible region uses the tempfile module. We do not need to introduce new imports, methods, or variable definitions.

[chemwolf6922]

Hi @snnn, While you are at this, could you please also add the nanobind type stub with nanobind_add_stub? Nanobind should produce better stub than what I'm trying to do in #1817