Support Python 3.14

[glatterf42]

Hi there :)

This week, Python 3.14 was released. So we've been trying to check if we can support it already in the packages we maintain.
One of these attempts is iiasa/ixmp4#209, which is now running into an error with pandera/pandas. This error is also described in pandas-dev/pandas#62711 (though not originally by me and the original description doesn't mention Python 3.14). It's just that the traceback seems very similar, so it might well be related.

Either way, this is an exemplary traceback I'm seeing (full logs can be found here):

Traceback

__ ERROR at setup of TestCoreIamcReadOnly.test_mp_tabulate_big_async[sqlite] ___

request = <SubRequest 'platform_med' for <Function test_mp_tabulate_big_async[sqlite]>>

    def platform_with_td(
        request: pytest.FixtureRequest,
    ) -> Generator[Platform, Any, None]:
        type = request.param
        postgres_dsn = request.config.option.postgres_dsn
        bctx = get_backend_context(type, postgres_dsn)
    
        with bctx as backend:
            platform = Platform(_backend=backend)
>           td.load_dataset(platform)

tests/conftest.py:175: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/fixtures/__init__.py:353: in load_dataset
    cls.load_datapoints(platform)
tests/fixtures/__init__.py:346: in load_datapoints
    cls.load_dp_df(platform, cls.datapoints)
tests/fixtures/__init__.py:342: in load_dp_df
    cls.load_run_datapoints(platform, (model, scenario, version), dps)
tests/fixtures/__init__.py:319: in load_run_datapoints
    run.iamc.add(annual, type=ixmp4.DataPoint.Type.ANNUAL)
ixmp4/core/iamc/data.py:63: in add
    df = AddDataPointFrameSchema.validate(df)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.14/site-packages/pandera/api/pandas/model.py:191: in validate
    cls.to_schema().validate(
.venv/lib/python3.14/site-packages/pandera/api/pandas/container.py:118: in validate
    return self._validate(
.venv/lib/python3.14/site-packages/pandera/api/pandas/container.py:139: in _validate
    return self.get_backend(check_obj).validate(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/container.py:104: in validate
    error_handler = self.run_checks_and_handle_errors(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/container.py:192: in run_checks_and_handle_errors
    error_handler.collect_error(
.venv/lib/python3.14/site-packages/pandera/api/base/error_handler.py:54: in collect_error
    raise schema_error from original_exc
.venv/lib/python3.14/site-packages/pandera/backends/pandas/container.py:227: in run_schema_component_checks
    result = schema_component.validate(
.venv/lib/python3.14/site-packages/pandera/api/dataframe/components.py:149: in validate
    return self.get_backend(check_obj).validate(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/components.py:142: in validate
    validated_column = validate_column(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/components.py:102: in validate_column
    error_handler.collect_error(
.venv/lib/python3.14/site-packages/pandera/api/base/error_handler.py:54: in collect_error
    raise schema_error from original_exc
.venv/lib/python3.14/site-packages/pandera/backends/pandas/components.py:78: in validate_column
    validated_check_obj = super(ColumnBackend, self).validate(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/array.py:74: in validate
    error_handler = self.run_checks_and_handle_errors(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/array.py:138: in run_checks_and_handle_errors
    error_handler.collect_error(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pandera.api.base.error_handler.ErrorHandler object at 0x7f6d4db479a0>
error_type = <ValidationScope.DATA: 'data'>
reason_code = <SchemaErrorReason.CHECK_ERROR: 'check_error'>
schema_error = SchemaError('Error while executing check function: KeyError("<class \'pandas.core.series.Series\'>")\nTraceback (most ...nput_data_type]\n         ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\nKeyError: <class \'pandas.core.series.Series\'>\n')
original_exc = KeyError(<class 'pandas.core.series.Series'>)

    def collect_error(
        self,
        error_type: ErrorCategory,
        reason_code: Optional[SchemaErrorReason],
        schema_error: SchemaError,
        original_exc: Union[BaseException, None] = None,
    ):
        """Collect schema error, raising exception if lazy is False.
    
        :param error_type: type of error
        :param reason_code: string representing reason for error
        :param schema_error: ``SchemaError`` object.
        """
        if not self._lazy:
>           raise schema_error from original_exc
E           pandera.errors.SchemaError: Error while executing check function: KeyError("<class 'pandas.core.series.Series'>")
E           Traceback (most recent call last):
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/components.py", line 242, in run_checks
E               self.run_check(
E               ~~~~~~~~~~~~~~^
E                   check_obj, schema, check, check_index, *check_args
E                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E               )
E               ^
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/base.py", line 115, in run_check
E               check_result: CheckResult = check(check_obj, *args)
E                                           ~~~~~^^^^^^^^^^^^^^^^^^
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/api/checks.py", line 234, in __call__
E               return backend(check_obj, column)
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/checks.py", line 349, in __call__
E               check_output = self.apply(check_obj)
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/checks.py", line 148, in apply
E               return apply_fn(check_obj)
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/checks.py", line 156, in apply_field
E               return self.check_fn(check_obj)
E                      ~~~~~~~~~~~~~^^^^^^^^^^^
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/api/function_dispatch.py", line 24, in __call__
E               fn = self._function_registry[input_data_type]
E                    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
E           KeyError: <class 'pandas.core.series.Series'>

.venv/lib/python3.14/site-packages/pandera/api/base/error_handler.py:54: SchemaError

These tests work fine with Python 3.13 (in both cases, I'm using pandas 2.3.1 and pandera 0.25.0), but they fail with Python 3.14.

Looking at the code here, I'm assuming it could be

pandera/pandera/engines/engine.py

Line 149 in 48ba659

cls._registry[cls].dispatch.register(source_dtype, _method)

or

pandera/pandera/api/extensions.py

Line 81 in 48ba659

check_dispatcher.register(fn) # type: ignore

or

pandera/pandera/api/base/checks.py

Line 106 in 48ba659

dispatcher.register(fn)

where the registration fails with Python 3.14 (the first one seems most likely to me because it mentions a dtype), but I'm not sure. It's just that the dtype in the first case is coming from typing.get_type_hints(), which wasn't officially changed in Python 3.14, but I know that e.g. pydantic had to adjust how they handle annotations in their metaclasses because of changes in 3.14. They now use the recommended new annotationlib for that, which might be necessary here, too.

I'm hoping that running the CI tests on Python 3.14, which this PR achieves, will point in the right direction and will allow you to officially support Python 3.14 :)

[glatterf42]

Some more investigation: it looks to me like https://github.com/unionai-oss/pandera/blob/main/pandera/api/function_dispatch.py#L20 is the only line adding functions to the Dispatcher._function_registry. This, in turn, relies on get_first_arg_type(), which uses inspect.signature() and typing_inspect. The former changed in Python 3.14; while this should just affect annotations from files with from __future__ import annotations, e.g. api/dataframe/container.py and dtypes.py have this line, so it could be important here.
The latter does not officially support 3.14 yet; in fact, the only thing I see about that is an open issue regarding a test failure with 3.14. So it may well be that support needs to be added there, first.

[JelleZijlstra]

The former changed in Python 3.14; while this should just affect annotations from files with from future import annotations, e.g. api/dataframe/container.py and dtypes.py have this line, so it could be important here.

The behavior under from __future__ import annotations should not have changed. (I implemented these changes in CPython.)

The latter does not officially support 3.14 yet

typing-inspect was written several years ago when the introspection capabilities of the typing module itself were much worse. I haven't checked your code but it may be easier to drop use of typing-inspect and use typing directly.

[glatterf42]

Thanks for chiming in here :)
The usage of typing_inspect in api/function_dispatch.py came indeed down to get_args() and get_origin(): two functions that by now also exist in typing. So I replaced calls to them with calls to their typing implementation.
Please note that this does not eliminate typing_inspect as a dependency entirely since the code still uses get_generic_bases(), is_optional_type(), and is_union_type(), which don't have a 1:1-replacement in typing (of course, these could still be replaced, but I don't know if you prefer using the existing logic from typing_inspect or duplicating it here to drop the dependency).

[JelleZijlstra]

Yes, replacing the rest will be a bit more work and refactoring. Not my decision whether that's worth doing, but in my opinion it's usually nice to minimize third-party dependencies.