Metaissue: 3.15 support

[clin1234]

This is to track test failures caused when compiling against 3.15. I know it's a bit early (even contradicting my own advice in CONTRIBUTING.md), but the sooner they're fixed before beta 1, the better.

• test_module_append_to_inittab failure with 3.15 #5611
• test_declarative_module failure with 3.15 #5612
• test_module failure with 3.15 #5613
• doc unit tests failure with 3.15 #5614

@ngoldbaum mind taking a peek?

[davidhewitt]

Basically all the erorrs are

SystemError: invalid PyModuleDef, extension possibly compiled for non-free-threaded Python

Maybe try nox -s ffi-check with a Python 3.15 alpha interpreter?

Alternatively this might be related to #4882 and the cache is broken.

[alex]

python/cpython@32b2c5d I think there might be an ABI change in PyModuleDef_HEAD_INIT?

[clin1234]

With nox -s ffi-check:

(venv) @clin1234 ➜ /workspaces/pyo3 (3.15) $ nox -s ffi-check
nox > Running session ffi-check
nox > Creating virtual environment (virtualenv) using python3.15-nogil in .nox/ffi-check
nox > cargo doc --manifest-path=pyo3-ffi-check/Cargo.toml -p pyo3-ffi --no-deps
   Compiling pyo3-build-config v0.27.1 (/workspaces/pyo3/pyo3-build-config)
   Compiling pyo3-ffi v0.27.1 (/workspaces/pyo3/pyo3-ffi)
 Documenting pyo3-ffi v0.27.1 (/workspaces/pyo3/pyo3-ffi)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 8.25s
   Generated /workspaces/pyo3/pyo3-ffi-check/target/doc/pyo3_ffi/index.html
nox > cargo run --manifest-path=pyo3-ffi-check/Cargo.toml
   Compiling pyo3-build-config v0.27.1 (/workspaces/pyo3/pyo3-build-config)
   Compiling pyo3-ffi v0.27.1 (/workspaces/pyo3/pyo3-ffi)
   Compiling pyo3-ffi-check v0.1.0 (/workspaces/pyo3/pyo3-ffi-check)
   Compiling pyo3-ffi-check-macro v0.1.0 (/workspaces/pyo3/pyo3-ffi-check/macro)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 29.85s
     Running `pyo3-ffi-check/target/debug/pyo3-ffi-check`
comparing pyo3-ffi against headers generated for 3.15.0a1
warning: ignoring zero-sized pyo3_ffi type PyCodeObject
warning: ignoring zero-sized pyo3_ffi type PyDictKeysObject
warning: ignoring zero-sized pyo3_ffi type PyDictValues
warning: ignoring zero-sized pyo3_ffi type PyFrameObject
warning: ignoring zero-sized pyo3_ffi type PyGenObject
warning: ignoring zero-sized pyo3_ffi type PyInterpreterState
warning: ignoring zero-sized pyo3_ffi type PyLongObject
warning: ignoring zero-sized pyo3_ffi type PyThreadState
warning: ignoring zero-sized pyo3_ffi type _PyInterpreterFrame
nox > Session ffi-check was successful in 41 seconds.

[davidhewitt]

Try repeating with a non-freethreaded interpreter

[davidhewitt]

Actually I guess the errors are for a freethreaded Python.

Based on what @alex has linked, probably we have the wrong definition of PyObject_HEAD_INIT, which ffi-check currently doesn't catch.

[clin1234]

Definition of PyObject and PyObject_HEAD_INIT on 3.15t:

// Objects that are not owned by any thread use a thread id (tid) of zero.
// This includes both immortal objects and objects whose reference count
// fields have been merged.
#define _Py_UNOWNED_TID             0

struct _object {
    // ob_tid stores the thread id (or zero). It is also used by the GC and the
    // trashcan mechanism as a linked list pointer and by the GC to store the
    // computed "gc_refs" refcount.
    _Py_ALIGNED_DEF(_PyObject_MIN_ALIGNMENT, uintptr_t) ob_tid;
    uint16_t ob_flags;
    PyMutex ob_mutex;           // per-object lock
    uint8_t ob_gc_bits;         // gc-related state
    uint32_t ob_ref_local;      // local reference count
    Py_ssize_t ob_ref_shared;   // shared (atomic) reference count
    PyTypeObject *ob_type;
};

#if defined(Py_GIL_DISABLED)
#define PyObject_HEAD_INIT(type)    \
    {                               \
        0,                          \
        _Py_STATICALLY_ALLOCATED_FLAG, \
        { 0 },                      \
        0,                          \
        _Py_IMMORTAL_REFCNT_LOCAL,  \
        0,                          \
        (type),                     \
    },
#else

[clin1234]

From pymacro.h

// _Py_ALIGNED_DEF(N, T): Define a variable/member with increased alignment
//
// `N`: the desired minimum alignment, an integer literal, number of bytes
// `T`: the type of the defined variable
//      (or a type with at least the defined variable's alignment)
//
// May not be used on a struct definition.
//
// Standards/compiler support for `alignas` alternatives:
// - `alignas` is a keyword in C23 and C++11.
// - `_Alignas` is a keyword in C11
// - GCC & clang has __attribute__((aligned))
//   (use that for older standards in pedantic mode)
// - MSVC has __declspec(align)
// - `_Alignas` is common C compiler extension
// Older compilers may name `alignas` differently; to allow compilation on such
// unsupported platforms, we don't redefine _Py_ALIGNED_DEF if it's already
// defined. Note that defining it wrong (including defining it to nothing) will
// cause ABI incompatibilities.
//
// Behavior of `alignas` alternatives:
// - `alignas` & `_Alignas`:
//   - Can be used multiple times; the greatest alignment applies.
//   - It is an *error* if the combined effect of all `alignas` modifiers would
//     decrease the alignment.
//   - Takes types or numbers.
//   - May not be used on a struct definition, unless also defining a variable.
// - `__declspec(align)`:
//   - Has no effect if it would decrease alignment.
//   - Only takes an integer literal.
//   - May be used on struct or variable definitions.
//     However, when defining both the struct and the variable at once,
//     `declspec(aligned)` causes compiler warning 5274 and possible ABI
//     incompatibility.
// - ` __attribute__((aligned))`:
//   - Has no effect if it would decrease alignment.
//   - Takes types or numbers
//   - May be used on struct or variable definitions.
#ifndef _Py_ALIGNED_DEF
#    ifdef __cplusplus
#        if __cplusplus >= 201103L
#            define _Py_ALIGNED_DEF(N, T) alignas(N) alignas(T) T
#        elif defined(__GNUC__) || defined(__clang__)
#            define _Py_ALIGNED_DEF(N, T) __attribute__((aligned(N))) T
#        elif defined(_MSC_VER)
#            define _Py_ALIGNED_DEF(N, T) __declspec(align(N)) T
#        else
#            define _Py_ALIGNED_DEF(N, T) alignas(N) alignas(T) T
#        endif
#    elif defined(_MSC_VER)
#        define _Py_ALIGNED_DEF(N, T) __declspec(align(N)) T
#    elif defined(__STDC_VERSION__) && __STDC_VERSION__ >= 202311L
#        define _Py_ALIGNED_DEF(N, T) alignas(N) alignas(T) T
#    elif defined(__STDC_VERSION__) && __STDC_VERSION__ >= 201112L
#        define _Py_ALIGNED_DEF(N, T)  _Alignas(N) _Alignas(T) T
#    elif (defined(__GNUC__) || defined(__clang__))
#        define _Py_ALIGNED_DEF(N, T) __attribute__((aligned(N))) T
#    else
#        define _Py_ALIGNED_DEF(N, T) _Alignas(N) _Alignas(T) T
#    endif
#endif

[ngoldbaum]

@ngoldbaum mind taking a peek?

Currently I've got my plate full trying to get Python 3.14 support working in packages. We'll probably want to start looking at 3.15 eventually, but probably not until next year.