Error in unicode entry scripts

[julienmalard]

In utils.py, the _ENTRYPOINT_REGEX regex does not recognise valid unicode Python identifiers (https://peps.python.org/pep-3131/), preventing the use of non-ascii file paths as entry points. Using the regex module as a drop-in replacement for re would solve this issue; please let me know if you would like me to submit a pull request.

[eli-schwartz]

Using the regex module as a drop-in replacement for re would solve this issue

This would be quite problematic since a major feature of this package was that it has no dependencies and can run purely based on the stdlib, making it suitable as the very bottom package in an ecosystem bootstrap.

It would also prevent being vendored into pip, though that is less about having dependencies and more about having dependencies that have a binary component.

[uranusjr]

Do we even need a regex engine though? The spec basically just requires splitting on the first equal sign, and checking some leading and trailing special characters. It can be hand-rolled.

[julienmalard]

Interesting...would something using the following functions to check for valid identifiers be useful?
If so, I could make a pull request.

def isalnum(x):
    if len(x) != 1:
        return all(isalnum(y) for y in x)
    return unicodedata.category(x) in ['Lu', 'Ll', 'Lt', 'Lm', 'Lo', 'Nl', 'Mn', 'Mc', 'Nd', 'Pc']


def isalpha(x):
    if len(x) != 1:
        return all(isalpha(y) for y in x)
    return unicodedata.category(x) in ['Lu', 'Ll', 'Lt', 'Lm', 'Lo', 'Mn', 'Mc', 'Pc']

[eli-schwartz]

Valid identifiers can be determined via x.isidentifier(), presumably.

[julienmalard]

Reproduced here (pull request with failing tests): https://github.com/pypa/installer/actions/runs/12326732550?pr=254

[julienmalard]

...and now fixed with PR: https://github.com/pypa/installer/actions/runs/12326868467/job/34408216136?pr=254

[pfmoore]

Do you have an actual reproducer here? The regex you're claiming is wrong, is also present in importlib.metadata and it handles Unicode just fine in that context.

More generally, the re module is perfectly capable of handling Unicode:

>>> re.match(r"\w", 'நான்')
<re.Match object; span=(0, 1), match='ந'>

[julienmalard]

@pfmoore Yes, please see https://bugs.python.org/issue1693050 as well as the test cases proposed in this PR. The tests do not work with the \w regex with Indic diactrics (running on நான் will not match the entire word, nor will வணக்கம்). This issue is fixed in the regex module, but this is not a solution for low-level packages such as installer that can't have other modules as dependencies.