Skip to content

Conversation

Eijebong
Copy link
Contributor

Python 3.13 dropped that in docker-library/python#895 Without this, running balrog locally results in

balrogadmin-1  |   File "/app/uwsgi/admin.wsgi", line 74, in <module>
balrogadmin-1  |     from auslib.web.admin.base import app as application  # noqa
balrogadmin-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
balrogadmin-1  |   File "/app/src/auslib/web/admin/base.py", line 29, in <module>
balrogadmin-1  |     .add_spec(path.join(current_dir, "swagger/api.yml"))
balrogadmin-1  |      ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/specsynthase/specbuilder.py", line 44, in add_spec
balrogadmin-1  |     spec = self._load_spec(file_name)
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/specsynthase/specbuilder.py", line 13, in _load_spec
balrogadmin-1  |     return yaml.load(spec_file, Loader=yaml.FullLoader)
balrogadmin-1  |            ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/__init__.py", line 79, in load
balrogadmin-1  |     loader = Loader(stream)
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/loader.py", line 24, in __init__
balrogadmin-1  |     Reader.__init__(self, stream)
balrogadmin-1  |     ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/reader.py", line 85, in __init__
balrogadmin-1  |     self.determine_encoding()
balrogadmin-1  |     ~~~~~~~~~~~~~~~~~~~~~~~^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/reader.py", line 124, in determine_encoding
balrogadmin-1  |     self.update_raw()
balrogadmin-1  |     ~~~~~~~~~~~~~~~^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/reader.py", line 178, in update_raw
balrogadmin-1  |     data = self.stream.read(size)
balrogadmin-1  |   File "/usr/local/lib/python3.13/encodings/ascii.py", line 26, in decode
balrogadmin-1  |     return codecs.ascii_decode(input, self.errors)[0]
balrogadmin-1  |            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
balrogadmin-1  | UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 773: ordinal not in range(128)
balrogadmin-1 exited with code 22

Because our swagger config contains some non ASCII characters so it cannot be read to a string if LANG is unset which defaults to POSIX (at least on my system).

I probably went overboard by setting it into every python container as some might not actually require it, but I feel like everyone would expect a UTF-8 locale to be set.

[eijemoz@plutonium] ~  >>> docker run -it python:3.11-slim /bin/bash
root@8ad1bc142b44:/# locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=
root@8ad1bc142b44:/#
exit
[eijemoz@plutonium] ~  >>> docker run -it python:3.13-slim /bin/bash
root@c1be2f814191:/# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

@Eijebong Eijebong requested a review from a team as a code owner June 23, 2025 16:35
Copy link
Contributor

@jcristau jcristau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please either set LC_CTYPE or LC_ALL, not LANG (which can be overridden by any of the other two).

Python 3.13 dropped that in docker-library/python#895
Without this, running balrog locally results in

```
balrogadmin-1  |   File "/app/uwsgi/admin.wsgi", line 74, in <module>
balrogadmin-1  |     from auslib.web.admin.base import app as application  # noqa
balrogadmin-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
balrogadmin-1  |   File "/app/src/auslib/web/admin/base.py", line 29, in <module>
balrogadmin-1  |     .add_spec(path.join(current_dir, "swagger/api.yml"))
balrogadmin-1  |      ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/specsynthase/specbuilder.py", line 44, in add_spec
balrogadmin-1  |     spec = self._load_spec(file_name)
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/specsynthase/specbuilder.py", line 13, in _load_spec
balrogadmin-1  |     return yaml.load(spec_file, Loader=yaml.FullLoader)
balrogadmin-1  |            ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/__init__.py", line 79, in load
balrogadmin-1  |     loader = Loader(stream)
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/loader.py", line 24, in __init__
balrogadmin-1  |     Reader.__init__(self, stream)
balrogadmin-1  |     ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/reader.py", line 85, in __init__
balrogadmin-1  |     self.determine_encoding()
balrogadmin-1  |     ~~~~~~~~~~~~~~~~~~~~~~~^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/reader.py", line 124, in determine_encoding
balrogadmin-1  |     self.update_raw()
balrogadmin-1  |     ~~~~~~~~~~~~~~~^^
balrogadmin-1  |   File "/usr/local/lib/python3.13/site-packages/yaml/reader.py", line 178, in update_raw
balrogadmin-1  |     data = self.stream.read(size)
balrogadmin-1  |   File "/usr/local/lib/python3.13/encodings/ascii.py", line 26, in decode
balrogadmin-1  |     return codecs.ascii_decode(input, self.errors)[0]
balrogadmin-1  |            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
balrogadmin-1  | UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 773: ordinal not in range(128)
balrogadmin-1 exited with code 22
```

Because our swagger config contains some non ASCII characters so it
cannot be read to a string if `LANG` is unset which defaults to `POSIX`
(at least on my system).

I probably went overboard by setting it into every python container as
some might not actually require it, but I feel like everyone would
expect a UTF-8 locale to be set.
@Eijebong Eijebong force-pushed the restore-utf8-locale-in-containers branch from 774cae5 to 125d34f Compare June 24, 2025 12:08
@Eijebong Eijebong changed the title Reintroduce LANG=C.UTF-8 in docker containers Set LC_ALL=C.UTF-8 in docker containers Jun 24, 2025
@Eijebong Eijebong requested a review from jcristau June 24, 2025 12:10
@Eijebong Eijebong merged commit 2b7d705 into mozilla-releng:main Jun 24, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants