Skip to content

Conversation

johnhaddon
Copy link
Member

On Windows, environment variable names are case-preserving but case-insensitive. Python's os.environ mapping has a botched emulation for this which implements the case-insensitive part by trashing the case and making everything upper-case. See https://bugs.python.org/issue28824.

This caused us problems where we were using os.environ.copy() as the source for a child environment we constructed for use with subprocess. The child environment now had all-upper-case environment variables instead of the originals. And that caused problems with Context.substitute( "${var}" ) in the child process, because context variable lookups are case-sensitive.

In the absence of a fix to Python itself, for now we add a Gaffer.environment() function that builds a dictionary from the ground truth of the process' environment table. And use that everywhere we're running subprocesses on behalf of the user.

Fixes #6371.

@johnhaddon johnhaddon requested a review from ericmehl April 14, 2025 13:01
@johnhaddon johnhaddon self-assigned this Apr 14, 2025
@github-project-automation github-project-automation bot moved this to Pending Review in Work in Progress Apr 14, 2025
@johnhaddon
Copy link
Member Author

I'm a little bit in two minds about including this in 1.5. It's a pretty central change, so if we were being conservative it might be more suitable for 1.6.

Copy link
Collaborator

@ericmehl ericmehl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks John! LGTM.

I think waiting for 1.6 is reasonable. It sounds like it's not too too far away and a workaround in the meantime is to use all caps for environment variables throughout.

Just needs a quick rebase to sort the merge conflict.

johnhaddon and others added 2 commits April 23, 2025 09:46
…on Windows

On Windows, environment variable names are case-preserving but case-insensitive.
Python's `os.environ` mapping has a botched emulation for this which implements
the case-insensitive part by trashing the case and making everything upper-case.
See https://bugs.python.org/issue28824.

This caused us problems where we were using `os.environ.copy()` as the source for a child environment we constructed for use with `subprocess`. The child environment now had all-upper-case environment variables instead of the originals. And that caused problems with `Context.substitute( "${var}" )` in the child process, because context variable lookups are case-sensitive.

In the absence of a fix to Python itself, for now we add a
`Gaffer.environment()` function that builds a dictionary from the ground truth of the process' environment table.

Fixes GafferHQ#6371.
Although the case-preservation fixed by `Gaffer.environment()` on Windows isn't needed here, it still makes sense to have the test code be an example of best practice.
@johnhaddon johnhaddon changed the base branch from 1.5_maintenance to main April 23, 2025 08:48
@johnhaddon johnhaddon merged commit c308703 into GafferHQ:main Apr 23, 2025
6 checks passed
@github-project-automation github-project-automation bot moved this from Pending Review to Pending release in Work in Progress Apr 23, 2025
@johnhaddon
Copy link
Member Author

Thanks Eric - I've rebased and merged to main.

@johnhaddon johnhaddon deleted the envVarCaseFix branch April 23, 2025 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Context.substitute only recognize uppercase env vars on Windows
2 participants