Bytewax Materializer Security Context Capabilities #3572
Description
Expected Behavior
Running the Bytewax materializer in a scaled and secured Kubernetes cluster should work, successfully running the materializer job.
Current Behavior
In a managed, secured cluster, setting the securityContext capabilities may not be permitted, as is enabled by default. This causes the materialization job/s to fail.
Steps to reproduce
Run the Bytewax materializer in an environment where securityContext capabilities cannot be specified/overriden in jobs.
Specifications
- Version: 0.30.2
- Platform: EKS
- Subsystem: Python, Bytewax, Snowflake, Dynamodb
Possible Solution
Add an optional boolean configuration parameter to the Bytewax materialization engine to dictate whether or not the (current default) security context options are included in the job spec. If set to False set the securityContext capabilities to None.