aws-glue-alpha: not possible to disable metrics on glue jobs

[pepdekpd]

Describe the bug

It seems that in the various glue spark job constructs, the argument --enable-metrics is always passed.
According to glue documentation though passing the argument regardless of the value enables metrics.
protected nonExecutableCommonArguments(props: SparkJobProps): {[key: string]: string} { // Enable CloudWatch metrics and continuous logging by default as a best practice const continuousLoggingArgs = this.setupContinuousLogging(this.role, props.continuousLogging); const profilingMetricsArgs = { '--enable-metrics': '' }; const observabilityMetricsArgs = { '--enable-observability-metrics': 'true' };

--enable-metrics
Enables the collection of metrics for job profiling for this job run. These metrics are available on the AWS Glue console and the Amazon CloudWatch console. The value of this parameter is not relevant. To enable this feature, you can provide this parameter with any value, but true is recommended for clarity. To disable this feature, remove this parameter from your job configuration.

As these metrics seem part of a cloudwatch custom namespace "Glue" we incur costs on usage type "EU-CW:MetricMonitorUsage".
So we like to disable this metric colletion, and see if we can control the cloudwatch costs.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

Somehow you should be able to disable the metric collection on spark glue jobs.

Current Behavior

Metrics are collected no matter what.

Reproduction Steps

`from aws_cdk import aws_glue_alpha as glue_alpha, aws_iam as iam, Stack

from constructs import Construct

class BugStack(Stack):

def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
    super().__init__(scope, construct_id, **kwargs)

    glue_job_role = iam.Role(
        self,
        "glue-job-role",
        assumed_by=iam.CompositePrincipal(
            iam.ServicePrincipal("glue.amazonaws.com"),
        ),
    )

    self.glue_job = glue_alpha.PySparkEtlJob(
        scope=self,
        id="blabla",
        job_name="blabla-job",
        glue_version=glue_alpha.GlueVersion.V5_0,
        script=glue_alpha.Code.from_asset(path="requirements.txt"),
        max_concurrent_runs=1,
        role=glue_job_role,
        max_retries=0,
        number_of_workers=2,
    )

`

If you synthesize this stack, you see in the cloudformation template:
"blablaE9B37712": { "Type": "AWS::Glue::Job", "Properties": { "Command": { "Name": "glueetl", "PythonVersion": "3", "ScriptLocation": { "Fn::Join": [ "", [ "s3://", { "Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}" }, "/5bc8938db6aed20519ef1629110549ff89b31be88860574351eed55b96b4a3fc.txt" ] ] } }, "DefaultArguments": { "--job-language": "python", "--enable-continuous-cloudwatch-log": "true", "--enable-metrics": "", "--enable-observability-metrics": "true" }, "ExecutionProperty": { "MaxConcurrentRuns": 1 }, "GlueVersion": "5.0", "JobRunQueuingEnabled": false, "MaxRetries": 0, "Name": "blabla-job", "NumberOfWorkers": 2, "Role": { "Fn::GetAtt": [ "gluejobroleAAC64F87", "Arn" ] },

Possible Solution

No response

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.208.0

AWS CDK CLI version

2.1023.0

Node.js Version

22.12.0

OS

ubuntu

Language

Python

Language Version

3.12

Other information

No response

[pahud]

Root Cause Analysis

The root cause is in the SparkJob.nonExecutableCommonArguments() method, which hardcodes metrics arguments:

const profilingMetricsArgs = { '--enable-metrics': '' };
const observabilityMetricsArgs = { '--enable-observability-metrics': 'true' };

This affects all Spark-based Glue job types:
• PySparkEtlJob
• ScalaSparkEtlJob
• PySparkStreamingJob
• ScalaSparkStreamingJob
• PySparkFlexEtlJob
• ScalaSparkFlexEtlJob
• RayJob

The implementation was designed with "best practices" in mind (enabling metrics by default), but it doesn't provide users with the flexibility to opt out for cost control purposes.

Document Evidence

• AWS Glue Programming ETL Arguments Documentation - Confirms that the presence of --enable-metrics
parameter enables metrics regardless of value
• SparkJob source code - Shows hardcoded metrics
arguments

Possible Fixes/Solutions

1. Add optional properties to SparkJobProps interface:
   • enableMetrics?: boolean (default: true for backward compatibility)
   • enableObservabilityMetrics?: boolean (default: true for backward compatibility)

2. Modify nonExecutableCommonArguments method to conditionally include metrics arguments based on these properties

3. Update all Spark job constructors to pass these new properties through

4. Maintain backward compatibility by defaulting to current behavior (metrics enabled)

---

Hi @pepdekpd,

Thank you for reporting this issue! You're absolutely correct that the current implementation of AWS Glue Spark job constructs in the aws-glue-alpha module hardcodes the --enable-metrics and --enable-observability-metrics arguments, making it impossible to disable metrics collection for cost control.

Looking at the SparkJob.nonExecutableCommonArguments() method, these arguments are indeed hardcoded:

aws-cdk/packages/@aws-cdk/aws-glue-alpha/lib/jobs/spark-job.ts

Lines 137 to 138 in cd9d69c

	const profilingMetricsArgs = { '--enable-metrics': '' };
	const observabilityMetricsArgs = { '--enable-observability-metrics': 'true' };

As you correctly noted from the AWS documentation, the presence of the --enable-metrics parameter (regardless of its value) enables metrics collection, and the only way to disable it is to remove the parameter entirely.

This affects all Spark-based Glue job types including PySparkEtlJob, ScalaSparkEtlJob, PySparkStreamingJob, etc.

Proposed Solution:
We should add optional boolean properties to the SparkJobProps interface:

• enableMetrics?: boolean (default: true for backward compatibility)
• enableObservabilityMetrics?: boolean (default: true for backward compatibility)

This would allow users to opt out of metrics collection when cost control is a priority, while maintaining the current "best practice" defaults.

We'll need a PR to get it addressed. The changes would need to:

1. Update the SparkJobProps interface to include the new optional properties
2. Modify the nonExecutableCommonArguments method to conditionally include metrics arguments
3. Update tests to cover both enabled and disabled scenarios
4. Ensure backward compatibility by defaulting to current behavior

This is a valid feature request that would provide users with the flexibility to control CloudWatch costs while maintaining sensible defaults.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.