feat: Add tecton_outputs module to write shared values.

[eshiferax]

Create a new module tecton_outputs and adding it to all root modules (controplane_rift, dataplane_rift, emr, databricks, dataplane_rift_with_emr, controlplane_rift_with_emr). The bucket policy allows the control-plane account read-only access.

When users terraform apply the root modules, the outputs are written in a json file (outputs.json) to one of three locations, depending on the location_config input:

1. new_bucket (default) – Create a dedicated bucket named <deployment_name>-tecton-outputs in the dataplane account. A bucket policy granting the Tecton control-plane account read access is applied.
2. offline_store_bucket_path – Write the outputs.json file to a key prefix (default: internal/tecton-outputs/) inside the Tecton offline-store bucket (created by the deployment module). No new bucket or policy is created.
3. tecton_hosted_presigned – Upload the file to a bucket owned by Tecton using a presigned upload URL. In this mode no AWS resources are created and Terraform uploads the JSON once during apply.

Also adding missing output emr_master_role_arn for emr/dataplane_with_emr modules.

Testing

• controlplane_rift module Terraform apply + default customer bucket outputs
  (No location_config specified - using default)

➜  aws-vault exec <control-plane>-- aws s3 cp s3://<dataplane_acct>-tecton-outputs/outputs.json .           
download: s3://<dataplane_acct>-tecton-outputs/outputs.json to ./outputs.json
➜  cat outputs.json | jq '.'
{
  "anyscale_docker_target_repo": null,
  "compute_arn": null,
  "compute_instance_profile_arn": null,
  "compute_manager_arn": null,
  "cross_account_external_id": "<EXTERNAL_ID>",
  "cross_account_role_arn": "arn:aws:iam::<ACCT_ID>:role/tecton-<CLUSTER_NAME>-cross-account-role",
  "databricks_workspace_url": null,
  "deployment_name": "<CLUSTER_NAME>",
  "emr_master_role_arn": null,
  "emr_security_group_id": null,
  "emr_service_security_group_id": null,
  "emr_subnet_id": null,
  "emr_subnet_route_table_ids": null,
  "kms_key_arn": null,
  "nat_gateway_public_ips": null,
  "notebook_cluster_id": null,
  "region": "us-west-2",
  "rift_compute_security_group_id": null,
  "spark_instance_profile_arn": null,
  "spark_instance_profile_name": null,
  "spark_role_arn": null,
  "spark_role_name": null,
  "vm_workload_subnet_ids": null,
  "vpc_id": null
}

• controlplane_rift module Terraform apply + offline-store bucket outputs

   location_config = {
     type                      = "offline_store_bucket_path"
   }

➜  aws-vault exec  <control_plane> -- aws s3 cp s3://<OFFLINE_STORE_BUCKET>/internal/tecton-outputs/outputs.json .
download: s3://<OFFLINE_STORE_BUCKET>/internal/tecton-outputs/outputs.json to ./outputs.json
➜  cat outputs.json | jq '.'
{
  "anyscale_docker_target_repo": null,
  "compute_arn": null,
  "compute_instance_profile_arn": null,
  "compute_manager_arn": null,
  "cross_account_external_id": "<EXTERNAL_ID>",
  "cross_account_role_arn": "arn:aws:iam::<ACCT_ID>:role/tecton-<CLUSTER_NAME>-cross-account-role",
  "databricks_workspace_url": null,
  "deployment_name": "<CLUSTER_NAME>",
  "emr_master_role_arn": null,
  "emr_security_group_id": null,
  "emr_service_security_group_id": null,
  "emr_subnet_id": null,
  "emr_subnet_route_table_ids": null,
  "kms_key_arn": null,
  "nat_gateway_public_ips": null,
  "notebook_cluster_id": null,
  "region": "us-west-2",
  "rift_compute_security_group_id": null,
  "spark_instance_profile_arn": null,
  "spark_instance_profile_name": null,
  "spark_role_arn": null,
  "spark_role_name": null,
  "vm_workload_subnet_ids": null,
  "vpc_id": null
}

• controlplane_rift module Terraform apply + presigned-url bucket outputs
  Created a special bucket (<bucket_for_presign_test>) in control-plane account and generated a presigned Upload URL

➜ python gen-put.py dev-drift-tst-presigned outputs.json
Generated PUT presigned URL: https://....

Applied

  location_config = {
    type                       = "tecton_hosted_presigned"
    tecton_presigned_write_url = "https://<bucket_for_presign_test>.s3.amazonaws.com/outputs.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=......."
  }

 aws s3 cp s3://<bucket_for_presign_test>/outputs.json .

 cat outputs.json | jq '.'
{
  "anyscale_docker_target_repo": null,
  "compute_arn": null,
  "compute_instance_profile_arn": null,
  "compute_manager_arn": null,
  "cross_account_external_id": "<EXTERNAL_ID>",
  "cross_account_role_arn": "arn:aws:iam::<ACCT_ID>:role/tecton-<CLUSTER_NAME>-cross-account-role",
  "databricks_workspace_url": null,
  "deployment_name": "<CLUSTER_NAME>",
  "emr_master_role_arn": null,
  "emr_security_group_id": null,
  "emr_service_security_group_id": null,
  "emr_subnet_id": null,
  "emr_subnet_route_table_ids": null,
  "kms_key_arn": null,
  "nat_gateway_public_ips": null,
  "notebook_cluster_id": null,
  "region": "us-west-2",
  "rift_compute_security_group_id": null,
  "spark_instance_profile_arn": null,
  "spark_instance_profile_name": null,
  "spark_role_arn": null,
  "spark_role_name": null,
  "vm_workload_subnet_ids": null,
  "vpc_id": null
}

[eshiferax]

• feat: Add tecton_outputs module to write shared values. #221 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[zhoujoetan]

Another nit: should we rename infrastructure.tf to main.tf to follow the Terraform convention?

[zhoujoetan]

See inline comments

[zhoujoetan]

Replied

[eshiferax]

Merge activity

• Jun 12, 6:30 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jun 12, 6:30 PM UTC: @eshiferax merged this pull request with Graphite.