Failure when plan has objects changed outside of terraform

[wyardley]

Describe the bug

I'm getting an error about labels (maybe an expected one?) and then a workflow failure at the Run # Post output. step when planning a few states (other states seem to be planning without error).
FWIW, we have a workflow that plans multiple states in parallel in different workflows using a shared workflow.

The common factor seems to be that the failing states have "objects changed outside of Terraform" warnings (see below).

Expected behavior

The state to plan without error

Screenshots

Finalizing artifact upload
Artifact terraform-245-5d8ad38495b4c9aeabbdccb2f90821e0.tfplan.zip successfully finalized. Artifact ID 2685240083
Artifact terraform-245-5d8ad38495b4c9aeabbdccb2f90821e0.tfplan has been successfully uploaded! Final size is 104125 bytes. Artifact ID is 2685240083
Artifact download URL: https://github.com/team-settle/terraform/actions/runs/13641722770/artifacts/2685240083
Run # Post output.
Error: Process completed with exit code 1.

In the details bit under Run # Post output., I don't see any useful output

Additional context

I am thinking the issue may be related to there being an "objects changed outside of Terraform" warning

 Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.staging_gke.google_container_cluster.primary has changed
  ~ resource "google_container_cluster" "primary" {
        id                                       = "projects/xxx/locations/us-central1/clusters/yyyy"
      ~ master_version                           = "1.30.9-gke.1046000" -> "1.30.9-gke.1127000"
        name                                     = "yyyy"
        # (34 unchanged attributes hidden)

      ~ node_pool {
            name                        = "default-pool"
          ~ version                     = "1.30.9-gke.1046000" -> "1.30.9-gke.1127000"
            # (7 unchanged attributes hidden)

            # (5 unchanged blocks hidden)
        }
      ~ node_pool {
            name                        = "pool-1"
          ~ version                     = "1.30.9-gke.1046000" -> "1.30.9-gke.1127000"
            # (7 unchanged attributes hidden)

            # (5 unchanged blocks hidden)
        }

        # (38 unchanged blocks hidden)
    }

I also see the labels error mentioned in #432, but I think it's a red herring.

To Reproduce

Try planning a state which has objects that have changed out of band since the state was last planned (for example, a GKE cluster version change from auto-updates). You could likely trigger this by changing a computed value outside of tf.

Existing config:
Other than the steps that checkout the code, run init / lint / tflint / etc. (we run those separately from the builtin hooks for that in this action), all we've got in the config is:

      - name: Provision TF
        uses: op5dev/tf-via-pr@v13
        with:
          working-directory: ${{ inputs.working-directory }}
          command: ${{ github.event_name == 'push' && 'apply' || 'plan' }}
          arg-lock: ${{ github.event_name == 'push' }}

[rdhar]

workflow failure at the Run # Post output. step when planning a few states (other states seem to be planning without error).

This is the most curious part. For the other states, is it the same set of inputs but different working-directory?

I also see the labels error mentioned in #432, but I think it's a red herring.

You can pass label-pr: false input to remove this from the equation altogether while testing.

[wyardley]

This is the most curious part. For the other states, is it the same set of inputs but different working-directory?

That's right, it's literally the same workflow file, just called with different working directories as input. The only consistent thing I can find is that the states that have "objects changed out of terraform" warnings are present in the 3 states that fail. The other states all plan just fine. Given that this shows up before the plan output, maybe it's just not a use case that's been tested and the output between the refresh output and the actual plan is causing some sort of issue?

I can try to dig into it a little if I get some time as well. I think we've talked on the OpenTofu slack before? So can chat there possibly sometime tomorrow as well.

[wyardley]

So one thing I noticed:

  # module.xxx.google_container_cluster.primary has changed
  ~ resource "google_container_cluster" "primary" {
        id                                       = "projects/xxx/locations/us-central1/clusters/yyy"
      ~ master_version                           = "1.30.9-gke.1046000" -> "1.30.9-gke.1127000"
        name                                     = "yyy"
        # (34 unchanged attributes hidden)
      ~ node_pool {
            name                        = "default-pool"
          ~ version                     = "1.30.9-gke.1046000" -> "1.30.9-gke.1127000"
            # (7 unchanged attributes hidden)
            # (5 unchanged blocks hidden)
        }
      ~ node_pool {
            name                        = "pool-1"
          ~ version                     = "1.30.9-gke.1046000" -> "1.30.9-gke.1127000"
            # (7 unchanged attributes hidden)
            # (5 unchanged blocks hidden)
        }
        # (38 unchanged blocks hidden)
    }

which I think is matching this line:

TF-via-PR/action.yml

Line 214 in bd7affe

-e 's/^ # \(.*\)/# \1/' > tf.diff.txt || true

tf.diff.txt contains the following

  # module.xyz.google_container_cluster.primary has changed

And there are also changes to outputs. But, at the same time, there are no changes to the resource that changed outside of terraform in the actual plan.

I think it's failing somewhere roughly within this range:

TF-via-PR/action.yml

Lines 314 to 339 in bd7affe

	# If steps.format.outcome failed, set syntax highlighting to diff, otherwise set it to hcl.
	syntax="hcl"
	if [[ "${{ steps.format.outcome }}" == "failure" ]]; then syntax="diff"; fi
	# Add summary to the job status.
	check_run=$(gh api /repos/${{ github.repository }}/check-runs/${{ steps.identifier.outputs.job }} --header "$GH_API" --method PATCH --field "output[title]=${summary}" --field "output[summary]=${summary}")
	# From check_run, echo html_url.
	check_url=$(echo "$check_run" | jq --raw-output '.html_url')
	echo "check_id=$(echo "$check_run" | jq --raw-output '.id')" >> "$GITHUB_OUTPUT"
	run_url=$(echo ${check_url}#step:${{ steps.identifier.outputs.step }}:1)
	echo "run_url=$run_url" >> "$GITHUB_OUTPUT"
	# If tf.diff.txt exists, display it within a diff block, truncated for character limit.
	if [[ -s tf.diff.txt ]]; then
	# Get count of lines in tf.diff.txt which don't start with "# ".
	diff_count=$(grep --invert-match '^# ' tf.diff.txt | wc --lines)
	if [[ $diff_count -eq 1 ]]; then diff_change="change"; else diff_change="changes"; fi
	# Parse diff of changes, truncated for character limit.
	diff_truncated=$(head --bytes=24000 tf.diff.txt)
	if [[ ${#diff_truncated} -eq 24000 ]]; then diff_truncated="${diff_truncated}"$'\n…'; fi
	echo "diff<<EODIFFTFVIAPR"$'\n'"$diff_truncated"$'\n'EODIFFTFVIAPR >> "$GITHUB_OUTPUT"
	diff="
	<details><summary>Diff of ${diff_count} ${diff_change}.</summary>

[rdhar]

Unfortunately, I'm unable to replicate this scenario which makes it tricky to test what, if any, impact the changed objects are having.

I'm curious if you can share a more complete plan, while obfuscating anything sensitive? With it, I could try running the shell script locally to check if borks out the same way.

[rdhar]

As followed up on OpenTofu's Slack, as yet unable to replicate the particular scenario which leads you to this issue. Perhaps sharing (obfuscated) raw workflow logs might give us a hint, otherwise closing out till we have a promising lead.

[wyardley]

Thanks. Feel free to keep closed for now - I will keep working on creating a repro case or plan file when I'm able.

Here's another example of a resource that's possible to induce this behavior for. Note: when the plan also has something to change, this output doesn't seem to cause an issue.

  # module.xyz.google_sql_database_instance.default has changed
  ~ resource "google_sql_database_instance" "default" {
        id                             = "xyz"
        name                           = "xyz"
        # (18 unchanged attributes hidden)

      ~ settings {
          ~ version                      = 136 -> 145
            # (16 unchanged attributes hidden)

            # (8 unchanged blocks hidden)
        }

        # (2 unchanged blocks hidden)

[wyardley]

And another:

  # google_redis_instance.memorystore_instance has changed
  ~ resource "google_redis_instance" "memorystore_instance" {
      ~ current_location_id      = "us-central1-a" -> "us-central1-c"
        id                       = "projects/xxxxx/locations/us-central1/instances/xyz"
        name                     = "xyz"
        # (32 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

The plan seems to get uploaded, and then still fail on the Run # Post output. bit.

Artifact terraform-350-bca4742d99af057166a2e736d770841b.tfplan.zip successfully finalized. Artifact ID 3117728410
Artifact terraform-350-bca4742d99af057166a2e736d770841b.tfplan has been successfully uploaded! Final size is 45048 bytes. Artifact ID is 3117728410

[rdhar]

Hi @wyardley, following the recent release of v13.5.0 (v13), could you update and retry to see if the issue is replicable? I think it might be resolved thanks to @jemc's contribution in #461.