show deprecation warning if `-state` is used in conjunction with `plan` or `apply`

[EugenKon]

Terraform Version

$ terraform --version
Terraform v1.8.5
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v5.62.0
+ provider registry.terraform.io/hashicorp/external v2.3.3
+ provider registry.terraform.io/hashicorp/local v2.5.1
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.2
+ provider registry.terraform.io/hashicorp/tls v4.0.5

### Terraform Configuration Files

```terraform
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }

    local = {
      source  = "hashicorp/local"
      version = "~> 2.5"
    }

    # Needed to upgrade old Portal projects
    null = {
      source  = "hashicorp/null"
      version = "~> 3.2"
    }
  }
}

provider "aws" {
  region = local.aws_region
  default_tags {
    tags = {
      Project = local.project_name
    }
  }
}

# Somewhere in other parts of a cluster configuration
resource "random_uuid" "consul_token" {
}
resource "tls_private_key" "pk" {
  algorithm = "ED25519"
}

Debug Output

│ Error: failed to read schema for module.private-cloud.random_uuid.consul_token in registry.terraform.io/hashicorp/random: failed to instantiate provider "registry.terraform.io/hashicorp/random" to obtain schema: unavailable provider "registry.terraform.io/hashicorp/random"

│ Error: failed to read provider configuration schema for registry.terraform.io/hashicorp/tls: failed to instantiate provider "registry.terraform.io/hashicorp/tls" to obtain schema: unavailable provider "registry.terraform.io/hashicorp/tls"

Expected Behavior

Terraform should load providers automatically for resources to delete them as it done for the same resources to create them.

Actual Behavior

I need to configure providers explicitly to delete resources not in configuration but in a state.

Steps to Reproduce

1. We had a configuration without 'tls_private_key' and 'random_uuid' resources and without 'tls' and 'random' providers.
2. We added into a configuration 'tls_private_key' and 'random_uuid' resources. Still a configuration does not have 'tls' and 'random' providers.
3. terraform plan/apply works fine
4. Then we decided to rollback cluster to the old configuration without 'tls_private_key' and 'random_uuid' resources and still without 'tls' and 'random' providers.
5. terraform plan/apply failed with the errors above

I assume, that terraform should connect tls and random providers automatically for resource delation as it was done to create these resources. For the first case, when resources are created, this is done from the information in a configuration. For the second case, when resources should be deleted, this could be done from the information in a terrafomr.tfstate file.

Additional Context

Because for the first case terraform core loaded tls and random providers automatically, I decided that the same should be done by terraform core for the second case. Thus this does not belongs to 'terraform-provider-tls' and/or 'terraform-provider-random'.

References

No response

[jbardin]

Hi @EugenKon,

That is definitely not the typical behavior here, and I'm not sure how you've managed to get resources which are not causing Terraform to initialize a provider. Even if you had configured providers within the module then removed them, which would cause an error because the provider config is not persisted, it would be a different error from this.

Can you show the output of terraform providers? That may give us some more information about the providers required by the configuration and state.

Thanks!

[EugenKon]

@jbardin Yes, sure.

This is how it looks on upgraded cluster:

$ terraform -chdir=derived-src/aws/ providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/external]
├── provider[registry.terraform.io/hashicorp/aws] ~> 5.0
├── provider[registry.terraform.io/hashicorp/local] ~> 2.5
├── provider[registry.terraform.io/hashicorp/null] ~> 3.2
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   ├── provider[registry.terraform.io/hashicorp/local]
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   ├── provider[registry.terraform.io/hashicorp/aws]
│   ├── provider[registry.terraform.io/hashicorp/random]
│   ├── provider[registry.terraform.io/hashicorp/tls]
│   └── provider[registry.terraform.io/hashicorp/local]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
└── module.XXX
    └── provider[registry.terraform.io/hashicorp/aws]

This is before downgrade (a cluster with configuration without tls_private_key and random_uuid resources:

terraform -chdir=derived-src/aws/ providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] ~> 5.0
├── provider[registry.terraform.io/hashicorp/local] ~> 2.5
├── provider[registry.terraform.io/hashicorp/null] ~> 3.2
├── provider[registry.terraform.io/hashicorp/external]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   ├── provider[registry.terraform.io/hashicorp/aws]
│   └── provider[registry.terraform.io/hashicorp/local]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
└── module.XXX
    └── provider[registry.terraform.io/hashicorp/aws]

Plan

terraform -chdir=derived-src/aws/  plan -state ../../state.d/terraform.tfstate -out ../../state.d/terraform.plan

╷
│ Error: failed to read schema for module.XXX.tls_private_key.pk in registry.terraform.io/hashicorp/tls: failed to instantiate provider "registry.terraform.io/hashicorp/tls" to obtain schema: unavailable provider "registry.terraform.io/hashicorp/tls"
│
│

After adding required providers into providers.tf

    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }

    random = {
      source  = "hashicorp/random"
      version = "~> 3.0"  
    }

$ terraform -chdir=derived-src/aws/ init
$ terraform -chdir=derived-src/aws/ providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] ~> 5.0
├── provider[registry.terraform.io/hashicorp/local] ~> 2.5
├── provider[registry.terraform.io/hashicorp/null] ~> 3.2
├── provider[registry.terraform.io/hashicorp/external]
├── provider[registry.terraform.io/hashicorp/tls] ~> 4.0
├── provider[registry.terraform.io/hashicorp/random] ~> 3.0
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   ├── provider[registry.terraform.io/hashicorp/aws]
│   └── provider[registry.terraform.io/hashicorp/local]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
└── module.XXX
    └── provider[registry.terraform.io/hashicorp/aws]

I ran plan again:

  # module.XXX.random_uuid.consul_token will be destroyed
  # (because random_uuid.consul_token is not in configuration)
  - resource "random_uuid" "consul_token" {
      - id     = "YYY-7b19-ce4f-XXX" -> null
      - result = "YYY-7b19-ce4f-XXX" -> null
    }
...
# module.XXX.tls_private_key.pk will be destroyed
# (because tls_private_key.pk is not in configuration)
- resource "tls_private_key" "pk" {
...
Plan: 1 to add, 5 to change, 12 to destroy.

After plan was applied here is output for already the downgraded cluster (I mean the cluster without tls and random resources at the configuration):

$ terraform -chdir=derived-src/aws/ providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] ~> 5.0
├── provider[registry.terraform.io/hashicorp/local] ~> 2.5
├── provider[registry.terraform.io/hashicorp/null] ~> 3.2
├── provider[registry.terraform.io/hashicorp/tls] ~> 4.0
├── provider[registry.terraform.io/hashicorp/external]
├── provider[registry.terraform.io/hashicorp/random] ~> 3.0
├── module.XXX
│   ├── provider[registry.terraform.io/hashicorp/aws]
│   └── provider[registry.terraform.io/hashicorp/local]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
├── module.XXX
│   └── provider[registry.terraform.io/hashicorp/aws]
└── module.XXX
    └── provider[registry.terraform.io/hashicorp/aws]

*PS. By the way, could it be possible to hide generated secret for "random_uuid" "consul_token" from the output?

  - resource "random_uuid" "consul_token" {
      - id     = "YYY-7b19-ce4f-XXX" -> null
      - result = "YYY-7b19-ce4f-XXX" -> null
    }

I can not find anything related to this here: https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/uuid

[jbardin]

Thanks for all the extra info @EugenKon! I'm guessing that it has something to do with the -state flag, since that is pretty rarely used. I'll try and see if I can come up with a reproduction around that feature.

(Also, there's unfortunately not any way to hide normal resource plan data from the output right now, unless that provider declares the attributes as sensitive in the resource schema)

[jbardin]

So I've managed to get a slightly different error, but I still think this is caused by the legacy -state flag.
If you are trying to run init again to fetch the providers for the root module, the init command doesn't know about the -state flag and can't tell what providers are only required by the state. This also applies to the providers command, which isn't going to see any existing state.

Does terraform work correctly if you configure the local backend to point to the state file?
I think it would be something like:

terraform {
  backend "local" {
    path = "../../../../state.d/terraform.tfstate"
  }
}

[EugenKon]

@jbardin The give configuration resolves the problem also. It works as expected: no error messages about 'tls' and 'random' providers. I hope my feedback helped you.

Thanks.

[jbardin]

Thanks @EugenKon. Since the -state flag was deprecated years ago, and will cause Terraform to initialize incorrectly in cases like you have (which is why it's deprecated), I'm going to close this as terraform is working correctly given the situation.

[EugenKon]

@jbardin It would be nice to warn about -state deprecation like this:

No deprecation warning for -state at the moment.

[jbardin]

That's a good point, it is deprecated in the documentation, but perhaps we could add a message if it's used in conjunction with plan or apply.