Network Interface private_ips in different order

[hashibot]

This issue was originally opened by @Shaiou as hashicorp/terraform#6750. It was migrated here as part of the provider split. The original body of the issue is below.

---

• aws_network_interface

Hi there,
I'm running into some issue when creating a network interface, I specified a list of private_ips and made sure the one I wanted as primary was listed first, however it does respect that order and the second one appears as primary:

Here's my ressource:

resource "aws_network_interface" "waf_1_pub" {
    subnet_id = "${aws_subnet.public.1.id}"
    security_groups = ["${aws_security_group.waf_prv.id}"]
    # We put the basic ip for sync and all and one IP per VS
    private_ips = ["${cidrhost(aws_subnet.public.1.cidr_block, var.waf_ip_offset)}", "${join(",",template_file.waf_1_vs_ips.*.rendered)}"]
    security_groups = [ "${aws_security_group.waf_pub.id}" ]
    attachment {
        instance = "${aws_instance.waf.1.id}"
        device_index = 1
    }
}

And the output of the terraform show after the apply:

aws_network_interface.waf_1_pub:
  id = eni-8ba85cf3
  attachment.# = 1
  attachment.3009111673.attachment_id = eni-attach-2baa9ae8
  attachment.3009111673.device_index = 1
  attachment.3009111673.instance = i-18be9790
  description = 
  private_ips.# = 2
  private_ips.3297441504 = 10.220.1.210
  private_ips.3717318561 = 10.220.1.200 => that was the first one in my list and
  security_groups.# = 2
  security_groups.1563906976 = sg-9a10b6fd
  security_groups.973784294 = sg-9910b6fe
  source_dest_check = true
  subnet_id = subnet-e2c50e86
  tags.# = 0

Can you help ?

[certnew]

Need this please

[arq-anthonyw]

Yep I just hit this while attempting to build a SQL availability group.

[ArturZurawski]

Yes, same for me. It gets a bit weird:

resource "aws_network_interface" "interface" {
...
  private_ips = [
     "10.113.0.4",
     "10.113.0.5"
  ]
}

produces:

private_ips.3103226348:     "" => "10.113.0.5"
private_ips.3488631162:     "" => "10.113.0.4"

but

resource "aws_network_interface" "interface" {
...
  private_ips = [
     "10.113.0.4",
     "10.113.0.6"
  ]
}

gives correct order:

private_ips.3488631162:     "" => "10.113.0.4"
private_ips.570304598:      "" => "10.113.0.6"

[foax]

I just discovered this behaviour when creating a number of private IPs on an ENI.

The ordering of private IP addresses in the Terraform state for aws_network_interface should match the ordering that is returned by the AWS API.

I ran a test to demonstrate how the ordering of the private_ips list differs from that returned by the AWS API. In this example I am using the private_ips_count argument to the aws_network_interface resource, to let AWS auto-assign IPs from the selected subnet.

resource "aws_network_interface" "test" {
  subnet_id         = "${local.subnet_id}"
  security_groups   = ["${local.sg_id}"]
  private_ips_count = 5
}

output "eni_list_of_private_ips" {
  value = "${aws_network_interface.test.private_ips}"
}

Check ordering of private_ips list in Terraform state and output:

$ terraform state show aws_network_interface.test

id                         = eni-fffffffffffffffff
attachment.#               = 0
description                = 
private_dns_name           = 
private_ip                 = 10.80.161.65
private_ips.#              = 6
private_ips.1075712318     = 10.80.162.20
private_ips.1185779547     = 10.80.161.65
private_ips.1590535948     = 10.80.162.115
private_ips.2028994980     = 10.80.163.205
private_ips.2489499117     = 10.80.161.153
private_ips.3399039115     = 10.80.160.6
private_ips_count          = 5
security_groups.#          = 1
security_groups.000000000 = sg-ffffffff
source_dest_check          = true
subnet_id                  = subnet-ffffffff
tags.%                     = 0

$ terraform output eni_list_of_private_ips

10.80.162.20,
10.80.161.65,
10.80.162.115,
10.80.163.205,
10.80.161.153,
10.80.160.6

Compare with ordering returned by AWS API:

$ aws ec2 describe-network-interfaces --network-interface-ids eni-fffffffffffffffff

{
    "NetworkInterfaces": [
        {
            "NetworkInterfaceId": "eni-fffffffffffffffff",
            "InterfaceType": "interface",
            "Groups": [
                {
                    "GroupName": "default",
                    "GroupId": "sg-ffffffff"
                }
            ],
            "SubnetId": "subnet-ffffffff",
            "PrivateIpAddress": "10.80.161.65",
            "PrivateIpAddresses": [
                {
                    "PrivateIpAddress": "10.80.161.65",
                    "Primary": true
                },
                {
                    "PrivateIpAddress": "10.80.162.115",
                    "PrivateDnsName": "",
                    "Primary": false
                },
                {
                    "PrivateIpAddress": "10.80.162.20",
                    "PrivateDnsName": "",
                    "Primary": false
                },
                {
                    "PrivateIpAddress": "10.80.160.6",
                    "PrivateDnsName": "",
                    "Primary": false
                },
                {
                    "PrivateIpAddress": "10.80.161.153",
                    "PrivateDnsName": "",
                    "Primary": false
                },
                {
                    "PrivateIpAddress": "10.80.163.205",
                    "PrivateDnsName": "",
                    "Primary": false
                }
            ]
        }
    ]
}

The ordering of private IPs in the Terraform state does not match the ordering returned by the AWS API. The order returned by AWS API matches what is displayed in the AWS console, and keeps the private IP address as the first IP in the list. The order of private_ips in the Terraform state is based on the number assigned to each private_ip entry (presumably some form of hash that Terraform performs).

Change interface count from 5 to 10:

resource "aws_network_interface" "test" {
  subnet_id         = "${local.subnet_id}"
  security_groups   = ["${local.sg_id}"]
  private_ips_count = 10
}

Check ordering in Terraform state and output again:

$ terraform state show aws_network_interface.test

id                         = eni-fffffffffffffffff
attachment.#               = 0
description                = 
private_dns_name           = 
private_ip                 = 10.80.161.65
private_ips.#              = 11
private_ips.1075712318     = 10.80.162.20
private_ips.1185779547     = 10.80.161.65
private_ips.1459346822     = 10.80.162.244
private_ips.1590535948     = 10.80.162.115
private_ips.2028994980     = 10.80.163.205
private_ips.2193114631     = 10.80.163.74
private_ips.2489499117     = 10.80.161.153
private_ips.3399039115     = 10.80.160.6
private_ips.565978087      = 10.80.163.0
private_ips.593506421      = 10.80.160.111
private_ips.7793821        = 10.80.161.105
private_ips_count          = 10
security_groups.#          = 1
security_groups.0000000000 = sg-ffffffff
source_dest_check          = true
subnet_id                  = subnet-ffffffff
tags.%                     = 0

$ terraform output eni_list_of_private_ips

10.80.161.105,
10.80.163.0,
10.80.160.111,
10.80.162.20,
10.80.161.65,
10.80.162.244,
10.80.162.115,
10.80.163.205,
10.80.163.74,
10.80.161.153,
10.80.160.6

5 new IPs have been added, but have been scattered through the existing list. The order of the existing IPs is maintained, but the new IP addresses are spread through the list.

However, the ordering of the private IPs returned by the AWS API is much more predictable. The existing list of private IPs are at the start of the list, with the new 5 IPs added to the end of the list:

$ aws ec2 describe-network-interfaces --network-interface-ids eni-fffffffffffffffff

{
    "NetworkInterfaces": [
        {
            "NetworkInterfaceId": "eni-fffffffffffffffff",
            "PrivateIpAddress": "10.80.161.65",
            "Groups": [
                {
                    "GroupName": "default",
                    "GroupId": "sg-ffffffff"
                }
            ],
            "SubnetId": "subnet-ffffffff",
            "InterfaceType": "interface",
            "PrivateIpAddresses": [
                {
                    "Primary": true,
                    "PrivateIpAddress": "10.80.161.65"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.162.115"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.162.20"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.160.6"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.161.153"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.163.205"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.163.0"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.162.244"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.161.105"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.163.74"
                },
                {
                    "PrivateDnsName": "",
                    "Primary": false,
                    "PrivateIpAddress": "10.80.160.111"
                }
            ]
        }
    ]
}

The problem I can foresee is that if I assign elastic IPs to these private IPs like this:

variable "eip_count" {
    default = "5"
}

resource "aws_eip" "test" {
    count = "${var.eip_count}"
    vpc   = "true"
}

resource "aws_eip_association" "test" {
    count                = "${var.eip_count}"
    allocation_id        = "${aws_eip.test.*.id[count.index]}"
    network_interface_id = "${aws_network_interface.test.id}"
    private_ip_address   = "${aws_network_interface.test.private_ips[count.index]}"
}

If the number of private IPs on the ENI increases, then the list of elastic IPs aws_eip.test.*.id will likely no longer match the existing elastic IP associations.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[thatsmydoing]

In practice, the API does seem to return the private IPs in a specific order. The web console also seems consistent with it. That said, nowhere does it say that they will be returned in a specific order. The EC2 API even goes to call it privateIpAddressesSet https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_NetworkInterface.html

That said, in Cloudformation, one of the outputs for a NetworkInterface is the list of private ip addresses https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-network-interface.html#aws-resource-ec2-network-interface-return-values and we've certainly used this for things like associating Elastic IPs to the addresses just based on order. Porting from Cloudformation to terraform does not work for that particular case.

On the other hand, changing the type in terraform from set to list would also certainly break a lot of things so I'm not really sure what a good option is.

A related PR is also still open #1672 with some discussion related to ordering but it looks like they're moving towards an alternate solution to the problem.