bug(AL2023): $Base ami created but instance not booting up as networking fails for IMDS calls

[Tavisca-Ashish-Sharma]

Summary:
On AL2023 (and CIS-hardened) AMIs, nodeadm-config.service is currently installed with Before=cloud-init.service. This causes nodeadm-config to run before cloud-init brings up networking/DHCP, so attempts to reach IMDS (169.254.169.254 / fd00:ec2::254) fail with "Network is unreachable" and node bootstrap fails.

Repro:

1. Use current amazon-eks-ami AL2023 build (build from repo).
2. Boot instance in a VPC (no special proxy).
3. Observe cloud-init logs: repeated HTTPConnectionPool(host='169.254.169.254', ...): Network is unreachable messages.
4. nodeadm-config.service fails because IMDS and network were not ready.

Root cause:
Unit ordering: nodeadm-config runs before cloud-init and before network-online.target. On AL2023 the network is initialized by cloud-init (and CIS-hardening may delay/alter network startup), so nodeadm-config must wait for network-online.

[mselim00]

Hey @Tavisca-Ashish-Sharma, can you share more details on the reproduction? The steps described are exercised by our tests, including the CI workflows we run on PRs against this repository.

The AMIs built from this repository also aren’t CIS-hardened by default though, is this perhaps intended for https://github.com/awslabs/security-hardened-amis-for-eks?

[Tavisca-Ashish-Sharma]

Hello @mselim00 ,
Steps to reproduce -

1. Take CIS Hardened AL2023 public image (to be used as base image to bake the ami)
2. Use the latest code from the repo
3. Build the ami (no customizations done in the upstream code)
4. Try to launch a instance with the created ami (for testing purpose)
5. It fails wit following error - cloud-init logs: repeated HTTPConnectionPool(host='169.254.169.254', ...): Network is unreachable messages.

Observations -
We are using CIS hardened images and nodeadm-config process runs before cloud-init. At this point networking is not setup so the calls to IMDSV2 endpoints for fetching EC2 instance metadata fails with above error.
Earlier we were using same CIS hardened images for Al2 images baking but it works fine as there is no nodeadm-config service in case of Al2

[mselim00]

Thanks for the steps.

I just went to try this out myself, only to discover a subscription is required for access to those images. Do you think you could share some logs instead?

Particularly sudo journalctl -u nodeadm-* (or specifically, nodeadm-config, nodeadm-run, and nodeadm-boot-hook) as well as sudo cat /var/log/cloud-init.log. Feel free to redact anything you feel is not relevant. A last working commit would also be helpful for narrowing down the change.

Also, full disclosure, this pattern is still not something we officially support. We can try to understand and address this case, if possible, but there is no guarantee there might not be some other incompatibility in the future. I would still strongly recommend using an alternative build source that documents support for this.

[Tavisca-Ashish-Sharma]

Hello @mselim00 ,
Thanks for helping . I have narrowed down the issue and able to successfully boot up an instance . Sharing all observations below. Kindly help me out in case you feel this is wrong approach .

Root cause of issue -

1. We were using public CIS hardened image ( ami-030cc88fead9b0eda ) as vanilla ami to bake eks related modules from above repo.
2. No modifications done to original code used in this repo .
3. Tried to boot up an instance with the baked ami but Instance reachability checks failed with continous errors of cloud-init logs: repeated HTTPConnectionPool(host='169.254.169.254', ...): Network is unreachable messages.
4. Upon further investigation i found a cleanup.sh script in the repo that is cleaning up some network related files . Upon skipping cleanup of this files the instance successfully boots up .
5. This means when nodeadm run happens before cloud-init it is not able to get network connection as files were cleaned up .

I am not sure whats the best practice to handle this scenario apart from skipping cleanup of these files and how other people are handling this issue .

Please suggest how to handle this

[mselim00]

If it works for you, that should be fine. That script is mainly to limit bloat from package caches and such as well as stopping unnecessary data like machine id propagating from the build instance (but most of that stuff should be overwritten on boot anyway)

[Tavisca-Ashish-Sharma]

Thanks @mselim00 . Appreciate your help