I would like to provision KOTS and kURL components using an AMI to bake these components in using something like Packer or Cloudformation.
But the one main issue I’m running into is the IP of the VM at the time of the AMI build is the IP that gets hardcoded in all the K8s configs/addons. This causes issue because when the AMI is used on a node with a different IP these configs do not get updated.
This is an excellent question, a lot of our vendors do this, may not be AMI exactly but OVA and the like.
Sadly, there is no easy way to “update” an IP in a K8s, not saying it’s impossible but its not straightforward, we’ve tried to do it once and were at it for several weeks and at the end we weren’t really sure if we “updated” all the right places with the new IP.
So how do the other folks do it? You can “semi-bake” the AMI. In other words, use the userdata and cloud-init to run the k8s.kurl.sh bash script at VM launch that way K8s will get configured with the IP of the VM that it’s being launched on. As you can imagine one of the downsides to this is it will take the VM 5-10 mins to come up based on the number of addons you have to install.