kURL: How can I bake an AMI?

Supporting your customers
1

Hello there,

I would like to provision KOTS and kURL components using an AMI to bake these components in using something like Packer or Cloudformation.

But the one main issue I’m running into is the IP of the VM at the time of the AMI build is the IP that gets hardcoded in all the K8s configs/addons. This causes issue because when the AMI is used on a node with a different IP these configs do not get updated.

How can I handle such a scenario?

Thanks!

2

This is an excellent question, a lot of our vendors do this, may not be AMI exactly but OVA and the like.

Overview

Sadly, there is no easy way to “update” an IP in a K8s, not saying it’s impossible but its not straightforward, we’ve tried to do it once and were at it for several weeks and at the end we weren’t really sure if we “updated” all the right places with the new IP.

Semi-bake

So how do the other folks do it? You can “semi-bake” the AMI. In other words, use the userdata and cloud-init to run the k8s.kurl.sh bash script at VM launch that way K8s will get configured with the IP of the VM that it’s being launched on. As you can imagine one of the downsides to this is it will take the VM 5-10 mins to come up based on the number of addons you have to install.

(Run commands on your Linux instance at launch - Amazon Elastic Compute Cloud)

Cloudformation Example

Here is a cloudformation example we worked on during a hackweek and the interesting bit for you is probably line 151.

Public Roadmap

All that being said this is one of the top items on our roadmap, here is the public url Trello

3

We have something that works in (REPLATS-514) Update kurl image to Redhat 8.4 by jpartlow · Pull Request #528 · puppetlabs/puppetlabs-packer · GitHub, but only for a specific set of components. Some of the challenges are:

  • CNI needs to be reset in a way that causes it to cleanup and recreate a bunch of IP mappings
  • You need to rename the node to match the new hostname if you want to be able to rerun the kURL script, which requires recreating any PVCs that were created as part of startup.
  • When creating the image, you want kubelet to be stopped so the image shrinking step doesn’t cause kubelet to think it’s running out of storage and start deleting images.