1

I deployed k8s cluster on bare metal using terraform following this repository on github

Now I have three nodes:

ewr1-controller, ewr1-worker-0, ewr1-worker-1

Next, I would like to run terraform apply and increment the worker nodes (*ewr1-worker-3, ewr1-worker-4 ... *) while keeping the existing controller and worker nodes. I tried incrementing the count.index to start from 3, however it still overwrites the existing workers.

resource "packet_device" "k8s_workers" {
  project_id       = "${packet_project.kubenet.id}"
  facilities       = "${var.facilities}"
  count            = "${var.worker_count}"
  plan             = "${var.worker_plan}"
  operating_system = "ubuntu_16_04"
  hostname         = "${format("%s-%s-%d", "${var.facilities[0]}", "worker", count.index+3)}"
  billing_cycle    = "hourly"
  tags             = ["kubernetes", "k8s", "worker"]
}

I havent tried this but if I do 

terraform state rm 'packet_device.k8s_workers'

I am assuming these worker nodes will not be managed by the kubernetes master. I don't want to create all the nodes at beginning because the worker nodes that I am adding will have different specs(instance types).

The entire script I used is available here on this github repository. I appreciate it if someone could tell what I am missing here and how to achieve this.

Thanks!

old_timer
  • 69,149
  • 8
  • 89
  • 168
Zstack
  • 4,046
  • 1
  • 19
  • 22
  • 1
    You need a **deployment strategy** and it must be implemented by your **provider** e.g. Auto Scaling Groups for AWS, https://medium.com/@endofcake/using-terraform-for-zero-downtime-updates-of-an-auto-scaling-group-in-aws-60faca582664 – Jonas Nov 13 '19 at 20:01
  • Can you try to `disallow deletion of the resource` that has been mentioned here: https://coderbook.com/@marcus/prevent-terraform-from-recreating-or-deleting-resource/ By adding this to your file: lifecycle { prevent_destroy = true } } – acid_fuji Nov 15 '19 at 10:16
  • @acid_fuji I tried but that doesn't for my situation, the problem is not just keeping the worker nodes but adding them to the existing k8s cluster. – Zstack Nov 15 '19 at 15:07

2 Answers2

2

Node resizing is best addressed using an autoscaler. Using Terraform to scale a nodepool might not be the optimal approach as the tool is meant to declare the state of the system rather than dynamically change it. The best approach for this is to use a cloud auto scaler. In bare metal, you can implement a CloudProvider interface (like the one provided by cloud such as AWS, GCP, Azure) as described here

After implementing that, you need to determine if your K8s implementation can be operated as a provider by Terraform, and if that's the case, find the nodepool autoscaler resource that allows the autoscaling.

Wrapping up, Terraform is not meant to be used as an autoscaler given its natures as a declarative language that describes the infrastructure.

The autoscaling features in K8s are meant to tackle this kind of requirements.

acid_fuji
  • 6,287
  • 7
  • 22
  • The cloud provider in this case doesn't support autos-caller, I have been doing that on other cloud providers. Here I was looking for way to get around this problem by modifying the terrafrom script. – Zstack Nov 18 '19 at 17:50
1

I solved this issue by modifying and removing modules, resources from the terraform.state: manually modifying it and using terraform state rm <--->.

  • Leaving out (remove state) the section that I want to keep as they are.

  • Modifying the sections in terraform.state that I want change when new servers are added.

  • Incrementing the counter to add new resources, see terraform interpolation.

I am using bare-metal cloud provider to deploy k8s and it doesn't support k8s HA or VA autos-calling. This is may not be optimal solution as other have pointed out but if it is not something you need to do quite often, terraform can do the job albeit the hardway.

Zstack
  • 4,046
  • 1
  • 19
  • 22