10

On NixOS is is easy to set up Kubernetes by a single line of config:

services.kubernetes.roles = ["master" "node"];

This installs both the master and node components on the local system and therefore creates a nice little working local kubernetes "cluster".

If I want to set up a "real" cluster I need to install it over multiple hosts, but I'm not sure about the intended way to connect them.

If I install only the master components on one host and only the node components on another node, how do I tell the node where to find its master?

There are quite a few configuration options, but I'm not sure how to use them correctly. Is anyone aware of some example setup?

michas
  • 25,361
  • 15
  • 76
  • 121

3 Answers3

1

Have a look at the latter part of Jaka Hudoklin/offlinehacker's NixCon '15 presentation about Kubernetes on NixOS at GateHub. It has an example configuration that configures docker to use a bridge interface. You can then use openvswitch to link the networks together.

Robert Hensing
  • 6,708
  • 18
  • 23
  • 2
    I am aware of that presentation. However he is using master and node at the *same* host exactly as I wrote above. The question is how I run master and node on *different* hosts. – michas Apr 22 '18 at 10:47
  • Hold on, he uses openvswitch for connecting those virtual networks. That's why you need to configure the bridge. I'm sorry, how is this not helpful? – Robert Hensing Apr 23 '18 at 08:59
  • How do you do that on different nodes? – MrR Oct 01 '18 at 17:42
  • @RobertHensing networking nodes is not the issue but how to distribute the shared cluster configuration to all the nodes. – itorres Jun 25 '19 at 19:05
1

I'm currently working to automate Kubernetes deployment with NixOS / NixOps. It works quiet well with multiple local VirtualBox nodes. Regarding AWS integration I still have to fix few things. Then I will try to integrate with other cloud providers.

You can have a look to this repository: NixOps Kubernetes. Do not hesitate to fork and help me improve it.

Thomas Pham
  • 11
  • 4
  • Thanks for sharing. A more verbose readme would be helpful. Tell what your are trying to achieve and what those commands are good for. They may be clear to you, but especially for people not too deep into AWS might be a bit puzzled. - You can find my repository at https://github.com/michas2/nix-qemu-kubeboot – michas May 20 '18 at 20:36
  • Sure, I'll improve the documentation. It's very early stage for the moment. I just wanted to shared it with you in case you need it asap. Thanks for sharing your repo. There is also interesting ideas for what I will need for scaling dynamically the cluster. I'll keep you posted as soon as I updated the doc. Thanks. – Thomas Pham May 21 '18 at 08:55
  • I think I understand most of your code, but how exactly do you connect the different nodes to a single cluster? What is the minimal change needed in my code to make the workers talk to the master to form an actual k8s cluster? – michas May 21 '18 at 09:35
  • There are several things that helped me understand all the component that compose a kubernetes cluster. You can have a look to the repository of Kelsey Hightower: https://github.com/kelseyhightower/kubernetes-the-hard-way. But regarding your code, maybe what's missing is how etcd client on the worker nodes connect to the etcd server on the masters to get updated on the configuration and topology of the cluster. An other thing maybe is also the need of certificates so kubernetes components are authorized to talk to each others. – Thomas Pham May 21 '18 at 10:08
  • The nixos code looks like it already handles most of the technical details. (After all you get a fully configured local cluster with just one line.) It feels like I'm missing only another config line to trigger the nix-magic to connect things in the right way. (Something like `master=$ip` or something.) – michas May 21 '18 at 10:14
  • try: `services.kubernetes.etcd.servers = [ "http://{ip_of_the_master}:2379" ];` and `environment.variables. ETCDCTL_PEERS = [ "http://{ip_of_the_master}:2379" ];` – Thomas Pham May 21 '18 at 10:36
  • Btw, I was inspired by their nix tests to do my integration. Have a look to https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/kubernetes/base.nix and I updated my repo with basic documentation. – Thomas Pham May 23 '18 at 05:28
  • Oh, I missed that one. The `mkKubernetesMultiNodeTest` looks pretty much exactly like the think I am looking for. Thanks! – michas May 23 '18 at 07:09
-1

Have you checked Kubeadm tool? You can check it out at - https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Himadri Ganguly
  • 715
  • 4
  • 11
  • 31