2016 is over. In Christmas 2015 I did something really useful: Prepare some tech recipes based on something that is was curious about: Containers. This year I would like to repeat the experience. Next station: Kubernetes. In this years’ Christmas tech recipe I am going to cook a 5 x Raspberry Pi 3 Kubernetes cluster. Come in and take a seat. Everything is in place and ready to cook. You’ll need some hardware and 2 hours and 20 minutes of your time.

img_3755

Introduction, motivation and credits

Getting back home from one of my business trips, I was thinking about how can easily explain concepts such containers or cloud computing. As such concepts are “new”, and very abstract people tend to reject them and prefer to stay with the old fashioned application server or IIS. Manually operated (with all the cons that this implies). But cloud computing sounds like something magic… and people don’t like magic. So it is a difficult fight. So I thought about having something I can carry on my bag and travel with. Put it on a table, turn it on and demo a private cloud. Explain the concepts and show there is no magic, but a real chance to improve how applications are design, implemented and operated.

On the other hand, I’ve been working with my colleagues from everis Zaragoza in some internal stuff. Juan Larriba (@compilemymind), my dear Kubemeister, opened me the door to this exciting world called Kubernetes. Free. Open Source. From Google (And recently adopted by Microsoft in Azure under the Azure Container Service stack).

So, why not? A Kubernetes cluster in a Raspberry Pi 3 I can carry and demo all around the world.

A fast pray to Google thrown results immediately. And the result was obvious: I was not the first with that idea:

  • http://blog.kubernetes.io/2015/11/creating-a-Raspberry-Pi-cluster-running-Kubernetes-the-shopping-list-Part-1.html
  • https://medium.com/google-cloud/everything-you-need-to-know-about-the-kubernetes-raspberry-pi-cluster-2a2413bfa0fa#.aa5kqwhw2
  • https://github.com/luxas/kubernetes-on-arm
  • https://opensource.com/life/16/2/build-a-kubernetes-cloud-with-raspberry-pi
  • … and so on

So thanks to Arjen Wassikin and Ray Tsang (@saturnism) for their shopping list, thoughts and the great video posted on the kubernetes blog.
Also remarkable the of Lucas Käldström (@luxas) and its kubernetes-on-arm project, that been officially merged to the official Kubernetes release. And is also collaborating in other interesting tools like kubeadm. Don’t miss Lucas’ history in this blog post. Congratulations.

During this Christmas recipe I have been playing with several options. However, all where outdated, manual and did not work after all. The best starting point I found was  Roland Huß‘s Ansible playbook. So I pick up his work, pulled out what I though not necessary, and then implemented an automated install following the step-by-step guide from kubeadm

You can find the result (and will need to clone it) at my GitHub repository kubernetes-raspberry-pi

All said, it is time to get back to work.

Ingredients

This recipe is not cheap at all. You will need some hardware. But the result worths it:

Amount Part Price
5 Raspberry Pi 3 5 * 39 EUR
5 Micro SD Card 32 GB 5 * 11 EUR
1 WLAN Router 24 EUR
1 100 MB Switch 12 EUR
5 Micro-USB wires 5 * 1 EUR
1 Power Supply 43 EUR
3 Multi stackable case 3 * 16 EUR
1 Bolt pack 6 EUR

And some open source tools you need to install:

Name Origin URL
HypriotOS GitHub hypriot/rpi-image-builder
flash GitHub hypriot/flash
Ansible Website Ansible installation guide

What you’ll get

It is really cool, doesn’t it?

img_3758

Assembling the toy (2 hours)

You will find the assembly quite straightforward. Bolt here and there. And not so much. Only two hacks that we have applied:

  1. We have removed the switch case, made some custom holes in the chasis and mount it with bolts, like the rest of the pi. This way it fits perfect within the structure.
  2. The have followed a similar process with the WLAN router (the small box at the top). Unmount, hole, bolt but, in this case, close the case again. The holes in the board where not suitable to fix it directly. So we preferred to make the hole in the case and left it like this.

Both router and switch are tight mounted.

img_3756

Installing Kubernetes (20 minutes)

Important: Please not that this playbook uses kubeadm. Kubeadm is currently in alpha state. I will try to periodically check that everything still works as the kubeadm team release the final version.

After days of research, try/fail, surf and so on, I finally found a clean and fast way to install Kubernetes on ARM: kubeadm. I made a first manual installation to check that everything was ok and the installation was smooth. And it was. So as mentioned before, I took Roland Huß‘s Ansible playbook and adjusted it to work with kubeadm. So,

Flash all SD cards

First of all, download the latest hypriotos image from GitHub. One downloaded, open your Terminal ,cd to where te image is and, using hypriot’s flash utility, flash all 5 SD cards with HypriotOS operating system:

In 50 seconds more o less, one SD is ready.

Now, put all the SD cards on the Pis, and turn it on.

Setup the router

The router is very important. For demo purposes, I have configured it in Hotspot mode:

  1. It connects to any existing WIFI network, and shares it with all computers connect either by LAN or WLAN
  2. It creates a new WIFI network, so you can connect to the cluster with your computer’s or your mobile phone WIFI.
  3. It acts as DHCP server

To setup it, connect to it (my original WIFI Network was TP-LINK_3224) and browse http://tplinkwifi.net (or http://192.168.0.1)

I strongly recommend you to, first of all, reconfigure the DHCP server to point out of the normal LAN IP range. I chose 172.16.X.X:

  1. Go to DHCP > DHCP Settings, an fill in the settings as follows:

dhcp-setup1

2. With all Pi flashed and connected to the network, go to DHCP > Client list and take note of their MAC addresses:

dhcp-setup3

3. Go to DHCP > Address reservation, and set an static (Reserved) IP address to each Pi. (The IP is important, because the Ansible playbook makes 172.16.0.10 = master, 172.16.0.11 node1… and so on).

dhcp-setup2

With the DHCP in place, is time for the Hotpost. The router includes a Quick Setup (that probably you’ve already completed in Router mode in first place). Anyway, you can rerun this setup again from the Quick Setup menu:

 

  1. In operation mode. Choose “Hotspot router”

hotspot-setup1

2. Dynamic IP will be the most common if your main network has a DCHP server.

hotspot-setup2

3. Look for the WIFI network you want to connect the hotspot to. This will be your gateway to Internet:

hotspot-setup3

4. Fill in the hotspot WIFI information (the network with internet you want to connect to)

hotspot-setup4

5. Ready to go (you may also want to change the Wireless Network Name (SSID) you share. It is done in the “Wireless” section):

hotspot-setup5

Reboot all the Raspberry Pi, so the DHCP assigns the newly defined static IP, and we are ready to deploy the software (You may check the IP is properly configured getting back to DHCP > DHCP Client List).

Getting the Ansible playbook

You should download, fork or whatever (pull requests welcome!)  from https://github.com/sesispla/kubernetes-raspberry-pi :

Now, copy the following files:

  • hosts is the file from Ansible will read the Pi address to configure
  • config.yml includes some configuration values I moved out the playbooks to make things easier

Edit hosts with your favourite text editor:

Under “Pis”, set all your Raspberry Pi devices you want to configure.
Under “Master”, set one (and only one) Pi that will act as master.
Under “Nodes”, set all the remaining Pi that will act as cluster nodes.

Important: Please note that I am using DNS name instead of IP addresses. I created 5 DNS entries in my /etc/hosts files, targeting every hostname to one of the static IP addresses.

My /etc/hosts file looks like this:

Finally, open and edit the config.yml file:

If you followed all steps and created a DNS entry in /etc/hosts this will be fine for you. Otherwise, please change your master ip/hostname.

Setting up all the Pis

We are almost at the end. Now, execute the playbook that makes all the Raspberry Pi base setup. It will take a while, but is worth it. You can check all the actions this playbook:

Please, note that as time I made this setup node 4 was not ready yet. I ran later all the setup with all 5 nodes.

Creating the Kubernetes master

Never has been that easy:

Internally, it performs a kubeadm init and install flannel, as described in the kubeadm documentation

Joining the nodes

Again, very easy:

And that’s all! Now we have a Kubernetes cluster up and running, with 1 master and X nodes (3, 4… or whatever, no matter the number the script will join them all)

Check the nodes

We might run a simple check with kubectl:

And, of course, deploy your first application!

With this information, you can open your browser, and navigate to http://NODE:NodePort (In my case, http://node3.cluster.local:32651)

rpi-busybox-httpd

We’re done with the recipe!

OPTIONAL: Connecting remotely using kubectl in your machine

This playbook also copies the kubectl config file to “./kubeconfig/clustername/etc/kubernetes”. You probably would like to copy it to ${HOME}/.kube/config
This will make kubectl target to this cluster by default.

OPTIONAL: Using kubectl proxy to access Kubernetes Dashboard

The “master” playbook include the “Dashboard” role, that automatically install the Kubernetes Dashboard for you. If you want to access to it, you’ll need to have kubectl configured in your machine, and launch:

Now, open surf to http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=default

The Dashboard should appear without problems:

dashboard

Thank you so much for your time.

Next station. Cross-compile .NET Core to run in Kubernetes in ARM 🙂