Major refactoring release with Kubernetes v1.3.6

This release is a complete reboot compared to earlier releases. Now, docker-multinode is used as the base, along with the official flannel, hyperkube, dashboard and kube-dns docker images. I've added arm, arm64 and ppc64le variants for the most important Kubernetes images, so now Kubernetes releases all binaries and images automatically in their release process.

Everything is written in line with my multi-platform proposal: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multi-platform.md

It's also possible to setup Kubernetes on ARM on Pine64 easily with the deb package!

Supported boards/OSes:

• HypriotOS v1.0.1
  • Raspberry Pi 1, 2, 3
  • Note: no reboot required! (DNS resolving on host won't work without a reboot though)
• Raspbian Jessie
  • Raspberry Pi 1, 2, 3
• Arch Linux ARM
  • The boards Arch Linux ARM supports should work, but haven't been tested
  • But here the docker pull bug is really messing things up. I haven't found a solution to that problem, see the bug referenced below.
• Debian Jessie
  • Pine64
  • Here again, the docker pull bug is really messing things up. I haven't found a solution to that problem, see the bug referenced below.
• Debian Jessie/Ubuntu Trusty on Scaleway
  • Works when the docker pull bug is fixed.

Changes

• Upgraded to Kubernetes v1.3.6 and flannel v0.6.1
• Using official Kubernetes binaries and docker images
  • They are built in line with my multi-platform proposal: kubernetes/kubernetes#26863
• Most of the code is "official" from kube-deploy/docker-multinode
• All binaries and images are now cross-compiled from amd64, no more releasing from Raspberry Pis!
• Using HypriotOS v1.0.1!
• It's fully possible to add amd64 nodes seamlessly with docker-multinode
  • Then you can control which nodes a pod should land on with the beta.kubernetes.io/arch label: kubernetes/kubernetes#23684
• Made the dns and the dashboard addons mandatory, they can't and shouldn't be disabled as they're required components of the core.
• The earlier skydns component has been replaced with the new kubedns component that is more integrated with Kubernetes and more efficent.
• Raspbian is supported out-of-the-box, just download the deb package and install on your Pi!
• Pine64 is now supported as a platform.
• Upgraded the addons:
  • dashboard to v1.1.1
  • kubedns to 1.5 (manifest version v17.1)
  • registry to v2.5.0
  • heapster to v1.2.0-beta.2
  • influxdb to v0.13.0
  • grafana to v3.1.1
• Total refactor of kube-config and of the whole project; as much as possible official Kubernetes code is used, which means the configuration for v0.8.0 is more generic than v0.7.0 which was more specialized.
• Added the helm package manager at version v2.0.0-alpha.3
• kube-registry-proxy is now added to the registry addon which makes every node expose the registry at localhost:5000
• Possible to use --containerized by adding export USE_CONTAINERIZED=true to /etc/kubernetes/k8s.conf
• Renamed kube-systemd to docker-multinode and the generic systemd OS to the generic debian OS
• Caveats:
  • Reboots aren't working nor supported. (Added to roadmap)
  • Arch Linux ARM, the Pine64 and Scaleway suffers from the docker pull mtu bug docker/docker#22635

Downloads

Kubernetes binaries:

# Generic
wget https://storage.googleapis.com/kubernetes-release/release/${VERSION}/bin/linux/${GOARCH}/${BINARY}

# Examples
wget https://storage.googleapis.com/kubernetes-release/release/v1.3.6/bin/linux/arm/kubectl
wget https://storage.googleapis.com/kubernetes-release/release/v1.3.6/bin/linux/arm/hyperkube
wget https://storage.googleapis.com/kubernetes-release/release/v1.3.6/bin/linux/arm/kubelet
wget https://storage.googleapis.com/kubernetes-release/release/v1.3.6/bin/linux/arm64/hyperkube

Install the latest .deb package:

wget https://github.com/luxas/kubernetes-on-arm/releases/download/v0.8.0/docker-multinode.deb
dpkg -i docker-multinode.deb

Roadmap for the future

I'm gonna continue to work hard on making this project unnecessary!

Roadmap

Major release with Kubernetes v1.2.0

A lot of new features are in the largest release of Kubernetes on ARM yet made. Highlights include officially built Kubernetes v1.2.0 ARM binaries, much better performance and cluster insight via the heapster addon.

Changes

• Upgrade to Kubernetes v1.2.0 (Changelog), dashboard v1.0.0, etcd v2.2.5, registry v2.3.1
• Using official binaries built from my Kubernetes PR: kubernetes/kubernetes#19769
• Added cluster monitoring! Heapster v1.0.0, influxdb v0.10.3 and grafana v2.6.0
• Changed to "native" kubelet mode. No more --containerized hack. Makes it possible to use Downward API
• Switched to host-gw instead of udp as the default flannel backend for improved performance as @larmog suggested. Also made the option FLANNEL_BACKEND in k8s.conf
• Compile flannel and registry statically. That reduces their total image size 147 MB
• Deprecated and removed luxas/raspbian in favor for plain resin/rpi-raspbian:jessie
• Added master.json to the hyperkube image so it's easy to spin up a one-node cluster as docker.md in official docs does.
• Using @hypriot's prebuilt Go tarballs for luxas/go. Will switch to official go1.6 soon.
• Added experimental RancherOS to sdcard/write.sh, but no Kubernetes rootfs is available yet
• Added sdcard/write support for Raspberry Pi 3, but yet only with armhf OSes.
• Made the storage driver easily changeable with the DOCKER_STORAGE_DRIVER in /etc/kubernetes/k8s.conf
• Changed indentation to spaces instead of tabs for the most of the files. Also trying to end all files with a newline.
• Replaced sleep based timeouts in kube-config for condition based loops, makes it faster and more reliable
• Added a debugging option in kube-config by specifying K8S_DEBUG=1 before the command.
• Fixed a HypriotOS/.deb-file issue where sudo modified the $PATH so kube-config couldn't find kubectl @DorianGray
• More reliable SD Card creation by using partprobe @DorianGray
• Lowered the nodeMonitorGracePeriod and the podEvictionTimeout as @saturnism suggested
• Changed proxying mode to iptables for better performance.
• Added a script for cross-compiling Kubernetes to ARM 64-bit on a amd64 host.
• Other minor enhancements, improvements and bug fixes are included too.

Downloads

Get the binaries:

curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.7.0/binaries.tar.gz | tar -xz -C /usr/bin

Load the images:

curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.7.0/images.tar.gz | gzip -dc | docker load

The images are also available on Docker Hub.

Install the latest .deb package:

wget https://github.com/luxas/kubernetes-on-arm/releases/download/v0.7.0/kube-systemd.deb
dpkg -i kube-systemd.deb

Stability and upgrade release

Upgrades to docker v1.10.0, and adds the loadbalancer and dashboard addons

Changes

• Add a http loadbalancer. Thanks for the help @larmog
• Upgrade to and support docker-1.10.0 only
• Kubernetes Dashboard UI added as an addon
• Do not depend on pacman, use a self-built statically linked ARMv6 docker-1.10 binary instead
• Revert to userspace proxying, since iptables proxying had some bugs in it
• Better documentation
• A worker may now reboot and come up again in some minutes
• Better UX when writing the hypriot image, caches the downloaded image for faster writes and installs unzip if not present
• Bug fixes
• Rearranged the addon manifests

Downloads

v0.6.5

Install the latest .deb package:

wget https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.5/kube-systemd.deb
dpkg -i kube-systemd.deb

Revisions of the .deb package:

• kube-systemd-1: The first revision of v0.6.5
• kube-systemd == kube-systemd-2: Second revision with Dashboard working OOTB

v0.6.2

The binaries and docker images haven't changed between v0.6.2 and v0.6.5.
Get the binaries:

mkdir k8s_binaries
curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/binaries.tar.gz | tar -xz -C k8s_binaries
ls -lh k8s_binaries

(Small exception: the service_loadbalancer binary is old there. The newest binary is in the kubernetesonarm/loadbalancer image)

Load the images:

curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/images.tar.gz | gzip -dc | docker load
docker pull kubernetesonarm/loadbalancer

All images are also available on Docker Hub.

(The grafana-server binary is just temporary hosted for the coming v0.7.0 release)

Stability and refactoring release

Better documentation, more stable .deb deployment, bugs fixed and it's now possible to edit DNS names

Changes

• Fix bugs and make the .deb file stable
• Add an experimental .tar.gz deployment for platforms that doesn't have dpkg
• Document the /etc/kubernetes/README.md better
• Refactor, fix bugs and remove unnecessary things
• Break out the DNS options cluster.local and 10.0.0.10 to /etc/kubernetes/k8s.conf
• Fixed so the master is able to collect logs from nodes @kyletravis
• Better DNS management on Arch Linux
• Use a shell for looking up paths to executables needed in .service files
• Easier to use on systemd, most of the packages is automatically installed

Downloads

v0.6.3

Install the latest .deb package:

wget https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.3/kube-systemd.deb
dpkg -i kube-systemd.deb

The .tar.gz package is experimental.

v0.6.2

The binaries and docker images haven't changed between v0.6.2 and v0.6.3.
Get the binaries:

mkdir k8s_binaries
curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/binaries.tar.gz | tar -xz -C k8s_binaries
ls -lh k8s_binaries

Load the images:

curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/images.tar.gz | gzip -dc | docker load

The images are also available on Docker Hub.

Major release with many new possibilities

Now HypriotOS is supported and you may run it anywhere (on a systemd based host) thanks to the .deb package.

Changes

• Support for Banana Pro
• .deb package deployment
• iptables proxying mode for kube-proxy should result in better performance
• docker is built statically for both ARMv6 and ARMv7. Optional to use in most cases.
• Support for HypriotOS
• Support for plain systemd OSes
• Enabled experimental Kubernetes by default, e.g. Jobs, DaemonSets
• k8s => 1.1.3, etcd => 2.2.2, flannel => 0.5.5, registry => 2.2.1
• Started to hack on mainline k8s: kubernetes/kubernetes#17981
• Renamed kube-archlinux to the more generic kube-systemd
• Small improvments and much better README
• Bug fixes
• Broke out the flannel subnet to the configuration file: /etc/kubernetes/k8s.conf supports now FLANNEL_SUBNET which defaults to 10.1.0.0/16

Downloads

Get the binaries:

mkdir k8s_binaries
curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/binaries.tar.gz | tar -xz -C k8s_binaries
ls -lh k8s_binaries

Load the images:

curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/images.tar.gz | gzip -dc | docker load

The images are also available on Docker Hub.

Install the latest .deb package:

wget https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/kube-systemd.deb
dpkg -i kube-systemd.deb

Revisions of the .deb package:

• kube-systemd-1: The first revision
• kube-systemd-2: The second revision with a bug fix for the general systemd OS
• kube-systemd == kube-systemd-3: Third revision with no README.md conflict on HypriotOS and kube-config enable-master working

Major release with many new features

This release brings many new features and bugfixes

Changes

• A new, more customizable way to write the SD Card, allows for more OSes in the future
  • Automates post-installation for cubietruck.
• Upgrade k8s => 1.1.2, flannel => v0.5.4, etcd => 2.2.1, registry => 2.2.0, go => 1.4.3
• Now it's possible to build the Kubernetes binaries with Go 1.5.1, but it's much slower so it's not default
• Add some test scripts for even more automation
• Fix the bug that makes this not run on armv6, e.g. Raspberry Pi 1
• Add windows downloads
• Now ServiceAccount secrets are working as they should
  • Make dns use ServiceAccount tokens
• kube-proxy runs in a container under kubelet on master
• Now it's possible to reboot and k8s restarts automatically
• Many small bugfixes and improvments

Downloads

Windows downloads:

• sdcard-rpi.zip: v0.6.0 SD Card image for Raspberry Pi 1. Command used: sdcard/write.sh rpi archlinux kube-archlinux
• sdcard-rpi-2.zip: v0.6.0 SD Card image for Raspberry Pi 2. Command used: sdcard/write.sh rpi-2 archlinux kube-archlinux

Get the binaries:

mkdir k8s_binaries
curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.0/binaries.tar.gz | tar -xz -C k8s_binaries
ls -lh k8s_binaries

Load the images:

curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.0/images.tar.gz | gzip -dc | docker load

The images are also available on Docker Hub.

The Banana Pro uboot file is also here, although Banana Pro isn't out-of-the-box supported. It will be in next release.

An experimental statically compiled docker daemon is uploaded here. It is compiled for both ARMv6 and ARMv7 unlike other daemons. Version 1.8.2

Mostly enhancements and some new features

Stability release, enhancements has highest priority in this release

Changes

• Enhanced the SD Card write process
• Now cubietruck is supported
• Kubernetes binaries are now downloaded from Github, this makes it also possible to use k8s out-of-docker
• Bugfixes, especially when running kube-config enable-*
• Now the installer also may download images from Github
• Added lots of new info to kube-config info
• Added kube-config upgrade for upgrading the system
• Added the sleep addon. README

Prerelease with bugfixes and some new experimental features

Mostly bugfixes are in this prerelease.
Just so may fetch the latest code.

Changes

• Added experimental support for Kube UI and Cubietruck
• Now memory accounting is enabled on RPi. No docker warnings are shown anymore.
• Updated README.md and added READMEs to the core Kubernetes images. More to come.
• Fixed the most of @nsteinmetz's bug reports. Thanks.
• Some bugfixes here and there
• Note: this is a prerelease, just for one to have newer code to hack on.

Changes:

• Added two addons: DNS and central image registry
• Fixed bugs
• Added support for Parallella (although it´s slow)
• Extended the README
• Published the Kubernetes images on Docker Hub
• Rewrite of the SD Card write process

Images on Docker Hub

Download and use the images:

curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.5.5/images.tar.gz | tar -xz
docker load -i images.tar

Get the binaries:

mkdir k8s_binaries
curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.5.5/binaries.tar.gz | tar -xz -C k8s_binaries
ls -lh k8s_binaries