Deploying on bare metal¶

Symplegma supports deploying on bare metal. It is actually the main use case.

symplegma-os_bootstrap supports bootstrapping python on Flatcar Linux but should also work with any OS manageable by Ansible as it installs binaries from sources and does not depends on distribution package manager. If you are using another distribution, just make sure python3-dev python3-pip or python-dev python-pip are installed and that the following kernel modules can be loaded.

---
pip_python_coreos_modules:
  - httplib2
  - six

override_system_hostname: true

bootstrap_python: false

kernel_modules:
  - overlay
  - br_netfilter
  - ip_vs
  - ip_vs_rr
  - ip_vs_wrr
  - ip_vs_sh

kernel_parameters:
  - net.bridge.bridge-nf-call-iptables
  - net.bridge.bridge-nf-call-arptables
  - net.bridge.bridge-nf-call-ip6tables
  - net.ipv4.ip_forward

bin_dir: /opt/bin

sysctl_file_path: "/etc/sysctl.d/99-kubernetes.conf"
module_load_path: "/etc/modules-load.d/99-kubernetes.conf"

etcd_version: 3.5.4
etcd_release_url: https://github.com/etcd-io/etcd/releases/download/v{{ etcd_version }}/etcd-v{{ etcd_version }}-linux-amd64.tar.gz
etcd_release_dir: /opt/etcd

jq_version: 1.6
jq_release_url: https://github.com/stedolan/jq/releases/download/jq-{{ jq_version }}/jq-linux64

You can contribute to the bootstrap role here

Requirements¶

Git clone Symplegma main repository:

git clone https://github.com/clusterfrak-dynamics/symplegma.git

Fetch the roles with ansible-galaxy:

ansible-galaxy install -r requirements.yml

Preparing inventory¶

Simply copy the sample inventory to another folder with the desired cluster name:

cp -ar inventory/ubuntu inventory/$CLUSTER_NAME

Create an inventory in a compatible format, for example in inventory/$CLUSTER_NAME/hosts file:

k8s-master-1.clusterfrak-dynamics.io
k8s-worker-1.clusterfrak-dynamics.io
k8s-worker-2.clusterfrak-dynamics.io
k8s-worker-3.clusterfrak-dynamics.io
k8s-worker-4.clusterfrak-dynamics.io

[master]
k8s-master-1.clusterfrak-dynamics.io

[node]
k8s-worker-1.clusterfrak-dynamics.io
k8s-worker-2.clusterfrak-dynamics.io
k8s-worker-3.clusterfrak-dynamics.io
k8s-worker-4.clusterfrak-dynamics.io

Your directory structure should be the following:

tree -I 'roles|contrib|docs|scripts|sample'
.
├── README.md
└── symplegma
    ├── CODEOWNERS
    ├── LICENSE
    ├── README.md
    ├── ansible.cfg
    ├── inventory
    │   └── $CLUSTER_NAME
    │       ├── group_vars
    │       │   └── all
    │       │       └── all.yml
    │       ├── host_vars
    │       └── hosts
    ├── kubeconfig
    │   └── $CLUSTER_NAME
    │       └── admin.conf
    ├── mkdocs.yml
    ├── requirements.yml
    ├── symplegma-init.yml
    ├── symplegma-reset.yml
    └── symplegma-upgrade.yml

Configuring the cluster¶

Cluster configuration is done in inventory/$CLUSTER_NAME/group_vars/all/all.yml:

---
bootstrap_python: false
# Install portable python distribution that do not provide python (eg.
# coreos/flatcar):
# bootstrap_python: true
# ansible_python_interpreter: /opt/bin/python

ansible_ssh_user: ubuntu

ansible_ssh_common_args: '-o StrictHostKeyChecking=no'
# To use a bastion host between node and ansible use:
# ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o ProxyCommand="ssh -o StrictHostKeyChecking=no -W %h:%p -q ubuntu@{{ ansible_ssh_bastion_host }}"'
# ansible_ssh_bastion_host: __BASTION_IP__

kubeadm_version: v1.24.1
kubernetes_version: v1.24.1
# If deploying HA clusters, specify the loadbalancer IP or domain name and port
# in front of the control plane nodes:
# kubernetes_api_server_address: __LB_HOSTNAME__
# kubernetes_api_server_port: __LB_LISTENER_PORT__

bin_dir: /usr/local/bin
# Change default path for custom binary. On OS with immutable file system (eg.
# coreos/flatcar) use a writable path
# bin_dir: /opt/bin

# Customize API server
kubeadm_api_server_extra_args: {}
kubeadm_api_server_extra_volumes: {}

# Customize controller manager scheduler
# eg. to publish prometheus metrics on "0.0.0.0":
# kubeadm_controller_manager_extra_args: |
#   address: 0.0.0.0
kubeadm_controller_manager_extra_args: {}
kubeadm_controller_manager_extra_volumes: {}

# Customize scheduler manager scheduler
# eg. to publish prometheus metrics on "0.0.0.0":
# kubeadm_scheduler_extra_args: |
#   address: 0.0.0.0
kubeadm_scheduler_extra_volumes: {}
kubeadm_scheduler_extra_args: {}

# Customize Kubelet
# `kubeadm_kubelet_extra_args` is to be used as a last resort,
# `kubeadm_kubelet_component_config` configure kubelet wth native kubeadm API,
# please see
# https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for
# more information
kubeadm_kubelet_component_config: {}
kubeadm_kubelet_extra_args: {}


# Customize Kube Proxy configuration using native Kubeadm API
# eg. to publish prometheus metrics on "0.0.0.0":
# kubeadm_kube_proxy_component_config: |
#   metricsBindAddress: 0.0.0.0
kubeadm_kube_proxy_component_config: {}

# Additionnal subject alternative names for the API server
# eg. to add aditionnals domains:
# kubeadm_api_server_cert_extra_sans: |
#   - mydomain.example.com
kubeadm_api_server_cert_extra_sans: {}

kubeadm_cluster_name: symplegma

# Do not label master nor taint (skip kubeadm phase)
# kubeadm_mark_control_plane: false

# Enable systemd cgroup for Kubelet and container runtime
# DO NOT CHANGE this on an existing cluster: Changing the cgroup driver of a
# Node that has joined a cluster is strongly not recommended. If the kubelet
# has created Pods using the semantics of one cgroup driver, changing the
# container runtime to another cgroup driver can cause errors when trying to
# re-create the Pod sandbox for such existing Pods. Restarting the kubelet may
# not solve such errors. Default is to use cgroupfs.
# systemd_cgroup: true

container_runtime: containerd

Some lines have been modified compared to the original file assumes you are running behind a bastion host. Here we connect directly to the host through SSH. kubeadm version and kubernetes version can also be customized here, please note by default kubernetes_version is equal to kubeadm_version but not the opposite :

---
bin_dir: /opt/bin

cri_socket: unix:///run/containerd/containerd.sock

kubeadm_version: v1.24.1

kubernetes_version: "{{ kubeadm_version }}"

kubernetes_api_endpoint_port: 6443

kubernetes_binaries_url: https://storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/amd64

kubeadm_exec_dir: /root
kubeadm_sync_dirs:
  - /etc/kubernetes
  - /etc/kubernetes/pki
  - /etc/kubernetes/pki/etcd

kubeadm_pod_subnet: 192.168.0.0/16

kubeadm_service_subnet: 10.96.0.0/12
kubeadm_service_dns_domain: cluster.local

kubeadm_api_server_extra_args: {}
kubeadm_api_server_extra_volumes: {}
kubeadm_api_server_cert_extra_sans: {}

kubeadm_controller_manager_extra_args: {}
kubeadm_controller_manager_extra_volumes: {}

kubeadm_scheduler_extra_args: {}
kubeadm_scheduler_extra_volumes: {}

kubeadm_kubelet_extra_args: {}
kubeadm_kubelet_component_config: {}

kubeadm_kube_proxy_component_config: {}

kubeadm_cluster_name: symplegma

kubeadm_force_init: false

kubeadm_mark_control_plane: true

systemd_cgroup: false

This allow to create custom deployments and custom upgrade logic (updating Kubelet before control plane for example).

Running the playbooks¶

Playbooks can be run from the symplegma directory of the repository:

sudo ansible-playbook -i inventory/$CLUSTER_NAME/hosts -b symplegma-init.yml -v

The following tags are also availabled to run specific roles:

bootstrap: bootstrap OS for ansible.
containerd: install Kubernetes default container runtime.
cni: install container network interface plugins.
kubernetes_hosts: install Kubernetes binaries and Kubeadm.
kubeadm-master: bootstrap Kubeadm master nodes
kuebadm-nodes: bootstrap Kubeadm worker nodes

Upgrading the cluster¶

There is another playbook symplegma-upgrade.yml which does the same thing as symplegma-init.yml but with a serial set to 1. It means that nodes will be upgraded one by one.

To update Kubernetes to another version, just change kubeadm_version and kubernetes_version in symplegma/inventory/$CLUSTER_NAME/group_vars/all/all.yml and re-run the playbooks for kubernetes_host and kubeadm-master.

ansible-playbook -i inventory/$CLUSTER_NAME/hosts -b symplegma-upgrade.yml -v --tags kubernetes_hosts
ansible-playbook -i inventory/$CLUSTER_NAME/hosts -b symplegma-upgrade.yml -v --tags kubeadm-master

Accessing the cluster¶

When the playbbok are done, a kubeconfig file is generated in symplegma/kubeconfig/$CLUSTER_NAME/admin.conf. This file contains the TLS certificates to access the cluster as an administrator.

By default, kubectl looks into ~/.kube/config. To use the generated kubeconfig file from the symplegma directory:

export KUBECONFIG=$(pwd)/kubeconfig/$CLUSTER_NAME/admin.conf

Check that you can access the cluster:

kubectl get nodes
NAME               STATUS   ROLES    AGE   VERSION
k8s-master-1   Ready    master   25h   v1.15.0
k8s-worker-1   Ready    <none>   25h   v1.15.0
k8s-worker-2   Ready    <none>   25h   v1.15.0
k8s-worker-3   Ready    <none>   25h   v1.15.0
k8s-worker-4   Ready    <none>   25h   v1.15.0

kubectl get pods --all-namespaces
NAMESPACE             NAME                                             READY   STATUS    RESTARTS   AGE
kube-system           calico-kube-controllers-6fb584dd97-wfj5r         1/1     Running   1          7d20h
kube-system           calico-node-554qz                                1/1     Running   1          7d20h
kube-system           calico-node-c28zf                                1/1     Running   1          7d20h
kube-system           calico-node-glqjd                                1/1     Running   1          7d20h
kube-system           calico-node-lm9hp                                1/1     Running   0          20h
kube-system           calico-node-qkw4j                                1/1     Running   1          7d20h
kube-system           coredns-5c98db65d4-kvsnz                         1/1     Running   2          7d20h
kube-system           coredns-5c98db65d4-s8zsb                         1/1     Running   2          7d20h
kube-system           etcd-k8s-master-1-dev                            1/1     Running   2          7d20h
kube-system           kube-apiserver-k8s-master-1-dev                  1/1     Running   2          7d20h
kube-system           kube-controller-manager-k8s-master-1-dev         1/1     Running   2          7d20h
kube-system           kube-proxy-dxllt                                 1/1     Running   1          7d20h
kube-system           kube-proxy-h97vq                                 1/1     Running   0          20h
kube-system           kube-proxy-kt9gc                                 1/1     Running   1          7d20h
kube-system           kube-proxy-ngw7p                                 1/1     Running   1          7d20h
kube-system           kube-proxy-rgh7t                                 1/1     Running   1          7d20h
kube-system           kube-scheduler-k8s-master-1-dev                  1/1     Running   2          7d20h
kube-system           nginx-proxy-k8s-worker-1-dev                     1/1     Running   1          7d20h
kube-system           nginx-proxy-k8s-worker-2-dev                     1/1     Running   1          7d20h
kube-system           nginx-proxy-k8s-worker-3-dev                     1/1     Running   1          7d20h
kube-system           nginx-proxy-k8s-worker-4-dev                     1/1     Running   1          20h

Conformance end to end test¶

The cluster has passed Kubernetes conformance testing. These tests are run with sonobuoy.

Install Sonobuoy¶

Install sonobuoy:

by downloading the binary -> go to https://github.com/heptio/sonobuoy/releases
alternatively, on a machine with Go installed:

go get -u -v github.com/heptio/sonobuoy

Run sonobuoy manually¶

To run sonobuoy (the right kubeconfig must be set as sonobuoy talks to the cluster):

sonobuoy run

The tests might take between 60 minutes and 2h. To check the status:

sonobuoy status

To retrieve the results once it is done:

sonobuoy retrieve

To inspect results:

sonobuoy e2e 201906261404_sonobuoy_4aff5e8e-21a8-448c-8960-c6565b92be91.tar.gz
failed tests: 0

Then delete cluster resources:

sonobuoy delete