Skip to content

tEKS

teks teks:mkdocs semantic-release FOSSA Status

tEKS is a set of Terraform / Terragrunt modules designed to get you everything you need to run a production EKS cluster on AWS. It ships with sensible defaults, and add a lot of common addons with their configurations that work out of the box.

This is our opinionated view of what a well structred infrastructure as code repository should look like.

⚠️ the v5 and further version of this project have been completely revamp and now offer a skeleton to use as a base for your infrastructure projects around EKS. All the modules have been moved outside this repository and get their own versioning. The old README is accessible here

⚠️ Terraform implementation will not be maintained anymore because of time, and mostly because it has become quite difficult to get feature parity with Terragrunt. Archive branch is available here

Terraform/Terragrunt

  • Terragrunt implementation is available in the terragrunt folder.

Contributing

Contribution are welcome, as well as issues, we are usually quite reactive. If you need more support for your project, do not hesitate to reach us directly.

Requirements

Terragrunt

Quickstart

Quickstart guide is available here or on the official documentation website

Main purposes

The main goal of this project is to glue together commonly used tooling with Kubernetes/EKS and to get from an AWS Account to a production cluster with everything you need without any manual configuration.

What you get

A production cluster all defined in IaaC with Terraform/Terragrunt:

Everything is tied together with Terragrunt and allows you to deploy a multi cluster architecture in a matter of minutes.

Curated Features

The additional features are provided by tEKS here as well as our curated addons module which support a bunch of various configuration.

Bottlerocket support

Bottlerocket OS is available for node groups (see example here). Bottle rocket is a container centric OS with less attack surface and no default shell.

AWS Session Manager by default

All the instances (Bottlerocket or Amazon Linux) are registered with AWS Session Manager. No SSH keys or SSH access is open on instances. Shell access on every instance can be given with SSM for added security.

aws ssm start-session --target INSTANCE_ID

From and to Zero scaling with EKS Managed Node Groups

tEKS support scaling to and from 0, even with using well know Kubernetes labels, there are a number of ongoing issues for support of EKS Managed node groups with Cluster Autoscaler. Thanks to automatic ASG tagging, tEKS adds the necessary tags on autoscaling group to balance similar node groups and allow you to scale to and from 0 and even to use well know labels such as node.kubernetes.io/instance-type or topology.kubernetes.io/zone. The logic can be extended to support other well known labels.

Automatic dependencies upgrade

We are using renovate to automatically open PR with the latest dependencies update (Terraform modules upgrade) so you never miss an upgrade and are alwasy up to date with the latest features.

Enforced security

  • Encryption by default for root volume on instances with Custom KMS Key
  • AWS EBS CSI volumes encrypted by default with Custom KMS Key
  • No IAM credentials on instances, everything is enforced with IRSA.
  • Each addons is deployed in it's own namespace with sensible default network policies.
  • Calico Tigera Operator for network policy.
  • PSP are enabled but not enforced because of depreciation.

Out of the box logging

Three stacks are supported: * AWS for Fluent Bit: Forward containers logs to Cloudwatch Logs * Grafana Loki: Uses Promtail to forward logs to Loki. Grafana or a tEKS supported monitoring stack (see below) is necessary to display logs.

Out of the box monitoring

  • Prometheus Operator with defaults dashboards
  • Addons that support metrics are enable along with their serviceMonitor
  • Custom grafana dashboard are available by default

Two stacks are supported: * Victoria Metrics Stack: Victoria Metrics is a Prometheus alertnative, compatible with prometheus CRDs * Kube Prometheus Stack: Classic Prometheus Monitoring

Long term storage with Thanos

With Prometheus, tEKS includes Thanos by default. Thanos uses S3 to store and query metrics, offering long term storage without the costs. For more information check out our article on the CNCF Blog

Support for ARM instances

With either Amazon Linux or BottleRocket, you can use a mix of ARM and AMD64 instances. Check out our example

Helm v3 provider

  • All addons support Helm v3 configuration
  • All charts are easily customizable

Other and not limited to

  • priorityClasses for addons and critical addons
  • lot of manual stuff have been automated under the hood

Always up to date

We always support the latest modules and features for our addons module.

Our cutting edges addons include (not limited to): * AWS EBS CSI Drivers: Support for Volume encryption by default, snapshot, etc * AWS EFS CSI Drivers: Use AWS NFS shares. * Secret Store CSI Driver: load secret from Secret Managers with aws-secret-store-csi-driver driver * Linkerd2 or Certificate Manager CSI for mTLS

Requirements

Terragrunt is not a hard requirement but all the modules are tested with Terragrunt.

Pre-commit

This repository use pre-commit hooks, please see this on how to setup tooling

ASDF

ASDF is a package manager which is great for managing cloud native tooling. More info here (eg. French).

Enabling plugins

for p in $(cut -d " " .tool-versions -f1); do asdf plugin add $p; done

Installing tools

asdf install

Examples

terragrunt/live folder provides an opinionated directory structure for a production environment.

Additional infrastructure blocks

If you wish to extend your infrastructure you can pick up additional modules on the particuleio github page. Some modules can also be found on the clusterfrak-dynamics github page.

Branches

  • main: Backward incompatible with v1.X but compatible with v2.X, releases bumped to v3.X because a lot has changed.
  • release-1.X: Compatible with Terraform < 0.12 and Terragrunt < 0.19. Be sure to target the same modules version.
  • release-2.X: Compatible with Terraform >= 0.12 and Terragrunt >= 0.19. Be sure to target the same modules version.

License

FOSSA Status