This post details the development and purpose behind the Custom AMIs for Amazon EKS available on the AWS Github.
Whether you are in a highly regulated industry, the government, or a security conscious organization you are most likely running hardened virtual machines within your environment. As more organizations, such as the United States Department of Defense, are adopting Kubernetes organization wide, their is a need for hardened AMIs that work with Amazon EKS and meet compliance requirements.
The Amazon Linux 2 EKS Optimized AMI is an optimized version of Amazon Linux 2 for use with Amazon EKS and is the preferred way of running Amazon EKS EC2 instances. Amazon has also announced Bottlerocket, a purpose built operating system designed for running containers. I believe purpose built container operating systems are the future for running Kubernetes.
For organizations like the US Federal Government, they have standards such as the Security Technical Implementation Guide (STIG), Criminal Justice Information Services (CJIS), and U.S. Government Commercial Cloud Services (C2S) in addition to the commercial Center for Internet Security (CIS) benchmarks. Operating systems such as Amazon Linux 2 and Bottlerocket, don’t always meet all of these compliance requirements. These compliance frameworks often include things like FIPS 140-2, SELinux, auditing rules, and specific file permissions.
In order to meet all of these requirements, I have developed a set of Packer and Ansible scripts that have been published to the AWS Samples Github. In this post, I will walk through how they were built and what they include. This post will focus on the ones supporting the compliance frameworks mentioned above.
To get started, we use Packer to orchestrate the build process within AWS and Ansible to configure our hosts. We also generated the hardening guidelines for STIG, CJIS, and C2S from the OpenSCAP tool with some slight modifications that documented within the repository. The Packer configuration will automatically encrypt the Elastic Blockstore (EBS) volumes that back the AMI, with the option to specify your own key. The Ansible scripts handle upgrading the operating system, hardening the system, installing Docker CE, and installing the EKS distribution of Kubernetes. Below are some of the hardening guidelines we apply to the images.
FIPS 140-2
The STIG hardening applied by this repository enables FIPS on the operating system following the recommended procedures from Red Hat. It adds the fips=1 to the boot command and ensures that packages like OpenSSL are in FIPS mode.
The AMI and backing EBS volumes are encrypted using AWS KMS. By default, it will use the default KMS keys within your account. You can also supply your own keys. KMS is validated up to FIPS 140-2 level 2.
SELinux
SELinux is installed and set to enforcing for CentOS and Red Hat. SELinux is also configured with the Docker daemon.
STIG / CJIS / C2S
Using the OpenSCAP project, I generated Ansible Playbooks based on the hardening guidelines. We also add the official US Government language to the SSH login banner. We did make slight modifications to support AWS, but these are documented in the repository and are minor.
Usage
This repository is meant to be starting place for organizations looking to deploy compliant environments. The resulting images should still be independently audited and scanned by your authorizing agency or internal security organization to ensure they meet the latest standards.
The AMI is designed to function like the Amazon Linux 2 EKS Optimized AMI, including the bootstrap.sh script. Once the AMI is deployed to your environment, you can simply run to join the node to the cluster:
/etc/eks/bootstrap.sh <cluster name>
In production, I recommend not using SSH and instead use the Amazon SSM Agent to securely access the host. This method leverages IAM to govern access to the instance rather than SSH keys. It also doesn’t require any ports to be opened.
Conclusion
If you have any feedback on how we can improve upon this image or if you find a configuration error don’t hesitate to reach out. If you have requirements that are not currently being met, feel free to file an issue on the Github Repository.