Using AWS Private Certificate Authority certificates to secure access to Kubernetes Helm

Helm is the Kubernetes package manager and a popular tool to use to deploy services onto Kubernetes. Helm recently became a top-level Cloud Native Computing Foundation Project which is sure to increase its popularity even more. While Helm simplifies the deployment of services onto Kubernetes, the security of the default configuration leaves something to be desired. In this blog post Joe Keegan, BlueChipTek Lead Cloud Services Architect, will show how you can utilize AWS Private Certificate Authority (PCA) to issue TLS certificates that can be used to secure access to Helm.

Helm and TLS

If you are reading this post, it’s best that you have a basic understanding of Helm. You can get up to speed quickly by reviewing the documentation and it’s especially recommend to review the Using SSL Between Helm and Tiller section.

Quick recap—Helm has a client/server architecture with the client called helm and the server called tiller. Tiller can be configured to only allow client requests with a certificate signed by a trusted Certificate Authority. This process provides some measure of authentication to requests made to tiller and it’s considered best practice in shared or production clusters. An extensive look at Helm security and how TLS fits can be seen in this excellent Bitnami Engineering blog post.

AWS Private Certificate Authority (PCA)

AWS recently launched the Private Certificate Authority service, which allows you to issue and manage private TLS certificates. Being an AWS services, it’s integrated with IAM and CloudTrail giving you all the permissions management and audit trails that you typically want with a Certificate Authority (CA). The PCA only currently supports being an intermediate CA, meaning that you need to have a Root CA setup before you can use PCA. I’ll start by showing you how to setup an OpenSSL based Root CA and then use that CA as part of the PCA setup.

I do want to mention that PCA has a flat fee, on top of a small cost for each certificate issued. So it may not be cost effective to use PCA for some use cases. For example, if you wanted to have a separate CA for each tiller deployment. In this case only a few certs would be issues from each CA and PCA may not be cost effective for this use case. It just depends on your PKI strategy, which is outside the scope of this post.

PCA Setup

To setup the RootCA, which will be used to sign the intermediate CA cert created in PCA, I’m following this OpenSSL Certificate Authority guide.  This involves downloading the openssl.cnf file mentioned in the guide to /root/ca/openssl.cnf.

Once the RootCA is setup, I’ll use the console to setup the CA in PCA. AWS Certificate Manager Launches Private Certificate Authority blog post provides good step-by-step screenshots of the process.

Once the CA is created I download the Certificate Signing Request (CSR) to /root/ca/pca.csr and use the following command to sign the intermediate CA cert. 

openssl ca -config openssl.cnf -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in pca.csr -out certs/pca.cert.pem -policy policy_loose

Note, the “-policy policy_loose” is important here, which I haven’t seen mentioned in other tutorials. If you don’t specify the policy argument then the Location is not included in the intermediate CA cert. The result you’ll get is the error message “The certificate subject contains a different subject than the CA certificate signing request.” when trying to upload it into PCA.

Once the cert is signed, you’ll upload it and the CA cert as shown in the AWS Blog post.

Helm Certs

Now that the intermediate CA is configured, we are ready to create the certs needed for Helm. A cert needs to be created for the tiller server, and then one or more certs to be used by helm clients. I’ll use OpenSSL to create the cert for tiller and generate a CSR to be signed by the intermediate CA.

openssl req -new -newkey rsa:2048 -nodes -days 365 \
     -keyout tiller.key.pem -out tiller.csr

Using the AWS CLI to generate a signed certificate from the CSR.

aws acm-pca issue-certificate \
  --certificate-authority-arn <CA ARN>\
  --csr file://tiller.csr \
  --signing-algorithm "SHA256WITHRSA" \
  --validity Value=365,Type="DAYS" \
  --idempotency-token 1234

This will return an ARN for the certificate. Use the following command to download the certificate.

aws acm-pca get-certificate \
--certificate-authority-arn <CA ARN>
--certificate-arn <Cert ARN> \
--output text --query 'Certificate' > tiller.cert.pem

You will need to do the same commands above but for helm, i.e. create a helm.key.pen, helm.csr and helm.cert.pem

Lastly, we will need to download the CA cert chain with the following command.

aws acm-pca get-certificate-authority-certificate \
--certificate-authority-arn <CA ARN>\
--output text > ca.cert.pem

After this you should have the following files: tiller.key.pen, tiller.csr ,tiller.cert.pem, helm.key.pen, helm.csr ,helm.cert.pem and ca.cert.pem

Install and Configured Helm

I’ll be deploying Helm into an EKS cluster, so the first thing I’ll need to do is create a Kubernetes service account to be used by tiller and a ClusterRoleBinding for the tiller account. The following manifest will be used an applied to the kube-system namespace.

apiVersion: v1
kind: ServiceAccount
  name: tiller
  namespace: kube-system

apiVersion: v1
kind: ClusterRoleBinding
  name: tiller-role-binding
  kind: ClusterRole
  name: cluster-admin
- kind: ServiceAccount
  name: tiller
  namespace: kube-system

Once the above is applied you can download Helm and use the following command to deploy tiller configured to use TLS into your cluster.

helm init --tiller-tls --tiller-tls-cert ./tiller.cert.pem \
--tiller-tls-key ./tiller.key.pem --tiller-tls-verify \
--tls-ca-cert ca.cert.pem --service-account tiller

This should complete with a message stating tiller is installed.

You now need to copy the keys used by helm into the Helm Home:

cp ca.cert.pem $(helm home)/ca.pem
cp helm.cert.pem $(helm home)/cert.pem
cp helm.key.pem $(helm home)/key.pem

We can tell helm to use TLS with the --tls argument.

$ helm ls --tls
$ helm ls
Error: transport is closing

As we can see in the commands above, when I include the --tls argument the commands work fine. But when I omit the --tls argument, I get the “Error: transport is closing” error. This error means tiller rejected my request because the TLS certificate (in this case no TLS cert) is invalid.   

We can tell helm to always use TLS by setting HELM_TLS_ENABLE to true.

export HELM_TLS_ENABLE=true
helm ls

This is something good to set in your .bashrc 

Tools, like Kubernetes and Helm, are enabling companies to be more agile but deploying them in a manner that can be used for production is still complicated. 

If you are looking to use Kubernetes and want help tackling these types of complex problems, then check out Jumpstart for Container Orchestration. We can come on site and show you how to get EKS up and running securely and ready for production.