Integrating Kubernetes with 1Password for infrastructure secrets

Don’t use the same password twice. Don’t write down your passwords on sticky notes. Use passwords that are difficult to guess. Yes, yes, and yes. And also: please give me a means to make all this secure behaviour easy for me. The solution: a password manager. And nowadays most people have some sort of password manager, whether it be integrated in their operating system or browser, or a dedicated password management tool like 1Password or LastPass. Larger enterprises, however, are often still struggling here and you’ll find password-protected Excel sheets, KeePass files and loads of shadow IT floating around. Infrastructure and systems are even worse. Passwords are floating around in config files, and it’s no exception to find plain-text credentials in a Git repository somewhere.

Now let’s add Kubernetes into the mix as well. Kubernetes offers Secrets as a clean abstraction to store sensitive information separate from deployment config, and apply access controls to it. Except that the secrets are not really encrypted (just base64 encoded), and actually storing them in Kubernetes is a bad idea for many reasons (long story short: infrastructure is unreliable and nobody wants to backup/restore Kubernetes). So quite often we’ll look for a way to store secrets outside of Kubernetes and then deploy them into a cluster. There are many solutions built specifically for secrets management, and a lot of them can actually be used with Kubernetes. Solutions like HashiCorp Vault or AWS Secrets Manager are great solutions for this. Or Bitnami SealedSecrets, which allows you to encrypt secrets properly so that you can store them in Git.

But this blog isn’t about any of those.

Today we’re looking at integrating Kubernetes with 1Password. “Why would you do that?!” you may ask. Well, maybe you’re working on a personal side project that needs to run on Kubernetes but setting up Vault feels like overkill, and AWS Secrets Manager isn’t free, and maybe SealedSecrets is a little too complex for your taste. Maybe you already use 1Password as your personal password manager. Or maybe you’re trying to kill two birds with one stone for your company, and using 1Password means introducing a good enterprise-ready password management solution that also works for infrastructure secrets.

Setting Up 1Password

The first thing we need to do is configure 1Password so that we can use it for infrastructure secrets. There are two parts to this:

1. Organizing the secrets
2. Setting up an Integration to use in Kubernetes

While the first part isn’t technically required, I think it’s a good idea to set up a separate 1Password Vault for infrastructure secrets at this point. For a more production-ready setup you would probably be looking at different Vaults per application (so you can configure access to the secrets easily based on teams), and probably per environment. As per Conway’s Law, your solution will heavily mirror your organization structure here. For this blogpost, I’m just going to add a single Vault named ‘KuberDemo’. I’m also going to add a few dummy secrets in there that we will use later.

The next step is to set up a 1Password Integration. You can set this up from the 1Password web UI. For personal accounts this will be at https://my.1password.com/integrations/directory/ typically. After logging in you should see something like the screenshot below.

Click on ‘Secrets Automation’ and you will be guided through a simple setup process where you’ll name your environment, set up the first access token, and grant access to one or more Vaults. For this blog, I’ve set up a token that only has ‘Read’ access to the ‘KuberDemo’ Vault. Given that the integration with Kubernetes will only consume secrets from 1Password there is no need for anything beyond ‘Read’ access. In the final step you can download the 1password-credentials.json file and copy the access token. We’re going to need both of them next.

Setting up Kubernetes

For this step I’m going to assume you have a Kubernetes cluster at your disposal. If you don’t, I recommend looking at local Kubernetes setups like minikube or k3d. You’ll also need Helm for this next step, so make sure you have that installed as well.

First, we’re going to add the 1Password Helm repository:

helm repo add 1password https://1password.github.io/connect-helm-charts/

The next step is to install the 1Password Connect server, and the 1Password Kubernetes Operator and CRDs to our cluster. For this blog, I have created a Kubernetes Namespace with the name opconnect and I will install everything there. By default, everything will go into your current Kubernetes namespace, and by default the Kubernetes Operator will only watch the current namespace. You can, however, configure it to watch multiple namespaces. For this blog, I will configure the Operator to watch the opconnect and default namespaces.

Make sure you run this command from the directory where you stored the 1password-credentials.json file, as we’ll be needing that.

# Put our 1Password Connect access token in a variable
OP_TOKEN="< paste your token here >"

# Use Helm to install and configure everything. It will use the 
# credential file and the ${OP_TOKEN} variable to use the integration
# we set up earlier.
helm upgrade --install connect 1password/connect --set-file connect.credentials=1password-credentials.json --set operator.create=true --set operator.token.value="${OP_TOKEN}" --set "operator.watchNamespace={opconnect,default}" --namespace opconnect

Let’s see if everything worked:

$ kubectl get pods -n opconnect
NAME                                           READY   STATUS    RESTARTS   AGE
onepassword-connect-operator-666677869-mpw7w   1/1     Running   0          1m
onepassword-connect-5484f8d4c4-dcwhl           2/2     Running   0          1m

Configuring 1Password Secrets in Kubernetes

Now that we’ve successfully integrated our Kubernetes cluster with 1Password, it’s time to get some sensitive information! When I set up the KuberDemo Vault earlier, I added a few items to that Vault. In this step, we’re going to make our Kubernetes cluster use the ‘Upsteam API key’ secret.

Let’s create the following Kubernetes manifest in upsteam-api-key.yaml:

apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
  name: upsteam-apikey
spec:
  itemPath: "vaults/KuberDemo/items/upsteam-apikey-prod"

Apply it to your cluster:

$ kubectl apply -f upsteam-api-key.yaml
onepassworditem.onepassword.com/upsteam-apikey created

So, now the following will happen: the 1Password Connect Operator will notice we have created a OnePasswordItem resource in a namespace it’s supposed to watch. It will now parse the resource, retrieve the ‘upsteam-apikey-prod’ item from the ‘KuberDemo’ Vault, and create a matching Kubernetes Secret with the name upsteam-apikey, matching the OnePasswordItem. Let’s check if it worked.

$ kubectl get onepassworditem,secret upsteam-apikey
NAME                                             AGE
onepassworditem.onepassword.com/upsteam-apikey   6m22s

NAME                    TYPE     DATA   AGE
secret/upsteam-apikey   Opaque   1      4m54s

Cool! We now have a matching Secret for our OnePasswordItem. Now as a final check, let’s see if the contents of this secret is what we expect. We can do this by retrieving the secret as JSON and decoding the base64 content.

$ kubectl get secret upsteam-apikey -o=jsonpath='{.data.credential}' | base64 -d
lovely-safely-frank-badger

That surely looks familiar.

Advanced usage of the 1Password Connect Operator

Now that we have the important bits working, let’s also look at some more advanced usage. For example, you can use annotations to define a 1Password secret to use inside a Kubernetes Deployment. The Operator will handle retrieval and cleanup automatically, and the secret will exist no longer than the Deployment does. The annotation works like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-example
  annotations:
    operator.1password.io/item-path: "vaults/<vault_id_or_title>/items/<item_id_or_title>"
    operator.1password.io/item-name: "<secret_name>"

Another feature offered by the 1Password Operator is handling of restarts when 1Password secrets change. This is something many people struggle with in Kubernetes, so it’s nice to see that the folks at 1Password spent some good time on getting this right. They allow you to configure whether or not to ‘auto restart’ on different levels: Namespace, Deployment, and OnePasswordItem. This gives you a lot of options to granularly configure the behaviour you need. Check out the documentation to learn more.

Closing thoughts

If you’re familiar with 1Password, you know it isn’t free. So the big elephant in the room when linking it to your infrastructure is pricing. By default you’ll get 3 free Vault Access Credits. This should be enough to evaluate it without getting any unwanted credit card charges. After that there’s a tiered pricing model up to 500 Vault Access Credits. If you need more, you can probably get some discount through their sales. The fixed tiers are:

• Up to 3 Vault Access Credits: Free
• Up to 25 Vault Access Credits: 29 USD/mo
• Up to 100 Vault Access Credits: 99 USD/mo
• Up to 500 Vault Access Credits: 299 USD/mo

Depending on your use case, this may mean that other paid alternatives like AWS Secrets Manager may be cheaper. However, if you don’t want to be tied to Amazon, or if you also need a proper password manager (and the nice workflow that comes with it), 1Password may be just what you need. Also, I don’t think their pricing is excessive.

As a 1Password user I can personally see myself applying this for some personal project. The setup on Kubernetes was pretty straightforward and the Operator works pretty well. Overall I quite like the user experience as well as the workflow for managing static credentials in 1Password. It also makes it a little easier to grant read-only access to certain infrastructure secrets via a password manager, which is certainly a lot nicer than having developers decode a Kubernetes secret to store it in their notes app. While I hadn’t really considered using my password manager for infrastructure it may actually make sense for a lot of companies.