ECR with on premises Kubernetes

Deploying Kubernetes on premises can be challenging enough.

You’ll quickly find out things you’d expect to work, simply don’t. That’s because cloud providers do lots of things for you (which is good).

I am currently storing docker images on ECR (AWS elastic container registry). It’s easy to use some simple commands to authenticate with your configured credentials then push / pull images when developing.

But this is something you are going to want to just do with Kubernetes. It is also essential for Kubernetes to be able to pull images from the cloud when it needs to. AWS ECR tokens expire every 12 hours so you will need a something that takes care refreshing tokens for you. Fortunately the registry-cred project on Github does just this.

Prerequisite: Create an AWS service account with programmatic access that can only interact with ECR. Be sure to save the access key id and secret access key.

Step 1: Log the host that you run the kubectl command from or you can copy / paste to the dashboard.

Step 2: git clone https://github.com/upmc-enterprises/registry-creds

Step 3: Enter the k8s sub directory within the cloned repo

Step 4: Edit the secrets.yaml file.

Replace the text YOUR_BASE64_HERE with a base64 encoded value for each field. You can accomplish this in bash by running echo -n “SECRET” | base64.

Then copy the output into each field as shown below.

apiVersion: v1
kind: Secret
metadata:
  name: registry-creds-ecr
  namespace: kube-system
  labels:
    app: registry-creds
    kubernetes.io/minikube-addons: registry-creds
    cloud: ecr
data:
  AWS_ACCESS_KEY_ID: YOUR_BASE64_HERE
  AWS_SECRET_ACCESS_KEY: YOUR_BASE64_HERE
  aws-account: YOUR_BASE64_HERE
  aws-region: YOUR_BASE64_HERE
type: Opaque

Run kubectl create -f secret.yaml to create the new secret.

Step 5: Edit replicationController.yaml and remove what you are not going to use.

apiVersion: v1
kind: ReplicationController
metadata:
  name: registry-creds
  namespace: kube-system
  labels:
    version: v1.6
spec:
  replicas: 1
  selector:
    name: registry-creds
    version: v1.9
  template:
    metadata:
      labels:
        name: registry-creds
        version: v1.9
    spec:
      containers:
      - image: upmcenterprises/registry-creds:1.9
        name: registry-creds
        imagePullPolicy: Always
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_SECRET_ACCESS_KEY
          - name: awsaccount
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: aws-account
          - name: awsregion
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: aws-region

Now you will see secrets magically appearing in all namespaces. They should be named: awsecr-cred