AWS EFS on Kubernetes with peered VPC
EFS storage “just works” with AWS EKS / Kubernetes. But there are a few quirks to be aware of when using a peered VPC.
You can easily install the AWS EFS CSI driver with Helm.
Link: https://github.com/kubernetes-sigs/aws-efs-csi-driver
But you run into a snag if you are using peered VPCs (with EKS / K8S on one VPC and EFS on another), which isn’t uncommon because the EKS tutorials encourage you to set up new VPCs based on cloud formation templates and you most likely already have EFS data on another VPC if you are migrating over workloads.
You could migrate the data or migrate EFS endpoints but that could lead to downtime or prove to be a bit time consuming. This is where VPC peering comes in, you can easily peer the two VPCs. But none of the DNS names will resolve on your peered VPCs.
The CSI driver project does provide some clues on how to fix this:
Link: https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/240
The solution is to hard code the EFS DNS endpoints as shown on the EFS file system network tab into the AWS CSI driver’s daemon set yaml using the built in hostAliases to resolve the static IPs or just define them with helm when deploying.
But this could lead to a sub-optimal situation. You most likely have at least two availability zones for your cluster and EFS will also be in two availability zones (AZ). But hostAliases just updates /etc/hosts, it doesn’t do anything clever. So the daemon set pods will always mount the first entry if you list both DNS endpoints and both IPs.
Thankfully the EFS CSI driver will automatically try to the next IP if the first one fails. You can then just setup multiple security groups and only allow connections from the IP range in the peered VPC (where your nodes are) that match the same AZ of the endpoint. This will stop nodes from a different AZ from connecting to an EFS endpoint in another AZ. The driver will just failover until it works and it happens very quickly.
VPC routing rules can also be used to ensure only the desired IPs can route to EFS in the same AZ. Just be careful, you might need to be specific with your routes or you could accidently break access to an RDS database that only has 1 endpoint in 1 AZ but needs to be accessed by all nodes. It’s always important to test this!
Speaking of testing, how do we know any of this works? It's possible to get the shell on a daemon set pod for AWS CSI in each AZ. Then ensure something is running that mounts EFS on at least two nodes in different AZs. You could make deployment with a simple image that does nothing but mounts EFS storage with a PV on each node using affinity to disallow pods on the same node or event select nodes.
Link: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
Then enter the shell for the AWS CSI daemon pods on at least two nodes in two different AZs. Run yum install net-tools and just use netstat -nputw | grep 2049 , to see what is connected and grep for the EFS TLS port. You should see nodes only connecting to EFS endpoints in the same AZ.
Now it’s also important to consider that this might not be what you need for your project. Allowing EFS to failover to different AZs might actually be a feature for you.
e.g. EFS is having issues in 1 AZ but not another. EFS CSI will just fail pods over to the available endpoint.
This is unlikely to be desired because you will should have several nodes in multiple AZs. You will most likely experience latency crossing AZs more than you ever see a long term outage. But it's worth thinking about because everyone’s projects have different needs.
