Verifying Service Mesh TLS in Kubernetes, Using ksniff and Wireshark

Alongside Nic Jackson from HashiCorp, I have recently presented at several conferences and webinars about the need for transport-level encryption that spans end-to-end, or “user to service”, within modern applications. TLS encryption (and termination) for traffic from a user’s browser to the application edge has been a long-standing feature of API gateways, CDNs and edge proxies, but only recently has service mesh technology made implementing TLS for service-to-service traffic a realistic approach for most of us. A lot of service mesh implementations promise low-touch TLS implementation, allowing operators to enable this with a single config option or a few lines in a YAML file.

However, how do you actually know your inter-cluster traffic is actually being encrypted successfully? Sure, you can fire up tcpdump within a Pod running in a Kubernetes cluster, but this can be tricky to manage, especially for those not super comfortable with Linux tooling. After a spate of recent service mesh investigation and TLS debugging, I bumped into the ksniff kubectl plugin from Eldad Rudich, and this has proved to be a very useful tool for examining traffic within a cluster.

I wanted to share my learnings from using ksniff, and also provide a couple of examples based on my recent investigation of TLS communication between an API gateway and the first internal hop to a service mesh.

Source: getambassador.io