Disabling mTLS in Istio for a NodePort service

Some of the services running in my local minikube environment are configured as NodePort services so that I can access them from applications running on my host system. Up until last week I was successfully able to connect to those services from a variety of tools. But last week I was experimenting with enabling Istio’s mTLS authentication for all pods in my cluster and noticed that whenever mTLS was enabled I was unable to connect to my NodePort services from my host system. Applications running in the cluster could still connect over the ClusterIP, so it took me a while to notice the problem. Any connection attempt over the NodePort resulted in an error stating the connection was closed or the remote service sent an invalid response. A quick Google search showed other people who have run into this problem..

For your reference, here is the PeerAuthentication configuration I put in the istio-system namespace to enable mTLS across the entire service mesh:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: mtls
spec:
  mtls:
    mode: STRICT

This behaviour was surprising to me initially, but made sense the more I thought about it, especially from a security perspective. When Istio injects its init container and sidecar container into a pod, the pod’s firewall rules are modified to force all ingress and egress traffic to go through the Istio sidecar proxy. When any client tries to connect to the application running in the pod, the sidecar proxy intercepts the connection. Additionally, when mTLS is enabled in STRICT mode, the Istio sidecar proxy requires the connecting client to present a valid client certificate. If an invalid certificate is presented then the sidecar proxy denies the connection.

When a client on my host system attempts to connect to an mTLS-protected NodePort service, the client doesn’t have a client certificate that is trusted by the Istio sidecar proxy. Therefore, the connection attempt is denied.

Ideally, Istio would be able to distinguish between internal traffic originating from other pods in the cluster and external traffic originating from the host system. In doing so it could require internal traffic to use a valid client certificate while letting NodePort traffic send data in plaintext. However, there is no secure way for Istio to distinguish such traffic. If Istio allowed non-mTLS connections from a NodePort then it would be possible for other pods in the cluster to pretend they were also connecting over a NodePort to bypass mTLS.

Fortunately, Istio allows mTLS to be configured in different policies with different scopes in the cluster. From the Istio security documentation:

Istio applies the narrowest matching policy for each workload using the following order:

1. workload-specific
2. namespace-wide
3. mesh-wide

Thus, the solution I implemented was to disable mTLS for services that were accessible from a NodePort using a workload-specific policy. This meant both internal traffic and external traffic would use unencrypted connections to the service. This reduced the security of the service mesh but I was comfortable with that because it was only for local development purposes. In production all traffic would remain protected with the STRICT mTLS mode.

Here’s an example of how I disabled mTLS for a single service when mesh-wide mTLS was set to STRICT mode:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: my-nodeport-service
spec:
  selector:
    matchLables:
      app: my-nodeport-app
  portLevelMtls:
    "443": # This must be a string due to a bug with kustomize.  See https://github.com/kubernetes-sigs/kustomize/issues/3446
      mode: DISABLE
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-nodeport-service
spec:
  host: my-nodeport-service
  trafficPolicy:
    tls:
      mode: DISABLE

Note that I had to disable mTLS on both the service (PeerAuthentication) and the clients connecting to the service (DestinationRule).

Instead of setting the mode to DISABLE I could have used PERMISSIVE. If I used PERMISSIVE then the DestinationRule would not be needed. However, I found it simpler to reason about the application knowing that all traffic was unencrypted going to my NodePort services.

Note that in the above PeerAuthentication yaml I had to use a string key instead of an integer key for the portLevelMtls port. That is due to a bug in kustomize. If you aren’t using kustomize then you can use an integer key instead.