Fixing Service Topology with nginx-ingress

TL;DR: Service Topology wasn't working with nginx-ingress cause the Ingress needs nginx.ingress.kubernetes.io/service-upstream=true.

In order to ensure that our users have their requests serviced as fast as possible, we're moving towards a multi-region deployment where users are serviced by the nearest workloads.

Kubernetes introduced the concept of Service Topology routing in 1.17 which enables Services to be able to define how they should be routed. My initial testing indicate that Service Topology worked as expected when using the Cluster IP of the Service, however nginx-ingress appeared to ignore Service Topology and would do its usual round robin behaviour.

I stumbled across a StackOverflow thread which explained that nginx-ingress looks at the Endpoint structures by default and thus bypasses the Service Topology mechanism provided by Service.

The magical fix for this was to add an annotation to the Ingress in question: nginx.ingress.kubernetes.io/service-upstream=true.

This was all that was needed for Service Topology to work.