I was not able to find this issue anywhere else on the internet, so here we go:
tl;dr; make sure your total HTTP response headers size of your health probe does not exceed 4096 bytes.
The problem
I was configuring my Azure Front Door (Premium) with a single origin group containing two origins. One was pointing to an Azure App Service in the North Europe region and the other was pointing to West Europe. For reasons (I won't get into) I wanted the traffic to be split equally between the two origins. When looking at the request volume hitting the App Services it was obviously preferring the North European app service over the West European.
The configuration
As I wanted the traffic to split equally over both origins, in accordance with the documentation, I configured my origins with:
- Both origins enabled
- A health probe pointing to the (healthy) readiness probe
- Equal priority
- High latency sensitivity (500ms) to make sure both origins would always be grouped together
- Equal weights
The fix
After contacting Microsoft Support they were able to determine that total response headers size of the health probes exceeded their undocumented limit of 4096 bytes. This caused an error on the points of presence (POPs) of the Azure Front Door network causing the POPs to mistake our health probe as unhealthy. These headers were there by mistake. I disabled the headers and traffic was being split equally again. Success!
I never really found out why the North Europe origin was preferred over West Europe though, both origins' health probes were sending out the same response with the same headers. Still, removing the unnecessary headers fixed the issue.