Figured it out, one of the network policies I had applied to make this deployment work was restricting traffic from external clients.
The policy was taken from here: Kubernetes - Api_request to the proxy failed with status code 599, retrying - #9 by dprateek1991
Had to change ingress to any peer to get it to work.