-
Notifications
You must be signed in to change notification settings - Fork 39.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Topology Aware Routing not working #124699
Comments
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
/sig network |
(my emphasis) If you'd like to enforce that traffic stays inside a particular zone, that would be a feature request @roman5595. Kubernetes is open source; if you'd like to improve the topology aware routing heuristics, we welcome code contributions. |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
@roman5595 how did you get the EndpointSlices for your Service? Maybe there's another EndpointSlice for the same Service that doesn't have hints assigned? |
Kubernetes 1.26 is EOL , latest supported one is 1.28 https://kubernetes.io/releases/ , please reopen if you are able to reproduce with a supported version /close |
@aojea: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What happened?
Despite enabling topology-aware routing, I'm still seeing traffic being routed from one AZ to another. This is not the expected behavior, as it should prioritize in-zone traffic.
Current setup:
3 AZ eu-west
3 pods in each AZ
3 endpoints per each AZ
What did you expect to happen?
When i enable topology aware routing I expect that traffic will stay in AZ where traffic was generated to maximize cost savings.
How can we reproduce it (as minimally and precisely as possible)?
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: nginx
apiVersion: v1
kind: Service
metadata:
annotations:
service.kubernetes.io/topology-aware-hints: Auto
name: nginx
spec:
ports:
protocol: TCP
targetPort: 80
selector:
app: nginx
type: ClusterIP
Anything else we need to know?
kubectl get events --field-selector involvedObject.kind=Service,involvedObject.name=nginx
16m Normal TopologyAwareHintsEnabled service/nginx Topology Aware Hints has been enabled, addressType: IPv4
Kube-Proxy logs:
I0505 16:46:38.759957 10 topology.go:171] "Skipping topology aware endpoint filtering since one or more endpoints is missing a zone hint"
I0505 16:46:38.105331 10 topology.go:171] "Skipping topology aware endpoint filtering since one or more endpoints is missing a zone hint"
I0505 16:49:27.896972 10 topology.go:181] "Skipping topology aware endpoint filtering since no hints were provided for zone" zone="eu-west-1c"
I0505 16:49:28.432488 10 topology.go:181] "Skipping topology aware endpoint filtering since no hints were provided for zone" zone="eu-west-1c"
Hints were assigned:
addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-06c703e12f8392a11
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-jdnd9
namespace: default
uid: a142d4d9-e078-44e9-9ce1-3956faa1e5ce
zone: eu-west-1c
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-06c703e12f8392a11
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-cr7f2
namespace: default
uid: a2bd185c-86d0-47a8-a5ab-45ff4e32a85d
zone: eu-west-1c
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-088e764bd881db13b
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-znbbm
namespace: default
uid: 204eaf01-79bb-4993-a37c-b7bb90fb2094
zone: eu-west-1b
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-088e764bd881db13b
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-lfxqp
namespace: default
uid: 47909cec-d536-4bee-ac95-5b021a29041b
zone: eu-west-1b
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-05ab65b1b0ba1ab03
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-mjvd5
namespace: default
uid: 777dceea-a87f-483d-ac7f-d30cd7812bb1
zone: eu-west-1a
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-05ab65b1b0ba1ab03
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-ltj2v
namespace: default
uid: 2a4d680c-8822-4a2c-b3b8-a56ceaff96e7
zone: eu-west-1a
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-06c703e12f8392a11
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-xwqlm
namespace: default
uid: b4fb49ed-630b-4f7c-8ea0-22cc11ab2c9d
zone: eu-west-1c
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-088e764bd881db13b
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-p92c5
namespace: default
uid: 48c29d22-98f4-4316-bc96-958d608a8ffc
zone: eu-west-1b
conditions:
ready: true
serving: true
terminating: false
hints:
forZones:
nodeName: i-0272547448c4bb6b7
targetRef:
kind: Pod
name: nginx-84bdbbf4bb-sqb79
namespace: default
uid: 3f206f30-184c-4c83-986f-29b7b4ad5f77
zone: eu-west-1a
kind: EndpointSlice
Kubernetes version
1.26.12
Cloud provider
AWS
OS version
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)
The text was updated successfully, but these errors were encountered: