-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conntrack entry mismatch leads to policy enforcement on reply packet for service loopback case #19406
Comments
maybe @pchaigno |
Did you check if the same bug also affects master? |
Yes, I tried |
I enabled debug logs which confirms that the reply packet is treated as a new packet:
Cluster IP - 10.105.185.194 |
So if I understand correctly, the issue here is that we're not using the correct port when creating the conntrack entry? Did you check the code to see why that is? |
I am using v1.12.0-rc1 for debugging. The conntrack entries are created correctly with this version. Cluster IP - 10.105.185.194
But the issue is with the conntrack lookup:
flags=0 which means the lookup was initialized from Line 664 in 964b034
Line 1078 in 964b034
Somehow packet fails lb4_lookup_service test and misses lb4_local. Does this summary looks accurate?. If so, how do I proceed with determining why service check fails for the reply packet?. |
Line 1974 in c5c7d41
|
I see, thanks. But why would it go through nodeport, it is a cluster IP service. Maybe it is coming from Line 1522 in 964b034
|
Oh, right. That makes more sense. So it's ingressing into the container and we're doing a CT lookup with the VIP instead of the loopback IP. So why wasn't the packet SNATed to the loopback IP on egress? |
Here is the full conntrack lookups for the trafffic. There was a lookup with loopback IP and the verdict was
|
This issue has been automatically marked as stale because it has not |
This issue has not seen any activity since it was marked stale. |
I wonder if the next step could be to create a new test in https://github.com/cilium/cilium-cli connectivity testsuite that configures a service with only one pod backing it, then initiates requests from that pod to the service with policy applied. This could make reproducing the issue more easy, which should also make it easier to debug and figure out a solution. |
If I understand the issue correctly, we already have a test case to verify this scenario - https://github.com/cilium/cilium/blob/master/test/k8s/services.go#L170-L170.
If this is indeed relevant, we should extend the test case. @skmatti Here is context for how we skip the policy enforcement for the service loopback + policy enforcement case - 52cd6da. This hopefully helps in debugging/fixing the issue further. |
This may be fixed by #27798 |
The issue is resolved on cilium
|
Is there an existing issue for this?
What happened?
Steps to reproduce:
It is important that the service port (81) is different from the container port (80). Issue does not occur if they are the same.
Send traffic to itself via cluster IP:
Cilium monitor output:
As per my understanding, a pod sending packets to itself via cluster IP should have skipped policy enforcement all together. Although, the first packet (pod -> itself via cluster IP) is skipped policy enforcement, the reply packet is not skipped (incorrectly subjected to policy enforcement). This is happening because the reply packet do not match any conntrack entries.
As per my understanding, the second entry is causing the problem:
10.0.0.208:55522 -> 10.98.138.60:80
- the destination IP address is the Cluster IP but the port is container port (80) instead of service port (81).The reply packet on the other hand has
10.98.138.60:81 -> 10.0.0.208:55518
which does not match the conntrack entry and is incorrectly subjected to network policy enforcement.Cilium Version
1.10.5
Kernel Version
x86_64
Kubernetes Version
1.22.8
Sysdump
cilium-sysdump-20220411-230655.zip
Relevant log output
No response
Anything else?
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: