Not seeing keep alive pings in grpc services with istio proxy in between #12382
Description
So I'm running into some weird issues with configuring keep alive. I have java based grpc client and server running in same namespace. Each has their own istio proxy running as side car.
Firstly when I look into tcp dump, I can see PINGS[0] from client to server but no PING[0] with ACK set back. The connections is not getting reset however. Even if the rate is less than keepalivepermit time im not seeing any GO AWAY[0]. I filtred out TCP type and checked all other packets. This packet was took from all of the interface from the pod like this
kubectl exec -n namespace -i podname -- tcpdump -i any -U -s0 -w - | wireshark -k -i -
Later i enabled trace logs of io.grpc. In that I can see the below in client logs
│ {"t":"2025-09-28T05:23:13.521Z","msg":"[id: 0x1ac0fbdb, L:/10.XX.XX.XX:55332 - R:grpc-server-dev.svc.cluster.local/172.XX.XX.XX:6868] OUTBOUND PING: ack=false bytes=1111","lgr":"io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler","trd":"gr │
│ pc-default-worker-ELG-1-2","lvl":"DEBUG"} │
│ {"t":"2025-09-28T05:23:13.521Z","msg":"[id: 0x1ac0fbdb, L:/10.XX.XX.XX:55332 - R:grpc-serve-dev.svc.cluster.local/172.XX.XX.XX:6868] INBOUND PING: ack=true bytes=1111","lgr":"io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler","trd":"grpc │
│ -default-worker-ELG-1-2","lvl":"DEBUG"}
To my surprise I can see in client logs the INBOUNG and OUTBOUND PING with its ACK. This means my client is getting ack for its PINGS. I still haven't found out why i didnt get this in my TCP dump
BUT I dont see the reverse of this in my server logs. It has also io.grpc trace logs enabled. I don't see the INBOUND PING with ack=false and OUTBOUND PING with ack=true in my server logs
And I don't see GO_AWAY also if permitkeepalive threshold is greater than my keepalive time.
This setup completely works fine when I run these two services in my local machine. Everything works fine. However in my cluster im facing these weird issues. Any help would be greatly appreciated, I'm breaking my head for a week now trying to find out what is happening