Abnormally high number of connections from go-redis when using ClusterClient

[Nashluffy]

Using the ClusterClient, we're seeing an extremely high number of connections opened to Redis. In the screenshot below, half the requests are MSets and half are MGets. All commands are wrapped in pipelines and we do not use hashtags. We have 15 nodes in the cluster. Usually both the client and Redis are inside an istio mesh, but we have pulled Redis outside the mesh for the purposes of this test. This is on a GKE cluster.

There's certainly additional overhead in clustered redis as go-redis breaks up a pipeline by node, so some fan-out is expected. However, there's an overwhelming amount of connections being opened to Redis, to the point the majority of our read requests are taking > 15ms and start to fail due to i/o timeout.

ClusterClient config immediately before client is created

Addrs: [redis-leader-0.redis-leader-headless:6379 redis-leader-1.redis-leader-headless:6379 redis-leader-2.redis-leader-headless:6379] 
ClientName: NewClient:nil 
MaxRedirects:0 
ReadOnly:false 
RouteByLatency:false 
RouteRandomly:false 
ClusterSlots:nil 
Dialer:nil 
OnConnect:nil 
Protocol:0 
Username: 
Password: 
CredentialsProvider:nil
CredentialsProviderContext:nil
MaxRetries:-1 
MinRetryBackoff:1ms 
MaxRetryBackoff:1ms 
DialTimeout:50ms 
ReadTimeout:15ms 
WriteTimeout:3s 
ContextTimeoutEnabled:false 
PoolFIFO:false 
PoolSize:0 
PoolTimeout:0s 
MinIdleConns:0 
MaxIdleConns:0 
MaxActiveConns:0 
ConnMaxIdleTime:0s 
ConnMaxLifetime:0s 
TLSConfig: nil 
DisableIndentity:false 

Interestingly the clients specified in the original config have significantly higher open connections. IIUC the client should be updated with all leaders in the cluster and round-robin between them

Example trace

[Nashluffy]

We noticed a high number of idle connections and decided to exclude redis traffic from Envoy on the client side. The number of idle connections immediately dropped. Both performance and the number of connections across all leaders is much more even now.

cat clients.txt | awk  '{print  $2}' | cut -d '=' -f2 | cut -d':' -f1 | sort | uniq -c
    344 10.136.15.57
    199 10.136.35.52
    248 10.136.44.100
    156 10.136.47.22
    446 10.136.47.23
    349 10.136.47.25
   1189 10.136.55.10
   1120 10.136.55.11
    414 10.136.55.12
     62 10.136.55.13
    544 10.136.55.5
    920 10.136.55.6
    469 10.136.55.7
   1089 10.136.55.8
    176 10.136.55.9
    229 10.136.58.187
    161 10.136.58.192
    404 10.136.62.16
    396 10.136.62.17
    435 10.136.62.18
      1 127.0.0.1