UnableToConnect on 10.x.x.x:10226/Subscription, Initializing/NotStarted

[vipinlalvs]

We are utilizing Azure Redis within a VNET configured with the following allowed port ranges: 15000-15999, 13000-13999, 6380, and 6379. Following a scale-up of our Azure Redis Cache, we are consistently encountering the following error from the connection multiplexer, as the specified port is not permitted within our VNET configuration.

"UnableToConnect on 10.x.x.x:10226/Subscription, Initializing/NotStarted, last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never (mid-beat), last-mbeat: 0s ago, global: 0s ago, v: 2.8.0.27420"

Could someone help clarify why StackExchange.Redis is attempting to connect to the 10221-10231 port range and explain its purpose?

@mgravell @rjmooney

[mgravell]

The library does what the server tells it. Do you have console access to the server to issue RESP commands via redis-cli or similar? If so: CLUSTER NODES will tell you what the server is declaring as the shard topology - I expect that is where these addresses are coming from. The SE.Redis library also has an inbuilt ExportConfiguration method to download the server topology etc as a zip file (of a bunch of related command outputs) if that is useful.

[vipinlalvs]

It never shows the 10226 port here in the "CLUSTER NODES"

@mgravell this is what we are getting.

d2b386b285e2a7f32202a35165b13e2c8b4eee02 10.x.x.x:15006 master - 0 1734612415000 11 connected 5462-8191 13653-16383
f60d1eb64a938a87eb1d42bef46d65e1833e5940 10.x.x.x:15001 slave 2077e68a8587515b5a15c8374a228a17cfa49c1c 0 1734612414567 4 connected
04b661f93b2dd6e99888c9680342d7e093d9f529 10.x.x.x:15008 slave d2b386b285e2a7f32202a35165b13e2c8b4eee02 0 1734612415000 11 connected
b4018e341cfff46024e844674dd7f1bc124ca392 10.x.x.x:15004 slave a0e8264650dfb4e0a90a3b53588f797f72085a8c 0 1734612415630 10 connected
a0e8264650dfb4e0a90a3b53588f797f72085a8c 10.x.x.x:15003 myself,master - 0 1734612414000 10 connected 8192-13652
2077e68a8587515b5a15c8374a228a17cfa49c1c 10.x.x.x:15002 master - 0 1734612414567 4 connected 0-5461
39a2e6b32489696339a4ce959e27842b1cbeb446 10.x.x.x:15005 slave a0e8264650dfb4e0a90a3b53588f797f72085a8c 0 1734612414364 10 connected
7a592ab48117de4574640e3efc90f821a648f797 10.x.x.x:15007 slave d2b386b285e2a7f32202a35165b13e2c8b4eee02 0 1734612414567 11 connected
1d2b7c37355dbf94d21ac3cedc85c5bbe3a3d32b 10.x.x.x:15000 slave 2077e68a8587515b5a15c8374a228a17cfa49c1c 0 1734612415000 4

[vipinlalvs]

We previously had a single-shard cluster and recently scaled up and scaled out to a 3-shard configuration. Following this change, we started encountering this issue.

Detailed logging is enabled on StackExchange.Redis, and we are observing errors. Could these errors be related to internal failures within the Redis cluster being propagated and reported by the client?

[mgravell]

well, that's weird; we don't make up ports, honest! So: they came from somewhere! is it possible that those ports were specified at some point during the resharding? Also, since this is azure redis - maybe @philon-msft can shed some light on anything weird that happened

[philon-msft]

@vipinlalvs, it does sound like there may be a problem in the cache itself. Could you send me the name of the cache? My microsoft address is the same my github ID minus "-msft"

[vipinlalvs]

@philon-msft We have been collaborating with Microsoft on this matter for some time. I can share the ticket ID if it would be helpful. I have sent you an email regarding the same.

Ticket: 2412190030004442

[philon-msft]

It looks like this has been resolved, and was caused by an unhealthy node in the AKS cluster where the client app was running.

[vipinlalvs]

It wasn't due to any issue in the cluster!.
"I suspect that the cache scale operation left it in a bad state where it advertised ports that did not match the actual available ports within the cache. The incident RCA should include more details from the engineer who mitigated the issue. "

this was the last update we had with MS support team. @philon-msft can you double check the ticket?

[philon-msft]

I was suspicious of the scale operation, but notes in the support ticket indicate that rebooting an AKS node resolved the issue.

[vipinlalvs]

@philon-msft we had similar behaviour today, after the scheduled maintenance of redis. And it says like the client is trying to connect 10266, which is never a shard port. I will email you the details along with the ticket!

I doubt there is some issue with redis after maintenance!

[philon-msft]

What port is specified in your Redis connection string?

[vipinlalvs]

It's on 6380.