@maan2003 I tried to dig into this today but the link you put here isn't working - do you mind updating it?

updated

there is no backoff in reconnect triggered by multiple concurrent requests to federation peer.

cc @dpc

@maan2003 I did a test. just devimint-env, then killall fedimintd, then I run env RUST_LOG=fm::client::net=debug fedimint-cli dev api status and see that timeouts are indeed increasing (thought I do see we need a fix).

there is no backoff in reconnect triggered by multiple concurrent requests to federation peer.

Hmm... I don't see how this could be happening. Multiple requests to FederationPeer<C>::request will synchronize on self.client.read() and self.client.write(), and trigger only one reconnect, then block and synchronize again on rclient.client.get_try().

So before I connect, can we confirm which version are we talking about here @maan2003 . These recent reconnection improvements are not part of 0.3.x and were unreleased yet.

Also, worth mentioning that reconnection tracking is part of a PeerClient<WsClient> and is not persisted anywhere. If the end application was somehow re-creating PeerClient<WsClient> it would keep getting reset.

Also, worth mentioning that reconnection tracking is part of a PeerClient<WsClient> and is not persisted anywhere. If the end application was somehow re-creating PeerClient<WsClient> it would keep getting reset.

Oh. I think I've found something.

Oh. I think I've found something.

Actually, #5587 is an improvement, but given it has backoff logic on its own, it shouldn't lead to hammering peers.

FYI. From my perspective everything looks good, after I've fixed the above issues. We had a call today where I showed more people what I looked at and why everything LGTM now.

Until there is more concrete details with latest code, I consider this fixed.

I am going to close this for now, will reopen if we hit this again.