High rate of "failed to connect to any candidate hosts" transient errors

get-help
21

I haven’t explicitly set the connect_timeout parameter, should default to 60s, right?

22

that’s correct

23

Hello @original-baboon , could you share what changes did you make exactly to fix the issue? We’re also seeing quite a lot of
KumoMTA internal: failed to connect to any candidate hosts: connect to [2607:f8b0:4004:c09::79]:25 and read initial banner: Skipping xx.xx.xx.106 because ha_proxy_source_address xx.xx.xx.145 address family does not match the destination address family [2607:f8b0:4004:c09::79]:25, connect to xx.xx.xx.121:25 and read initial banner: reading banner: Not connected

24

Ok, we just tell Kumo not to use IPv6

25

Our situation was different, we were getting those errors when it was trying to send to ipv4. The problem was that the proxy service was running out of file descriptors, made some changes in /etc/security/limits.conf and /etc/sysctl.conf on the proxy servers to increase the limits and that took care of it.

26

‘kumod 2024.06.18-308d4301’ is super old. The last stable release was November. I highly recommend upgrading to the latest.

27

Sorry for bumping this thread, but it was the only one that looked relevant to the issue. We also seemed to have an issue with the proxy:

[2025-06-25T21:59:52Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.3.71:54654: early eof: early eof
[2025-06-25T21:59:54Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.2.144:40672: early eof: early eof
[2025-06-25T21:59:54Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.3.71:54668: early eof: early eof
[2025-06-25T21:59:56Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.2.144:40676: early eof: early eof
[2025-06-25T21:59:56Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.3.71:38842: early eof: early eof
[2025-06-25T21:59:58Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.2.144:40678: early eof: early eof
[2025-06-25T21:59:58Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.3.71:38856: early eof: early eof
[2025-06-25T22:00:00Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.2.144:40684: early eof: early eof
[2025-06-25T22:00:00Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.3.71:38868: early eof: early eof
[2025-06-25T22:00:02Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.2.144:40688: early eof: early eof
[2025-06-25T22:00:02Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.3.71:38872: early eof: early eof
[2025-06-25T22:00:04Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.2.144:54138: early eof: early eof
[2025-06-25T22:00:04Z ERROR proxy_server] proxy session error: failed to read client handshake from 10.20.3.71:38884: early eof: early eof
[2025-06-25T22:00:06Z ERROR proxy_server] accept failed: Too many open files (os error 24)```

After the Too many open files, the process crashed - it's running but refusing all connections. Running `Version: 2025.05.06.055544.b29689af` via apt on the proxies.

I noticed the sockets are being left open:

```# ss -ant | awk '{print $1}' | sort | uniq -c
    247 CLOSE-WAIT
      2 ESTAB
    247 FIN-WAIT-2
      6 LISTEN
      1 State```

Though I am not too familiar with these, asking AI tells me CLOSE-WAIT's indicate the clients are not correctly closing the connection. Is this possibly a bug or something we have to fix by changing configuration?

I will raise the value in sysctl, but it feels to me that this only postpones the issue until that new limit gets hit
28

Just out of curiosity, since I’ve dealt with in the past with connection issues, socket and open file limits on some systems as well as on my old MTAs:

What kernel version and settings do you have for:
tcp_tw_reuse
tcp_fin_timeout
tcp_keepalive_time

29

I haven’t had this issue since I updated kumo proxy back then and made some changes to sysctl, the exception is when mailbox provider is actually dropping the connections for whatever reason. For example, if you don’t back off when Yahoo is responding with IPTS* / TSS* / TS* errors they will start dropping connections from your IP and you’ll start seeing early eof or other similar errors in the logs.

30 
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_keepalive_time  = 300
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 6

Sorry for the late reply - but recently updated the values to these. However, still seeing an increasing amount of CLOSE-WAITs:

    154 CLOSE-WAIT
      2 ESTAB
    154 FIN-WAIT-2
      7 LISTEN
      1 State
     32 TIME-WAIT```
31

Ah, interesting. I’m not seeing bounces, so that’s odd… but a good one to keep in mind