Help Needed: KumoMTA WebSocket Errors and IO Issues

get-help
1

I’m experiencing some issues with my KumoMTA setup and could use some help. I’ve encountered a series of errors in the logs that seem related to WebSocket communication and IO operations. I’ve included the relevant log entries below for reference:

Sep 10 10:40:28 test.dummy-domain.com kumod[3555350]: 2024-09-10T10:40:28.258517Z  INFO localset-0 kumod::spool: start_spool: still enumerating. 427794 items in 3870.165372425s 111/s
Sep 10 10:40:34 test.dummy-domain.com kumod[3555350]: 2024-09-10T10:40:34.862137Z ERROR localset-0 kumod::http_server::admin_trace_smtp_client_v1: error in websocket: IO error: Broken pipe (os error 32): IO error: Broken pipe (os error 32): Broken pipe (os error 32)
Sep 10 10:40:44 test.dummy-domain.com kumod[3555350]: TSA Error, will retry in 30 seconds        false        WebSocket protocol error: Connection reset without closing handshake
Sep 10 10:40:44 test.dummy-domain.com kumod[3555350]: stack traceback:
Sep 10 10:40:44 test.dummy-domain.com kumod[3555350]:         [C]: in local 'poll'
Sep 10 10:40:44 test.dummy-domain.com kumod[3555350]:         [string "?"]:4: in method 'recv'
Sep 10 10:40:44 test.dummy-domain.com kumod[3555350]:         /opt/kumomta/share/policy-extras/shaping.lua:191: in function </opt/kumomta/share/policy-extras/shaping.lua:180>
Sep 10 10:40:44 test.dummy-domain.com kumod[3555350]:         [C]: in function 'pcall'
Sep 10 10:40:44 test.dummy-domain.com kumod[3555350]:         /opt/kumomta/share/policy-extras/shaping.lua:206: in function </opt/kumomta/share/policy-extras/shaping.lua:200>
3

Looks to me that the kumo-tsa-daemon is not running and kumod cannot connect to it. But in order to help better we would need to see some bits of the configuration.

4

Thanks for pointing that out! It does seem like the kumo-tsa-daemon might not be running properly.

Could you let me know which specific configuration files or sections you’d like to see in order to help troubleshoot this issue further? I can share any relevant details you need.

5

Always share all configs.

6

Shaping.toml

["default"]
connection_limit = 10
max_connection_rate = "100/min"
max_deliveries_per_connection = 100
max_message_rate = "100/s"
idle_timeout = "60s"
data_timeout = "30s"
data_dot_timeout = "60s"
enable_tls = "Disabled"
consecutive_connection_failures_before_delay = 100

["gmail.com"]
max_deliveries_per_connection = 50
connection_limit = 5
enable_tls = "Required"
consecutive_connection_failures_before_delay = 5

["yahoo.com"]
max_deliveries_per_connection = 20

[["yahoo.com".automation]]
regex = "\\[TS04\\]"
action = "Suspend"
duration = "2 hours"

["comcast.net"]
connection_limit = 10
max_deliveries_per_connection = 250
enable_tls = "Required"
idle_timeout = "30s"
consecutive_connection_failures_before_delay = 24

["mail.com"]
max_deliveries_per_connection = 100

["example.com"]
mx_rollup = false
max_deliveries_per_connection = 100
connection_limit = 3

["example.com".sources."sources"]
connection_limit = 5
7

tsa_init.lua

local tsa = require 'tsa'
local kumo = require 'kumo'

kumo.on('tsa_init', function()
  tsa.start_http_listener {
    listen = '0.0.0.0:8008',
    trusted_hosts = { '127.0.0.1', '::1' },
  }
end)

local cached_load_shaping_data = kumo.memoize(kumo.shaping.load, {
  name = 'tsa_load_shaping_data',
  ttl = '5 minutes',
  capacity = 4,
})

kumo.on('tsa_load_shaping_data', function()
  local shaping = cached_load_shaping_data {
    '/opt/kumomta/share/policy-extras/shaping.toml',
    --    '/opt/kumomta/etc/policy/shaping.toml',
  }
  return shaping
end)
8

Have you followed Configuring Traffic Shaping - KumoMTA Docs ?

9

Hey, thanks for sharing the link!

I’ve gone through the Traffic Shaping section and the debugging tips in the KumoMTA docs. I’ve been working on configuring traffic shaping for different ESPs and MBPs like Gmail, Yahoo, Outlook, AOL, and some others, but I’m still hitting roadblocks when it comes to:

  1. Max connections: Should I set the same limits across all ESPs (like Gmail, Yahoo, Outlook, AOL, etc.) or fine-tune these for each provider based on feedback? For instance, Gmail seems stricter with temporary blocks, while Yahoo sometimes has other issues like TS04 blocks.

  2. Shaping across different MBPs: How do I optimize the max connection and message rate for MBPs (like Rackspace, Amazon WorkMail, etc.)? I want to ensure I hit the ideal limits for each one.

  3. Could you share any example shaping.toml configurations that you’ve found to be effective across these ESPs/MBPs?

  4. Additionally, if you have an example of a tsa_init.lua config that’s optimized for traffic shaping, that would be super helpful as well!

Any advice or tips for fine-tuning traffic shaping for various ESPs and MBPs would be greatly appreciated!

Thanks again for your help!

10

I was asking if you verified the tsa daemon was running.

11

Sorry Mike, my bad No tsa-daemon is not running

12

Ok then the troubleshooting section says how to start it.

13

As for what to put in your TSA configs, I’ll leave that one to the community. We already give the software away for free, our experience has to be kept for the sponsors.

14

my tas-daemon status

● tsa-daemon.service - TSA Daemon for KumoMTA
     Loaded: loaded (/etc/systemd/system/tsa-daemon.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-09-10 16:22:32 UTC; 25s ago
   Main PID: 3579265 (tsa-daemon)
      Tasks: 8 (limit: 18910)
     Memory: 8.0M
        CPU: 259ms
     CGroup: /system.slice/tsa-daemon.service
             └─3579265 /opt/kumomta/sbin/tsa-daemon --policy /opt/kumomta/etc/policy/tsa_init.lua

Sep 10 16:23:02 test.dummy-domain.com tsa-daemon[3579265]:     unable to open database file (code 14)
Sep 10 16:23:02 test.dummy-domain.com tsa-daemon[3579265]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Sep 10 16:23:02 test.dummy-domain.com systemd[1]: tsa-daemon.service: Main process exited, code=dumped, status=3/QUIT
Sep 10 16:23:02 test.dummy-domain.com systemd[1]: tsa-daemon.service: Failed with result 'core-dump'.
Sep 10 16:23:02 test.dummy-domain.com systemd[1]: tsa-daemon.service: Scheduled restart job, restart counter is at 20899.
Sep 10 16:23:02 test.dummy-domain.com systemd[1]: Stopped TSA Daemon for KumoMTA.
Sep 10 16:23:02 test.dummy-domain.com systemd[1]: Started TSA Daemon for KumoMTA.
Sep 10 16:23:02 test.dummy-domain.com tsa-daemon[3580134]: 2024-09-10T16:23:02.749643Z  INFO localset-0 kumo_server_runtime: localset pool starting with 1 threads
Sep 10 16:23:02 test.dummy-domain.com tsa-daemon[3580134]: 2024-09-10T16:23:02.784333Z  INFO localset-0 kumo_server_common::http_server: http listener on 0.0.0.0:8008
Sep 10 16:23:02 test.dummy-domain.com tsa-daemon[3580134]: 2024-09-10T16:23:02.784395Z  INFO localset-0 kumo_server_common::start: initialization complete
15

And now a systemctl status kumo-tsa-daemon

16

Also can you curl against http://127.0.0.1:8008/get_config_v1/shaping.toml to see what comes back?

17

Hey @yearning-hyena – Sorry to bring up an old thread, but this is somewhat relevant. Can you confirm that the tsa-daemon will autostart in the docker container?

From what I can see, only the init.lua is started with the docker-runner.sh script. I had to manually bring it up with the following command in the container:

root@a585b6081922:/opt/kumomta/sbin# ./tsa-daemon --policy /opt/kumomta/etc/policy/tsa_init.lua. to stop this error:

kumo-sink  | TSA Error, will retry in 30 seconds        false   callback error
kumo-sink  | stack traceback:
kumo-sink  |    [C]: in local 'poll'
kumo-sink  |    [string "?"]:5: in field 'connect_websocket'
kumo-sink  |    /opt/kumomta/share/policy-extras/shaping.lua:184: in function </opt/kumomta/share/policy-extras/shaping.lua:180>
kumo-sink  |    [C]: in function 'pcall'
kumo-sink  |    /opt/kumomta/share/policy-extras/shaping.lua:206: in function </opt/kumomta/share/policy-extras/shaping.lua:200>
kumo-sink  | caused by: IO error: Connection refused (os error 111)
18

New thread pls.