Any way to enrich a message at the start_esmtp_listener stage?

get-help
1

Hi,
Note, just a comparison to understand if I’m not looking in the right direction to achieve what I want

Momentum guy here :slightly_smiling_face: I’ve recently resumed my research and POCs with KumoMTA after a few months

My need is to activate multiple Listeners, the messages received on these listeners must be identifiable later. Is there a way to add “context variables” to messages as they come in through the Listeners?

Doesn’t seem possible from the reference start_esmtp_listener

Example from “Momentum”:

ESMTP_Listener {
  Listen "172.23.123.233:25" {
    Peer "0.0.0.0/0" {
        context = [
           message_type = "A",
           customer_id = "FOO"
        ]
    }
  }
}
ESMTP_Listener {
  Listen "192.168.45.176:25" {
    Peer "0.0.0.0/0" {
        context = [
           message_type = "B",
           customer_id = "BAR"
        ]
    }
  }
}

With this setup, I can later retrieve these context variables at any stage of email processing in Momentum:

vctx_customer_id = vctx_conn_get "customer_id";
vctx_message_type = vctx_conn_get "message_type";
ec_header_add "X-My-Customer-ID" "${vctx_customer_id}";
ec_header_add "X-My-Message-Type" "${vctx_message_type}";

This approach is really useful as it allows me to avoid writing additional code or maintaining extra mappings

I know I could extract this information from the, for example, received_via metadata, but that would require setting up and maintaining a mapping like:

172.23.123.233:25 message_type A
172.23.123.233:25 customer_id FOO
192.168.45.176:25 message_type B
192.168.45.176:25 customer_id BAR

to be used in the 'smtp_server_message_received' function(msg, conn) stage, however, this would mean maintaining an additional database, handling connections, caching, exceptions, and memory usage in order to retrieve this data for every single message

Is there a simpler/alternative way to achieve this in KumoMTA that I might have overlooked?

4

Aah yeah, I forgot about the context setting in Listeners from Momentum. At this time that is a mapping you would do after the fact, but I’ll add a ticket to the roadmap for that, I remember building out a few customers with that functionality.

5

Thanks for the interest, let’s make extensive use of this function across different listeners, I will continue at this point, for testing purposes, with the ‘manual’ implementation mentioned above.

6

Yeah, given this is the first time it’s been requested it’s a Needs Funding, but it’s in: Support for configuring predefined metadata in Listener definition · Issue #355 · KumoCorp/kumomta · GitHub

7

While you cannot set that in the listener, you can do it right after. The connection vars you are looking for are probably held in conn:meta which is valid even before message_received event fires.

once the message hits the “received” event, that conn:meta is automatically rolled into the message:meta.
IE: in Momentum terms: Connection Context gets included into Message Context.

SO for instance, if you wanted to drop a message at the EHLO based on IP address you can do that without wasting time processing the message:

8

I have also used that for reporting later, so that when you get to the log reporting stage, you can add the meta data for received_via which will record which listener IP and port KumoMTA received it on.

9

When migrating Momentum configs (which I am happy to help with in a PS engagement) you can think of most of the Connection and Message VCTX data as KumoMTA metadata.

10

Hi, yes I know I can use, for instance, the received_via variable.

like:

kumo.on('smtp_server_message_received', function(msg, conn)
  
  print(conn:get_meta("received_via"))
  print(conn:get_meta("hostname"))
end)

But that’s exactly what I want to avoid, as described in my original post.

The context variables on the Listener are needed so I can perform subsequent operations (routing, headers, processing, etc.) without translate nothing from the “connection”, and they are key/value pairs that I can use freely

Thanks for the feedback

11

AH, I see

12

You can do this today with a bit of lua glue, something like this:

local LISTENERS = {
  {
    params = {
       listen = "172.23.123.233:25",
       -- insert any other params for start_esmtp_listener here
    },
    context = {
      ["0.0.0.0/0"] = {
          message_type = "A",
          customer_id = "FOO",
      }
    }
  }
}

local function make_listener_lookup()
   local listener_map = {}
   for _, listener in pairs(LISTENERS) do
      listener_map[listener.params.listen] = listener.context
   end
   return kumo.cidr.make_map(listener_map)
end

-- You could kumo.memoize this if needed
local function lookup_context(via, peer)
  local listener_map = make_listener_lookup()
  local info = listener_map[via] or {}
  local peer_map = kumo.cidr.make_map(info)
  return peer_map[peer] or {}
end

kumo.on('init', function()
  for _, listener in pairs(LISTENERS) do
     kumo.start_esmtp_listener(listener.params)
  end
end)

kumo.on('smtp_server_ehlo', function(_domain, conn_meta)
  local via = conn_meta:get_meta 'received_via'
  local peer = conn_meta:get_meta 'received_from'
  local peer_ip, peer_port = policy_utils.split_ip_port(peer)
  local data = lookup_context(via, peer_ip)
  for k, v in pairs(data) do
     conn_meta:set_meta(k, v)
  end
end)
13

Thanks @free-spirited-yorksh for your suggestion!

I was actually implementing something quite similar. I was more inclined to retrieve these variables from a mapping file in .toml style via lua and bootstrap the listeners by reading that file ( following specs TOML: English v1.0.0 )

Something Like:

[listener.'172.23.123.233:25']
# default for listener
banner = '...'
hostname = '...'
# extra for listener
context.message_type = "A"
context.customer_id = 'FOO'

[listener.'192.168.45.176:25']
# default for listener
banner = '...'
hostname = '...'
# extra for listener
context.message_type = "B"
context.customer_id = 'BAR'

You’ve given me some tips for the implementation.

Thanks!

15

Hi, I ended up creating a prototype based on what was discussed above. I hope it can be generally useful for someone else.

I created a file called “listeners.toml”:

[listener.'172.23.123.233:25']
# default for listener
params.listen = '172.23.123.233:25'
params.banner = '...'
params.hostname = '...'
params.relay_hosts = [ '127.0.0.1', '192.168.1.0/24', '172.23.123.232' ]
# extra for listener
context.message_type = "A"
context.customer_id = 'FOO'

[listener.'172.23.123.234:25']
params.listen = '172.23.123.234:25'
params.banner = '...'
params.hostname = '...'
params.relay_hosts = [ '127.0.0.1', '192.168.1.0/24', '172.23.123.232' ]
# extra for listener
context.message_type = "B"
context.customer_id = 'BAR'

I imported in init.lua:
local utils = require 'policy-extras.policy_utils'

Then, I loaded and bootstrapped the listeners automatically like this:

kumo.on('init', function()
  local test = utils.load_json_or_toml_file('/tmp/listeners.toml');
  -- print(value_dump(test));
  for k, listener in pairs(test.listener) do
      kumo.start_esmtp_listener(listener.params)
  end
end)

And I read the “context variables” on the EHLO:

kumo.on('smtp_server_ehlo', function(_domain, conn_meta)
  local test = utils.load_json_or_toml_file('/tmp/listeners.toml');
  -- print(value_dump(test));

  local via = conn_meta:get_meta 'received_via'
  -- conn_meta:set_meta(k, v)

  print("customer_id: " .. test.listener[via].context.customer_id);
  print("message_type: " .. test.listener[via].context.message_type);
end)

Next: swaks --to dummy-to-tenant-nope@example.com --from dummy-from@example.com --server 172.23.123.234 --port 25
Results in:

Mar 21 15:54:41 -- kumod[324188]: customer_id: BAR
Mar 21 15:54:41 -- kumod[324188]: message_type: B

as expected

This way, everything worked as I had envisioned.

Of course, this is just a prototype and can be improved in various ways, but it does what I need.

16

Implemented in smtp_server: add `meta` parameter to start_esmtp_listener · KumoCorp/kumomta@8253417 · GitHub give it a few minutes to build.

17

Docs at:

and if you want things to be more dynamic:

18

Oh, this is really good news! Thanks for the direct implementation in the source code! :folded_hands:

I’m applying as a beta tester for the function :grinning_face_with_smiling_eyes: in this week I build and test

19

Hi, It works! Thanks!

Initially, I tried using meta and managed to bootstrap listeners from my custom TOML file to avoid a massive Lua configuration file in case of many of dedicated listener IPs.

Here’s my TOML configuration:

[listener.'172.23.123.233:25']
# Default for listener
params.listen = '172.23.123.233:25'
params.banner = '...'
params.hostname = '...'
params.relay_hosts = [ '127.0.0.1' ]
params.meta = { message_type = "A", customer_id = "FOO" }

[listener.'172.23.123.234:25']
# Default for listener
params.listen = '172.23.123.234:25'
params.banner = '...'
params.hostname = '...'
params.relay_hosts = [ '127.0.0.1' ]
params.meta = { message_type = "B", customer_id = "BAR" }

And the corresponding Lua script:

kumo.on('init', function()
  local test = kumo.serde.toml_load('/tmp/listeners.toml')
  print(value_dump(test))
  for k, listener in pairs(test.listener) do
    kumo.start_esmtp_listener(listener.params)
  end
end)

kumo.on('smtp_server_message_received', function(msg, conn)
  print(string.format("customer_id: %s", conn:get_meta("customer_id")))
  print(string.format("message_type: %s", conn:get_meta("message_type")))
end)

test variable is correctly dumped as follows:

{
  "listener": {
    "172.23.123.234:25": {
      "params": {
        "listen": "172.23.123.234:25",
        "relay_hosts": [
          "127.0.0.1",
          "192.168.1.0/24",
          "172.23.123.232"
        ],
        "hostname": "...",
        "banner": "...",
        "meta": {
          "customer_id": "BAR",
          "message_type": "B"
        }
      }
    }
<cut>

To test it, I sent an email:

$ swaks --to dummy-to-tenant-nope@example.com --from dummy-from@example.com --server 172.23.123.233 --port 25

The output:

Mar 24 11:55:58 - - kumod[176504]: customer_id: FOO
Mar 24 11:55:58 - - kumod[176504]: message_type: A

Next, I’ll proceed with some stress tests and the other via, peer, dy

Thanks again

20

note that you can also use kcli trace-smtp-server and it will show you metadata as it changes