How using separate egress_pools for different ESPs defined in queue.toml?

I already know that in shaping.toml we can set different speeds based on provider — it’s really useful.

Now, in my use case: I need certain tenant(s) to use dedicated egress_pools for specific providers such as Outlook mailboxes.
For example, client A considers Yahoo very important, so messages to Yahoo should be sent through the yahooip egress_pool. In other words, all traffic to Yahoo (this ESP) should use the yahooip egress_pool for delivery.

[tenant.'test']
egress_pool = 'test'


[queue.'yahoo.com'.'test'] 
egress_pool = 'yahooips'

[queue.'yahoo.com.br'.'test'] 
egress_pool = 'yahooips'

[queue.'yahoo.com.au'.'test'] 
egress_pool = 'yahooips'

[queue.'yahoo.com.in'.'test'] 
egress_pool = 'yahooips'

[queue.'yahoo.com.mx'.'test'] 
egress_pool = 'yahooips'

[queue.'yahoo.com.vn'.'test'] 
egress_pool = 'yahooips'

[queue.'yahoo.com.jp'.'test'] 
egress_pool = 'yahooips'

.....



That approach could work, even if I use Lua to implement it. However, many corporate mailboxes have MX records that also point to Yahoo, and I have no way to exhaustively cover all of them.

Is there a better solution? At the queue stage, MX resolution probably hasn’t happened yet, so it may indeed be difficult to handle.

The queue helper has a function you can use for doing provider-specific lua, one example:

local queue_module = require 'policy-extras.queue'

kumo.on('smtp_server_message_received', function(msg)
   local domain = msg:recipient().domain
   local provider = queue_module:resolve_provider(domain)
   if provider == "undesirable_provider_name" then
      kumo.reject(550, "some reason to pass back to injector")
   end
end)

Thanks for your reply. I’ll run detailed tests.

When I write in the smtp_server_message_received method, whatever I set there gets overridden by queue_helper, even if I place it below the queue_helper code.

 if tenant == 'test' then
    local domain = msg:recipient().domain
    local provider = queue_module:resolve_provider(domain)
    if provider == "google" then
      msg:set_meta('egress_pool', 'google_pool')
    end
  end
  -- Call the queue helper to set up the queue for the message.
  queue_helper:apply(msg)

or

-- Call the queue helper to set up the queue for the message.
queue_helper:apply(msg)
 
if tenant == 'test' then
    local domain = msg:recipient().domain
    local provider = queue_module:resolve_provider(domain)
    if provider == "google" then 
      msg:set_meta('egress_pool', 'google_pool')
    end
  end
  

I might need to try it in get_queue_config.

if tenant == 'test' then
    local provider = queue_module:resolve_provider(domain)
    if provider == "google" then  
      return kumo.make_queue_config {  
        egress_pool = 'google_pool'
      }
    end
end

I feel it’s not quite appropriate to handle resolver_provider here.The main concern is performance overhead.

However, I’m concerned that it could disrupt the normal operation of the queue helper.

Thank you for your reply.

You will need to override the get_queue_config handling so that you can return your provider-specific configuration for that domain.

Note that get_queue_config is called according to the refresh strategy you define on that queue configuration, and that any DNS resolution performed to resolve the provider is also cached, so performance is not likely to be an issue.

If you were not using the queue helper to set scheduled queue configuration, you don’t need to call queue_helper:apply(msg), you can use the resolve_provider function independent from using the rest of the queue helper module.

If you are using it, then you’ll probably want to use skip_queue_config_hook = true when you queue_module:setup_with_options, then you can use queue_helper:resolve_config(domain, tenant, campaign) in your get_queue_config handler to extract the queue configuration from the helper.

In most cases, I directly use the queue helper, and only when it’s truly impossible to use the helper do I write it in Lua code.

I need to delve deeper into what you said; it might be a bit challenging for me.

I’m thinking that once I become familiar with KumoMTA, I could contribute best-practice documentation on GitHub to help make Kumomta easier to use.

Thank you!