Automatic certificate reloading

get-help
1

It seems like there is no way to reload the certificate other than restarting Kumo.

in init.lua

  kumo.start_esmtp_listener {
    listen = '0.0.0.0:587',
    relay_hosts = { '0.0.0.0/0' },
    max_recipients_per_message = 50,
    tls_certificate = "/opt/kumomta/etc/ssl/certs/fullchain.crt",
    tls_private_key = "/opt/kumomta/etc/ssl/private/key.pem",
    trace_headers = {
      -- add the Received: header
      received_header = true,

      -- add the supplemental header
      supplemental_header = true,

      -- the name of the supplemental header
      header_name = 'X-AhaSendRef',
    },
  }

Certificate expiration date shown by openssl:

openssl s_client -starttls smtp -connect send.ahasend.com:587 -servername send.ahasend.com 2>/dev/null | openssl x509 -noout -dates -subject -issuer
notBefore=Nov 21 06:43:46 2024 GMT
notAfter=Feb 19 06:43:45 2025 GMT
subject=CN=mx.ahasend.com
issuer=C=US, O=Let's Encrypt, CN=E6

Certificate details on disk:

openssl x509 -in /opt/kumomta/etc/ssl/certs/fullchain.crt -noout -enddate
notAfter=Apr 20 11:32:17 2025 GMT

I love that Kumo is so stable that I don’t have to restart unless I do an upgrade, but I’m using Let’s Encrypt to get the certs, which means that I’ll have to restart Kumo at least once every three months to make sure it doesn’t continue using the same certificate after it has expired. Would be great if there was an option to tell Kumo to reload the certificate every 24 hours - or if it’d automatically reload the certificate periodically itself without having to tell it in the config.

3

Yes, there is no explicit restarting of a listener. It is one of the few things that requires a restart.

4

It is technically possible to make the certs reload without a restart of the whole listener, but it needs a bit of engineering effort. I think there’s also a story around handling integrated let’s encrypt processing here too via HTTP and/or DNS challenges, as well as just reloading when the files change on disk, so I would consider both together when sitting down to design this.

5

(the integrated let’s encrypt is better suited to the http listener, of course, not the smtp listener)

6

I was thinking of something like this in lua:

local cached_file = kumo.memoize(load_file, {
  name = 'certs',
  ttl = '24 hours',
  capacity = 1,
})
local cached_pkey = kumo.memoize(load_file, {
  name = 'privkeys',
  ttl = '24 hours',
  capacity = 1,
})

local function load_file(file)
  local f = io.open(file, "r")
  local data = f:read("*a")
  f:close()
  return data
end

kumo.on('init', function()
  kumo.start_esmtp_listener {
    tls_certificate_func = function()
      return cached_cert("/opt/kumomta/etc/ssl/certs/fullchain.crt")
    end,
    tls_private_key_func = function()
      return cached_pkey("/opt/kumomta/etc/ssl/private/key.pem")
    end,
    -- ...
  }
end)

But of course this also needs some engineering work to support getting the cert/private key by calling a function.

7

that’s a bigger sort of change than I had in mind. Due to limitations in the lua bindings, we cannot directly reference functions in the way that’s sketched out there, we’d need to use kumo.on to set up the event handling.

What I was thinking was honestly a bit simpler: periodically check the mtime of the certs when the KeySource (the cert and key parameters are KeySource objects: object: keysource - KumoMTA Docs) is a local file.

That could be expanded to work more like the dkim signer type, that is essentially a key source + ttl.

Having a callback function seems powerful, but I don’t know how many folks will actually use it, as they’re most likely dealing with regular files on disk for this stuff.

8

Yeah, it’s probably overkill here. I actually would prefer it to be automatic based on mtime like you described.

What I wrote above - or better yet, a kumo.on event as you mentioned - might make sense for situations where the MTA is being used for multiple domains on the same port, thinking of what Office365 / Google Workspace do, where the customer can access SMTP on customer-domain.com:587. The callback method could allow the system to provide a separate cert based on the domain (if the domain is passed to the func as a parameter). but that’s not what I’m doing anyway.

9

Just wanted to bump this - had an incident last night with expired certs again, our monitoring systems caught it and I quickly fixed it by restarting kumo. I could write a cron job to restart kumo every 30 days, but for some reason kumo almost always takes longer than 5 minutes to shutdown and get killed (SIGKILL) by systemd in the end, so that would mean at least a 5 minute downtime every 30 days.

10

(I’ll be booking a consulting call to investigate the shutdown hanging issue and see if it’s related to our config)

11

Well usually let’s encrypt will update certs really in time… so if you would restart kumomta once a week you should be serted?
And if you wanna make it real fancy: a check script to find all end_dates of certificates which can be used to calculate when certificates will need updating. To then restart kumomta only when it is really needed.
But in production with tons of certificates for clients it will restart more frequently then probably, so a weekly restart of Kumo would then still be the easiest option.

Oh and if you have more then one Kumomta running restart them on different times and use load-balancing to keep the SMTP flow going.

12

Yeah, I was thinking along the same lines, but I’ll have to figure out the long shutdown time before I can do automatic weekly/monthly restarts

13

as of smtp_server: refactor tls_config · KumoCorp/kumomta@9de685a · GitHub (Mar 22), certificates have a 5 minute ttl and reload when it expires

14

re: shutdown timeouts, check out system_shutdown_timeout - KumoMTA Docs and note that the dev build has this in the changelog:

Shutdown could take longer than the 300s permitted by kumomta.service when lua delivery handlers are experiencing delays, leading to systemd issuing a SIGKILL.

15

Thanks Wez!