Unable to connect to HashiCorp Vault from KumoMTA (empty VAULT_ADDR and VAULT_TOKEN env variables)

get-help
1

I have Kumo and Vault setup in Docker in the same network and they’re both working successfully.

I’ve followed the instructions in the documentation to provide the necessary environmental variables, but when trying to retrieve secrets from the Vault an error is thrown:

kumod::smtp_server: Error in SmtpServer: Vault { vault_address: None, vault_token: None, vault_mount: "secret", vault_path: "dkim/..." }: kv2::read vault_mount=secret, vault_path= Vault { vault_address: None, vault_token: None, vault_mount: "secret", vault_path: "dkim/..." }: The Vault server returned an error (status code 404)

The troubleshooting and debug output I’ve put in place leads me to believe it may not be a configuration issue, but how the environment variables are being passed into the Vault client.

3

I’m using the Docker container provided by KumoMTA ghcr.io/kumocorp/kumomta-dev:latest and have the service running locally on macOS.

I can connect to the Vault using the CLI and web UI, but despite setting the vault_addr and vault_token variables everywhere I can think of, KumoMTA shows both variables as being empty.

4

Post your configs pls.

5

Here’s the relevent configuration files:


VAULT_ADDR=http://vault:8200
VAULT_TOKEN=password


services:
    kumo:
        container_name: kumo
        image: ghcr.io/kumocorp/kumomta-dev:latest
        volumes:
            - ./docker/kumomta/etc:/opt/kumomta/etc
            - ./docker/kumomta/kumomta.service:/usr/lib/systemd/system/kumomta.service
        ports:
            - 2025:25
            - 8888:8000
        environment:
            - VAULT_ADDR=${VAULT_ADDR}
            - VAULT_TOKEN=${VAULT_TOKEN}
        networks:
            - proxy


[Service]
Environment="VAULT_ADDR=http://vault:8200"
Environment="VAULT_TOKEN=password"


VAULT_ADDR = os.getenv 'VAULT_ADDR' or 'http://vault:8200'
VAULT_TOKEN = os.getenv 'VAULT_TOKEN' or 'password'

print('VAULT_ADDR', VAULT_ADDR)
print('VAULT_TOKEN', VAULT_TOKEN)
6

I can echo out the environmental variables being passed from .env into the running Docker container and they are correct.

I can SSH into the Docker container and verify the variables are set:

root@kumo:/# printenv | grep 'VAULT'

VAULT_ADDR=http://vault:8200
VAULT_TROKEN=password

A cURL request to the Vault container returns a 200 response, indicating Docker sees the hostname as valid:

root@kumo:/# curl -IL http://vault:8200/ui/

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: no-store
Content-Length: 996281
Content-Security-Policy: default-src 'none'; connect-src 'self'; img-src 'self' data:; script-src 'self'; style-src 'unsafe-inline' 'self'; form-action  'none'; frame-ancestors 'none'; font-src 'self'
Content-Type: text/html; charset=utf-8
Service-Worker-Allowed: /
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
Date: Thu, 13 Feb 2025 20:28:07 GMT
7

And those prints in init.lua?

8

Yes, those variables are available within init.lua and shown in the console output:

https://imgur.com/a/hYp2vWX

$ docker compose up

[+] Running 1/1
 ✔ Container kumo  Recreated                                                                                                                                                                                                                                            0.1s 
Attaching to kumo
kumo  | + KUMO_POLICY=/opt/kumomta/etc/policy/init.lua
kumo  | + exec /opt/kumomta/sbin/kumod --policy /opt/kumomta/etc/policy/init.lua --user kumod
kumo  | VAULT_ADDR    http://vault:8200
kumo  | VAULT_TOKEN    password
kumo  | 2025-02-13T18:45:56.285110Z  INFO localset-0 kumod: NodeId is f004ff34-2afc-4e1c-82fc-4d69bb80f50e
9

I tried looking through the KumoMTA source code in GitHub for additional examples and any potential integration test cases, but wasn’t able to find anything helpful in this regard.

10

After doing a bit more troubleshooting, it seems the connection to the Vault fails when using the DKIM Signer Policy Helper (dkim_data.toml):

[base]
# If these are present, we'll use hashicorp vault instead of reading from disk
vault_mount = "secret"
vault_path_prefix = "dkim/"

local dkim_sign = require 'policy-extras.dkim_sign'
local dkim_signer = dkim_sign:setup { '/opt/kumomta/etc/policy/dkim_data.toml' }

kumo.on('smtp_server_message_received', function(msg)
  dkim_signer(msg)
end)

However, it succeeds (!) when used in Policy (init.lua), either by supplying the credentials manually or from the environmental variables:

kumo.on('smtp_server_message_received', function(msg)
  print('VAULT_ADDR', VAULT_ADDR)
  print('VAULT_TOKEN', VAULT_TOKEN)

  local vault_signer = kumo.dkim.rsa_sha256_signer {
    domain = msg:from_header().domain,
    selector = 'default',
    headers = { 'From', 'To', 'Subject' },
    key = {
      vault_mount = 'secret',
      vault_path = 'dkim/' .. msg:from_header().domain,

      -- Specify how to reach the vault; if you omit these,
      -- values will be read from $VAULT_ADDR and $VAULT_TOKEN

      -- vault_address = "http://vault:8200",
      -- vault_token = "password"

      -- vault_address = VAULT_ADDR,
      -- vault_token = VAULT_TOKEN
    },
  }

  msg:dkim_sign(vault_signer)
end)
11

Based on my testing, it appears there’s a bug in the DKIM Policy Extras code that does not reference the Vault environment variables: VAULT_ARR and VAULT_TOKEN.

Connections to the Vault seem to work elsewhere, but have not been tested extensively.

12

I’ll check it out.

13

Great, thank you.

Let me know if I can help out in any way.

14

The Vault server returned an error (status code 404)

That suggests that it did make a request to the vault server, but the vault server returned a 404 error. Are the mount and path set correctly?

15

Is this the full content of your dkim_data.toml?

[base]
# If these are present, we'll use hashicorp vault instead of reading from disk
vault_mount = "secret"
vault_path_prefix = "dkim/"

you need at least a [domain] stanza in there to activate signing.

Note that if you do not explicitly set the filename in the domain stanza, the automatically derived vault_path that is set into the signing object will be %s/%s/%s.key, that is the prefix (default: dkim), the domain and the selector, with .key appended. In your straight lua code example, it appears as though you don’t have a .key suffix, so perhaps you need to set the filename in the toml explicitly to exclude that.

16

Here’s a stripped-down version of the DKIM TOML file that exhibits the buggy behavior:

[base]
vault_mount = "secret"
vault_path_prefix = "dkim"

selector = "default"

headers = ["From", "To", "Subject"]

[domain."example.test"]
headers = ["From", "To", "Subject", "Date", "Sender"]

When using the above configuration, the Docker logs shown an error:

kumo         | 2025-02-13T21:15:48.013512Z ERROR  smtpsrv-4 run{socket=PollEvented { io: Some(TcpStream { addr: 172.18.0.2:25, peer: 192.168.65.1:57193, fd: 43 }) }}: kumod::smtp_server: Error in SmtpServer: Vault { vault_address: None, vault_token: None, vault_mount: "secret", vault_path: "dkim/example.test/default.key" }: kv2::read vault_mount=secret, vault_path=dkim/example.test/default.key Vault { vault_address: None, vault_token: None, vault_mount: "secret", vault_path: "dkim/example.test/default.key" }: The Vault server returned an error (status code 404)
kumo         | stack traceback:
kumo         |     [C]: in local 'poll'
kumo         |     [string "?"]:4: in main chunk
kumo         |     (...tail calls...)
kumo         |     /opt/kumomta/share/policy-extras/dkim_sign.lua:283: in upvalue 'do_dkim_sign'
kumo         |     /opt/kumomta/share/policy-extras/dkim_sign.lua:356: in upvalue 'dkim_signer'
kumo         |     [string "/opt/kumomta/etc/policy/init.lua"]:59: in function <[string "/opt/kumomta/etc/policy/init.lua"]:56>

A few things from the above error log catch my eye:

  • Empty values for the vault_address and vault_token variables
  • Invalid path to the Vault secret (e.g. dkim/example.test/default.key)

It would appear the derived path to the Vault secret is appending the domain selector private key filename (e.g. dkim/example.test/default.key), rather than the secret path Vault expects (e.g. dkim/example.test).

Using the Vault CLI from a Terminal, I’m able to verify the correct path and data returned:

$ vault kv get -mount="secret" "dkim/example.test"
17

One observation thru debugging is when I used the suggested Vault path prefix from the documentation, the error log showed double slashes in the path:

# /opt/kumomta/etc/policy/dkim_data.toml

[base]
vault_mount = "secret"
vault_path_prefix = "dkim/"

selector = "default"

Having the above configuration results in the following error:

Error in SmtpServer: Vault { vault_address: None, vault_token: None, vault_mount: "secret", vault_path: "dkim//example.test/default.key" }
18

thanks for confirming what I suspected above. I just pushed dkim helper: fixup vault usage · KumoCorp/kumomta@f210a95 · GitHub

19

Nice! It looks like that commit will fix the Vault secrets path issue.

20

Do you have any insight into why the error log is showing the Vault address and token as being empty?

21

that’s expected: you didn’t specify one at that layer, so the default behavior will then be to read values from the environment