log_hooks:new_json does not include custom header in the webhook payload

get-help
1

I have the following configuration in init.lua - the log_parameters includes X-CUSTOMER-ID:

log_hooks:new_json {
  name = "webhook",
  -- log_parameters are combined with the name and
  -- passed through to kumo.configure_log_hook
  log_parameters = {
    headers = { 'Subject', 'X-CUSTOMER-ID' },
    meta = { 'tenant' }
  },
  -- queue config are passed to kumo.make_queue_config.
  -- You can use these to override the retry parameters
  -- if you wish.
  -- The defaults are shown below.
  queue_config = {
    retry_interval = "1m",
    max_retry_interval = "20m",
  },
  -- The URL to POST the JSON to
  url = "http://192.168.0.100:4242/log",
}
-- the rest of the config
kumo.on('http_message_generated', function(msg)
  local tenant = cached_tenant_id(msg:from_header().domain)
  if tenant then
    msg:set_meta('source', 'http')
    msg:set_meta('tenant', tenant)
    msg:append_header('X-CUSTOMER-ID', tenant)
  else
    kumo.reject(
      500,
      string.format("from domain '%s' is not allowed.", msg.from_header().domain)
    )
  end
  print(msg:get_data())
  queue_helper:apply(msg)
  dkim_signer(msg)
end)

The X-CUSTOMER-ID header is being added to the message on http_message_generated (I see it in the output of print(msg:get_data())), but the payload sent to the webhook does not contain the header:

{"type":"Reception","id":"5cab1f3d92d111eea4dc080027be1609","sender":"noreply@example.com","recipient":"recipient@example.com","queue":"test@example.com","site":"","size":883,"response":{"code":250,"enhanced_code":null,"content":"","command":null},"peer_address":{"name":"","addr":"192.168.0.100"},"timestamp":1701713839,"created":1701713839,"num_attempts":0,"bounce_classification":"Uncategorized","egress_pool":null,"egress_source":null,"feedback_report":null,"meta":{"tenant":"test"},"headers":{"Subject":"This is the subject"},"delivery_protocol":null,"reception_protocol":"HTTP","nodeid":"6c48c1dc-5f6a-487c-ab67-c3e6f426d6a6"}
2

I’ve added tenant to log_parameters.meta and that’s being included in the webhook payload, so I can take care of my usecase, but was curious as to why it’s not working with the header.

3

Does it appear in the delivery log entry? I’m wondering if it’s not considered part of the reciption data but then becomes part of the delivery.

4

No, the delivery log entry doesn’t include any headers (reception and bounce include the subject header) - also, my webhook handler does not receive the delivery log, the entry below is from the kumo log files:

{"type":"Delivery","id":"14fa851f92dd11eea92d080027be1609","sender":"noreply@example.com","recipient":"recipient@example.com","queue":"webhook.log_hook","site":"unspecified->webhook.log_hook@lua:make.webhook.log_hook","size":695,"response":{"code":200,"enhanced_code":null,"content":"200 OK: OK","command":null},"peer_address":{"name":"Lua via make.webhook.log_hook","addr":"0.0.0.0"},"timestamp":1701718873,"created":1701718873,"num_attempts":0,"bounce_classification":"Uncategorized","egress_pool":"unspecified","egress_source":"unspecified","feedback_report":null,"meta":{"tenant":"test"},"headers":{},"delivery_protocol":"Lua","reception_protocol":"LogRecord","nodeid":"6c48c1dc-5f6a-487c-ab67-c3e6f426d6a6"}
5

Question: why are you trying to log the tenant header at all? You have the tenant name in the meta in the log already. And you did a server restart? I’m curious why subject was logged and the X-Header was not if this is the active config.

6

Yeah, I started with trying to log the X-CUSTOMER-ID header and when it didn’t work, I logged the tenant meta instead and can take care of my use case - was just wondering why it the X-header didn’t work and thought maybe it’s a bug. Yes, I’ve been restarting kumo after every config change

7

I’ll leave the decision on it being a bug up to @free-spirited-yorksh.

8

if you use the same headers config in kumo.configure_local_logs, does the header show up in the log file?

9

changed it to:

  kumo.configure_local_logs {
    log_dir = '/var/log/kumomta',
    -- Flush logs every 10 seconds.
    -- You may wish to set a larger value in your production
    -- configuration; this lower value makes it quicker to see
    -- logs while you are first getting set up.
    max_segment_duration = '10s',
    meta = { 'tenant' },
    headers = { 'X-CUSTOMER-ID', 'Subject' }
  }

still only ‘Subject’ header in the webhook payload.

10

FWIW, this log record is for the delivery of the webhook payload to the webhook, not the message you injected via http. You can tell because the queue here is "webhook.log_hook" and "delivery_protocol":"Lua"

11

I’d like to see what is logged in the delivery logs in /var/log/kumomta with that config in place, to see if it is different from what you see in the webhook

12

/var/log/kumomta: https://paste.mozilla.org/Jw8uv1Vs

13

The configure_local_logs config was in place and kumomta was restarted before I copied the logs above.

14

it’s also weird to me to see that Subject is present, and has an uppercase first character

15

what version are you running? kumod --version

16

if you inject a message via SMTP, does it behave any better?

17

the uppercase first letter is troubling me, because we lowercase all the header names before we put them in the JsonLogRecord

18

kumod 2023.11.28-b5252a41

19

Same situation with SMTP injection method - webhook payload:

{"type":"Reception","id":"47af579092e411ee9699080027be1609","sender":"someone@example.com","recipient":"me@example.com","queue":"test@example.com","site":"","size":389,"response":{"code":250,"enhanced_code":null,"content":"","command":null},"peer_address":{"name":"moto","addr":"127.0.0.1"},"timestamp":1701721964,"created":1701721964,"num_attempts":0,"bounce_classification":"Uncategorized","egress_pool":null,"egress_source":null,"feedback_report":null,"meta":{"tenant":"test"},"headers":{"Subject":"hello!"},"delivery_protocol":null,"reception_protocol":"ESMTP","nodeid":"6c48c1dc-5f6a-487c-ab67-c3e6f426d6a6"}

{"type":"Bounce","id":"47af579092e411ee9699080027be1609","sender":"someone@example.com","recipient":"me@example.com","queue":"test@example.com","site":"test-ip->@smtp_client","size":389,"response":{"code":556,"enhanced_code":{"class":5,"subject":1,"detail":10},"content":"Recipient address has a null MX","command":null},"peer_address":null,"timestamp":1701721964,"created":1701721964,"num_attempts":0,"bounce_classification":"Uncategorized","egress_pool":"test-pool","egress_source":"test-ip","feedback_report":null,"meta":{"tenant":"test"},"headers":{"Subject":"hello!"},"delivery_protocol":null,"reception_protocol":"ESMTP","nodeid":"6c48c1dc-5f6a-487c-ab67-c3e6f426d6a6"}
20

/var/log/kumomta: https://paste.mozilla.org/anaqey9H