Log Questions

get-help
1

I tested webhook’s way of sending logs to kafak today, and after sending, I found that the local log(/opt/kumomta/sbin/tailer --tail /var/log/kumomta) had also changed, which disturbed me.

One scenario I envision is that when a tenant uses KumoMTA, I judge the tenant and pass a specific type of data back to the tenant.

How to keep the original log (local log) in case of forwarding some logs to kafka.
Here is my hypothetical code

3

log_hooks:new {
name = “kafka”,
batch_size = 1,
constructor = function(domain, tenant, campaign)
if tenant == ‘some tenant’
local connection = {}
local producer = kumo.kafka.build_producer {
[“bootstrap.servers”] = “xxx.xxx.xxx.xxx:xxxx”,
[“queue.buffering.max.ms”] = 1000,
}
local topics_map = {
[‘Reception’] = “kumo_received”,
[‘Delivery’] = “kumo_delivered”,
[‘Bounce’] = “kumo_bounced”,
[‘TransientFailure’] = “kumo_transient”,
[‘Feedback’] = “kumo_feedback”,
[‘AdminBounce’] = “kumo_admin_bounced”,
[‘Expiration’] = “kumo_expired”,
[‘OOB’] = “kumo_oob”,
}

        function connection:send(message)
            local log_record = message:get_meta('log_record')
            if not log_record then
            kumo.reject(400, 'Missing log record metadata')
            return
            end
            
            local topic = topics_map[log_record.type] or "pmta_other"
            local status, partition, offset = pcall(producer.send, producer, {
            topic = topic,
            payload = message:get_data(),
            })
            
            if status then
            return string.format('Message sent successfully to topic %s', topic)
            else
            kumo.reject(400, string.format('Failed to send to topic %s: %s', topic, tostring(partition)))
            end
        end

    function connection:close()
        producer:close()
    end

    return connection
end,

}

4

After webhook forwarding is configured, the system logs become the logs of whether the forwarding to the webhook is successful.

5

I’ve come up with several solutions.

  1. Save the current logs to the ELK system.
  2. Write a send back script for the ELK log to send back tenant data.
    The second step is no longer KumoMTA’s job.

What I would like to ask is whether KumoMTA had any previous considerations that I did not find in the documentation. For example, I originally thought webhook was just a copy of a log, but instead of a replacement.

6

I was confused. I waited a little longer and it turned out to be correct. I made sure I restarted KumoMTA for each change

7

Please post your full code.

8

@yearning-hyena This is my init.lua file. For other files like sources.toml etc ,it’s all default.
init.lua (6.76 KB)

9

Now I can see full logs . and kafka also rev message . I only want to know if what I’m doing is within the best practice

10

Did you previously omit configure_local_logs from your init event?

11

Sure, I can always check logs. /opt/kumomta/sbin/tailer --tail /var/log/kumomta

12

Then I guess I missed the issue.

13

ohh i never omit coonfigure_local_logs

14

maybe the log slow ? but now it work well.

15

The logs flush at an interval or message count threshold. It’s not slow, it’s just not continuously written to conserve IO.

16
17

yeah , good. I think i will first webhook the log to ELK , Then, data is retrieved from ELK and returned to the tenants. This may reduce the impact on KumoMTA.