Log Event Processing and Retrieval Inquiry

get-help
21

Hi @yearning-hyena This is not working . Please help me to push the log events to amqp. The configuration is attached below

22

Due to large file size . split and upload file

**file - /opt/kumomta/etc/policy/init.lua **
`–
local kumo = require ‘kumo’
local shaping = require ‘policy-extras.shaping’

local sources = require ‘policy-extras.sources’
sources:setup { ‘/opt/kumomta/etc/policy/sources.toml’ }
local queue_module = require ‘policy-extras.queue’
local queue_helper = queue_module:setup ({ ‘/opt/kumomta/etc/policy/queues.toml’ })

–[[ Start of INIT section ]]

kumo.on(‘init’, function()
kumo.set_diagnostic_log_filter ‘kumod=debug’
kumo.start_esmtp_listener {
listen = ‘0:25’,
hostname = ‘kumo.domain.com’,
relay_hosts = {‘127.0.0.1’,‘127.99.9.99’,‘139.88.8.888’},

}

kumo.start_esmtp_listener {
listen = ‘0.0.0.0:587’,
hostname = ‘kumo.domain.com’,
relay_hosts = { ‘127.99.9.99’ },
tls_certificate = ‘/opt/kumomta/etc/ssl/kumomesagescafe-cert.pem’,
tls_private_key = ‘/opt/kumomta/etc/ssl/kumodomain-key.pem’,
}
kumo.start_http_listener {
listen = ‘127.0.0.1:8000’,
hostname = ‘kumomta.domain.com’,
tls_certificate = ‘’,
tls_private_key = ‘/etc/letsencrypt/live/kumomta.domain.com/privkey.pem’,
use_tls = false
}

kumo.define_spool {
name = ‘data’,
path = ‘/var/spool/kumomta/data’,
}

kumo.define_spool {
name = ‘meta’,
path = ‘/var/spool/kumomta/meta’,
}

kumo.configure_local_logs {
log_dir = ‘/var/log/kumomta’,
}

end)
–[[ End of INIT Section ]]
`

23

`–[[ Start of Non-INIT level config ]]

kumo.on(‘get_queue_config’, function(domain, tenant, campaign, routing_domain)
print (“enter get_queue_config”)
if domain == ‘amqp’ then
return kumo.make_queue_config {
protocol = {
custom_lua = {
constructor = ‘make.amqp’,
},
},
}
end
return kumo.make_queue_config {}
end)

kumo.on(‘make.amqp’, function(domain, tenant, campaign)
print (“Process make.amqp”)
local client = kumo.amqp.build_client ‘amqp://username:password@127.99.9.99:5672’
local confirm = client:publish {
routing_key = ‘logging’,
payload = message:get_data(),
}
local result = confirm:wait()
if result.status == ‘Ack’ or result.status == ‘NotRequested’ then
return
end
kumo.reject(500, kumo.json_encode(result))
end)

local AUTH_DETAILS = kumo.json_load ‘/opt/kumomta/etc/auth_details.json’`

24

`kumo.on(‘smtp_server_message_received’, function(msg)

if checkEmailAllowed(AUTH_DETAILS,"arunkvinam@gmail.com",msg:recipient().email) then
msg:set_meta(‘queue’, ‘null’)
end

queue_helper:apply(msg)
msg:append_header(“X-Server-Id”, “Vinam-Kumo”)
msg:append_header(“X-Sender-Id”, string.format(‘5.6.0 %s’, result.status) )
msg:append_header(“List-Unsubscribe”, kumo.encode.base64_encode( msg:recipient().email .. ‘/’ .. msg:id() ))
msg:append_header(“List-Unsubscribe-Post”, “List-Unsubscribe=One-Click”)
local signer = kumo.dkim.rsa_sha256_signer {
domain = ‘domain.com’,
selector = ‘kmdk’,
headers = { ‘From’, ‘To’, ‘Subject’ },
key = ‘/opt/kumomta/etc/dkim/domain.com/kumo1.pem’,
}
msg:dkim_sign(signer)

local double_signer = kumo.dkim.rsa_sha256_signer {
domain = msg:from_header().domain,
selector = ‘kmdk’,
headers = { ‘From’, ‘To’, ‘Subject’ },
key = ‘/opt/kumomta/etc/dkim/’ .. string.gsub(msg:from_header().domain,‘kumo.’,‘’) .. ‘/kumo1.pem’,
}
msg:dkim_sign(double_signer)

end)

local AUTH_CONFIG = kumo.json_load ‘/opt/kumomta/etc/auth_file.json’
kumo.on(‘smtp_server_auth_plain’, function(authz, authc, password)
if password == ‘’ then
return false
end
return AUTH_CONFIG[authc] == password
end)

kumo.on(‘get_egress_path_config’,
shaping:setup { ‘/opt/kumomta/etc/shaping.json’ }
)

– PLEASE read https://docs.kumomta.com/ for extensive documentation on customizing this config.
–[[ End of Non-INIT level config ]]

function checkEmailAllowed(details,from_email,to_email)
if details[from_email].blocked_emails[to_email] then
return true
else
return false
end
end`

25

file - /opt/kumomta/etc/policy/sources.toml

`[source.“ip-1”]
source_address = “139.99.9.99”
ehlo_domain = ‘kumo.domain.com’

Pool containing just ip-1, which has weight=1

[pool.“pool-1”]
[pool.“pool-1”.“ip-1”]`

26

file - /opt/kumomta/etc/policy/queues.toml

`scheduling_header = “X-Schedule”
tenant_header = “X-Tenant”
remove_tenant_header = true

campaign_header = “X-Campaign”
remove_campaign_header = true

default_tenant = “default-tenant”

[tenant.‘default-tenant’]
egress_pool = ‘pool-1’

[queue.‘gmail.com’]
max_age = ‘22 hours’
retry_interval = ‘17 mins’
`

27

** file - /opt/kumomta/etc/auth_details.json**

{ "aru*****.com": { "blocked_emails": { "1aruff@*******.com": "arufff@*******.com", "geetffa@*******.com": "geeffa@*******.com" } } }

28

file - /opt/kumomta/etc/auth_file.json

{"aru**************.com":"123456789"}

29

file - /opt/kumomta/etc/shaping.json

{ "default": { "replace_base": true, "connection_limit": 3, "max_connection_rate": "100/min", "max_message_rate": "50/min" }, "domain.com": { "connection_limit": 3, "max_message_rate": "10/min", "sources": { "ip-1": { "connection_limit": 5, "max_message_rate": "2/min" } } } }

30

Hi @yearning-hyena Iam expecting to push following logs to AMQP/RabbitMQ using above configuration. but unfortunately this not working.

1.Reception
2.Bounce
3.TransientFailure
4.Expiration
5.AdminBounce
6.OOB
7.Feedback"

31
  1. Please re-post all that in https://gist.github.com/ so that it’s easier to read.
  2. When you say it is not working, please elaborate. How is it not working? What did you expect vs what it did, and post any system logs or other errors.
32

Ian trying to push logs in to amqp

33

Hi @yearning-hyena Iam expecting to push following logs to AMQP/RabbitMQ using above configuration. but unfortunately this not working.

1.Reception
2.Bounce
3.TransientFailure
4.Expiration
5.AdminBounce
6.OOB
7.Feedback"

34

Adding to @yearning-hyena 's comments above, tailing a debug journal as mentioned at Discord would will give you more detailed error information to share. It is really hard to fix “it’s broken” without much more detail.

35

There is no error logged

36

Iam adding a custom lua ‘make. amqp’.that custom lua not working.

37

Inside the custom lua iam writing code to push logs to amqp

38

Have a look at this thread, there is an updated code of the make.ampq and also the follow-up discussion: https://discord.com/channels/1072980126737907824/1184990215576764517/1185147121431752755

39

Hi @faithful-ostrich @courteous-yorkshire Using below code in init.lua . getting an error

Code
kumo.on('make.amqp', function(domain, tenant, campaign) print ("Process make.amqp") local client = kumo.amqp.build_client 'amqp://username:Kpd.password@199.99.9.99:5672' local confirm = client:publish { routing_key = 'logging', payload = message:get_data(), } local result = confirm:wait() if result.status == 'Ack' or result.status == 'NotRequested' then return end kumo.reject(500, kumo.json_encode(result)) end)

Error

Dec 22 05:05:39 development.domain.com kumod[839839]: 2023-12-22T05:05:39.829710Z DEBUG logger kumod::logging: waiting until deadline=None for a log record
Dec 22 05:05:39 development.domain.com kumod[839839]: 2023-12-22T05:05:39.830119Z DEBUG localset-5 kumod::ready_queue: Error in Dispatcher::run for unspecified->amqp@lua:make.amqp: runtime error: [string “/opt/kumomta/etc/policy/init.lua”]:113: attempt to index a nil value (global ‘message’)
Dec 22 05:05:39 development.domain.com kumod[839839]: stack traceback:
Dec 22 05:05:39 development.domain.com kumod[839839]: [string “/opt/kumomta/etc/policy/init.lua”]:113: in function <[string “/opt/kumomta/etc/policy/init.lua”]:108> (consecutive_connection_failures=2)

40

This is relevant:

runtime error: [string "/opt/kumomta/etc/policy/init.lua"]:113: attempt to index a nil value (global 'message')
Check to see if it maybe should be “msg” instead?