Hello team, I have couple questions about Kafka implementation. Basically I want to use it to deliver the webhooks to DynamoDB (I’m using AWS: MSK + lambda + DynamoDB, also considering Clickhouse)
I found this one:
https://docs.kumomta.com/userguide/policy/kafka/?h=kafka, looks like I need to implement both code snippets into my init.lua script right? How can I debug messages that I’m sending to Kafka?
Also I found this https://docs.kumomta.com/userguide/operation/webhooks/, second example. Should I implement this full call to the log_hooks.lua helper for the Kafka based webhooks sending? Sorry for the dumb questions. Thanks in advance.
Hey there @dapper-yeti, thanks for posting. Please read the “Troubleshooting” and “How to Ask for Help” buttons below. If you would like a 1:1 support session from the KumoMTA team, details are at the “Book a Support Session” button below.
use the second example, but swap out the http client with the kafka client
@free-spirited-yorksh thanks for the help! I’m using AWS MSK. Any chance I can connect to AWS MSK using SASL/SCRAM or IAM from the kumo kafka module?
I have no direct experience with that, so I don’t know
@free-spirited-yorksh ok, thanks anyways - in general based on your experience whats the best practices with the webhooks using Kumo? Like at the peaks you can receive thousands of events per minute, how to proceed all of them without data loss and system outage? Did you have any experience with that? Thanks.
each individual webhook record is queued in the same way that other kinds of messages are queued. The regular kumomta queue management applies to webhook dispatching, including setting concurrency limits and retries. If the destination is unavailable or returns an error, that webhook event will be re-submitted based on its retry_interval.
Well I didn’t manage to get it working…Now I’m trying to use Amazon MQ (RabbitMQ) for the webhooks, I can’t find anything about how can I connect to RabbitMQ via secure protocol (ampqs in a connection string), anything specific should I know? In CloudWatch I see:
TLS server: In state hello at tls_handshake.erl:354 generated SERVER ALERT: Fatal - Insufficient Security
TLS server: In state hello at tls_record.erl:561 generated SERVER ALERT: Fatal - Unexpected Message
TLS server: In state hello at tls_handshake.erl:401 generated SERVER ALERT: Fatal - Protocol Version
Ok, I figured it out, not sure why exactly it started working, I’ll post answer later, but I did one more test on Single-instance broker, not the latest RabbitMQ version, also I URL-encoded my password with special characters
It’s just a matter of using the secure version of the connection
(Looking for a recent example…)
local client = kumo.amqp.build_client 'amqps://USER:PASSWD@RABBIT-FQDN:PORT'
Thanks @faithful-ostrich !
So now I can see some weird behaviour - I’m sending email from the API:
curl --user xxx:xxx -H 'Content-Type: application/json' 'https://api.xxx.org/api/inject/v1' -d '{
"envelope_sender": "test@erlangjobs.com",
"content": {
"text_body": "NEW PART",
"html_body": "<p>NEW PART</p>",
"from": {
"email": "test@erlangjobs.com",
"name": "Jobs"
},
"subject": "Example Send",
"reply_to": {
"email": "test@erlangjobs.com",
"name": "Help"
}
},
"recipients": [
{
"email": "xxx@gmail.com"
}
]
}'
{"success_count":1,"fail_count":0,"failed_recipients":[],"errors":[]}%
And I can see in my lambda logs that I’m always receiving like 30+ webhook calls, event about other email sends (see schreenshot, my lambda is listening to the RabbitMQ events, and I assume I should receive only 2 for Reception and Delivery)
Can anyone help? Still receiving 30+ messages for any API call 
You haven’t provided the configs and whatnot recommended in the first reply in this thread, without your configs we’re not going to be able to help.
Omg so sorry - I was debugging and just forgot about it, sure: https://gist.github.com/maratgaliev/7161536f8dc9a6d61d6df6fe3536cba9 (put my configs there)
Cloud: AWS
Version: kumod 2024.06.10-84e84b89
Rocky Linux: 5.14.0-427.24.1.el9_4.x86_64
Pls do a gist of the log lines as well instead of a screenshot.
Sorry about that: https://gist.github.com/maratgaliev/2fc94858f62cbc1f209cba196a4898c3
Basically what I see now - on API injection call I’m receiving around 30-40 messages in RabbitMQ even with my previous API calls (different subjects/body etc)
Those lines all have "reception_protocol": "HTTP" and you are not trapping that. You are trapping "reception_protocol": "log_record" . You may need to adjust your logic.