webhook log_hook message count

get-help
1

My queue webhook.log_hook got very high (>500k). This was a result due to high memory usage on the server:

Nov 26 11:43:25 mta01 kumod[2423]: 2024-11-26T10:43:25.022608Z ERROR   qmaint-0 shrink_ready_queue_due_to_low_mem: kumod::ready_queue: did shrink 0 of out 20002 msgs in ready queue unspecified->webhook.log_hook@lua:make.webhook.log_hook due to memory shortage, and will requeue 76 due to hitting constraints

And now the queue is only slowly getting lower. I already tried to tweak the settings for the queue:

    retry_interval = '1m',
    max_retry_interval = '24h',
    max_message_rate = '100000/s',
  },

esp. the max_message_rate parameter, but it seams not to have any positive impact. it still only pushes 100 msg /sec.
max_retry_interval was set very high in order to not loose any message.

Do you have any good idea how to publish more messages?

I’m using current latest stable kumomta version: kumod 2024.11.08-d383b033

3

Is the webhook collector capable of accepting more than 100/s?

4

In the most recent development release we added better batching capability so you can send fewer but larger webhook batches. You might consider rewriting your webhook code to batch a thousand at a time.

5
6

You might also consider LO meeting the max_ready on the other messages so that webhooks code gets more memory.
Or you might consider increasing RAM.

7

we already have 128 GB of RAM

8

so increasing is not an option :grinning_face_with_smiling_eyes:

9

Yeah that is not likely the problem

10

behind the hook we have an AMQP message broker

11

(rabbitmq)

12

What is max_ready set to for normal mail and for webhooks?

13

and it can accept more than just 100msg/s

14

Oh. Why not just write to AMQP directly?

15

sorry maybe wrote that part wrong. we write to AMQP by using the webhook

16

max_ready is only set in the default

17

So you doing something like this?

18

correct

19

And what version are you on now?

20

kumod 2024.11.08-d383b033

21

Perfect.
You may want to lower the max_ready to say 512 it may increase your scheduled queue but will reserve less Ram.