Insert_delayed message Not be Empty

KumoMTA Administration
1

Hi teams.

Issue Report

KumoMTA version: kcli 2025.12.02-67ee9e96
Host: kumo01

Error message

2026-02-03T01:45:50.861517Z ERROR qmaint-1 insert_ready{context=InsertContext([DueTimeWasReached]) deadline=None}:requeue_message{increment_attempts=No delay=Some(TimeDelta { secs: 26, nanos: 0 }) response=Response { code: 451, enhanced_code: Some(EnhancedStatusCode { class: 4, subject: 4, detail: 4 }), content: "KumoMTA internal: myqueuename throttled message rate, delay=TimeDelta { secs: 26, nanos: 0 }", command: None } context=InsertContext([DueTimeWasReached, MessageRateThrottle])}:requeue_message_internal{increment_attempts=No delay=Some(TimeDelta { secs: 26, nanos: 0 }) context=InsertContext([DueTimeWasReached, MessageRateThrottle])}:insert{context=InsertContext([DueTimeWasReached, MessageRateThrottle]) deadline=None}:insert_delayed{context=InsertContext([DueTimeWasReached, MessageRateThrottle])}: kumod::queue::queue: while shrinking: 44c65abffcd911f0a92fbc241172448d: message data must not be empty

System memory usage

               total        used        free      shared  buff/cache   available
Mem:            84Gi        20Gi        38Gi       1.5Gi        27Gi        63Gi
Swap:          5.9Gi          0B       5.9Gi

KumoMTA metrics

curl -s http://localhost:8000/metrics.json | jq '.memory_limit, .memory_usage, .memory_low_count, .memory_over_limit_count'

Output:

{
  "help": "soft memory limit measured in bytes",
  "type": "gauge",
  "value": 68009585664
}
{
  "help": "number of bytes of used memory (Resident Set Size)",
  "type": "gauge",
  "value": 8878993408
}
null
null
3

It seems like KumoMTA is generating repeated errors during message queue operations:

message data must not be empty

This happens in the insert_delayed / requeue_message_internal process, after a 451 response with throttling (MessageRateThrottle).

Memory usage looks healthy — only ~8.8 GB RSS out of ~68 GB configured limit, so it doesn’t appear to be a memory exhaustion issue.

Questions

  1. What could cause the "message data must not be empty" error during requeue operations?
  2. Could this be related to rate throttling or message expiration (DueTimeWasReached)?
  3. Is there any recommended way to debug or inspect the affected queue (44c65abffcd911f0a92fbc241172448d)?
4

kcli inspect-message - KumoMTA Docs or kcli inspect-sched-q - KumoMTA Docs might be interesting to apply to that message/queue

5

"message data must not be empty" is raised when the data for the message is completely empty, which is not something that should happen in normal operation. Do you have the Reception and various Delayed/TransientFailure logs for that particular message?

6

I’m not sure what happened — a few hours ago, I restarted kumomta, and it became normal again; those error logs disappeared. Before the restart, there were about 3-5error logs per second.

When I restarted it, I encountered a log message that I had never seen before. memory_low_thresh=50.67 GB ;

I suspect it might have been triggered by memory_low_thresh, which then caused the subsequent tsa unresponsiveness and related issues.

image.png
image.png1363×148 64.4 KB

7

After restart the server . I have not found this message .

image.png
image.png1244×884 124 KB

8

New finding: I checked the historical logs and found they all share the same message ID. I’m thinking that if the message is empty, we should stop looping; otherwise, it could overwhelm the system.

9

Do you have any code that operates on the body of the message? (anywhere?)

10

I am sure NO from komomta.