messages have been around 5000
Please upgrade to a more recent build; in particular this commit: add re_memory accounting allocator · KumoCorp/kumomta@1dddd75 · GitHub has some stuff that could help to pinpoint what is happening
Ok, will do and come back to you
installed now: /opt/kumomta/sbin/kumod --version
kumod 2024.12.09-8689d4da
added some stats from that new endpoint
stats.txt (3.66 MB)
for the first question: why doesn’t systemctl’s memory output match the metric reported by kumo? We’re reporting RSS of the kumod process only. systemd is reporting some other metric for the cgroup which contains kumod. They are different things. I couldn’t find a succinct description of the differences, but a longer general write up of how referring to memory usage is a hard problem, because there are many dimensions to it: https://superuser.com/a/1571475/45480
the stats that you shared don’t look bad, but it is hard to evaluate those without also the RSS value (that you see as Memory in kcli top) because that can help to understand the level of fragmentation
I don’t have recollection of any context on the size of your system either so I don’t know what your limits really are.
I made a little utility in add a little utility to summarize memory stats · KumoCorp/kumomta@5812d84 · GitHub that summarizes the call stacks from the stats output to something a bit more succinct and focused.
I’ve run your stats output through it and attached it to this message
summarized.txt (380 KB)
The top items are the pre-allocated memory for your various ready queues; that directly correlates with what you’ve configured as max_ready x the number of ready queues in the system. In the data you shared there are a few GiB of ready queues allocated (some from reception time, some from spooling in, so they have different call stacks).
The next highest ram usage was message bodies, but it looks to be less than 1GiB total in message bodies
Then there’s a few hundreds of MiB of shaping data.
The shaping data is larger than I would expect, and is likely due to having a couple of copies hanging around for use from different threads. That is something I will look into, however, it is still less memory than the ready queue usage
I would suggest that you review your max_ready values to make sure that they aren’t over-provisioned for your throughput. I wrote up Memory Management - KumoMTA Docs a couple of weeks ago with some general guidance for this
sure i can review the setting max_ready. the only thing that i can say is on the server we didnt had soo many messages laying around and the memory usage was around >40 GB.
we had some issues that messages that have been ready not got delivered
and by increase max ready we could fix it
especially during black friday and christmas time we have peaks
hmm, I just added some aggregation into the summary tool; I didn’t put it there originally because I assumed that the re_memory crate would do a good job at aggregating. It shows that the top usage is actually the shaping data:
5.2 GiB in 17811264 allocations (312 B each)
...
kumo_api_types::shaping::_::<impl serde::de::Deserialize for kumo_api_types::shaping::PartialEntry>::deserialize kumo-api-types/src/shaping.rs:979:10
<kumo_api_types::shaping::_::<impl serde::de::Deserialize for kumo_api_types::shaping::ShapingFile>::deserialize::__Visitor as serde::de::Visitor>::visit_map kumo-api-types/src/shaping.rs:969:10
kumo_api_types::shaping::_::<impl serde::de::Deserialize for kumo_api_types::shaping::ShapingFile>::deserialize kumo-api-types/src/shaping.rs:969:10
kumo_api_types::shaping::from_toml kumo-api-types/src/shaping.rs:451:8
kumo_api_types::shaping::Shaping::load_from_file::{{closure}} kumo-api-types/src/shaping.rs:592:13
kumo_api_types::shaping::Shaping::merge_files::{{closure}} kumo-api-types/src/shaping.rs:622:74
kumo_api_types::shaping::register::{{closure}}::{{closure}} kumo-api-types/src/shaping.rs:1417:22
mod_memoize::register::{{closure}}::{{closure}}::{{closure}} mod-memoize/src/lib.rs:457:73
config::async_call_callback::{{closure}} config/src/lib.rs:369:46
config::LuaConfig::async_call_callback::{{closure}} config/src/lib.rs:227:50
kumod::ready_queue::ReadyQueueManager::compute_config::{{closure}} kumod/src/ready_queue.rs:277:14
kumod::ready_queue::ReadyQueue::perform_config_refresh::{{closure}} kumod/src/ready_queue.rs:845:10
kumod::ready_queue::ReadyQueue::perform_config_refresh_if_due::{{closure}} kumod/src/ready_queue.rs:827:56
kumod::ready_queue::ReadyQueueManager::check_config_refresh::{{closure}} kumod/src/ready_queue.rs:411:18
kumod::ready_queue::ReadyQueueManager::queue_config_maintainer::{{closure}} kumod/src/ready_queue.rs:387:67
i will sent you the current TSA shaping file