webhook logs stuck in scheduled queue

This started today on kumod 2024.10.16-6359820d. kcli queue-summary shows that there are ~8000 messages in the scheduled queue for webhook.log_hook and it’s increasing (fortunately a it’s a quiet day here for us with low volumes). The service that accepts the log hooks is up and running and it’s receiving another call from KumoMTA that sends the message data, but it’s not receiving any requests on the webhook endpoint.
kumod --user kumod --validate says:

Issues found in the combined set of shaping files:
 - /opt/kumomta/share/policy-extras/shaping.toml
 - /opt/kumomta/etc/policy/config/shaping.toml
 - http://127.0.0.1:8008/get_config_v1/shaping.toml
WARNING: Ignoring http://127.0.0.1:8008/get_config_v1/shaping.toml because skip_remote is set to true
WARNING: multiple domain blocks alias to the same site: (mx1|mx2)?.(mxge|mx1a1|mx1c1|mx1h1|mx2a1|mx2c1|mx2h1).comcast.net: comcast.net, comcast.net
WARNING: multiple domain blocks alias to the same site: (mx00|mx01).mail.com: mail.com, mail.com
WARNING: multiple domain blocks alias to the same site: smtp-in.orange.fr: orange.fr, orange.fr
WARNING: domain outlook.com is also matched by provider(s): outlook
WARNING: domain gmail.com is also matched by provider(s): google
WARNING: domain yahoo.com is also matched by provider(s): yahoo
WARNING: domain live.com is also matched by provider(s): outlook

(not sure what skip_remote is about, I don’t have that anywhere in my configuration files as far as I can tell)

I did restart both kumomta and the log consumer service (a few times), but nothing changed. Everything was working fine until this morning, not sure what changed.

I also enabled the debug logs, attached to this ticket.

For now I’ve downgraded to the latest stable build and things went back to normal and the queued webhook logs were processed and delivered successfully.
logs.txt (2.46 MB)

OK, seems like the log I posted is irrelevant and it was actually processing the webhooks after I restarted kumomta, but I was seing errors in my consumer service because of the large number of backed up logs and the new webhooks getting queued after the ones from before, causing them to arrive late. I’ll see if I can reproduce this with debug logs enabled by default so I can catch the issue in the logs.