HTTP API: Webhook events replace some characters with in subject

get-help
1

We have “Subject” as one of the headers that should be passed with the webhook logs:

log_hooks:new_json {
  name = "webhook",
  -- log_parameters are combined with the name and
  -- passed through to kumo.configure_log_hook
  log_parameters = {
    headers = { 'Subject', 'From', 'Message-Id', 'References', 'In-Reply-To' },
    meta = { 'tenant', 'domain_id', 'customer_id', 'direction', 'metadata_retention', 'data_retention', 'tags' }
  },
  -- queue config are passed to kumo.make_queue_config.
  -- You can use these to override the retry parameters
  -- if you wish.
  -- The defaults are shown below.
  queue_config = {
    retry_interval = "1m",
    max_retry_interval = "24h",
    max_message_rate = '100000/s',
  },
  -- The URL to POST the JSON to
  url = LOG_WEBHOOK_URL
}

Sending the following payload using the HTTP API results in some characters getting replaced with � in the webhook event payload:

{
  "envelope_sender": "no-reply@email.ahasend.com",
  "content": {
    "text_body": "test",
    "html_body": "test",
    "subject": "بوت‌كمپ قدرت نوشتن رهنماکالج",
    "from": {
      "email": "noreply@email.ahasend.com",
      "name": "آها سند"
    }
  },
  "recipients": [
    {
      "email": "hf.farhad@gmail.com",
      "name": "Farhad Hedayatifard"
    }
  ]
}
3

The webhook event fired for this message:

{
    "type": "Reception",
    "id": "fa6f3f38caac11efaedc960002cafe7c",
    "sender": "fa6f3f38caac11efaedc960002cafe7c@psrp.email.ahasend.com",
    "recipient": "hf.farhad@gmail.com",
    "queue": "8b9dcfcf-94f7-477c-8372-0178618ef975@gmail.com",
    "site": "",
    "size": 2346,
    "response": {
        "code": 250,
        "enhanced_code": null,
        "content": "",
        "command": ""
    },
    "peer_address": {
        "name": "",
        "addr": "127.0.0.1"
    },
    "timestamp": 1736002975,
    "created": 1736002975,
    "num_attempts": 0,
    "bounce_classification": "Uncategorized",
    "egress_pool": "",
    "egress_source": "",
    "meta": {
        "domain_id": "72703b13-166a-4f41-ab96-7333c61608f5",
        "customer_id": "8b9dcfcf-94f7-477c-8372-0178618ef975",
        "direction": "outbound",
        "metadata_retention": null,
        "data_retention": null,
        "tags": ""
    },
    "headers": {
        "Subject": "بوت‌كمپ قد��ت نوشتن رهن��اکالج",
        "From": "آها سند <noreply@email.ahasend.com>",
        "Message-Id": "",
        "References": "",
        "In-Reply-To": ""
    },
    "delivery_protocol": "",
    "reception_protocol": "HTTP",
    "nodeid": "2f6489ba-9d9d-47d5-83bb-8492ac3fdd93",
    "feedback_report": null
}
4

That being said, the message subject does end up being rendered correctly in Gmail

image.png
image.png613×183 13 KB

5

but it does not render correctly in KMail (the KDE mail client)

image.png
image.png2198×322 39.3 KB

6

Sending the same message using SMTP fixes the issue (both the subject included in the webhook event and shown in KMail are OK).

{
    "type": "Reception",
    "id": "c60c9affcaae11efb514960002cafe7c",
    "sender": "c60c9affcaae11efb514960002cafe7c@psrp.email.ahasend.com",
    "recipient": "farhad@taskulu.com",
    "queue": "8b9dcfcf-94f7-477c-8372-0178618ef975@taskulu.com",
    "site": "",
    "size": 2639,
    "response": {
        "code": 250,
        "enhanced_code": null,
        "content": "",
        "command": ""
    },
    "peer_address": {
        "name": "farhad-83dg",
        "addr": "217.142.21.126"
    },
    "timestamp": 1736003746,
    "created": 1736003746,
    "num_attempts": 0,
    "bounce_classification": "Uncategorized",
    "egress_pool": "",
    "egress_source": "",
    "meta": {
        "domain_id": "72703b13-166a-4f41-ab96-7333c61608f5",
        "customer_id": "8b9dcfcf-94f7-477c-8372-0178618ef975",
        "direction": "outbound",
        "metadata_retention": null,
        "data_retention": null,
        "tags": ""
    },
    "headers": {
        "Subject": "بوت‌كمپ قدرت نوشتن رهنماکالج",
        "From": "آها سند <no-reply@email.ahasend.com>",
        "Message-Id": "<DlpLff99xakZENfgl9vNAebOPmeyTpkYO6XrYBYH6Vg@farhad-83dg>",
        "References": "",
        "In-Reply-To": ""
    },
    "delivery_protocol": "",
    "reception_protocol": "ESMTP",
    "nodeid": "2f6489ba-9d9d-47d5-83bb-8492ac3fdd93",
    "feedback_report": null
}

image.png
image.png2198×322 34.5 KB

7

This might be related to this:

8
9

Where HTTP injected messages could be having a problem with UTF-8 headers. Will try to investigate in the next few days, but it sounds like your last comment confirms that this is not a problem when injecting with SMTP.

10

Thanks Tom, I tested everything with the latest dev version from today btw.

11

Is this resolved with the latest version?
(kumod 2025.01.06-2bb7e4c0)

12

No, it’s still hapenning:

{
    "type": "Reception",
    "id": "a6ef94d6cc5311efa1e8960002cafe7c",
    "sender": "a6ef94d6cc5311efa1e8960002cafe7c@psrp.email.ahasend.com",
    "recipient": "farhad@taskulu.com",
    "queue": "8b9dcfcf-94f7-477c-8372-0178618ef975@taskulu.com",
    "site": "",
    "size": 2345,
    "response": {
        "code": 250,
        "enhanced_code": null,
        "content": "",
        "command": ""
    },
    "peer_address": {
        "name": "",
        "addr": "127.0.0.1"
    },
    "timestamp": 1736184512,
    "created": 1736184512,
    "num_attempts": 0,
    "bounce_classification": "Uncategorized",
    "egress_pool": "",
    "egress_source": "",
    "meta": {
        "domain_id": "72703b13-166a-4f41-ab96-7333c61608f5",
        "customer_id": "8b9dcfcf-94f7-477c-8372-0178618ef975",
        "direction": "outbound",
        "metadata_retention": null,
        "data_retention": null,
        "tags": ""
    },
    "headers": {
        "Subject": "بوت‌كمپ قد��ت نوشتن رهن��اکالج",
        "From": "آها سند <noreply@email.ahasend.com>",
        "Message-Id": "",
        "References": "",
        "In-Reply-To": ""
    },
    "delivery_protocol": "",
    "reception_protocol": "HTTP",
    "nodeid": "2f6489ba-9d9d-47d5-83bb-8492ac3fdd93",
    "feedback_report": null
}

# /opt/kumomta/sbin/kumod --version
kumod 2025.01.06-2bb7e4c0

image.png
image.png2196×268 39.3 KB

13

should be resolved by mailparsing: fix qp_encode with UTF-8 subsequences containing 0x20 · KumoCorp/kumomta@7390f31 · GitHub

14

Confirmed, it’s resolved.