Is it possible to set trusted_hosts for /metrics endpoint only?

get-help
1

I have basic auth configured and working. The trusted hosts on my listener are set to localhost only at the moment.

    kumo.start_http_listener({
        listen = "0.0.0.0:8000",
        trusted_hosts = { "127.0.0.1", "::1" },
    })

Is it possible to set the trusted hosts for the /metrics api endpoint separately? My desired end state is to allow an internal prometheus to scrape the metrics endoint without auth while maintaining basic auth on all of the other endpoints.

2

Hey there @yogic-koala, thanks for posting. Please read the “Troubleshooting” and “How to Ask for Help” buttons below. If you would like a 1:1 support session from the KumoMTA team, details are at the “Book a Support Session” button below.

3

Any host not in the trusted hosts list with have to auth, so setting the prometheus host to trusted but no other hosts should do it, unless you mean you don’t want the prometheus host to touch any other endpoint.

4

My prometheus IPs are not stable. It runs as a statefulset in Kubernetes and the pods get a new IP each time they are restarted.

5

Aah.

6

Somewhat related, if the /metrics endpoint honored the basic auth config this would be a on issue as I can simply configure prometheus to use basic auth for scraping.

7

Any idea if its supported to configure multiple http listeners?

8

Hmm? Are you saying that a call to /metrics didn’t trigger http_server_validate_auth_basic?

9

correct

10

https://docs.kumomta.com/reference/http/metrics/

Access to this endpoint requires Trusted IP authentication. HTTP authentication is not permitted.

11

Aah yes. I don’t know that starting another listener would help, let me check on a couple of things.

12

I’m testing a work around for now, though I’d like to clean it up later. I am running in Kumo in Kubernetes. What I did was start a 2nd http listener on a separate port and a different trusted_hosts config.

    -- Configure HTTP Listeners for injection and management APIs.
    -- See https://docs.kumomta.com/userguide/configuration/httplisteners/
    kumo.start_http_listener({
        listen = "0.0.0.0:8000",
        trusted_hosts = { "127.0.0.1", "::1" },
    })
    -- Separate HTTP listener to use for prometheus metrics. The metrics endpoint only supports
    -- the trusted_hosts and does not work with basic auth. https://docs.kumomta.com/reference/http/metrics/
    -- This is a workaround. They key is that this port is not exposed through the kubernetes service or ingress. It should be available in cluster.
    kumo.start_http_listener({
        listen = "0.0.0.0:8080",
        trusted_hosts = { "10.32.0.0/16" },
    })

That port is exposed inside the k8s cluster, but is not available to any outside parties. This enables prometheus to scrape it without issue. The big downside is that anything in the Kubernetes cluster would allowed to hit the APIs. Thats not ideal but workable for the moment.

13

the back story here is that we consider metrics (and various operator commands) to be privileged, and since the current shape of our http auth hooks doesn’t allow for role based authorization, we use trusted IPs as the way to decide whether a given request has sufficient privilege.

For the moment, I would recommend configuring the listener (or a secondary) with trusted hosts set to match your network environment.

Longer term, we can look at expanding the authentication events for this sort of thing; we’d need to introduce some way to indicate whether an auth id has a particular role and some other helpers for eg: parsing a bearer token and extracting the auth id and so on, so that it is possible to use something like JWT or similar and plug in to other existing authentication services that might be present. There’s reasonable amount of stuff to design and consider around this, so we’ve been in a bit of a holding pattern to see what sorts of things customers need, so that we can avoid over-designing the wrong thing.