Monitoring & Observability

Running an ACME server in production requires visibility into its health, request volume, and error rates. acme-proxy offers three surfaces for that: a health endpoint, structured logging, and a Prometheus exporter.

Health checks

• Endpoint: GET /health
• Response: 200 OK with the body {"healthy": true}

The handler takes no application state: it does not query the database, and it does not consult the signer. It answers 200 whenever the process is alive and able to accept a connection, and it has no other failure mode. Treat it as a liveness probe, not a readiness probe — it will not tell you that the disk holding sqlite.db filled up.

What makes it worth polling is where it is mounted. /health lives on the root router, not inside a profile, which means it is deliberately outside:

• the admission-control layer, so a saturated server still answers the probe (inside the limit, the probe was starved exactly when it mattered, and a load balancer would go on reporting the node healthy right up to the point where the probe could no longer get a slot);
• every profile’s filter chain, so an IP allowlist never has to be widened for your load balancer;
• the Replay-Nonce middleware, so probing costs no database write; and
• the Link: rel="index" middleware.

GET / redirects to /health.

Metrics

GET /metrics serves the OpenMetrics text format (application/openmetrics-text; version=1.0.0), which Prometheus reads natively. It is off by default and lives on a listener of its own — a third socket beside the ACME and admin ones, configured by [metrics]:

[metrics]
enabled      = true
bind_address = "127.0.0.1:3002"

The separate port is the access control. A scrape carries no credential and none is checked: reaching the port at all is the permission, so restrict it the way you restrict any other internal service — a firewall rule, a network policy, or leaving it on loopback and scraping from the same host. The endpoint is absent from the ACME and admin sockets entirely, so exposing the ACME listener to the internet does not expose this.

$ curl -s localhost:3002/metrics
# HELP acme_proxy_requests Requests served, by endpoint, matched route and response status.
# TYPE acme_proxy_requests counter
acme_proxy_requests_total{role="acme,admin,worker",profile="default",route="/newOrder",status="201"} 42
acme_proxy_requests_total{role="acme,admin,worker",profile="none",route="/health",status="200"} 8613
# HELP acme_proxy_request_duration_seconds Time to answer a request, by endpoint and matched route.
# TYPE acme_proxy_request_duration_seconds histogram
# UNIT acme_proxy_request_duration_seconds seconds
acme_proxy_request_duration_seconds_sum{role="acme,admin,worker",profile="default",route="/newOrder"} 0.84
acme_proxy_request_duration_seconds_count{role="acme,admin,worker",profile="default",route="/newOrder"} 42
acme_proxy_request_duration_seconds_bucket{role="acme,admin,worker",le="0.005",profile="default",route="/newOrder"} 3
…
acme_proxy_request_duration_seconds_bucket{role="acme,admin,worker",le="+Inf",profile="default",route="/newOrder"} 42
# HELP acme_proxy_certificates_issued Certificates signed, by endpoint.
# TYPE acme_proxy_certificates_issued counter
acme_proxy_certificates_issued_total{role="acme,admin,worker",profile="default"} 41
# HELP acme_proxy_certificate_issue_failures Issuance attempts the CA refused, by endpoint and ACME problem type.
# TYPE acme_proxy_certificate_issue_failures counter
acme_proxy_certificate_issue_failures_total{role="acme,admin,worker",profile="default",reason="badCSR"} 1
# HELP acme_proxy_certificate_issue_duration_seconds Time from an accepted finalize to a stored certificate, by endpoint.
# TYPE acme_proxy_certificate_issue_duration_seconds histogram
# UNIT acme_proxy_certificate_issue_duration_seconds seconds
acme_proxy_certificate_issue_duration_seconds_sum{role="acme,admin,worker",profile="default"} 45.0
acme_proxy_certificate_issue_duration_seconds_count{role="acme,admin,worker",profile="default"} 41
…
# HELP acme_proxy_database_pool_connections Connections in the database pool.
# TYPE acme_proxy_database_pool_connections gauge
acme_proxy_database_pool_connections{role="acme,admin,worker",state="idle"} 4
acme_proxy_database_pool_connections{role="acme,admin,worker",state="busy"} 1
# EOF

A counter’s # TYPE line names its family without _total, as OpenMetrics requires; its series, which is what a query names, keep the suffix.

Six things are worth knowing about the numbers.

role names the roles that process runs, comma-separated — acme,admin,worker for an all-in-one deployment, which is the default. The counters are per-process memory by design, so a split deployment is several scrape targets reporting the same family names, and this is what tells them apart. Each process also needs its own metrics.bind_address.

route is the matched route pattern, not the URI. A request for /profile/le/order/9f3c… is counted under route="/order/{id}", and the endpoint it reached becomes the profile label. Anything that matched no route at all — a scanner, a typo — collapses into a single route="<unmatched>" series rather than one series per path tried. Root-router requests such as /health carry profile="none".

reason is the ACME problem type the CA refused with (badCSR, serverInternal), the same vocabulary acme-proxy audit list --event certificate_issue_failed prints. Both are rendered from one record, so the metric and the trail cannot disagree about what happened.

The two histograms time different things. acme_proxy_request_duration_seconds runs from the request reaching the router to its response head, with buckets from 5 ms to 10 s; it has no status label, because a histogram multiplies every label by its bucket count. acme_proxy_certificate_issue_duration_seconds runs from the finalize request being accepted to the certificate being stored, whichever process signs it, with buckets from 1 s to 1 h. It is measured from job timestamps, which are whole seconds, and for a relayed order it includes the upstream CA’s own validation. It is observed by the process that signs, so in a split deployment it is on the worker’s scrape target.

The counters survive a reload but not a restart. SIGHUP rebuilds the routers and keeps the registry, so a configuration change does not read as a counter reset; a restart genuinely is a new process and starts from zero, which is what rate() expects.

The pool gauge is read from the pool. state="idle" is connections the pool holds but has not checked out and state="busy" those in use, so busy is in-flight database work rather than a number this server maintains in parallel.

A minimal scrape configuration:

scrape_configs:
  - job_name: acme-proxy
    static_configs:
      - targets: ['ca.internal:3002']

A dashboard over all six families ships in the repository — see Grafana Dashboard.

Logging

acme-proxy uses the tracing crate, configured by the six keys in [logging] — filter, JSON or human-readable output, stdout or stderr, ANSI colour, span timing, and whether JSON fields sit at the top level. Every one of them is validated at startup: an unknown value is a refusal to start with a message naming the key, never a silent fallback.

Two of them are worth setting deliberately in production:

[logging]
# Structured fields become first-class keys rather than text to be re-parsed.
json_format = true
# ... and at the top level, which is what most pipelines want.
flatten_event = true

The rest of this page is about what those records contain.

Log levels

RUST_LOG=acme_proxy=info   # operational logging (recommended for production)
RUST_LOG=acme_proxy=debug  # per-request detail, challenge validation steps,
                           # database writes, truncated response previews
                           # from http-01

There is no trace level to reach for: the crate emits nothing below debug.

RUST_LOG replaces the whole filter, so a bare RUST_LOG=debug also turns on debug logging for every dependency. acme-proxy does not configure SQL statement logging itself; to see sqlx’s own statement logs you must ask for them explicitly, e.g. RUST_LOG=acme_proxy=info,sqlx=debug.

Everything on this page is the server’s log stream. The admin commands emit none of it unless asked, with --log-level or a non-empty RUST_LOG, and what they then emit goes to stderr rather than into the output a script is parsing.

Request correlation

Every request passes through one server-wide middleware that reads an incoming x-request-id header or generates a UUID v7 when absent, opens the request span, and echoes the id back on the response. Every log line emitted while handling that request is nested under that span and carries the id, so one client’s failing renewal can be pulled out of a busy log in a single query — and a reverse proxy that already assigns request ids will have its value preserved rather than replaced. A header whose bytes are not valid ASCII cannot be echoed back, so it is replaced by a generated id and a request_id_header_invalid line at debug says so.

The span carries method, uri, version, request_id, client_ip and — for a request that reached an ACME endpoint rather than /health — profile.

client_ip is the peer address of the connection, replaced by the address resolved from filter.forwarded_header where filter.trusted_proxies says the peer is a reverse proxy. Both are per-profile settings, so a request that never reaches a profile — /health, the http-01 responder, anything admission control sheds — always shows the peer. A request arriving over a socket with no peer address at all leaves the field absent rather than reporting a placeholder.

The access line

Each request closes with exactly one request_completed record carrying status and latency_ms. It is emitted under this crate’s own target, so it is visible at the default filter. Its level varies:

This is the only event name emitted at more than one level, and the only one whose outcome comes from the response rather than from its own name. Liveness probes are at debug deliberately: at one probe per second per node they would otherwise be most of the log. Set RUST_LOG=acme_proxy=debug to see them.

Structured events

Every log record carries two fields you can build alerting on, and their shape is enforced by a test rather than by convention (tests/logging_convention.rs).

event is a stable, greppable name, shaped <subsystem>_<object>_<outcome> — always a literal in the source, so a name in a log greps straight back to the line that wrote it. The subsystem prefix comes from a closed list, so a family grep finds the whole family: event = "certificate_revoke reaches every revocation refusal, db_ reaches the storage layer, challenge_http_01_ reaches one validator. Challenge types are always spelled with separators (http_01, dns_01, tls_alpn_01).

outcome is one of four values, and it exists so that “show me everything that broke” is an exact match instead of a suffix glob. Failure is spelled a dozen ways across the several hundred event names (_failed, but also _invalid, _mismatch, _missing, _unauthorized, _rejected), so globbing on the name silently misses most of it:

Every error-level record is outcome = "failure"; an advisory sits at warn.

The events worth building alerts on:

Only with [admin] enabled — see Web Admin:

The full set is much broader than this table — several hundred names, of which the ones above are the curated subset worth an alert. Because the subsystem prefix is drawn from a closed list, an ad-hoc investigation can grep a whole family: event = "certificate_revoke for every revocation refusal, event = "replaces_ for RFC 9773 correspondence, event = "db_ for the storage layer.

Suggested alerts

Two sources, and they are complementary rather than alternatives. The metrics are cheap to alert on and answer “how much”; the log stream answers “which one, and why”, and covers everything the metrics do not.

With [metrics] enabled:

# Issuance is failing, whatever the reason.
rate(acme_proxy_certificate_issue_failures_total[15m]) > 0

# Nothing has been issued over a window longer than your shortest renewal
# interval -- silent breakage, and the one nothing else will tell you.
increase(acme_proxy_certificates_issued_total[6h]) == 0

# The server is shedding load: undersized, or a client is in a retry loop.
rate(acme_proxy_requests_total{status="503"}[5m]) > 0

# 5xx as a share of everything: this server failing, not clients being refused.
sum(rate(acme_proxy_requests_total{status=~"5.."}[5m]))
  / sum(rate(acme_proxy_requests_total[5m])) > 0.01

# The pool is saturated, so requests are queueing on a connection.
acme_proxy_database_pool_connections{state="idle"} == 0

# Requests are slow: the 95th percentile over a second, on any route.
histogram_quantile(0.95,
  sum(rate(acme_proxy_request_duration_seconds_bucket[5m])) by (le, route)) > 1

The up metric Prometheus synthesises per target also gives a liveness alert for free, though /health remains the better probe for a load balancer since it is on the ACME listener itself.

From the log stream, which reaches further. The first is the general case and subsumes most of the rest; the others are worth splitting out because each has a different response:

• outcome = "failure" above its usual rate → something broke. This is the one query that catches every refusal, whatever the event is called, and it is the reason the field exists.
• request_shed above zero for more than a few minutes → the server is undersized, or a client is in a retry loop.
• challenge_validation_failed rising against a flat local_ca_leaf_issued rate → clients are asking and failing; check egress and the responder ports.
• Any upstream_relay_failed → issuance is broken for every client behind the relay, not just one.
• No local_ca_leaf_issued at all over a window longer than your shortest renewal interval → silent breakage.
• Any audit_write_failed → the audit trail is silently incomplete, and nothing else will tell you. See Audit Trail.
• server_fatal_error, db_migration_failed, profile_init_failed → page immediately.