Troubleshooting

This section covers common issues when operating acme-proxy in production.

The server refuses to start

Several configuration mistakes are deliberately fatal at startup rather than silently degrading at runtime. The log line names the problem in each case.

Failures specific to a hardware CA key — PIN, token, slot and mechanism problems — have their own table in Hardware Keys (PKCS#11).

A wildcard order is rejected

• Symptoms — newOrder for *.example.com returns rejectedIdentifier naming dns-01.
• Cause — Wildcards can only be proven with a DNS challenge, so acme-proxy accepts them only when dns-01 is among challenge.enabled.
• Fix — Add dns-01 to challenge.enabled. See Challenge Validation.

A client suddenly cannot order anything

• Symptoms — Every order-side request from one client returns unauthorized.
• Cause — The account has been deactivated — by the client itself, or by acme-proxy account deactivate. Deactivation is permanent and blocks all issuance.
• Fix — The client must register a new account.

An order sits in processing and never finishes

• Symptoms — A client polls an order that reached processing and stays there. Only the relay backend defers like this; local_ca and custom answer finalize inline.
• Cause — The issuance is a background job, and it is either waiting out a backoff after a retryable upstream failure or has been retired for good. The order object itself will not say which.
• Fix — Ask the queue:

# What is the job doing, and how many attempts has it spent?
acme-proxy jobs list --kind signer_relay_issue --limit 20

# The upstream's own error text, and the URLs it was talking to.
acme-proxy jobs show <job-id>

A job still ready is waiting for its next attempt at the run_at it prints — acme-proxy jobs run-now <id> pulls that forward. A failed one has spent its budget: run-now grants exactly one more attempt, and acme-proxy jobs cancel <id> abandons the ACME order so the client stops polling and can order again. See Job queue, and job_run_abandoned in Monitoring for the log line that says it happened.

SQLite database locks

• Symptoms — The server logs show database is locked errors during high concurrency order creation.
• Cause — The database may not be utilizing Write-Ahead Logging (WAL) or your filesystem does not support proper locking mechanisms (e.g., NFS).
• Fix — Ensure acme-proxy is running on a local filesystem and journal_mode = WAL is applied. (The server automatically attempts to enable WAL on startup).

Reading the database directly

When the CLI cannot answer a question — usually “what does the row actually say?” — the database is readable while the server runs, because WAL allows a reader alongside the writer.

# Which migrations have run, and did they all succeed?
sqlite3 sqlite.db "SELECT version, description, success FROM _sqlx_migrations;"

# Everything this server refused in the last day, and why.
sqlite3 sqlite.db "SELECT datetime(created_at,'unixepoch'), event, profile, client_ip, reason
                     FROM audit_log WHERE outcome = 'failure'
                      AND created_at > strftime('%s','now','-1 day');"

# An order that will not progress: its status and its authorizations'.
sqlite3 sqlite.db "SELECT o.status, a.identifier, a.status, c.type, c.status, c.error
                     FROM orders o JOIN authorizations a ON a.order_id = o.id
                     JOIN challenges c ON c.authz_id = a.id WHERE o.id = '…';"

Read, do not write. The CHECK constraints will catch an impossible status, but nothing re-syncs the in-memory state a running handler is holding. Use the Admin CLI to change anything.

Backups must include the WAL. Copying sqlite.db on its own gives you a database missing every recent write. Either take all three files (sqlite.db, -wal, -shm) with the server stopped, or run sqlite3 sqlite.db ".backup backup.db", which is consistent by construction and safe against a running server.

The schema itself — every table, constraint and index, and why each is shaped the way it is — is documented in Database Schema.

Upstream Let’s Encrypt rate limits

• Symptoms — Order finalization fails with HTTP 429 Too Many Requests from the upstream CA.
• Cause — When using the relay signer backend, all internal clients share a single external ACME account. Let’s Encrypt applies rate limits (e.g., 50 certificates per registered domain per week).
• Fix — Request a rate limit increase for the root domain you are relaying, or implement careful caching mechanisms on your internal servers to prevent excessive renewals.

Network challenge blockages

• Symptoms — Order stays in pending state, or challenge verification fails.
• Cause — If challenge.bypass = false, the server must reach the internal client over HTTP (port 80) or DNS to verify domain ownership. Firewalls might be blocking this internal callback.
• Fix — Ensure the host running acme-proxy has egress network access to reach the internal servers requesting certificates.

EAB registration fails

• Symptoms — Client receives an error stating External Account Binding is required.
• Cause — The client is connecting to a profile that requires EAB, but did not provide the kid and hmac credentials.
• Fix — Create EAB credentials using the acme-proxy eab create CLI command and configure the client to use them.

Order finalization fails (413 payload too large)

• Symptoms — Finalizing the order returns a 413 error.
• Cause — The encoded Certificate Signing Request (CSR) exceeds the server.max_body_bytes limit (default 128 KiB).
• Fix — Ensure your client is not generating excessively large CSRs or increase the limit in your configuration.

Order finalization fails (badCSR: identifier mismatch)

• Symptoms — Finalizing the order returns 400 badCSR complaining about the requested identifiers.
• Cause — The DNS Subject Alternative Names in your CSR are not exactly the set of names the order authorized. The comparison is set equality, so an extra name fails just as an omitted one does.
• Fix — Configure your ACME client to put every ordered domain — and nothing else — into the CSR’s SANs.

Order finalization fails (badCSR: common name)

• Symptoms — 400 badCSR, “CSR common name is a domain the order does not cover”.
• Cause — The CSR’s Subject Common Name looks like a DNS name that the order does not authorize. acme-proxy does not ignore the CN: a domain-shaped CN must be covered by the order, precisely so a name cannot be smuggled past the identifier filters by moving it out of the SANs.
• Fix — Either add that name to the order, or drop the CN. A CN that is not domain-shaped — a human label like rcgen self signed cert — is tolerated and ignored. Note that the local_ca signer empties the subject entirely on the issued certificate, so a CN is never carried through to the leaf.

Order finalization fails (badCSR: non-DNS SAN)

• Symptoms — 400 badCSR from the local_ca signer for a CSR containing an IP, email or URI SAN.
• Cause — local_ca accepts DNS SANs only, and rejects anything else outright rather than stripping it.
• Fix — Remove the non-DNS SANs from the CSR.

Connection refused or 503 service unavailable

• Symptoms — High-throughput ACME clients receive HTTP 503 errors during bursts.
• Cause — The server’s load shedder activated because the number of concurrent in-flight requests exceeded server.max_concurrent_requests.
• Fix — Increase max_concurrent_requests and admission_wait_ms, or configure your client to retry with exponential backoff.