Rate limit ACME requests

[tashian]

Sometimes ACME clients can misbehave and it's pretty easy to DoS step-ca in that case.

@MCWertGaming discovered an interaction between Caddy and step-ca that causes a flood of ACME requests, possibly triggered by the CA being unable to do a DNS lookup of the requested domain. See #598 for step-ca context and logs, and see also caddyserver/caddy#4186 for details of the Caddy side of things (potential issue with Caddy's ACME client).

Note: Rate limiting in ACME needs to return a rate limit error as defined in the RFC.

[staxfur]

As I said earlier, I'd suggest that one domain can only have something like 5 requests per hour and unfinished request should probably get cleaned up after some time, because if they are not finished after 5 minutes, the client probably already tried to order a new one anyway.

[staxfur]

I provided more Informations here: caddyserver/caddy#4186 (comment).

[TheSecMaven]

We run into this issue ourselves when a client is not being a good actor. Rate limiting is part of the RFC https://datatracker.ietf.org/doc/html/rfc8555#section-6.6

[dopey]

Hey, we had the chance to discuss this issue in depth this morning. I'll do my best to transcribe our thoughts.

Our main question during the discussion was: what are we actually trying to prevent?

• bad actors trying to DOS the server?
• good actors that are in a bad state?

The "fixes" would be different depending on which of these scenarios we're trying to prevent.

A level deeper - we're weary of rate limiting because we don't want to introduce unintended consequences. e.g. Suppose we rate limit based on client IP. Well if the CA is being used in RA mode then we may expect a single client to making requests on behalf of many domains. Suppose we rate limit based on domain name, then I could DOS you from getting a cert if your CA were public and I knew your domain names. We could list scenarios for a while (and we did this morning), but our conclusion was that we didn't want to start fixing something without a clearer understanding of the underlying problem (is the server running out of memory, disk, open sockets, etc.). And, more generally, we believe that a client making too many requests is a client side bug and should be fixed on that side.

We'll leave the issue open because we want to collect more info about use cases and actual server issues (OOM, segfault, etc), but we're not planning any work here yet.

[staxfur]

For me the problem was that the open requests made the internet database grow (for me 200mb was the limit, something like 206mb and the database was broken as described before). In the moment on which the db gets to that size, step-ca starts to use 100% CPU while doing nothing. The website still works, like ca.domain.com/health, but acme requests are not handled anymore. After a restart of step-ca either the db still works and step-ca is just running normally after, or badger is complaining about some unsafe config option I would need but it's not telling a reason. The more precise explanation is in the discussing if it would help you in any way.

Thanks for getting deeper into this!

[staxfur]

Hey, any update on this? My CA broke again due to too many pending acme requests that couldn't succeed because the DNS entry was wrong or missing.

[staxfur]

Caddy has fixed the bug already where it's flooding the step-ca server, but it's still a problem when the caddy server is creating requests over days.

[Skinner927]

I've encountered the same "friendly" DoS attack where a DNS entry was incorrectly changed, step-ca could no longer resolve the domain, and Traefik ACME spammed step-ca until the drives on the step-ca host hit 100% and essentially bricked the host.

Rate limiting friendlies and/or some type of automatic logrotate or log pruning to prevent something like this would be welcomed.

[dopey]

@MCWertGaming @Skinner927 can you be more specific about what exactly failed? Did the DB of step-ca fill up? Was it the logs of step-ca that filled up?