This is a surprisingly common issue. I’ve had it happen at least once in every job I’ve worked. This is usually the responsibility of the devops or devsec teams, and they are usually heavily underfunded since they are cost centers that do not bring in profit.
I’m young enough that I never had to experience anything but let’s encrypt/ACME. Manually renewing certs sounds like such a major PITA that I’d switch to it as soon as I could…
I work in DevOps, this is one of the easier things to automate. It’s common for certs to be issued on a 90 day basis these days, no way that would be maintainable without automating.
The issue here is this is a client certificate, issued within the League client, for seemingly local<->local traffic. This ain’t no typical HTTPS ceritifcate, it’s bundled into the client build. See from the source “League client’s hard-coded certificate meant someone at Riot would’ve needed to remember it required updating before its expiration date.”
So, not quite as easy as configuring an ACME CRON, but something that’d need to be remembered or have some kind of internal reminder for.
Certbot / LE has to be running on some machine and that machine can be accidentally turned off, payments not fulfilled, was supposed to be moved but the new instance doesn’t work, gateway configuration changed, etc.
Automation requires maintenance and that introduces human error
This is a surprisingly common issue. I’ve had it happen at least once in every job I’ve worked. This is usually the responsibility of the devops or devsec teams, and they are usually heavily underfunded since they are cost centers that do not bring in profit.
There are tools to actually remind you to do this on a timely fashion… Also, some of them go as far as doing auto renewal. Is this such a hard thing?
I’m young enough that I never had to experience anything but let’s encrypt/ACME. Manually renewing certs sounds like such a major PITA that I’d switch to it as soon as I could…
Yeah. Oh it’s 10 years until we gotta replace it. That’s someone else’s job
I work in DevOps, this is one of the easier things to automate. It’s common for certs to be issued on a 90 day basis these days, no way that would be maintainable without automating.
The issue here is this is a client certificate, issued within the League client, for seemingly local<->local traffic. This ain’t no typical HTTPS ceritifcate, it’s bundled into the client build. See from the source “League client’s hard-coded certificate meant someone at Riot would’ve needed to remember it required updating before its expiration date.” So, not quite as easy as configuring an ACME CRON, but something that’d need to be remembered or have some kind of internal reminder for.
I’m aware, but it should have been part of their build system and they should have, at the very least, had alarms for this.
Even the simplest things fail sometimes
That’s what alarming is for.
The problem sometimes is the automation failing for some reason.
Have you had Certbot or LE fail on prod for you before?
I’m sure stuff happens, but I usually view them as one of the most robust moving parts on a server.
E: I don’t mean to express disbelief at all; just curious to learn about possible footguns.
Certbot / LE has to be running on some machine and that machine can be accidentally turned off, payments not fulfilled, was supposed to be moved but the new instance doesn’t work, gateway configuration changed, etc.
Automation requires maintenance and that introduces human error