The Bleeding Hearts Club: Heartbleed Recovery for System Administrators
The Heartbleed SSL vulnerability presents significant concerns for users and major challenges for site operators. This article presents a series of steps server and site owners should carry out as soon as possible to help protect the public. We acknowledge that some steps might not be feasible, important, or even relevant for every site, so the steps are given in order both of their importance and the order they should be carried out.
1. Update Your Servers
If you haven't yet, update any and all of your systems that use OpenSSL for TLS encrypted communications. This includes most web servers, load balancers, cache servers, mail servers, messaging and chat servers, VPN servers, and file servers, especially those running on Linux, Unix, BSD, Mac OS X, or Cygwin.
The vulnerable OpenSSL version numbers are 1.0.1 through 1.0.1f and 1.0.2-beta1. The flaw is fixed in OpenSSL 1.0.1g. However, some operating systems have introduced the fix to earlier branches of OpenSSL, and may instruct you to install packages with minimum versions such as 1.0.1e-2+deb7u5 (in the case of Debian GNU/Linux).
If your operating system has not yet released an updated package, download openssl-1.0.1g.tar.gz directly from https://www.openssl.org/source/ and follow the instructions in the INSTALL text file to compile the new version locally.
After installing a fixed version of OpenSSL, be sure to restart all services that depend on it. On your sysytem this might include web and proxy servers such as apache, nginx, pound, and squid, caches such as memcached and redis, databases like mysql and postgres, and mail services like postfix, exim, and dovecot. When in doubt, reboot the entire server if possible.
If you manage systems with custom operating systems like switches and routers, you may need to ask your vendor for a patch directly.
If you haven't updated your systems yet, stop reading and do it now. If this is the only step you can carry out in your environment, you will still have done the most important thing by far.
2. Test Your Servers
It's important to verify that the hole has been closed, especially if you have multiple servers and services to stitch up. The bad news is that this vulnerability is relatively easy to exploit. The good news is that means there are a few tools available to see if you're safe.
The SSL Server Test from Qualys SSL Labs will let you know if your web server remains vulnerable. If you have servers running on other ports to test, or STARTTLS mail servers, you can try the hb-test.py script. The hbcheck script can help you test an internal network using nmap. Finally, if you have a large number of hostnames to test, my hb-batch.py script might be helpful.
Please note these tests might not be completely reliable, and running them against servers you do not own might not always be considered polite.
3. Be Safer Next Time
This is the worst and biggest security flaw we've seen recently, but it won't be the last. Putting good practices into play for Heartbleed can help you prepare for anything else that might come down the pike next.
One of the strongest protections you can have against TLS vulnerabilities is Perfect Forward Secrecy. This is not simple to configure, and does not yet have global browser support. However, it is the encryption technology that provides the best defense against attacks with the potential to steal your private key and use it to decrypt your traffic.
You should also make sure you're practicing good password discipline. Use a password vault, use strong passwords, change them regularly, and don't reuse them.
Practice least authority for certificates, too. If you don't need to give everyone root access to every server, you probably don't need to give every server a certificate for *.example.com.
Finally, make sure you have reliable (if not automated) process for providing all of your servers with security updates quickly. After all, the only thing worse than getting pwned by a zero-day vulnerability is getting pwned by a one-day.
4. Consider Rekeying Your Servers
One of the worst things about the Heartbleed vulnerability is that it makes it theoretically possible for an attacker to recover your server's private key. Fortunately, the probability of this being possible on a given server appears quite low. Unfortunately, we can't yet be completely sure if that's true.
Key theft is a terrible attack because it tends to be undetectable by you, the server operator. Worse still is the harsh truth that, unless all your connections are served with Perfect Forward Secrecy, this would allow such an attacker not only to decrypt any newly intercepted traffic but to decode records of past traffic. If you run a server that intelligence agencies are likely to attack, this is a serious problem.
[Update: Cloudflare's Heartbleed Challenge has shown private key recovery to be achievable in a realistic scenario. This was done with a more work-intensive and discoverable attack than things like Heartbleed password and session recovery, but it might not be the only real-world attack targeting private key material.]
That means you may wish to consider revoking and regenerating your existing SSL certificates using new keys. Doing so will protect against the possibility of passive traffic decryption (if you don't use PFS) and man-in-the-middle attacks with a stolen key.
Because private key compromise via Heartbleed currently appears to be quite rare, this may not need to be a priority except for high value services (large or sensitive email and messaging systems, software distribution points, banks). Other services may not need to panic and rush to rekey quite so urgently. For most threat scenarios, adopting PFS provides greater overall protections than rekeying so we will remind you to make PFS a priority.
The details of the rekeying process will vary depending on the Certificate Authority you use to generate certificates and/or manage domain names. Some will allow you to regenerate in one step. Some will require you to revoke the old certificate before requesting a new one.1 Most will have a prominent link in their control panels, and many will waive their normal fees right now.
If you are given the option during the certificate regeneration process, it's a good idea to create a .csr file (Certificate Signing Request) and private key locally on your server using the openssl command. It might seem strange to prefer trusting OpenSSL at the moment, but it's still a safer bet than trusting a third party with your private key right off the bat.
5. Consider Changing Passwords
Unlike private key compromises, Heartbleed leakage of recently-used passwords from server processes linked to OpenSSL appears to have been quite common. Unfortunately, this could affect not just your operators and staff but your users.
This means you should perform risk assessment and determine which categories of passwords on your servers and services may need immediate resets, user-reset-on-next-login, or advisories suggesting resets. Variables in the risk assessment include how quickly you were able to patch your servers after the vulnerability was publicly disclosed at around 17:30UTC on 2014-04-07,2 the sensitivity and value of potentially accessible accounts, whether accounts had been used recently (meaning their passwords were in RAM), and the probability that random or specific people on the Internet might have found your servers to be interesting targets.
You should determine which passwords are of sufficient value to deserve precautionary resets, and perform these after the steps above, in order to offer the new passwords proper protection. (If you've decided to rekey because of a concern about private key exposure, that is another reason to change users' passwords.)
You should also consider changing CSRF and OAUTH authentication tokens, invalidating session cookies, and rotating authentication cookies. These steps can be performed independently of passwords changes and may be far less disruptive.
6. Update Your Users
Your users have already heard of this scary Internet password thing, and chances are they're concerned about how it affects your site. Let them know what you've done, what you will do, and what the remaining risks are. Don't try to give them a false sense of security. Knowing that you're working on it and reaching out to them at all will work wonders.
7. Turn on Perfect Forward Secrecy
Because you skipped it in step three, didn't you? That's okay. There's still time. We'll wait.
- 1. The infrastructure for revocation of certificates is quite broken, but that's a story for another post.
- 2. According to a recent article in Bloomberg, it appears that NSA knew about and used the Heartbleed vulnerability for at least two years before its public discovery. It is not currently known if any other attackers had knowledge of the bug before April 7 2014.