Akka gRPC with Let’s Encrypt

The Electric Frontier Foundation (EFF) has recommendations about encrypting the web; there is no reason to be running servers over unencrypted HTTP any longer. It is irresponsible to your users and unnecessary, as such there is a formal mechanism called HTTP Strict Transport Security (HSTS) that enforces HTTPS for all requests at the domain level. Taking it further, modern browsers include a set of domains which can only work over HTTPS, it started with Google TLDs such as .dev and .app, but it is growing https://hstspreload.org/.

Let’s Encrypt is the most popular way to get a free TLS certificate. (Many still refer to these incorrectly as the outdated an SSL certificate, leaving documentation pedantically incorrect and libraries such as OpenSSL unfortunately named) While cost is no longer a factor preventing HTTPS adoption, there continue to be additional configuration difficulties. Even glossing over certificate revocation, certificate expiration is a plague in most enterprises. In my experience managing TLS certificates has mostly been a manual process, especially when self-signed certs are used in non-production environments. In microservice setups, there are often hundreds (!) of services each with a separate certificate making it assured there is always one expiring at an inopportune moment.

The Let’s Encrypt Certbot tool automates renewal, and it is arguably better to use the Let’s Encrypt practice of 90 day expiry even when offered the choice for more. DevOps rust without use and a brisk 90-day repeating process essentially forces automation. Fortunately, it is a smooth process for anyone familiar with security practices, and if your organization lacks this talent you have much larger issues to worry about.

According to the Akka documentation: Akka relies on the JVM TLS implementation, which is sufficient for many use cases, but is planned to be replaced with conscrypt or netty-tcnative. I think this means instead of using the public/private PEM key pair generated by Certbot directly, such is done with other software such as Nginx, there are additional steps necessary to create a PKCS12 file. PKCS12 files are common with JVM software and while this is an extra step (and another password to manage), it is hardly extra work within an automated process.

Creating a PKCS12 file

Setting up a running certbot is outside the scope of this article; it really depends on your domain and environment setup. In the end, it will create the key files: cert.pem, chain.pem, fullchain.pem and privkey.pem in the /etc/letsencrypt/live/[domain] directory for your [domain].

Side note: While working through commands to generate the PKCS12 file, there is a minor issue as noted at https://community.letsencrypt.org/t/cannot-verify-domain-with-openssl/11545, that the cert.pem chain is incomplete. This makes it necessary to pass the fullchain.pem to -in below, some documentation may incorrectly suggest cert.pem be used. Basically consumers can’t traverse from the child cert.pem to a root Certificate Authority (CA) certificate in the normal way, so it is necessary to provide an explicit path/chain.

openssl pkcs12 -export \
	 -in /etc/letsencrypt/live/[domain]/fullchain.pem \
	 -inkey /etc/letsencrypt/live/[domain]/privkey.pem \
	 -out /etc/letsencrypt/live/[domain]/cert.pkcs12 \
	 -name [your-url] \
	 -CAfile /etc/letsencrypt/live/[domain]/fullchain.pem \
	 -caname "Let's Encrypt Authority X3" \
	 -password pass:[password]

This cert.pkcs12 is now the only file needed to enable HTTPS server-side encryption - Akka won’t be using any of the PEM files directly. Copy the PKCS12 file to where your Akka application can securely read it and remember your [password]!

Akka HTTP / gRPC

Enabling HTTPS on a gRPC or HTTP server in Akka is done by calling enableHttps during the Http builder to specify a httpsConnectionContext.

Http(actorSystem)
  .newServerAt(interface, port)
  .enableHttps(httpsConnectionContext)
  .bind(routes)

The httpsConnectionContext is generated using a JVM Keystore, which contains private keys to TLS certificates and should not be confused with a Truststore such as cacerts which contains only public keys.

To create an HttpsConnectionContext generate a new Keystore, load the cert.pkcs12 file into it, and use it to create a SSLContext:

def serverHttpsContext(pkcs12File: File, pkcs12Password: String): Try[HttpsConnectionContext] = {
  Using(new FileInputStream(pkcs12File)) {
    contentStream =>
      val ks = KeyStore.getInstance("PKCS12")
      ks.load(contentStream, pkcs12Password.toCharArray)
      ks
  }.map {
    keyStore =>
      val keyManagerFactory = KeyManagerFactory.getInstance("SunX509")
      keyManagerFactory.init(keyStore, pkcs12Password.toCharArray)
      val context = SSLContext.getInstance("TLS")
      context.init(keyManagerFactory.getKeyManagers, null, new SecureRandom)
      ConnectionContext.httpsServer(context)
  }
}

If Akka starts without error it still doesn’t ensure full functionality. As mentioned above it is possible to have incomplete chains due to self-signed certificates or misconfiguration, it is still important to ensure full functionality with a cryptographic tool such as OpenSSL:

openssl s_client -connect <hostname>:<port>

Certificate Renewal

Certbot certificates expire in 90 days, so repeating this process in an automated way is key. Once the certbot is initially run, certbot renew will create another set of PEM certificates without user input. Createing the PCKS12 file and more importantly restarting the Akka application will test your zero-downtime engineering.