My Faults My Own (and other rossry.net and r-y.io subdomains) are now available over HTTPS, with certificates from Let's Encrypt. (cf. https://blog.rossry.net/https)

The setup took nontrivial effort, so I've narrated it here for my or your future reference. I don't think there's anything technically novel here, and there may even be an HTTPS-setup guide for 2019 somewhere else that dominates mine for usefulness, but there wasn't one easy-to-find enough that I found it, so here we are.

(0)

First, the dramatis personae:

Let's Encrypt (hereafter "LE"), a project of the nonprofit Internet Security Research Group, issues free TLS (née SSL) certificates; they recommend that site administrators with shell access use the LE client Certbot, a project of the Electronic Frontier Foundation.

My Faults My Own, and other rossry.net and r-y.io subdomain services, are happily hosted by Digital Ocean (this turns out not to matter), running nginx on Ubuntu 14.04. (Certbot supports many other servers and OS setups as well; I'm listing my specifics here just as context for the following narration, especially as this combination specifically has some tricky issues.)

My domain registrar and DNS provider, Namecheap, sadly does not really support Certbot's automated DNS-based authentication (necessary for a "wildcard certificate", which will cover all of *.rossry.net), and I don't want to manually mess with DNS records every 90 days to get new certificates issued.

Fortunately, joohoi/acme-dns is a lightweight DNS server intended specifically to help automate ACME DNS challenges, and even comes with a certbot hook by the author. The dns-01 challenge protocol of the Automatic Certificate Management Environment standard involves setting _acme-challenge.rossry.net to a specified TXT record to prove that I am the controller of *.rossry.net.

(1)

I followed Certbot's Ubuntu/nginx install directions and installed go (from binary), sqlite3 for acme-dns, and python-requests for joohoi's acme-dns client hook.

To get my acme-dns up, I needed to tweak the config.cfg so that it was listening for API requests (to localhost) on a non-80 port (since 80 was already in use for http traffic; I chose 8053), which meant that I would later have to tweak acme-dns-auth.py to likewise send its calls to http://127.0.0.1:8053.

For the dns-01 challenge, though, recall that I needed acme-dns to be able to serve a specific TXT record at _acme-challenge.rossry.net. The way that acme-dns handles this is clever, I thought, involving three DNS records with Namecheap that I'll hopefully never need to change again:

• A record for auth.rossry.net → 162.243.94.179, my standard IP address (where acme-dns is listening on port 53)
• NS record for auth.rossry.net → auth.rossry.net. (indicating that auth.rossry.net is an authority for any *.auth.rossry.net records)
• CNAME record for _acme-challenge.rossry.net → 3b5[...]6d3.auth.rossry.net, an arbitrary subdomain of auth.rossry.net that acme-dns can set to an appropriate TXT record, as an authority for all of *.auth.rossry.net.

After checking that this was up ($ dig auth.example.com is your friend!), the Certbot command I used to get my certificate from LE was:

$ sudo certbot certonly \
    --manual \
    --manual-auth-hook ~/w/auth/acme-dns-auth.py \
    --preferred-challenges dns \
    --debug-challenges \
    -d rossry.net \
    -d \*.rossry.net

...at which point I had a certificate(!), but no nginx config setup.

(2a)

Now, for a brief digression about HTTP/2. HTTP/2 is a new transport protocol for HTML-based content, with headline advantages of multiplexing, deserialized pipelining, HTTP header compression, server-pushed content, and on-wire binary serialization for HTML. All major browser clients support HTTP/2 only over TLS.

nginx has supported HTTP/2 since version 1.5.9. Unfortunately, the latest version available in Ubuntu 14.04's stable repositories is 1.4.6. Separately, nginx needs to be compiled with OpenSSL 1.0.2 in order to support the Application-Level Protocol Negotiation (ALPN), which Chrome requires for HTTP/2 -- but nginx’s own PPA compiles for 14.04 with OpenSSL 1.0.1f.

At this point, I could either upgrade my OS or re-install an upgraded nginx built with up-to-date OpenSSL. 14.04's end-of-life is coming in April 2019, and I'll need to upgrade to 18.04 soon, but I wanted to stop this project from mushrooming too far out of control, so I decided to find a way to upgrade nginx only.

Fortunately, Ondřej Surý (better known for officially packaging PHP for Debian), packages nginx built with up-to-date OpenSSL at ppa:ondrej/nginx. I added the PPA and re-installed upgraded nginx with:

$ sudo service nginx stop
$ sudo apt-get remove nginx
$ sudo add-apt-repository ppa:ondrej/nginx
$ sudo apt-get update
$ sudo apt-get install nginx
$ sudo rm /etc/nginx/sites-enabled/default
$ sudo service nginx start

...whereupon $ nginx -V reported nginx version: nginx/1.14.2 and built with OpenSSL 1.1.1, as desired.

(2b)

I'm going to run optional-HTTPS for a while before switching over as default, so first I wanted to add HTTPS to my server on top of the existing HTTP. Here's what that looked like for static.rossry.net, a subdomain I use to serve files and static pages:

server {
    listen 80;
    server_name static.rossry.net;

    location / {
        alias /home/rry/w/static/;
    }
}

server {
    listen 443 ssl http2;
    server_name static.rossry.net;

    ssl_certificate /etc/letsencrypt/live/rossry.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/rossry.net/privkey.pem;

    location / {
        alias /home/rry/w/static/;
    }
}

$ service nginx reload && service nginx restart, and now https://static.rossry.net/papers/sp-continuous_geb2018.pdf serves you Jagadeesan et al. (2018) over HTTPS.

Adding a similar stanza for blog.rossry.net: (n. b.: the reverse-proxy forwarding to localhost:$PORT is standard practice for a Ghost server)

server {
    listen 443 ssl http2;
    server_name blog.rossry.net;

    ssl_certificate /etc/letsencrypt/live/rossry.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/rossry.net/privkey.pem;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host      $http_host;
        proxy_pass       http://127.0.0.1:2368;
    }
}

...and trying to hit https://blog.rossry.net almost worked, except that some absolute links to http://blog.rossry.net/assets/ broke, as Content Security Policy meant my browser refused to downgrade security by linking them. I had to go through my custom Ghost theme's HTML generators and change external links to use https:// and clean up internal links to use Ghost's helpful {{asset 'css/foo'}} Handlebars helper. (Internal relative links would probably have worked, but I already preferred to use Ghost's asset helper for features like caching and cache-busting query strings.) Chrome's console was helpful in finding what things were not being loaded.

(3)

With the manual registration workflow figured out, I wanted an automated re-registration script I could chuck into my crontab and forget about. Jeff Kaufman suggests that this works in a root crontab:

26  2 * * * certbot renew --quiet --post-hook "service nginx restart"
26 14 * * * certbot renew --quiet --post-hook "service nginx restart"

...which I'm willing to believe, though I haven't seen it do the hard part yet. I suppose I'll try to come back and edit this section when I do. (I picked 2:26 randomly, in order to keep the load on LE's servers more consistent. Someone somewhere on the internet (cite lost, unfortunately) recommended twice-daily renewal attempts, and since certbot will only trigger a renewal for certificates with <30 days to live, it seems pretty cheap.)

I deferred setting up HTTP-to-HTTPS 301 redirects until I'd had time to actually see the fully-automated renewal job work once. Regardless, I added a redirect for the Ghost admin portal specifically, so I could eat my own dogfood and be the first to encounter errors:

server {
    listen 80;
    server_name blog.rossry.net;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host      $http_host;
        proxy_pass       http://127.0.0.1:2368;
    }
    
    location /ghost/ {
        rewrite ^/(.*) https://blog.rossry.net/$1 permanent;
    }
}

(I also got started on an HTTPS-served Jupyter server, which was my real motivation for this whole thing, but that's a story for another day.)

(4)

Finally, I wanted to take a second look at my TLS settings. (Since I probably wasn't going to touch these for a few more years, I figured I might as well get up-to-date as of 1Q2019.)

My primary reference here was Aditya Gupta @ Medium | How to get an ‘A+’ in SSL Labs Server Test with NginX configuration and Qualys's SSL Labs server test; I cross-checked the blog post against a few top Google hits, but they didn't turn anything else up. These changes covered:

• Refuse the insecure protocols SSLv3 and TLSv1.0.
• Generate a 4096-bit Diffie-Hellman key ($ openssl dhparam -out /etc/nginx/dhparam.pem 4096) and use it for key exchange.
• Use better cipher suites (use the server's cipher preferences instead of the client's). Since I know ~nothing about cipher strength, I'm following Gupta's recommendation to disable {null ciphers, ciphers with low security, known-vulnerable ciphers (e. g. MD5, RD4), little-used ciphers (e. g. Camellia, Seed)}, preferring PFS, preferring AES128 to AES256, and including DES3 (for IE8).
• OCSP stapling, appending a signed, time-stamped, response from the CA to the initial TLS handshake to save the client a query.
• SSL session resumption, to reduce the number of handshakes re-visiting clients need to perform.

I added these to the http block of my nginx.conf, rather than the server blocks, since I wanted them to apply to every server using TLS and didn't care to duplicate them:

http {
    # additional security
    ssl_protocols TLSv1.3 TLSv1.2 TLSv1.1 TLSv1;
    ssl_dhparam /etc/nginx/dhparam.pem;
    ssl_ciphers 'kEECDH+ECDSA+AES128 kEECDH+ECDSA+AES256 kEECDH+AES128 kEECDH+AES256 kEDH+AES128 kEDH+AES256 DES-CBC3-SHA +SHA !aNULL !eNULL !LOW !kECDH !DSS !MD5 !RC4 !EXP !PSK !SRP !CAMELLIA !SEED';
    ssl_prefer_server_ciphers on;
    
    # additional ssl performance
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/letsencrypt/live/rossry.net/fullchain.pem;
    resolver 8.8.8.8 8.8.4.4 valid=300s; #Google DNS
    resolver_timeout 10s;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    ...

I declined to activate HSTS or a stricter CSP, partly because I found them confusing and no one seemed to unequivocally recommend them, and partly because some of the warnings about bricking a site with HSTS misconfigurations seemed pretty bad.

Finally, I set a Certificate Authority Authorization record with my DNS provider, to declare that only letsencrpyt.org is allowed to issue certificates for rossry.net. In Namecheap's UI, this meant adding three CAA records:

• CAA @ 0 issue "letsencrypt.org"
• CAA @ 0 issuewild "letsencrypt.org"
• CAA @ 0 iodef "mailto:ross@r-y.io" (to tell the world who to contact if they see bad certificates running around)

(5)

I made one-time $40 donations to each of the ISRG and EFF in gratitude for the public goods they've provided. Certbot's notes suggest donating with earmarks to support these organizations' LE work, but I felt better making unrestricted donations, as I've been a sideline fan of the EFF's work for a while and the ISRG is 1/1 for starting amazing projects -- if either of them want to spend my gratitude-dollars on something other than more LE for the world, I expect to be happy for it on average.

I don't, however, consider these donations part of my most-effectively-doing-good budget; rather, I consider them "only fair" compensation for providing a valuable service to the public below cost. I'm happy to pay more out of my 'personal' budget than I strictly have to (i. e. $0) in order to do my small part in aligning the world's incentives with my preferences.

(Bibliography)

• JeffTK | HTTPS Support
• JeffTK | Moving to HTTPS
• Certbot | Nginx on Ubuntu 14.04 (trusty)
• IETF/NWG | draft-ietf-acme-acme-03
• Github | joohoi/acme-dns
• Github | joohoi/acme-dns-certbot-joohoi
• Sal.Farrarello | Chrome Clear Redirect Cache -- a slick solution for when you've accidentally pointed a 301 at the wrong place and need to hit the original page to reset your browser's cache.
• Wikipedia | Transport Layer Security
• EFF | ETS Isn't TLS and You Shouldn't Use It
• Aditya Gupta @ Medium | How to get an ‘A+’ in SSL Labs Server Test with NginX configuration
• SSL Labs | SSL Server Test