Many new cars infotainment systems come with a WLAN hotspot by default. Owners can use it for media consumption and internet access on the road and of course never change the SSID. Such access-points are quite noisy and constantly broadcast their SSID, MAC addresses, a fact that might be interesting for multiple reasons:
Polluting frequencies within areas that are already short on channels
Tracking individual vehicles using a set of APs or large networks like Freifunk or even a TelCo
Checking if a certain vehicle gets close to certain “etablissements” (hello cheaters)
Estimating the worth of equipment when selecting a car to steal
Checking the last 30 days of my access-points “neighbouring access points” log did reveal quite interesting data about drive-by cars:
77 Mercedes Benz (SSID: “MB WLAN XXXXX” or “MB Hotspot XXXXXX”, MAC vendor “Harman/B”)
73 Opel (SSID “WiFi Hotspot XXXX”, MAC vendor “MitsumiE”)
14 Skoda (SSID “SmartGate_XXXXXX”, MAC vendor “Universa”)
7 Audi (SSID “Audi_MMI_XXXX”, MAC vendor “WistronN”)
6 Volvo (SSID “MyVolvoXXXX”, MAC vendor “Actia”)
When using a WLAN enabled car, the first thing to do would be changing the SSID, disable broadcasting or change the MAC but since cars are not quite hacker-friendly such options are most certainly disabled.
Securing HTTP has luckily become a commodity over the last couple of years, even for mainstream sites which only did enable secure connections for authentication processes before, if at all. Browsers start to implement more features (HPKP, CSP, HSTS…) with even better usability and start to get rid of legacy protocols and ciphers. At this corner of the internet, the state of TLS looks really good and everyone feels great about green lock-signs. But whats about less obvious areas of secure communication?
E-Mail for example has seen lots of good initiatives to enhance security primarily with the goal of reducing Spam (DKIM, Greylisting, Domain validation…). Apart from that the situation still looks quite ugly. There are indeed options for skilled end-users to do end-to-end encryption using SMIME, PGP for mail clients and to use browser-based crypto or server-side solutions for webmail applications. However, these solutions are either too complicated or only cover niches. Ultimately users could do something to improve their security when sending mail, but there is no solution to demand receiving mail securely - while incoming mail represents the vast majority of John Doe’s mail traffic.
By design E-Mail - more specifically SMTP - depends on servers talking to each other. Clients (like Outlook, GMail or Apple Mail) submit their messages to the configured server and let SMTP figure out a suitable route to deliver the mail across the internet and local networks. This leads to a quite robust infrastructure but takes away influence on security if the user did not encrypt the message. There is no way to tell or define what route a mail will take and if those servers communicate in a secure way with each other. Even when running a well configured mail server yourselves there is always a chance that a incoming mail was transferred in plain-text between two other servers. The recipient can only figure this out after receiving the mail, which might have been compromised by then. Keeping in mind what kind of sensitive data gets sent around multiple billion times per day, this is really troubling.
Security vs. cost
Operators of larger mail services usually enable strong security features but the majority of smaller operators does not care as much. With an almost bullet-proof configuration, there is a huge chance of not being able to communicate to many services. For example, while GMail or Outlook.com may use top notch security, many mails are sent by non-public mail operators such like your bank, random businesses or small online-shops. These guys often have no clue about proper SMTP configuration or have to depend on ancient mail system configurations for the sake of compatibility. Having a substantial error rate, say 5%, when sending or receiving invoices, general purpose or business communication leads to high cost.
With this in mind, most operators chose to stick with a less secure but more compatible configuration. For the more security focused operators, this means they can’t remove less secure configuration, a classical chicken-and-egg problem. There have been some regional projects to raise security (e.g. “E-Mail made in Germany”) which however appear to exist for marketing purposes instead of establishing real security. The ridiculous name aside, this initiative just makes sure that the biggest consumer mail operators in a given country talk TLS to each other. Those popular operators had TLS for incoming and outgoing SMTP enabled anyway and such campaigns avoid to mention that the other 99% of operators don’t know or care.
Mails sent by humans via huge mail services are actually a minority, most are generated by automated systems. If your local lawyers office or hotel booking system sends data to GMail without encryption it does not matter that GMail and Yahoo talk securely with each other. Those major services simply don’t force encryption on incoming mails to become a commodity, instead they implement a good-enough configuration to avoid being blamed and seem to hope the situation will enhance by itself. This led to the obscene situation that nowadays cat pictures on the web are better secured than critical information transferred via mail.
Doing the test
Lets get into some practical examples. I chose to enable a secure incoming SMTP configuration only allowing commonly used ECDHE ciphers (which are expected to be “unbreakable” in the forseeable future) for my own mail server for a month and checked what kind of issues would happen. “Funny” enough, most practical issues affected financial institutions and large corporations. My assumption is that those guys have a tight corset of policies in place that suppress innovation or even keeping up with reality when it comes to secure configuration. Some examples of failed incoming delivery:
In most cases the worst thing that happens is that mail does not get delivered. American Express (“AMEX”) is a little different though. The topic has been discussed with one of their “Senior Security Analyst Cyber Security Investigations” (so much for title porn) but did not lead to any real change. I guess two months after disclosing the issue to American Express and not getting any more response it’s fair to get the word out.
The AMEX situation
What AMEX like most financial institutions does is checking for geo-location, IP and other fraud relevant metadata when accessing a account. If someone successfully logs in from France and just 10 minutes later from Japan thats reasonable suspicious to block access. Good idea. They then send a some-digit “authentication code” to the account holders mail address as additional factor to authorize the login. The same happens quite often when accessing the same account simultaniously from mobile and stationary internet connections since the IP and geo-location information changes, which gave me the opportunity to do research without traveling.
When the authentication code is delivered, AMEX does not handle ECDHE ciphers for SMTP and delivery fails.
Dec 23 16:01:05 box postfix/smtpd: SSL_accept error from extmta1-new.aexp.com[18.104.22.168]: -1
Dec 23 16:01:05 box postfix/smtpd: warning: TLS library problem: error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher:s3_srvr.c:1440:
OK, so no authentication code for me, sad. But wait!
Dec 23 16:29:16 box postfix/smtpd: connect from extmta1-new.aexp.com[22.214.171.124]
Dec 23 16:29:17 box postfix/smtpd: 1FBCDAE88B: client=extmta1-new.aexp.com[126.96.36.199]
Dec 23 16:29:17 box postfix/cleanup: 1FBCDAE88B: message-id=<UMS255338020161223042913.DEUDEUMSPWR0001.ENG-ALERTS@welcome.aexp.com>
Some minutes later the same MTA connects again and succeeds. So what happened? Did they fixed their MTA in the meantime and trigger a retry? Looking at the received mails headers suggests otherwise…
# Received: from welcome.aexp.com (extmta1-new.aexp.com [188.8.131.52])
# by mx.heiland.io (Postfix) with ESMTP id 1FBCDAE88B
# for <email@example.com>; Fri, 23 Dec 2016 16:29:16 +0100 (CET)
The absence of TLS information at the header indicates that AMEX indeed has a fall-back for failed mail delivery - but one which drops any kind of encryption. This means my second factor to validate legitimate access to my credit-card account just got sent through the internet as plain-text since outgoing AMEX MTAs are incapable of handling 2015-ish security configurations. Ugh.
So what could be done with that? At the very least it makes passive attacks and sniffing shared networks a viable option to acquire this secure token. Some attacker could look out for reasonably secure mail servers to which AMEX is incompatible to and look out for re-tried unencrypted mail without launching an active man-in-the-middle attack. This might be a hard thing to pull off between AMEX and GMail, but certainly when thinking about smaller corporate mail services. Legitimate recipients may not notice the issue since they still get their authentication code, just a bit later. AMEX is quite huge in the corporate credit-card business so there is a good chance to hit some high-profile accounts with interesting data. I did not get into more detail with this but there certainly are some scenarios where this issue degrades passive security for the authentication code to near-zero. AMEX is just the example that came to my attention, there is no reason to believe this would be a unique case.
So what are my results from this one-month experiment?
Using current generation encryption for SMTP leads to failed deliveries.
It does not only affect small but also large companies. Public mail providers were unaffected.
Forcing high security standards may even lead to drastically reduced security.
HTTP Public Key Pinning (HPKP) is a mechanism to make sure a HTTP client (e.g. web browser) only trusts a pre-defined set of certificates when establishing a TLS (“https”) connection. If the certificate presented by the server does not (or no longer) match the previously defined hashes, the connection does not get established. This is useful to counter man-in-the-middle attacks where someone intercepts the connection and in cases where DNS records or the certificate gets compromised. HPKP information is provided as a HTTP header with a list of hash values of certificate fingerprints as content. The client then checks the certificate used for the TLS connection against this list and will only trust the specified list of certificates for future connections to the domain. This highlights a important detail: Since HPKP operates on HTTP level and certificates are exchange on the (lower) TLS protocol level, the first connection to the host will not check for certificate fingerprints since the client does not yet know about them or the fact that the server offers HPKP. This also means there is no way to tell the client that a HPKP setting has changed prior to establishing a TLS connection and talking HTTP.
While HPKP is useful to enhance security, it’s a two-sided sword as well. Incorrect configuration may render your domain useless if clients look for the wrong certificate fingerprint. HPKP information is usually preserved by the client for two months or longer before getting refreshed. While clearing a clients history or settings will force a reset, there is no way to communicate this to users since the client will not even try to establish a HTTP connection. Worst case, the domain won’t be valid to serve content for the defined refresh interval. That being said, careful selection about what gets “pinned” at the header information helps to avoid most of the trouble.
The header is designed to contain a number of base64 encoded hash values which are valid for the specific domain and includes an obligation to pin “backup” hashes in case the “primary” hashes do not match anymore. A client will just iterate through the full trust-chain for provided certificate and look if any certificates fingerprint matches any of the hashes provided by HPKP headers. If any certificate matches, the check passes and the TLS connection is established. Now, what should be pinned at HPKP headers? The obvious answer would be “the certificate!” but wait - certificates may change quite often, especially when using short-living validity like Let’s Encrypt does. In this case the new certificates hash is not yet part of HPKP information stored with the client but the old one is expired, as a result the client would not be able to connect until the client re-requests HPKP information. More information can be checked at RFC 7469.
Lets start with the “primary” hashes. Adding the hash of the domains certificate will certainly not harm, however that means the header information needs to be updated as soon as a new certificate gets issued for the domain. Next should be the Intermediate certificate(s) of the CA in case the CA uses them, this means any other certificate for the domain issued by the CA and signed with the same intermediate certificate will be valid for the host. Third is the Root certificate of the CA itself, which means that any certificate issued directly or via any Intermediate certificate by the CA would be trusted as well. Taking Let’s Encrypt for example, the following certificate hashes would get pinned:
DST Root CA X3
Let’s Encrypt Authority X3
Great, now what happens if your domain certificate expires and “Let’s Encrypt Authority X3” or the “DST Root CA X3” go out of business or get banned from the clients trust stores (hello WoSign, hello DigiNotar…)? The client would not accept any of the certificates because they are either removed from its trust store (Root, Intermediate) or not covered by HPKP information (new domain certificate). This is where pinned “backup” hashes come into play.
Since there is no limitation in hashes, except the 8192 bytes limit for HTTP headers, it’s possible to pin hashes of Intermediate and Root certificates of other CAs which may be an option in case your primary CA gets into trouble. The next best thing to Let’s Encrypt might be Comodo, which means pinning Comodos Intermediate certificates or their Root CA in addition to Let’s Encrypt would flag their certificates to be valid as well.
There is another way though. Since what we’re pinning is not the hash of the actual certificate but the “SPKI Fingerprint”, we can also pin fingerprints of one or more Certificate Signing Request (CSR) which are not yet issued as a certificate. With this, a future certificate issued for this CSR is already pinned, regardless which CA signs the certificate. So in case of a problem with the current CA, that CSR is used to create a certificate at an arbitrary other CA. The certificates fingerprint would then already be part of the HPKP information since it matches the CSR fingerprint.
HPKP transfers the SHA256 hash of the SPKI fingerprint of a certificate or CSR, generated with OpenSSL and no other hash algorithms are supported.
When pinning a CAs root or intermediate certificate, the first step is to acquire the correct public key. Usually CAs offer support pages where those can be obtained. The information about which root or intermediate certificates are used, can easily be looked up from the domain certificate, for example by inspecting it at a browser.
In this case the “DST Root CA X3” is the CAs root certificate and “Let’s Encrypt Authority X3” is a intermediate certificate. Searching for those names leads to the download page https://letsencrypt.org/certificates/ where the public keys can be downloaded as PEM format.
Creating the SHA256 hashes for the CAs root or intermediate certificates SPKI fingerprint is quite straight forward:
To create a CSR and a hash of its SPKI fingerprint, a private key is required to start with. Both the private key and the created CSR must be stored at a safe place for future use. The hash of the CSRs fingerprint can be used immediately for HPKP though.
Create private key
$ openssl genrsa -out backup1.key2048
Create CSR for single or wildcard domain name
Note that CAs may require or check the provided data of a CSR, for example the legitimization of the specified organisation or address. At the very least however, the “Common Name” is critical since it must contain the domain name for which the certificate will be created for.
Common Name (e.g. server FQDN or YOUR name) : sub.domain.tld OR *.domain.tld (for wildcard certificates)
Create CSR for SNI
In case our certificate should contain multiple domains at its Server Name Indication (“SNI”) information, the default OpenSSL configuration needs to be tweaked a bit. The following additional parameters are required for the OpenSSL configuration file which comes with Debian GNU/Linux.
$ cp /etc/ssl/openssl.cnf backup.cnf
$ vim backup.cnf
[ req ]
req_extensions = v3_req
[ v3_req ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = www.domain.tld
DNS.2 = domain.tld
DNS.3 = something.domain.tld
The the configuration file gets included when creating the CSR.
HQZ03DioNrXVV7/zEuQONyO8cwUo3ncA71fzLO+o/d8= is the base64 encoded SPKI fingerprint of the CSR.
Creating the HPKP header
The header is a quite straight forward list of hashes and adds max-age information to define the maximum time a client (e.g. browser) will cache HPKP information once it got provided. This sample pins the CAs root certificate, two intermediate certificates and several CSRs as backup. The client will validate the presented certificate against all entries at this list, regardless of its order. Maximum age should be set to two months (expressed in seconds) or longer.
HTTP Servers like nginx allow to simply add those headers at the respective sites configuration:
$ vim /etc/nginx/sites-enabled/martin.heiland.io.conf
General availability of the header can be checked by using a browsers development console and look for response headers. There are some more sophisticated sites that check syntax, content and validity of that headers information
Before deploying to production, HPKP settings need to be double-checked since they may lead to unavailability of the site in case their content is incorrect. If the syntax is wrong, a client may just ignore the header and not add any security.