Postfix mail relay

Since spying on citizens’ daily communication has shifted from being a rumour for paranoids to being a proven fact, some people think a lot harder about what they can do to preserve their privacy. With E-Mail still being the number one form of communication at the internet, this is one of the most obvious starting points. Due to the nature of the SMTP protocol, spying cannot be shut-down completely, even with content encryption. However, one aspect that can be worked on, is mail storage. Spying on each mail transferred via SMTP is quite expensive for an attacker while just fetching information from a multi-GB mailbox with years worth of social links and information is very cheap.

When relying on secure physical storage, most hosted mail services are not acceptable. They either make money from user data, closely work together with intelligence agencies, cannot provide an acceptable privacy policy - or simply are too expensive for personal usage. A possible solution is to use a local “at home” mail storage. Sadly, just putting some machine as a IMAP/SMTP host to a residential internet line means a couple of drawbacks. For one, most consumer internet connections will use dynamic IP addresses and force regular reconnections/assignments. Second, most telcos don’t offer proper reverse-resolution of the IP, many SMTP hosts use that to identify spammers as well as blocking whole IP ranges which are known for consumer internet connections.

Dynamic IP addressing could be worked around by using popular DynDNS services. However, a DNS MX entry must not be a CNAME but a “real” address. Some mail providers (e.g. GMX) check for this and deny taking mail from such hosts. The issue regarding reverse-resolution can be tackled by buying some mail re-direction services (e.g. dnsexit, dnsmadeeasy) that will relay your mail, but will set you back at least $25 a year with limited confidence to the “privacy” concern and limitations to mail throughput/features.

For me, the best solution has been using a small virtual server (VPS) as a SMTP relay. This gives full control on mail services, features and security. At home, i use a Synology DS214 for IMAP storage and SMTP submission. This box sends all mail to my VPS which then relays that mail to the destination recipient. The other way around it works the same way: incoming mail for my domain is relayed to the DS214 at home. The VPS costs me about €3 a month and of course allows more than just relaying mail. I’d like to share the configuration details with you.

Network hosts

client
The clients mail application uses “ds214.example.com” as IMAP and SMTP server.

ds214.example.com
This host is located at home and uses a residential ADSL line. It uses a dynamic DNS service to update its dynamic IP for “ds214.example.com”. Ports 25 and 993 are forwarded by the ADSL router to allow access by the client and the mail relay (vps.example.com).

vps.example.com
This host does the mail relaying. It is added as DNS MX entry for “example.com”, it’s static IP address reversely resolves to example.com

mx.otherhost.com
Other mail providers hosts just communicate with “vps.example.com”, not knowing anything about it’s relay functionality.

DS214 configuration

This is Synology specific, but may easily be achieved with other MTAs as well.

Synology SMTP config

Postfix configuration

The MTA is running Postfix and Postgrey at vps.example.com is configured like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no
append_dot_mydomain = no
readme_directory = no
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
myhostname = vps.example.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = vps.example.com, localhost.example.com, localhost
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = 123.123.123.123 # external IP address
transport_maps = hash:/etc/postfix/transport
relay_domains = hash:/etc/postfix/transport
relay_recipient_maps = hash:/etc/postfix/recipients
unknown_relay_recipient_reject_code = 550
smtpd_client_restrictions = permit_sasl_authenticated, reject_unauth_destination, reject_unlisted_recipient, reject_rbl_client zen.spamhaus.org
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, check_policy_service inet:127.0.0.1:10023
smtpd_sasl_local_domain = example.com
smtpd_sasl_auth_enable = yes
broken_sasl_auth_clients = yes

Relay domains and target hosts are configured at /etc/postfix/transport. In this case, all mail to “example.com” is relayed to ds214.example.com.

1
example.com smtp:[ds214.example.com]

Valid recipients for this domain are configured at /etc/postfix/recipients. In this case, only mail to any “example.com” address is allowed to get relayed.

1
@example.com OK

Make sure these files are converted to a binary format.
relayed.

1
2
postmap /etc/postfix/recipients
postmap /etc/postfix/transport

When relaying outgoing mail (from ds214.example.com), SMTP authentication is used to make sure no unprivileged users use this server as mail relay. In this case, i’ve used saslauthd for authentication. Postfix is advised to use saslauthd and accept “PLAIN” password authentication at /etc/postfix/sasl/smtpd.conf.

1
2
pwcheck_method: saslauthd
mech_list: PLAIN LOGIN

Restart Postfix after applying this configuration

1
service postfix restart

saslauthd also needs to be configured to accept PLAIN authentication. As the user/pass database, sasldb should be used. This can be done at /etc/default/saslauthd:

1
2
3
START=yes
MECHANISMS="sasldb"
OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"

Restart saslauthd and make sure the path specified with the -m option can be accessed. This is required since Postfix is supposed to run in a “chroot” environment.

1
2
service saslauthd restart
dpkg-statoverride --add root sasl 710 /var/spool/postfix/var/run/saslauthd

Now we create a user within sasldb which is used to authenticate a client which is relaying mail through this MTA.

1
saslpasswd2 -u example.com -a smtpauth mailrelay

Make sure the user “mailrelay” and the given password is used at the SMTP client, which is ds214.example.com in this case.

Web application load testing - and why you should consider Gatling

When it comes to load testing a web application, usually JMeter comes up as the go-to solution. It has a huge amount of features, protocol support, is opensource, establishment - and boy it has a GUI! There are some other alternatives like pyLot, locust, Grinder, radview, wapt or LoadUI which are either commercial or not as feature rich and less established.

Lately, some hosted solutions like loader.io, neotys, blitz.io, blazemeter, loadstorm etc. joined the party. These offerings are nice for testing the whole stack from an end-user perspective or running limited tests. The downside is, that these are typically sold by bandwidth or threads (users) which is a good business model but comes unhandy when really stress testing an application over a long period of time or testing within an internal network.

Usability

One of the biggest benefits of JMeter, in many peoples minds, has been its graphical user interface. Well, it’s kinda powerful and it kinda allows easy test creation. People tend to think that a GUI makes everything easier since many concepts are well known.

A GUI makes everything easier, right?

In my opinion the opposite is true, at least for JMeter. Its GUI is quite bloated, which is natural given its complexity and amount of features, but in the end is some button-input-and-knob representation of an XML structure. Under the hood, JMeter generates massive amounts of XML, based on what tests the user defined at the GUI. By itself, this is not bad. However, it shuts down any serious attempt on manual interaction on an XML level. This becomes very visible when using JMeter tests while working within a team. We use git for source control and due to a lack of good and powerful GUI based tools, primarily work with the command line implementation. Reviewing commits or merging is really a pain when you have to diff and compare on a XML level.

Even with great syntax highlighting, this is where you probably just want to go home.

Much better

Readability of code is usually determined by the skills of the author, in this case the author really doesn’t do anything wrong - but by focusing on a GUI, readability for all other forms of representation suffers. Even if you manually tame the XML output, JMeter will just overwrite it using machine-readable-only code. XML is very well structured but apart from being structured, there are other requirements to good code. Even easy tasks like replacing some parameter or defining properties becomes a pain using the GUI since you have to browse it and manually spot elements. Either you got almost everything within variables, or you’re doomed.

In my opinion, JMeter is nice to rapidly create one-shot tests that can be shown around for education, validation or audit. For any kind of sustainable agile development or testing, it’s next to useless.

Gatling on the other hand uses Scala to define load tests. Scala is probably not as established as XML but (as programming languages in general) it allows to code much cleaner and use the power of a functional programming language. Being object oriented and relatively up to date, it allows re-using code to avoid redundancy and pretty much everything one would expect from a modern programming language. While coding Scala requires some specific toolchain, the compiled code runs within a standard JVM. This makes it very easy to deploy and leverage existing workflows and tools. If you’re a programmer you probably can start with Scala right away. If you’re not familiar with programming, some hurdle may exist but learning a programming language while creating load tests sounds a good deal, right? There is no need to learn each and every bit of Scala right away, tests usually consist of the same fragments that just need to get put together.

Since Gatling is under very active development and currently transitioning from version 1 to version 2, some API vs. documentation hickups and bugs may happen from time to time. The core developers and community is very responsive and most issues get covered within hours.

Performance

The funny thing about load tests is, that many environments actually test the efficiency and performance of the test runner, rather than the application thats supposed to be tested. In projects i’ve seen people building monstrous phalanxes of test clients to stress-test a single server. Distributed test clients and down-sizing both the servers spec and the test coverage seem to be appropriate measures to get the server under load. Lets check some real-world example.

At work we got a nice Intel Modular Server box dedicated for load testing. This baby sports a chassis with redundant PSU, networking, storage and six identical blades loaded with dual Xeons and lots of memory. In a nutshell, this is a datacenter within 6HE where testing can happen without external influences. Earlier, we used one virtual machine as test director and result-parser, 4 machines running JMeter and one machine running the application (including all infrastructure) we wanted to test. JMeter has been configured to spread the test scenario to all 4 machines, effectively cutting the number of threads by 4 and feed them to the test clients to execute them simultaneously. While this worked well, it really felt odd to have 4 specialised machines hammering on one specialised machine to push it to its limits.

JMeter system load

What we see at this screenshot is one machine running about 100 concurrent threads (“virtual users”) occupying 4 CPUs. The server on the other hand is also quite stressed but keep in mind there are 4 test machines with 4 CPUs each running 400 concurrent threads in total. Even these 4 test machines did not manage to create significant load at the server, to find out its diminishing or even tensile point. One major drawback with JMeter is that it uses one system thread per user, handling large numbers of active threads is very expensive on the CPU.

When using Gatling, we easily manage to get 1200 concurrent threads running at just 25% CPU load of one CPU of a single virtual machine. This is about 200x more efficient than JMeter (1/64th of the CPU load while creating 3x the load). The server is also stressed quite well and we’re able to push load testing far beyond its tensile point.

Gatling system load

Reporting

When running JMeter using its GUI, reports are fairly nice, real time and the tool offers some help to dig through the results. However, when not being able to use the GUI (e.g. for unattended testing, continuous integration), you get a bunch of .jtl files, either XML or CSV. These can then be put into JMeter again for analysis or get processed using XSLT or tools that understand CSV (yeah… Excel). Thats all good and at this point highly structured data makes a lot of sense. There is a variety of tools that help with graphing, charting and analysing of its data. The downside is that you almost always need some kind of extra tool to make JMeter reports understandable - and you always have to wait until the test run finished. At least to my knowledge, there is no realtime graphing apart of the JMeter GUI.

Gatling also creates machine readable data files, but already provides a really nice report module that generates HTML pages with charts and graphs. In addition, the integrated Graphite feeder allows real-time charting of test runs to tools like Graphite or Grafana. This becomes really powerful for showcases or unattended test runs. Overall i think the built-in reporting of Gatling outperforms JMeter by large, even if JMeter reporting may be more accurate and comprehensive on a scientific level.

Default Gatling graphs

Graphite realtime graphs

As with every discussion, there is no “using the wrong tool” - it just depends on the job that needs to be done and some thinking outside the box. There are a lot of great tools that are not (yet) mainstream but help with every days work and contribute to getting better tests, results and software.

Spy my Mac

Apple Inc. builds great computers. Well, at least the hordes of slave workers over at Asia do. Kudos to them for building astonishing precision and the design team to create beautiful machines that feel great when working with them. Shame on this $500bn+ company for taking advantage on such cheap labour, spending .5% of their margin on working conditions would provide hundreds of thousands of families a much better perspective and fair work. But lets put that aside for a moment.

A Mac can easily be sold for over 60% if its initial price two years from now. Try this with an Asus, Sony, Acer, HP, what have you. Almost all other Laptops i’ve seen so far - since IBM dumped the Thinkpad - look like a piece of junk after two years of transportation and serious usage (like 8-12 hours a day)

Now this is where the issue starts: Macs are re-sold regularly and people are trying to get their hands on a pre-owned device to cut some of the initial cost. On the other hand, Apple has built the perfect golden cage for it’s users. Services, hardware, software, complementary products - all very well integrated and made to work with each other quite smoothly (well, most times at least…). When buying a Mac, one does not just buy a computer but enters the realm of multiple services spun around the users “digital lifestyle” that tries to keep the user at the chosen platform. While hardware may and is meant to become obsolete, these services stay and are assigned to a user (via it’s Apple-ID) rather than to a specific piece of hardware. Obviously Apple is very successful in retaining users to proprietary services so they won’t flee back to good ol’ Windows or Android world.

One service that is tightly integrated with many Apple devices, is “Find My iPhone/Mac” (FMI from now on). In a nutshell, it geo-locates devices that are registered with the users Apple-ID in case a device got lost. This works for both phones/tablets by using GPS tracking as well as Laptops using wireless network information for approximate location. The service becomes very valuable over time and more than once it helped to give a good idea on where i’ve left my phone. Other than locating a device on a map it allows basic remote management features like making the device audible, lock it to avoid misuse and wipe it’s data in case it got potentially compromised.

Now, what’s the catch? Well, back in 2011 i bought a MacBook Air and used it for almost two years before selling it online. While using it, i had FMI activated and the device has been assigned to my Apple-ID. Also, i enabled FileVault which is an encryption feature of OSX that encrypts the whole computer storage, which by the way everybody should use. Additionally i’ve wiped all of that storage data to hand over a plain computer to the new owner. There is close to no chance that anybody outside of an intelligence agency or Apple could access my old (account) data. I sold it to one of these fixed-price dealers where you don’t get to know the buyer nor care about what happens with the machine.

Roughly half a year after i sold that machine, i occasionally checked by FMI to see if there have been updates to the service. To my astonishment, a machine called “Luisa’s Mac Book Air” showed up at my list of machines.

Hello there, Luisa

It was not online at the time i checked so i opted for “Notify me when found”. Shortly after, i received an E-Mail. My old Mac Book Air has been found. Yay! Interestingly enough that it has still been linked to my Apple-ID, all the location and remote management feature were still in place. Based on the location information and her name used to generate the computer name, i was able to look up Luisa on various online services. She’s a student and based on the transportation habit of her (my old) Laptop it should be easy to find out where she’s living, whom she’s meeting and what courses she’s taking… Well i’m not interested in stalking people so i took this much information to verify it’s real. Given a bit of criminal energy, i could lock her machine and blackmail her to re-gain access to her data. Not interested in this as well, just sayin’.

Lucky me.

There are stories of guys that recovered their stolen machines by hacking into it remotely, track it, identify the thief and all that stuff. Apple, pursuing the idea of “everything simple”, has just taken it a step further. In a nutshell, i got privileged remote access to a machine that has been wiped-clean, re-installed and i had no user account nor information about. Well, that’s one hell of a privilege escalation. Luisa for sure is not using my Apple-ID and has no clue that she can still be tracked.

Knock, knock...

There are various scenarios on what has happened. She uses FMI and all that iCloud stuff, that would point out a permission issue within iCloud when linking machines to Accounts. Or, she does not use FMI, which is worse since it implies this service is available regardless of the users choice made when setting up the machine. In both cases the new owner of the machine does not seem to have any possibility to opt-out from being tracked by the previous owner.

Digging deeper, i started to suspect that Apple is not really using the owners user or account credentials to get access to the machine. While this may be a logical implementation to track/manage machines where the user is not logged in, it’s quite critical in terms of privacy. A user may enter a strong password and connect the machine to it’s Apple-ID - but in the end it’s just identified and accessed by some kind of unique hardware identifier. At least the FMI web app uses such (hashed) identifiers. It’s a good guess that these IDs are used to bypass authentication at her Laptop. In this case, it is plausible that Apple has remote access to location data of all devices as well as powerful remote management capabilities (wipe, lock…). At the very least, remote access to the machine is not secured using credentials that the user has chosen at it’s Apple-ID but rather a static ID and vague trust-relationship with Apple. Authentication by shared secrets is an issue for itself, however it’s still better than obviously having no secret within the authentication process.

Now, i’ve contacted the Apple product-security contact and described the issue. The contact is very responsive, which definitely is a good sign for taking such issues seriously. However, they came back with the statement, that there is no actual security issue. I (as the previous owner) could just remove my old machine from my FMI account to stop tracking it. Well yeah, that may “help” me not tracking somebody - but it’s definitely not helping the person that gets tracked. Furthermore they pointed to a “how-to” document that describes what actions should be done before selling a Mac (e.g remove it from all iCloud services). This document of course is optional and the new owner has no way of verifying that the previous owner has removed the machine from these kind of services.

So what is this? Bad luck? Poor design? Wrong expectations? One could argue that if a machine gets stolen and logged out, the original user still needs to be able to wipe it. That’s true and valid, but does not require the machine to be bound to the original users Apple-ID for it’s whole lifetime. As soon as somebody wipes the storage or starts over using a different Apple-ID, the connection with FMI should be save to be reset. After all, FMI is not a thievery protection system, it’s a service to find a temporarily lost device. Is Apple doing this kind of hardware-service-lock-in to prevent users from reselling their hardware? Well that could be, but i don’t think they do it on purpose or to push their own “used hardware” service.

IMHO it’s a follow-up issue of the attitude to retain customers and all their credentials, data and devices to a closed ecosystem. This is just one more example where “simplicity over security” has a significant backlash. Also, it illustrates that open source software for infrastructure is not the only key to a more trustworthy and secure environment. Web services are already much more relevant to end-users than operating systems or infrastructure services are. Many of these underlying services have already become invisible. Trusting both web and infrastructure services with sensitive information like unrestricted access to physical location data is a huge problem if the service is running as a complete black box without the user in control of his or her own data.

There is a pressing need to discuss, spread and implement the definition of “who owns what” within the web services world. Right now, most people are extremely naive when blindly accepting these 70 page’ish TOS, since there is no real socially acceptable alternative to many of these services. Users must become sensitive to this an be in a position to claim their ownership on their data. Giving it away to industry giants for free will manifest the current situation and ultimately lead the way to a world with few players dominating not just the market but also their users habits and (digital) life.