Home server upgrades, ZFS, Optane and more...

I recently started to replace the HDD storage of my home server since my three WD RED 4TB drives got rather old and i required more space. After lots of experimenting i ended up with ZFS, three new HGST 10TB drives and a shiny Optane 900p. Here is my story so far.

What is ZFS?

There are many videos, articles and other documentation out there, describing in detail what ZFS is. Lets make this brief. ZFS is a copy-on-write file system created by Sun Microsystems for Solaris and available under a Open-Source (-ish) license for Linux and other operating systems. It combines the abilities of a volume manager (like LVM) with a file system (like ext4). Compared to most other file systems, it natively handles multi-device situations by creating all kinds of stripes, mirrors and parity based constructs for data redundancy. Unlike any other file system (yes i know BTRFS…) it sets its priority on data consistency, self-healing capabilities, error prevention and has a proven track-record in the enterprise storage industry.

ZFS works best when exposing disks directly to it, favouring a “JBOD” configuration over RAID controllers. It strictly is NOT “Software RAID / Ghetto RAID”, in fact it offers feature no other file system or hardware RAID controller can offer. Lets face it, RAID controllers are just expensive, optimized computers with crappy, often incompatible firmware and a bunch of SATA/SAS connectors. Since i evaluated multiple solutions (Linux MD, a LSI 9260-8i hardware controller, BTRFS and ZFS) i dare to have an opinion on that topic. The only thing ZFS does not have is a battery-backup unit (“BBU”), however the risk of losing any data during a power outage is extremely low and data corruption can not happen with ZFS. A external UPS is a lot cheaper than a entry level RAID controller with BBU. This only leaves PSU failures, cable errors and software bugs as risk.

As usual there are concessions to make - for ZFS that was higher resource usage (and subsequently potentially lower performance), compared to file systems that care less about data integrity. It has to go many extra miles to make sure data is not just received from disks but the data is actually the correct one, intact, unmodified and gets repaired in case its corrupted. This by the way means using ECC RAM is a very good idea, as faulty data in RAM would lead to “incorrectly repaired” (aka. corrupted) data. Optional features like compression, de-duplication and encryption take an extra toll. ZFS has intelligent caches which are quite memory hungry and can easily use 16GB of available RAM even on small systems. That being said, unused RAM is wasted RAM and its important to understand what ZFS is using it for. To offload some of this resource usage, ZFS allows a second level of caching being written to non-volatile memory called the L2ARC (“Level 2 adaptive replacement cache”) which acts similar to a “read cache”. Next there is a mechanism called ZIL (“ZFS intent log”) which is similar to a “write cache” that collects and streamlines write operations and ZFS then flushes them to disk every couple of seconds.

Performance of ZFS can be greatly enhanced by using a SLOG (“separate log device”) for ZIL and also offload L2ARC to high-speed, low-latency storage. Since DRAM is volatile it’s not a consideration, except some super expensive battery/capacitor buffered DRAM devices. SSDs are a lot more affordable, non-volatile by nature and really fast compared to hard drives. However, compared to DRAM, SSDs are several multitudes slower. Just recently a new technology has been released, claiming to fit between DRAM and traditional SSDs and therefor be an obvious choice for ZIL and L2ARC: Intel Optane.

What is Optane?

  • It’s a product range based on 3D-XPoint memory
  • It’s built for very specific use-cases like caching
  • It’s cheaper than DRAM but more expensive as typical SSDs
  • It uses proprietary memory tech from Intel and Micron
  • It’s NOT a typical SSD, since it’s not based on NAND flash
  • It’s NOT DRAM, since it’s non-volatile

3D-XPoint “3D cross-point” memory technology was announced years ago and first products, called “Optane”, hit the market in early 2017. The first release was a datacenter-grade memory product called “Optane SSD DC P4800X”, available as 375GB and 750GB capacities and as U.2 drive and PCIe card formats. Roughly at the same time some much more consumer oriented “Optane Memory” M.2 cards became available as 16GB and 32GB configuration. In late 2017 Intel released the “Optane SSD 900p” product with capacities of 280GB and 480GB as PCIe card and U.2 drive.

While all Optane products are based on 3D-XPoint memory, their scope and performance varies a lot. Those small “Optane Memory” M.2 cards are meant to serve as system cache/accelerator for HDD-based desktop and mobile computers, while the P4800X and 900P are targeting server and enthusiast desktop computing. The latter two use much more power but also deliver significantly better performance as they pack more 3D-XPoint modules and speedier controllers. The P4800X is Intels top-of-the-line offering and comes with more integrity checks, capacitor based buffer to avoid data loss and better durability. Performance-wise it’s rather close to the 900p, and both share stunning specs.

  • 2500MB/s read, 2000MB/s write
  • 500.000 IOPS read and write
  • 10usec latency for read and write
  • 5PBW, 1.6M hours MTBF
  • 1 sector per 10^17 bits read uncorrectable

Intel claims that those cards require a 7th generation Intel Core CPU, which is just half of the truth. In fact those drives use the NVMe protocol and can be used as regular block device with any current CPU and platform. To run Intels software for automated caching indeed a 7th generation Intel Core CPU is enforced, which appears to be a sales oriented decision. Anyway, for my use-case the 900p meets a 5th generation Xeon E3 CPU on a C232 chipset - and it just works fine.

Now, whats the fuzz about? Why is Optane spectacular? When looking at the typical benchmarks, Optane based products deliver okay-ish performance compared to NAND-based NVMe SSDs like a Samsung 960 Pro - but come as a steep price premium. SSD Benchmarks usually assume large block sizes (>=1M) and high queue-depth (>=16). These values do not represent typical server workloads, in fact i dare to claim they represent almost no relevant workload and are made up by vendors to present large numbers. NAND based SSDs are great in producing high throughput when reading large quantities off many NAND chips in parallel (sequential access), and this is a good thing. However, the fun starts at small block sizes (e.g. 4K) and low queue depths (e.g. 2 or 4) often seen at server workloads like databases. Consumer grade NAND SSDs are usually also terrible at random write performance. Intel claims Optane can fix that.

Benchmarking the beast

Disclaimer: I’ve not received any freebies or been in contact with any of the brands mentioned here. All stuff has been bought from my own money. I understand benchmarks can be in-comprehensive and i admit that the SM951 was in use for some years so it might not produce perfect results anymore. Also the system was running some load during the benchmark and potentially lacking optimization. While my results might not be scientifically perfect, they represent a real-world configuration.

Lets have a look at a Samsung SM951 running at the same system as a Intel Optane SSD 900p, both connected via PCIe x4:

1M blocksize, QD16, random read
$ fio --name=test1M --filename=test1M --size=10000M --direct=1 --bs=1M --ioengine=libaio --iodepth=16 --rw=randread --numjobs=2 --group_reporting --runtime=5

* 900p: 2563 IOPS, 2536 MB/s, 1247 usec avg. latency
* SM951: 2005 IOPS, 2005 MB/s, 1594 usec avg. latency

So far so good, both products are almost toe to toe while the 900p delivers a bit more performance justifying its higher price point. Note that both products appear to maxed out regarding bandwidth. Now, lets write some data.

1M blocksize, QD16, random write
$ fio --name=test1M --filename=test1M --size=10000M --direct=1 --bs=1M --ioengine=libaio --iodepth=16 --rw=randwrite --numjobs=2 --group_reporting --runtime=5

* 900p: 2152 IOPS, 2152 MB/s, 1485 usec avg. latency
* SM951: 399 IOPS, 409 MB/s, 7981 usec avg. latency

Things start to become interesting as the 900p suddenly pulls away with 5x higher IOPS while still being maxed out and bandwidth. Write intense workloads are obviously an issue for consumer NAND SSDs.

As said before, 1M block sizes and a queue-depth of 16 are unusual for server workloads, lets lower the block size to 4K:
4K blocksize, QD16, random read
$ fio --name=test4k --filename=test4k --size=10000M --direct=1 --bs=4k --ioengine=libaio --iodepth=16 --rw=randread --randrepeat=1 --rwmixread=75

* 900p: 310227 IOPS, 1211 MB/s, 51 usec avg. latency
* SM951: 177432 IOPS, 710 MB/s, 90 usec avg. latency

Again, the SM951 does a good job in reading, however the gap becomes a lot bigger. The 900p now delivers 75% better throughput. Let’s write some data…

4K blocksize, QD16, random write
$ fio --name=test4k --filename=test4k --size=10000M --direct=1 --bs=4k --ioengine=libaio --iodepth=16 --rw=randwrite --randrepeat=1 --rwmixread=75

* 900p: 188632 IOPS, 755 MB/s, 84 usec avg. latency
* SM951: 22012 IOPS, 88 MB/s, 712 usec avg. latency

While 22k IOPS are still very respectable from the SM951, the 900p again obliterates it, now producing about 9x higher performance.

Conclusion

Those numbers being crunched, NAND based SSDs remain to be great products, just not for every workload and use-case. 3D-XPoint clearly defines a new standard for such workloads, somewhere in between DRAM and NAND.

Back to specs, the 900p’s endurance is rated as 5PBW (five petabytes written) compared to 400TBW (four hundred terabytes written) of the SM951. The datacenter focused P4800X is even rated at 20PBW. To be fair on specs, the 900p uses a lot more power (5W idle, 14W load) compared to 40mW idle and 5W load of the Samsung and other NAND SSDs.

Both the latency advantage and higher durability make 3D-XPoint based products a very interesting device for enterprise workloads and caching. Therefor i decided to get a 900p and use it as cache device for my home server. Before doing so yourself, consider that Optane is a 1st generation product, there are likely to be improved cards around the corner.

Upgrading my home server

The server runs a bunch of KVM managed by Proxmox, sports a E3-1260L CPU, 32GB of DDR4 ECC memory and a P10S-I board.

Spinning up ZFS

Creating the primary storage pool is quite straight forward:
$ zpool create -O compression=lz4 -O normalization=formD -o ashift=12 storage raidz1 ata-HGST_HUH721010ALN600_1SJ5HXXX ata-HGST_HUH721010ALN600_1SJ5JXXX ata-HGST_HUH721010ALN600_1SJ6KXXX

Explanation:

  • compression=lz4 means LZ4 compression is used on compressible data. ZFS will find out if a block is actually compressible.
  • normalization=formD means file names are stored as normalized UTF-8
  • ashift=12 means native 4K blocks are used, which my drives feature
  • raidz1 means the provided drives are organized in a way traditional RAID5 does, storing a parity as redundancy to allow recovering one failed drive

Tuning

ZFS is quite reasonably configured by default, however there are a few useful knobs to adjust to both workload and hardware. Please always verify that a change has positive impact and adjust, there is no perfect universal config otherwise this would be the default anyway. I’ll write a separate post about file-system tuning in a broader scope.

Adding Optane ZIL/L2ARC

To use the Optane 900p as caching devices, i created a GPT partition table with a 10GB ZIL (“log”) and 120GB L2ARC (“cache”) partition. Adding them to the pool is easy:

1
2
$ zpool add storage log nvme-INTEL_SSDPED1D280GA_PHMXXX2301DU280CGN-part1
$ zpool add storage cache nvme-INTEL_SSDPED1D280GA_PHMXXX2301DU280CGN-part2

Now my pool looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ zpool status -v
pool: storage
state: ONLINE
scan: scrub repaired 0B in 20h37m with 0 errors on Sun Feb 11 21:01:09 2018
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-HGST_HUH721010ALN600_1SJ5HXXX ONLINE 0 0 0
ata-HGST_HUH721010ALN600_1SJ5JXXX ONLINE 0 0 0
ata-HGST_HUH721010ALN600_1SJ6KXXX ONLINE 0 0 0
logs
nvme-INTEL_SSDPED1D280GA_PHMXXX2301DU280CGN-part1 ONLINE 0 0 0
cache
nvme-INTEL_SSDPED1D280GA_PHMXXX2301DU280CGN-part2 ONLINE 0 0 0
errors: No known data errors

Migrating images

I was previously using the “qcow2” disk format on ext4, which is now a bad idea since ZFS is already a copy-on-write system. Those images can easily be transformed to RAW images and dd’ed back to the ZFS dataset.

1
2
$ qemu-img convert -f qcow2 -O raw vm-100-disk-1.qcow2 vm-100-disk-1.raw
$ dd if=vm-100-disk-1.raw of=/dev/zvol/storage/vm-100-disk-1 bs=1M

ZFS allows to create sparse datasets, which will only grow if their space is actually used. Since zeros are highly compressible, writing and deleting a large “zero file” within the VMs can actually free up ZFS storage. After moving to RAW images, run the following within the VM:

1
2
$ dd if=/dev/zero of=zerofile bs=1M
$ rm zerofile

Swapping to Optane

Since i’m running virtual machines, there is another thing which should go to low-latency storage: swap. I try to conserve as much memory as possible, which means VMs sometimes use their swap space, which gets horribly slow in case it resides on spinning disks. For that reason i created another partition, created a separate ZFS pool and created disk images that will hold the VMs swamp data.

Creating a new pool is very simple and as i don’t need redundancy on swap it will just be one “device”, actually a partition. Using unique hardware identifiers instead of device paths (e.g. “/dev/nvme0n1p3”) is quite helpful as PCIe enumeration and partition order may change.

1
$ zpool create -O normalization=formD -O sync=always swaps INTEL_SSDPED1D280GA_PHMXXX2301DU280CGN-part4

Now new virtual disks are created on this ZFS pool and get attached to their virtual machine.

1
2
3
4
5
$ zfs list
swaps 33.1M 96.8G 24K /swaps
swaps/vm-100-disk-1 30K 96.8G 30K -
swaps/vm-101-disk-1 1.02M 96.8G 1.02M -
...

Replacing old swap and re-claiming that space for the root partition is easy if the VMs are using LVM. /dev/sdb is the new virtual device available to the VM, stored at the ZFS “swaps” pool on Optane.

Add the new swap space to LVM:

1
2
3
$ pvcreate /dev/sdb
$ vgcreate vm-optane /dev/sdb
$ lvcreate -l 100%FREE -n swap vm-optane

Create the swap file system and use the UUID as device identifier in /etc/fstab:

1
2
$ mkswap /dev/vm-optane/swap
$ vim /etc/fstab

Disable and remove the old swap partition:

1
2
$ swapoff /dev/vm-system/swap
$ lvremove /dev/vm-system/swap

Extend the root partition and file system to use the free’d up space:

1
2
$ lvextend -l +100%FREE /dev/vm-system/root
$ resize2fs /dev/vm-system/root

…and reboot the VM, just to be sure the file system is undamaged.

Camera Buying Guide

With the holiday season at the doorsteps, some may look for a new camera and following up a discussion i recently had, i’d like to share some advice on what to buy. While owning Canon cameras and lenses, i try to be as vendor agnostic and unbiased as possible. In fact i will not propose a specific camera but try to provide a list of useful parameters as a basis for you to decide. The goal is to avoid wasting money on cameras and lenses which would be much better spent on a vacation to see the world and actually take photos.

Basic decisions

First, do you need a dedicated camera for still photos? Ask yourself the following questions:

  • Are you able and willing to spend at least €1000 on a camera body and lens?
  • Are you ready for a (temporary) world of pain when moving beyond automatic mode?
  • Are you limited by your smartphones camera abilities?

If the answer to any of those questions is “no”, please don’t waste your time and money and continue using your phones camera. It will be do an exceptional job for capturing still photo and video. Why?

Today there really is no camera market at the sub-€500 section, those simple point and shoot cameras got pretty much extinct by phones. There are lots of offers between €500 and €1000, so-called “bridge cameras” which provide a very small upgrade from your phone but have a fixed lens and cheap “system cameras” that are not really better either but allow to exchange the lens. By “system cameras” i mean DSLR and mirrorless cameras alike, the differentiation to “bridge cameras” really is to swap the lens and other accessories. A good camera and lens combination to get started will cost about €1000, regardless the vendor. Those cameras have good specs, can shoot RAW images and offer creative access to exposure. Looking for used gear is a great option but note that lenses are quite stable in price.

If you stick with automatic exposure mode, there really is not much use in switching to a dedicated camera either. Chances are that the automatic mode of your phone is much more powerful and constantly improving at a pace the camera industry currently is not. For the beginning its totally normal to use automatic exposure to get some useful images, however the power of a dedicated camera system lays in the way how the photographer can influence image composition, exposure and details in a creative way which is not possible in automatic mode. And no, “creative” does not mean bokeh simulation or subject isolation that current high-end phones offer.

In case you’re taking photos to post them on social networks or show them on your phone, there is a high probability that a dedicated camera will not really be used anyway. Getting photos off those cameras is still not trivial and a phone is the natural place to share and thus take selfies or food porn.

Marketing metrics

Lets assume the answer to any of those questions is “yes”, the next logical question is “what camera to buy?”. For 99% of photographers stepping up from smartphone photography, the answer is simple: it does not matter. Nowadays its impossible to buy a bad camera when choosing a well known vendor (Canon, Nikon, Sony, Fuji, Sigma, Olympus, Panasonic, Leica, Pentax…). Anyway, there has to be something to narrow down the camera market to take a decision. Typically this “something” is a list of technical parameters each camera has and is therefor easy to compare. Vendors and customers like large numbers, so the following parameters are often used for this decision:

  • Resolution (Megapixels)
  • ISO range (something between 50 and 51200, extended by some phantasy number)
  • Dynamic Range (measured in “stops”)
  • Video resolution and frame rate (like 4K@120fps)
  • Still photo frames per second (between 4 and 20fps)

While those parameters have impact on photography performance, they do not really matter nowadays and are vastly overrated.

Resolution

Any camera uses sensors exceeding 12 megapixels and except for special cases this is far more than needed to capture great images. The first professional digital cameras resolved at 4 megapixels and the resulting images have been printed to cover whole buildings. Really, megapixels is the last thing that should be considered for a decision. Except for wasting storage, very specific usages and “pixel peeping” there is no significant difference between a sensor resolving at 12 or 50 megapixels.

ISO range

ISO range is a similar story, unless taking pictures in really dark scenarios any modern camera will provide relatively clean images up to ISO 3200. If you know how to influence your other exposure parameters (lens aperture and shutter speed) you almost always can work around choosing high ISO settings that will mess up images regardless of the camera. Only when comparing a €1000 camera to a €4000 camera there will be noticeable difference. 35mm sensors (so-called “full frame”) tend to produce less noisy images but the benefit is not huge while the difference in price is significant and you need more expensive and heavy lenses.

Dynamic range

Dynamic range or “DR” is often used to rate a sensor. It describes the range of light intensities from the darkest shadows to the brightest highlights. A practical example is the ability to resolve details at areas with high contrast. Just like megapixels, there are situations where higher DR leads to better results but those are often neglectable and can be fixed by using proper exposure, filters or post processing. Any relevant camera today has a DR of at least 12 stops, which will do the job just fine. To be very clear, don’t buy gear solely by comparing results at DxOmark, many of their test results have no practical relevancy and only provide a very isolated rating.

Video

When looking for a camera to take still photos, video capabilities should be low in priority for most users. In fact, your phone will produce better video than a sub-€2000 still photo camera in most cases. Those who really need pro-grade video will spend a lot more but probably not choose a still photo camera. For the rest of us, like semi-pro video blogging, there are great and relatively inexpensive video cameras out there.

FPS

Last but not least frame rate can be of utmost importance for specialized photography like sports. But when stepping up from a phone camera, faster frame rates are quite overrated. There really is not much reason to take 40 photos (thats a 2-second burst on a Sony a9) of a single subject. Cameras with high fps (>8) need to have exceptional data throughput, mechanical stability and focusing capabilities which makes them really expensive. Anything around 5 fps will perfectly do its job for 99% of cases.

Useful metrics

Now, what should you look for instead? The list i propose is rather non-technical:

  • Ergonomics
  • User experience
  • Battery life
  • Size and weight
  • Native lenses
  • Shutter lag
  • Filter size
  • Noise
  • Buffer capacity

Ergonomics and UX

The first two items directly and critically influence your photography. Your camera has to be an extension of your hand and mind. If after some practice the camera still gets into your way and things are not happening intuitively, your photos will be far from what you could achieve. No additional megapixels or ISO settings could ever fix this. You’ll ultimately abandon photography if you’re uncomfortable holding and interacting with your camera. This is true for button placement, shape as well as how intuitive and responsive the software user interface is. Never underestimate the impact of details like the ability to operate the camera while writing to card and so on. Therefor always try out a camera for several days and different scenarios before deciding to buy it. For the same reason cameras tend to be terrible presents as only the photographer can decide which camera works best for individual requirements.

Battery life

Battery life has become an issue again with the advent of (semi-)professional mirrorless systems (“MILC”), which use power hungry components (sensor, screens) that are active very often while maintaining a small form factor. The physically unavoidable result is small batteries and less photos per charge. A typical DSLR will do about 1000 photos on one charge, while a MILC will struggle getting out more than 400. Battery tech and power conservation is improving a lot so this downside may vanish sometime in the future. Camera body “grips” can provide space for more than one battery at the expense of adding bulk and weight.

Size and weight

The topic of battery life and ergonomics is directly linked to size and weight. If you take a lot of photos with a power hungry system you’ll have to buy and carry a lot of extra batteries. Cameras with lots of features are usually bigger and have more knobs and dials than less sophisticated devices. If carrying a camera becomes a burden, there is a high probability not to take it with you but fall back to your phone again. Therefor always consider a small and light camera over a huge and heavy one, even if this means sacrificing features or image quality. The same is true for lenses, especially when deciding towards a 35mm system which always requires larger lenses compared to APS-C or even MFT. There is no magic sauce that can shrink lenses while maintaining a certain focal range and aperture for a given sensor size. That being said the weight and size of the camera body becomes less relevant when mounting a big ass lens, say a 600mm f/4 @ 4kg. Quite to the contrary, a large camera body (or a normal body with grip extension) is very often beneficial with regards to balancing and shake reduction when dealing with large and heavy lenses.

Native lenses

When focusing on image quality while being on a budget, always choose to invest in a better lens rather than a better camera body. Each camera vendor offers a set of lenses that fit their cameras natively, which means they “just work” and don’t require any adapters. Producing lenses is a different discipline than producing camera bodies. Sigma for example has a huge range of lenses but produces relatively few models of bodies. On the other hand Sony produces lots of bodies but has to catch up with their lens selection. Building great lenses is really hard and takes a lot of resources. Optics stay optics and replacement cycles for lenses are usually decades, compared to a few years for camera bodies. Therefor the “old guys” like Nikon or Canon have a large variety of bodies as well as lenses for each and every use case. Some lenses are unique per vendor, for example the EF 11-24 f/4 while other more typical designs are implemented exceptionally well by one specific vendor, like the Zeiss Otus line. When adapting third-party lenses, features like autofocus or CA correction may get lost or work less perfect and ergonomics can suffer. Some combinations with third-party lenses work great and are both cheap and beneficial to creativity, however buying a Canon body to exclusively use Sigma lenses should make you think.

Shutter lag

Shutter lag is the delay between triggering and the actual process of exposure, not including focusing. Good cameras should have less than 100ms of shutter lag to make sure moving subjects are captured and in-focus. Cameras with high shutter lag will significantly increase the amount of “missed” or out-of-focus subjects and mess up image composition in extreme cases. Vendors do usually not provide numbers of this metric, some for “good” reasons.

Filter size

Some use-cases require to use filters in front of a lens, for example to reduce incoming light without stopping down (ND filters), to physically modify the image (polarized filters) or just safeguard the front element of a lens. Those filters fit exactly one diameter which gets defined by the lenses size and build. Good filters are typically expensive and having multiple lenses with the same filter size allows to save a lot of money. A very popular filter size ist 77mm which certain lens makers use for professional lenses from 16mm up to 200mm focal range.

Noise

When recording video, taking pictures at a wedding or similar, noise level is a critical factor. The best gear in the world won’t take pictures if you’re disturbing the cermeony. Mirrorless camera models obviously have an advantage since the flapping noise of the mirror is absent. Still there are multiple components that potentially create noise, for example shutters, lens motors and case cracking. When choosing a camera body and lens always try how loud those components are and decide if they are fit for the job. Many cameras offer a “silent” mode which reduces noise at the cost of frames-per-second.

Buffer capacity

Last but not least buffer capacity can be a real pain for sports photography and similar disciplines. There are cameras that take a lot of pictures per second but cannot store them well. This is related to both the interface type of the storage card and the internal buffer of the camera. For example a Canon 1DX Mark II with proper storage can take up to 170 RAW photos before slowing down while a Canon 70D gets slow after 15 photos. As the 1DX takes pictures at a rate of 14fps, this means 12 seconds of continuous shooting compared to 7fps or 2 seconds at the 70D.

Mechanical Keyboards & Reviews

History

During the past years mechanical keyboards have been re-appearing at the consumer IT market. While by far most keyboards sold and used are based on a membrane (“rubber-dome”), especially the gaming industry established keyboards with mechanical switches as a “new” gold standard for input devices. Looking back in the history of personal computing, such mechanical keyboards were the norm until the late 1990’s and got replaced by much cheaper, less complex and more light-weight rubber-dome desktop and scissor-style laptop keyboards during the 2000’s. While for laptops, the “thin and flat” paradigm was and is the reason to integrate such keyboard, the reason to use them at desktops and workstations was simply the race to the bottom of cost.

The lower tier of desktop computing did suffer a lot from tablets and notebooks, entry-level users rather opt for a cheap notebook than for a clunky stationary computer with peripherals. The market which continued to expand is gaming, government and professional grade workstations - rather expensive machines. However, even though enthusiasts spend a fortune on powerful GPUs, complex case-mods and electricity, peripherals like screens and keyboards were not considered to be relevant for a long time. The same is true for professional users. Employers provide high-end workstations well above the range of €5000 but equip it with a €200 screen and a €20 keyboard+mouse combo that lasts two years. At the first glance, this makes sense from a business side since broken peripherals can be replaced with low maintenance effort.

Health

In reality however peripherals which are already past their lifespan continue to be used for a long time, with disastrous impact on health and productivity. When thinking about how much time one uses such peripherals, ergonomics and hygiene are a extremely relevant but often ignored fact. Losing productivity at the end of a working day or even getting sick due to finger fatigue or bad eyesight has huge productivity impact. One would expect that employers realize this and start investing in proper peripherals, which is quite the opposite of expensive when looking at productivity gain. High-quality screens, mice and keyboards can outlast typical upgrade-cycles of computers many times, so on the long run it may even be financially cheaper to invest into proper peripherals.

Going mechanical

Fortunately the mindset and product ranges started to change some years ago and manufacturers expanded their portfolio with mechanical keyboards, ergonomic mice and great screens again. While this kind of equipment never really vanished at the government sector, to most PC users it appears as something entirely new. Now what makes a keyboard mechanical? There are lots of in-detail articles on the web about this topic so lets keep it brief:

“A mechanical keyboard uses some kind of electronic switch that get actuated by the force of a key press. Rubber-dome keyboards rely on actuating a rubber membrane to close a electronic circuit when pressing a key.”

In theory this sounds like a marginal difference but the practical difference is like comparing a Ford Model T to a Tesla Model S. Manufacturers did realize that people are willing to spend a multitude of money on a good mechanical keyboard compared to a rubber-dome keyboard. This did not only led to greater choice of mechanical keyboards but also allowed those manufacturers to differentiate by adding features, premium materials and actually invest into research instead of just throwing the same damn product on a dead low-margin market where the only differentiation is a brand name.

Regardless of features like connectivity, back lights or media keys, the main ingredients for a good mechanical keyboard are its switches and key caps. When moving to mechanical keyboards its perfectly normal to feel a bit more fatigued in the beginning since pressing a key usually requires more force. Rubber-dome veterans are used to bottom-out their keystrokes, which is not necessary on a mechanical keyboard but adds to fatigue. This is however temporary and gets diminished for the more pleasant typing feedback in the long run and getting used to the characteristics of a mechanical keyboard.

Of course there are many subtypes, yet there are some general characteristics:

Rubber-dome

  • Usually require to bottom-out the key press to actuate, soft feedback
  • Hard “confirmation” of the key press when hitting the bottom PCB layer
  • Low lifespan when using it many hours a day
  • Inconsistent typing feedback once the rubber wears out
  • Usually “pad printed” or “laser etched” key cap legends that wear out quickly
  • Hard or impossible to clean or change key caps, no real choice in caps
  • Cheap build quality of key caps and case due to overall low price point
  • Price point: €10 - €80

Mechanical

  • Consistent “actuation” point which does not require to bottom out, very precise and spot-on feedback
  • Optional characteristics like “tactile” (feel the actuation point) and “clicky” (acoustic feedback of the actuation point)
  • Long lifespan, typically 30-50 million keystrokes per key (~20 years at 8 hours usage 5 days a week)
  • Premium key cap materials (ABS, PBT) with robust legends (dye-sublimation, double-shot)
  • Translucent key caps and switches which allow back light, color LED effects
  • Different types of key caps, which can be removed, cleaned, replaced and customized
  • Overall better build quality due to higher price point, more rigid quality control
  • Price point: €60 - €350

Financials

Spending more than €200 on a keyboard is certainly a bit excessive and can’t be justified financially when looking at it from a neutral point of view. For some people however mechanical keyboards are a “hobby” and even “collectors items” when thinking about vintage or rare keyboards or artisan key caps. There are great mechanical keyboards at the sub-€100 range, especially from gaming peripherals makers. Those really expensive keyboards usually target the typists and software development community - people that type for a living.

When discussing the cost of mechanical keyboards, it has to be noted that both switches and high quality key caps are very expensive - and a keyboard needs 100+ of them. The next cost driver is the fact that most keyboards are using a 104-key ANSI layout. If you’re used to a 105-key ISO layout there is a good chance to either pay a premium for “exclusivity” or just not getting ISO-style keyboard off the shelf at all. As an example, a full set of high quality ABS double-shot or PBT dye-sublimated key caps costs €100+ alone, add €0,50 per switch, electronics, cables, casing. Last but not least high-end keyboards are not a mass-market product that get produced by the millions. All this puts a €100+ price tag for a keyboard into perspective, another way to view it is: Good keyboards cost good money, we just got used to low quality cheap keyboards.

Switches

There are two major types of mechanical switches: “Cherry MX” made by Cherry GmbH in Germany and “Topre electrostatic capacitive” made by Topre Corporation in Japan. Several clones of those switch types are being manufactured but should be avoided since they usually don’t get made with same amount of precision and quality requirements. Other less common switch types, like ALPS or vintage “buckling spring”, are built with different specification. Such specifications define the actual switch characteristics, the housing dimensions as well as the stem and key cap mount format. While Cherry MX have “+”-type stems, Topre have “o”-type stems which are incompatible to each other - with the exception of special-made stems for Realforce RGB and Novatouch (discontinued) keyboards that fit both key cap mounts. On top of these stems sits the actual key cap made of plastic, which is usually removable. A word on big numbers - it does not matter if a switch has a projected lifespan of 30 or 50 million keystrokes, in either way it will easily outlive the case, keycaps or even the connector. Try finding a recent computer with AT or PS2 connectors which were used 20-30 years back.

Cherry MX are traditional switches with a spring that requires a certain force to compress and move parts into place to close a electric circuit. Depending on the type (MX “color”), switches are either tactile or linear, clicky or silent. Their basic concept stays the same though. Topre is a very different story, in fact they are more rubber-dome than mechanical. The tactile feedback when pressing a key is provided by a high quality rubber-dome and beneath that dome a spring gets compressed. The spring is however not used to provide force but to indicate capacity. Beneath the spring a sensor registers the electrostatic capacity of the compressing spring and once a threshold is passed a actuation is registered. No traditional “mechanical” switch is used which is good for durability since the system is more “sealed” and suffers less from friction. The typing characteristics and durability are similar to a Cherry MX Brown (tactile, silent) but again completely different. Pressing a Topre switch feels more like pushing a piece of metal through a magnetic field and the push-back when releasing the key is significantly “different”. That being said, both switch types provide a extraordinary typing experience. Before spending lots of money on a new switch type though, definitely consider getting a sample switch or switch-tester to evaluate.

Reviews

Over time i bought several mechanical keyboards for different use-cases, so far none of them did break or has ever shown severe issues. This naturally reduces the amount of items i could compare.

  • Steelseries 6Gv2
  • Filco Majestictouch 2 TKL
  • Topre Realforce 105UB
  • Uniqey Q100

Disclosure: I bought all those boards from my own money and did not get any sponsored samples for review, neither has any of the manufacturers contacted me.

Steelseries 6Gv2

Steelseries 6Gv2
The 6Gv2 is a full-sized keyboard priced at an entry-level €70 and aims for the gaming community. It uses plate-mounted Cherry MX Black switches (rated at 50M keystrokes) which are “linear” meaning the actuation point can’t be felt when pressing a key. Resistance to the key is 60 grams which makes it a bit harder to press but also reduces unintended key presses. The case has a acceptable build quality and weights in at about 1.3 kilograms, big rubber feet make sure it stays in place like its glued to the table.

When using a USB-to-PS2 Adapter, and assuming the computer still offers a PS2 port, this keyboard allows n-key-rollover (“NKRO”) which means actuation of all 105 keys can be signaled to the computer at the same time. USB is limited to 6KRO by design on the other hand. The keyboard comes with a 2 meter non-detachable rubber USB cable, PS2 adapter and can be tilted.

Overall the keyboard is basic in functionality but does a great job when gaming is a priority - for typing and fatigue-reduction there are better alternatives though. The key caps start to get shiny after a few years and the white color at the laser-etched legends starts to fade. Since the cost-benefit ratio is excellent, its a great keyboard to get into the mechanical keyboard experience.

Filco Majestictouch 2 TKL

Filco Majestictouch 2 TKL
This is a premium (IMHO overpriced) €150 ten-keyless keyboard (“TKL”) which means the numblock is missing to make it more compact. While this might be a deal breaker for typists its great for gaming since the mouse-hand is much closer to the keyboard-hand which reduces fatigue. Most games won’t need the numblock anyway. The Majestictouch is available with different plate-mounted Cherry MX keys (Brown, Black, Blue, Red) which translates to different characteristics that can be looked up easily on the web. The build is very sturdy, despite its compact size and 88 keys it weights about 1 kilograms and has great rubber feet that fix it to the desk.

Just like the Steelseries it supports NKRO when using PS2, has a non-detachable rubber USB cable as well as the ability to be tilted. Key caps are made of ABS with laser-etched legends and are of good but not outstanding quality since they wear out after some years, especially when using the “WASD” section excessively. The cost-benefit ratio is quite bad compared to the Steelseries and it only makes sense when specifically looking for a nicely built ten-keyless Cherry MX gaming keyboard without any fancy features. On the other hand ten-keyless layouts are a niche configuration within a niche product range, which makes them even more exclusive and counting keys to justify the price does not take that into respect.

For personal preference i use the Cherry MX Brown switch option (tactile, non-clicky) of the keyboard. With regards to noise this is a good example how switches do affect noise but are not the only factor to take into consideration. Compared to other MX Brown keyboards, the Filco Majestictouch is much louder during typical use. This is caused by inherent keycap “rattling” and quite loud clicks when bottoming out the keys which is related to the material and quality of the keycaps. If i’d to chose again i’d probably go for a Corsair K65 which offers a similar form factor and quality for half the price.

Topre Realforce 105UB

Topre Realforce
Topre Corporation makes keyboards using their proprietary electrostatic capacitive switches, rated at 30M actuations. The “Realforce” brand is their high-end keyboard series which targets business users. While there are ten-keyless options and even numblock-only devices available, the “full-size” 104 or 105 key models with standard layout are most popular. There are few distributors and a ISO layout is rather hard to find. As described earlier the typing experience is one of a kind due to the specific switch characteristics. The keyboard is made of high quality plastic, including key caps made of PBT instead of ABS. PBT is more durable and less prone to getting shiny over time. The keys legends are dye-sublimated since there currently is no way to make double-shot PBT key caps and the space bar is made of ABS since large PBT key caps are also hard to manufacture. However, there are replacement space bars made of PBT available.

The Realforce comes with a solid 1,5 meter non-detachable rubber USB cable and some cable routing tunnels beneath the board, it can be tilted as well. Its weight of 1,5 kilograms and two rubber feets at the bottom make sure that the keyboard stays in place quite well. A nice feature is variable actuation weight, which makes sure keys usually triggered by less dominant fingers require 35 or 45 grams while more dominant fingers have to overcome 55 grams for their keys.

The overall build quality is flawless and as good as a plastic product could possibly be. So far i use this keyboard 8 hours per day at most days of the week for 6 years straight, it did never let me down and is a real workhorse. The only thing i noticed is that macOS sometimes does not recognize the keyboard after getting back from hibernation and i had to unplug/plug it in again. Over time even those PBT key caps got a bit shiny and some often-used keys start to fade. However, considering the heavy usage its the most durable keyboard i came across so far. The asking price including taxes and shipping from England is €260 in Germany, which is quite an investment.

Just recently Topre introduced the “Realforce RGB” keyboard which is more gaming oriented and sacrifices some of the typical Realforce features (like PBT keycaps) for more stylish things like illumination which requires ABS keycaps with shine-through legend. I had the opportunity to check it out for some minutes and if budget is not a limitation this would be a natural choice for a great gaming keyboard. The electrostatic capacitive switches allow a unique feature which is per-key actuation point configuration since the actuation is determined by measuring capacity instead of a mechanical switch. Illuminated keyboards are not my thing so i’d rather swap the ABS keycaps that come with it with some PBT caps which is easy since the Realforce RGB comes with key stems that are compatible to both Topre and Cherry MX keycaps. If only they would make a ISO layout of the RGB…

Uniqey Q100

Uniqey Q100
The Uniqey Q100 is the latest addition to my collection and appears to be pretty much unknown on the Internet and particularly among reviewers. The brand just started shipping their keyboards in late 2016 and is backed by the german industrial equipment manufacturer “GMK electronic design”, well respected for their high-end keycaps. Besides offering some pre-configured keyboards, their unique selling point is customization and their high quality double-shot ABS keycaps. ISO and ANSI layouts are offered whereas sections of the key layout can be customized with different colors and legends. Uniqey offers the full range of PCB-mounted Cherry MX switches and adds their own “QMX-Clips” as an option which reduce the noise of a key press significantly. Note that those are far better than rubber o-rings, which heavily influence typing feedback. The biggest difference to other keyboards are the materials used for the keyboards case. Every Q100 is made of a anodized aluminum body and can be customized with side-panels made of wood or anodized aluminum.

While other keyboards are said to be built like a tank, the Uniqey Q100 literally IS a tank. Its made of metal and weights 1,5 kilograms, has modular detachable rubber feet, a highly flexible detachable Micro-USB cable for power supply and data and offer a auxiliary USB port to connect mice. On top of USB connectivity it comes with an integrated Bluetooth module which allows to type on three different devices like tablets, phones and laptops and can easily switch the paired device on the fly. I opted for Cherry MX Brown (tactile, non-clicky) switches and added QMX clips, combined with the sturdy build quality and thick ABS keycaps typing is extremely silent even when bottoming out keys. By choosing switches with specific charactersitics the Q100 can be configured to be a perfect typing keyboard or a great gaming keyboard.

While using the keyboard for multiple months i could not find any downside of this product, its a truly remarkable piece of engineering and a prime example for quality stuff “Made in Germany”. That being said the price is just as remarkable with €265 to €338 depending on configuration. However, the product delivers just perfectly, after a few minutes one will understand why this thing costs as much as a entry-level laptop. It operates perfectly and the look and feel comes very close to the enclosure of a Mac Book Pro or comparable high-end devices. At the same time its visual appearance is very unobtrusive due to the absence of illumination and brand logos. In my personal opinion the Uniqey Q100 provides a better overall package than any other keyboard i have used, including the best-of-the-rest: Topre Realforce.

Conclusion

As always there is no “best” product but if budget is not a concern and you’re looking for the ultimate keyboard then figure out your preferred switch type, get a Uniqey Q100 or Topre Realforce and look no further. If fanciness is what you’re into, get a Topre Realforce RGB. If you’re on a budget or uncertain if mechanical keyboards are a thing for you, get a Corsair K65 or Steelseries G6v2. In any way, the typing experience and impact on health will be worth the investment, especially if your occupation means a lot of typing.