When it comes to load testing a web application, usually JMeter comes up as the go-to solution. It has a huge amount of features, protocol support, is opensource, establishment - and boy it has a GUI! There are some other alternatives like pyLot, locust, Grinder, radview, wapt or LoadUI which are either commercial or not as feature rich and less established.

Lately, some hosted solutions like loader.io, neotys, blitz.io, blazemeter, loadstorm etc. joined the party. These offerings are nice for testing the whole stack from an end-user perspective or running limited tests. The downside is, that these are typically sold by bandwidth or threads (users) which is a good business model but comes unhandy when really stress testing an application over a long period of time or testing within an internal network.

Usability

One of the biggest benefits of JMeter, in many peoples minds, has been its graphical user interface. Well, it’s kinda powerful and it kinda allows easy test creation. People tend to think that a GUI makes everything easier since many concepts are well known.

A GUI makes everything easier, right?

In my opinion the opposite is true, at least for JMeter. Its GUI is quite bloated, which is natural given its complexity and amount of features, but in the end is some button-input-and-knob representation of an XML structure. Under the hood, JMeter generates massive amounts of XML, based on what tests the user defined at the GUI. By itself, this is not bad. However, it shuts down any serious attempt on manual interaction on an XML level. This becomes very visible when using JMeter tests while working within a team. We use git for source control and due to a lack of good and powerful GUI based tools, primarily work with the command line implementation. Reviewing commits or merging is really a pain when you have to diff and compare on a XML level.

Even with great syntax highlighting, this is where you probably just want to go home.

Much better

Readability of code is usually determined by the skills of the author, in this case the author really doesn’t do anything wrong - but by focusing on a GUI, readability for all other forms of representation suffers. Even if you manually tame the XML output, JMeter will just overwrite it using machine-readable-only code. XML is very well structured but apart from being structured, there are other requirements to good code. Even easy tasks like replacing some parameter or defining properties becomes a pain using the GUI since you have to browse it and manually spot elements. Either you got almost everything within variables, or you’re doomed.

In my opinion, JMeter is nice to rapidly create one-shot tests that can be shown around for education, validation or audit. For any kind of sustainable agile development or testing, it’s next to useless.

Gatling on the other hand uses Scala to define load tests. Scala is probably not as established as XML but (as programming languages in general) it allows to code much cleaner and use the power of a functional programming language. Being object oriented and relatively up to date, it allows re-using code to avoid redundancy and pretty much everything one would expect from a modern programming language. While coding Scala requires some specific toolchain, the compiled code runs within a standard JVM. This makes it very easy to deploy and leverage existing workflows and tools. If you’re a programmer you probably can start with Scala right away. If you’re not familiar with programming, some hurdle may exist but learning a programming language while creating load tests sounds a good deal, right? There is no need to learn each and every bit of Scala right away, tests usually consist of the same fragments that just need to get put together.

Since Gatling is under very active development and currently transitioning from version 1 to version 2, some API vs. documentation hickups and bugs may happen from time to time. The core developers and community is very responsive and most issues get covered within hours.

Performance

The funny thing about load tests is, that many environments actually test the efficiency and performance of the test runner, rather than the application thats supposed to be tested. In projects i’ve seen people building monstrous phalanxes of test clients to stress-test a single server. Distributed test clients and down-sizing both the servers spec and the test coverage seem to be appropriate measures to get the server under load. Lets check some real-world example.

At work we got a nice Intel Modular Server box dedicated for load testing. This baby sports a chassis with redundant PSU, networking, storage and six identical blades loaded with dual Xeons and lots of memory. In a nutshell, this is a datacenter within 6HE where testing can happen without external influences. Earlier, we used one virtual machine as test director and result-parser, 4 machines running JMeter and one machine running the application (including all infrastructure) we wanted to test. JMeter has been configured to spread the test scenario to all 4 machines, effectively cutting the number of threads by 4 and feed them to the test clients to execute them simultaneously. While this worked well, it really felt odd to have 4 specialised machines hammering on one specialised machine to push it to its limits.

JMeter system load

What we see at this screenshot is one machine running about 100 concurrent threads (“virtual users”) occupying 4 CPUs. The server on the other hand is also quite stressed but keep in mind there are 4 test machines with 4 CPUs each running 400 concurrent threads in total. Even these 4 test machines did not manage to create significant load at the server, to find out its diminishing or even tensile point. One major drawback with JMeter is that it uses one system thread per user, handling large numbers of active threads is very expensive on the CPU.

When using Gatling, we easily manage to get 1200 concurrent threads running at just 25% CPU load of one CPU of a single virtual machine. This is about 200x more efficient than JMeter (1/64th of the CPU load while creating 3x the load). The server is also stressed quite well and we’re able to push load testing far beyond its tensile point.

Gatling system load

Reporting

When running JMeter using its GUI, reports are fairly nice, real time and the tool offers some help to dig through the results. However, when not being able to use the GUI (e.g. for unattended testing, continuous integration), you get a bunch of .jtl files, either XML or CSV. These can then be put into JMeter again for analysis or get processed using XSLT or tools that understand CSV (yeah… Excel). Thats all good and at this point highly structured data makes a lot of sense. There is a variety of tools that help with graphing, charting and analysing of its data. The downside is that you almost always need some kind of extra tool to make JMeter reports understandable - and you always have to wait until the test run finished. At least to my knowledge, there is no realtime graphing apart of the JMeter GUI.

Gatling also creates machine readable data files, but already provides a really nice report module that generates HTML pages with charts and graphs. In addition, the integrated Graphite feeder allows real-time charting of test runs to tools like Graphite or Grafana. This becomes really powerful for showcases or unattended test runs. Overall i think the built-in reporting of Gatling outperforms JMeter by large, even if JMeter reporting may be more accurate and comprehensive on a scientific level.

Default Gatling graphs

Graphite realtime graphs

As with every discussion, there is no “using the wrong tool” - it just depends on the job that needs to be done and some thinking outside the box. There are a lot of great tools that are not (yet) mainstream but help with every days work and contribute to getting better tests, results and software.

Apple Inc. builds great computers. Well, at least the hordes of slave workers over at Asia do. Kudos to them for building astonishing precision and the design team to create beautiful machines that feel great when working with them. Shame on this $500bn+ company for taking advantage on such cheap labour, spending .5% of their margin on working conditions would provide hundreds of thousands of families a much better perspective and fair work. But lets put that aside for a moment.

A Mac can easily be sold for over 60% if its initial price two years from now. Try this with an Asus, Sony, Acer, HP, what have you. Almost all other Laptops i’ve seen so far - since IBM dumped the Thinkpad - look like a piece of junk after two years of transportation and serious usage (like 8-12 hours a day)

Now this is where the issue starts: Macs are re-sold regularly and people are trying to get their hands on a pre-owned device to cut some of the initial cost. On the other hand, Apple has built the perfect golden cage for it’s users. Services, hardware, software, complementary products - all very well integrated and made to work with each other quite smoothly (well, most times at least…). When buying a Mac, one does not just buy a computer but enters the realm of multiple services spun around the users “digital lifestyle” that tries to keep the user at the chosen platform. While hardware may and is meant to become obsolete, these services stay and are assigned to a user (via it’s Apple-ID) rather than to a specific piece of hardware. Obviously Apple is very successful in retaining users to proprietary services so they won’t flee back to good ol’ Windows or Android world.

One service that is tightly integrated with many Apple devices, is “Find My iPhone/Mac” (FMI from now on). In a nutshell, it geo-locates devices that are registered with the users Apple-ID in case a device got lost. This works for both phones/tablets by using GPS tracking as well as Laptops using wireless network information for approximate location. The service becomes very valuable over time and more than once it helped to give a good idea on where i’ve left my phone. Other than locating a device on a map it allows basic remote management features like making the device audible, lock it to avoid misuse and wipe it’s data in case it got potentially compromised.

Now, what’s the catch? Well, back in 2011 i bought a MacBook Air and used it for almost two years before selling it online. While using it, i had FMI activated and the device has been assigned to my Apple-ID. Also, i enabled FileVault which is an encryption feature of OSX that encrypts the whole computer storage, which by the way everybody should use. Additionally i’ve wiped all of that storage data to hand over a plain computer to the new owner. There is close to no chance that anybody outside of an intelligence agency or Apple could access my old (account) data. I sold it to one of these fixed-price dealers where you don’t get to know the buyer nor care about what happens with the machine.

Roughly half a year after i sold that machine, i occasionally checked by FMI to see if there have been updates to the service. To my astonishment, a machine called “Luisa’s Mac Book Air” showed up at my list of machines.

Hello there, Luisa

It was not online at the time i checked so i opted for “Notify me when found”. Shortly after, i received an E-Mail. My old Mac Book Air has been found. Yay! Interestingly enough that it has still been linked to my Apple-ID, all the location and remote management feature were still in place. Based on the location information and her name used to generate the computer name, i was able to look up Luisa on various online services. She’s a student and based on the transportation habit of her (my old) Laptop it should be easy to find out where she’s living, whom she’s meeting and what courses she’s taking… Well i’m not interested in stalking people so i took this much information to verify it’s real. Given a bit of criminal energy, i could lock her machine and blackmail her to re-gain access to her data. Not interested in this as well, just sayin’.

Lucky me.

There are stories of guys that recovered their stolen machines by hacking into it remotely, track it, identify the thief and all that stuff. Apple, pursuing the idea of “everything simple”, has just taken it a step further. In a nutshell, i got privileged remote access to a machine that has been wiped-clean, re-installed and i had no user account nor information about. Well, that’s one hell of a privilege escalation. Luisa for sure is not using my Apple-ID and has no clue that she can still be tracked.

Knock, knock...

There are various scenarios on what has happened. She uses FMI and all that iCloud stuff, that would point out a permission issue within iCloud when linking machines to Accounts. Or, she does not use FMI, which is worse since it implies this service is available regardless of the users choice made when setting up the machine. In both cases the new owner of the machine does not seem to have any possibility to opt-out from being tracked by the previous owner.

Digging deeper, i started to suspect that Apple is not really using the owners user or account credentials to get access to the machine. While this may be a logical implementation to track/manage machines where the user is not logged in, it’s quite critical in terms of privacy. A user may enter a strong password and connect the machine to it’s Apple-ID - but in the end it’s just identified and accessed by some kind of unique hardware identifier. At least the FMI web app uses such (hashed) identifiers. It’s a good guess that these IDs are used to bypass authentication at her Laptop. In this case, it is plausible that Apple has remote access to location data of all devices as well as powerful remote management capabilities (wipe, lock…). At the very least, remote access to the machine is not secured using credentials that the user has chosen at it’s Apple-ID but rather a static ID and vague trust-relationship with Apple. Authentication by shared secrets is an issue for itself, however it’s still better than obviously having no secret within the authentication process.

Now, i’ve contacted the Apple product-security contact and described the issue. The contact is very responsive, which definitely is a good sign for taking such issues seriously. However, they came back with the statement, that there is no actual security issue. I (as the previous owner) could just remove my old machine from my FMI account to stop tracking it. Well yeah, that may “help” me not tracking somebody - but it’s definitely not helping the person that gets tracked. Furthermore they pointed to a “how-to” document that describes what actions should be done before selling a Mac (e.g remove it from all iCloud services). This document of course is optional and the new owner has no way of verifying that the previous owner has removed the machine from these kind of services.

So what is this? Bad luck? Poor design? Wrong expectations? One could argue that if a machine gets stolen and logged out, the original user still needs to be able to wipe it. That’s true and valid, but does not require the machine to be bound to the original users Apple-ID for it’s whole lifetime. As soon as somebody wipes the storage or starts over using a different Apple-ID, the connection with FMI should be save to be reset. After all, FMI is not a thievery protection system, it’s a service to find a temporarily lost device. Is Apple doing this kind of hardware-service-lock-in to prevent users from reselling their hardware? Well that could be, but i don’t think they do it on purpose or to push their own “used hardware” service.

IMHO it’s a follow-up issue of the attitude to retain customers and all their credentials, data and devices to a closed ecosystem. This is just one more example where “simplicity over security” has a significant backlash. Also, it illustrates that open source software for infrastructure is not the only key to a more trustworthy and secure environment. Web services are already much more relevant to end-users than operating systems or infrastructure services are. Many of these underlying services have already become invisible. Trusting both web and infrastructure services with sensitive information like unrestricted access to physical location data is a huge problem if the service is running as a complete black box without the user in control of his or her own data.

There is a pressing need to discuss, spread and implement the definition of “who owns what” within the web services world. Right now, most people are extremely naive when blindly accepting these 70 page’ish TOS, since there is no real socially acceptable alternative to many of these services. Users must become sensitive to this an be in a position to claim their ownership on their data. Giving it away to industry giants for free will manifest the current situation and ultimately lead the way to a world with few players dominating not just the market but also their users habits and (digital) life.

At work, we run a lot of automated integration, unit and system tests. This requires virtual machines that can be started up, get deployed with the software we want to test, run tests against it, shut it down and reset to a non-tainted state for the next test. While we’re working on a OpenStack based solution for that kind of internal deployment, i’ve had the joy of working with a, by all means, legacy Xen paravirtual hypervisor setup. It basically runs on a old machine with old debian and old XEN. The XEN guests (aka. domU’s) are stored within LVM on an old storage system.

The requirement in my case is, that all partitions exist as raw LVs without a partition table, so i can continue using existing tools for backup/restore. Sadly virt-install does not allow that (from what i found out at least) so i needed quite a workaround. To start with, i used virt-install to install the domain to a temporary image file.

Hint: virt-install requires a valid locale, just in case there is none set:

1
$ export LC_ALL=C

When installing, i’m not using a separate /boot or SWAP partition, just for the sake of ease. However, the provided steps may be used for more complex partition setups as well. When installing, do NOT create a LVM based storage setup within the VM.

In this case, i’m going to use Debian Wheezy, but when following some tricks, it’s possible to install every Linux Distro using this workflow. See “Other Distros” for more details. Since the old hardware does not support Hardware-assisted virtualization (HVM) either provided by AMD-V or Intel VT-x, i’ve to stick with “paravirt”.

1
$ virt-install --paravirt --name debian7-install --ram 1024 --file /root/debian7-install.img --file-size 5 --nographics --location http://ftp.de.debian.org/debian/dists/wheezy/main/installer-amd64/

Virt-install may print some error output, however in most cases it still succeeds. Just jump into the domU. For SLES, please see the “Other Distros” section.

1
$ xm console debian7-install

Observe the image partition table, sector size and starting block of a partition

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ fdisk -lu debian7-install.img
You must set cylinders
You can do this from the extra functions menu.
Disk debian7-install.img: 0 MB, 0 bytes
181 heads, 40 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b3bee

Device Boot Start End Blocks Id System
debian7-install.img * 2048 10483711 5240832 83 Linux
Partition 1 has different physical/logical beginnings (non-Linux?):
phys=(0, 32, 33) logical=(0, 51, 9)
Partition 1 has different physical/logical endings:
phys=(652, 180, 40) logical=(1448, 55, 40)

In this case, the sector size is 512b and the primary (only) partition starts at block 2048.

Now mount the image file to a loop device, starting at the offset calculated based on sector size and start block.

1
$ losetup /dev/loop0 debian7-install.img -o $((2048 * 512))

Check if the partition has been mounted correctly

1
2
$ file -s /dev/loop0
/dev/loop0: Linux rev 1.0 ext4 filesystem data, UUID=18af6c1e-1c7e-4ccc-9828-a9a46770d0f8 (extents) (large files) (huge files)

As we see, the mounted data is a valid ext4 partition.

Create a new LV based on the size of the partition, calculated by start and end block, multiplied by the sector size.

1
$ lvcreate -L $(((10483711 - 2048) * 512))b --name debian7-install-disk vg

Now use partimage to create an image from the mounted partition.

1
$ partimage -z2 -o -d -M -b save /dev/loop0 /root/backups/debian7-install.new

Compression and use of partimage will save a lot of space since only used blocks are stored within the new image.

1
2
$ ls -lh /root/backups/debian7-install.new.000
-rw------- 1 root root 192M Nov 28 10:38 /root/backups/debian7-install.new.000

Unmount the loop device after getting the partition image

1
$ losetup -d /dev/loop0

You may also move the original image to another location and remove the domain from XEN

1
2
$ mv debian7-install.img debian7-install.img.old
$ xm delete debian7-install

Create a SWAP partition for later usage

1
2
3
4
5
$ lvcreate -L 512M --name debian7-install-swap vg
$ mkswap /dev/vg/debian7-install-swap
mkswap: /dev/vg/debian7-install-swap: warning: don't erase bootbits sectors on whole disk. Use -f to force.
Setting up swapspace version 1, size = 524284 KiB
no label, UUID=46d4b88f-1731-459d-9b98-edf8b2066717

Now restore the image created on the original image and it’s partition to the LV

1
$ partimage restore /dev/vg/debian7-install-disk /root/backups/debian7-install.new.000

Fetch the LVs UUIDs, these are required for the next steps

1
2
3
4
$ blkid
[...]
/dev/mapper/vg-debian7--install--disk: UUID="36aa2e94-9f2f-4ab1-b013-fd2cb960b55a" TYPE="ext4"
/dev/mapper/vg-debian7--install--swap: UUID="46d4b88f-1731-459d-9b98-edf8b2066717" TYPE="swap"

The image has been successfully restored to LVM, so the hard work is done. Next, some modifications within the image need to be performed, to make the system bootable.

For that, just mount the LV to an arbitrary mount point.

1
2
$ mkdir /mnt/debian7-install
$ mount /dev/vg/debian7-install-disk /mnt/debian7-install/

XEN will use pygrub to boot the machine. Pygrub only understands the menu.lst style GRUB configuration files, not the more recent grub.cfg files. Therefor we have to create a new menu.lst file based on the new partition settings. The UUID value for the / disk is used here.

1
2
3
4
5
6
7
8
9
10
11
12
13
$ vim /mnt/debian7-install/boot/grub/menu.lst
default 0
timeout 2

title Debian GNU/Linux 7
root (hd0,0)
kernel /boot/vmlinuz-3.2.0-4-amd64 root=UUID=36aa2e94-9f2f-4ab1-b013-fd2cb960b55a ro
initrd /boot/initrd.img-3.2.0-4-amd64

title Debian GNU/Linux 7 (Single-User)
root (hd0,0)
kernel /boot/vmlinuz-3.2.0-4-amd64 root=UUID=36aa2e94-9f2f-4ab1-b013-fd2cb960b55a ro single
initrd /boot/initrd.img-3.2.0-4-amd64

Adjust the kernel path, version and name based on the files located in /boot

Also adjust the fstab file, use the UUID of the / and SWAP disk.

1
2
3
$ vim /mnt/debian7-install/etc/fstab
UUID=36aa2e94-9f2f-4ab1-b013-fd2cb960b55a / ext4 errors=remount-ro 0 1
UUID=46d4b88f-1731-459d-9b98-edf8b2066717 none swap sw 0 0

Unmount the LV after modifying these settings

1
$ umount /mnt/rhel6-install/

Create a new XEN configuration file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ vim /etc/xen/debian7-install.cfg
bootloader = '/usr/lib/xen-default/bin/pygrub'

vcpus = '1'
memory = '1024'

root = '/dev/xvda2 rw'
disk = [
'phy:/dev/vg/debian7-install-disk,xvda2,w',
'phy:/dev/vg/debian7-install-swap,xvda1,w',
]

name = 'debian7-install'
vif = [ 'bridge=eth0,mac=00:16:36:1d:0f:b6' ]

on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'

In some cases, the journal got corrupted, for the sake of consistency, run fsck on the newly created disk:

1
$ fsck /dev/vg/debian7-install-disk

Now we’re ready to boot the new XEN domain from LVM:

1
$ xm create -c /etc/xen/debian7-install.cfg

Other Distros

Debian 6 (Squeeze)

For Squeeze, just modify the location URL to point to a different dist.

1
$ virt-install --paravirt --name debian6-install --ram 1024 --file /root/debian6-install.img --file-size 5 --nographics --location http://ftp.de.debian.org/debian/dists/squeeze/main/installer-amd64/

CentOS6

Just call virt-install using a mirror repository:

1
$ virt-install --paravirt --name centos6-install --ram 1024 --file /root/centos6-install.img --file-size 5 --nographics --location http://mirror.netcologne.de/centos/6.4/os/x86_64/

Note that the default text-mode CentOS installer is somewhat limited. It does not allow to modify the partition table, for example not using LVM but just a plain partition. If that’s the case for you, use the VNC (launched from the text-mode installer) or virt-viewer to launch the more sophisticated (GTK’ish) installer of CentOS. This will allow you to work around LVM.

RHEL6

For this, you need the binary DVD images of RHEL6 and make them accessible via HTTP (aka. create your own installation mirror). Therefor, you require a HTTP server and need to mount it to a directory which gets served:

1
$ mount -o loop /root/rhel-server-6.4-x86_64-dvd.iso /var/www/rhel/

Then, use virt-install to start the installation process from this HTTP site:

1
$ virt-install --paravirt --name rhel6-install --ram 1024 --file /root/rhel6-install.img --file-size 5 --nographics --location http://example.com/rhel/

Note that the default text-mode RHEL6 installer is somewhat limited. It does not allow to modify the partition table, for example not using LVM but just a plain partition. If that’s the case for you, use the VNC (launched from the text-mode installer) or virt-viewer to launch the more sophisticated (GTK’ish) installer of RHEL6. This will allow you to work around LVM.

When booting, this errors tend to show up, however they won’t keep the domU from booting

1
2
3
4
PCI: Fatal: No config space access function found
Could not set up I/O space
Could not set up I/O space
Could not set up I/O space

SLES11

For SLES, the process is similar to the one used at RHEL, but enhanced by the pitfall of SLES requiring some kind of “real” graphical interface rather than just a XEN console. In my case i had no X running at the XEN dom0, so i had to improvise quite a bit.

xm console just stopped responding at:

1
2
3
4
5
6
[    9.479067] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[ 9.479084] EDD information not available.
[ 9.646300] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[ 9.646317] EDD information not available.
[ 9.813624] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[ 9.813641] EDD information not available.

For this, you need the binary DVD images of SLES11 and make them accessible via HTTP (aka. create your own installation mirror). Therefor, you require a HTTP server and need to mount it to a directory which gets served:

1
$ mount -o loop SLES-11-SP3-DVD-x86_64-GM-DVD1.iso /var/www/sles/

Then, use virt-install to start the installation process from this HTTP site:

1
$ virt-install --paravirt --name sles11-install --ram 1024 --file /root/sles11-install.img --file-size 5 --nographics --location http://example.com/sles/ --vnc

Now the fun part starts. Usually there is a VNC console available from virt-install. This did not work for me when using a “real” remote VNC client. However, using “virt-viewer” did the trick. To use any kind of VNC, i’ve added the “vnc” parameter to the virt-install command.

Right after executing virt-install, call virt-viewer to jump into the installation process. In my case, i had to use a client computer running X, install xserver-xorg, xauth at the dom0 to run “virt-viewer”:

1
$ apt-get install xauth xserver-xorg

Check that SSHd on dom0 allows X forwards:

1
2
3
4
5
$ vim /etc/ssh/sshd_config
[...]
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost no

Connect to the dom0 machine from a client using X and enable forwarding:

1
2
$ ssh -X root@your.dom0
$ virt-viewer sles11-install

When installing SLES, make sure to install the bootloader to MBR (the default is the “root” partition!) and select the option to install within a paravirtualizied environment. After installation, you’ll also need vnc to boot the machine since SLES uses an awkward “graphical” boot console. You can do that by adding this to your domU config file:

1
vfb         = [ 'type=vnc' ]

Later on, modify the /boot/grub/menu.lst file and set

1
splash=0

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×