Custom VMWare Solutions

VMWare and other virtualisation systems provide strong web-based management UIs, which allow administrators to start, stop and suspend individual VMs. However, in applications such as software testing, the end-users often require a more tailored management interface.

Blueberry Customer ABC Ltd undertakes software localisation and localisation testing for large software development companies. Delivering localisation testing requires that ABC has access to test machines with all possible combinations of Windows OS version and language. With perhaps 4 current Windows versions and 30 languages, this means they need to have up to 120 variants available. In practice, users also need to install additional software on their test machines, and retain the test machine for the duration of the project. Because of this a template system is required: firstly 120 templates are created in all possible OS combinations. When a new project starts, an end user specifies which combinations are required, and the system makes copies of the virtual machines for the project. An engineer can then install the software under test on each VM. End users (testers) then access the VMs remotely using Terminal Services (RDP) or VNC.

To deliver this functionality, Blueberry built a custom VMWare management system for Rubric. Based on the free edition of VMWare, the web-based management system provides an interface that is structured in terms that ABC’s users understand – such as projects and languages. The management system also enforces business rules: end users are not allowed to see or interact with the template machines, and some level of load-balancing is introduced – if a VM is not used for 5 minutes, it is automatically suspended.

The ABC custom management system was implemented using a number of technologies. The web-based management interface was built in ASP.NET and SQL Server. On each VM host, a special script was written to provide a web-services wrapper for VMWare’s own perl API. Custom code was added to automatically generate NAT mappings for new VMs, so that end-users could transparently connect.

This system has been used successfully on a number of high-profile testing projects, and is currently managing three large VM servers.

Blueberry customer, UPC Broadband, based in the Netherlands, includes a software test team responsible for testing internal software systems. These systems normally include multiple database and application servers. With multiple versions of these systems deployed, and many different systems in development, the test team needed a way to manage testing resources. They were already using VMWare as a platform to host test systems.

For UPC, Blueberry created a custom management UI similar to the ABC system, but with a different focus. UPC needed to manage systems at the “cluster” level – they wanted to be able to start, stop and suspend a group of VMs that together made up a test setup for a particular application. They also had specific requirements for IP address allocation: because their systems often used load balancing, some of the servers within a cluster needed to have fixed IPs during a test. The UPC system used two main VMWare servers, and one control server running the management UI.

Blueberry customer ABC wanted to upgrade their application server hosting facility, and also achieve a greater level of redundancy after experiencing problems with server stability.

At the time, VMWare did not have the high-availability solutions that it has today, so Blueberry investigated whether there was some way to use VMWare to achieve server redundancy.

The solution adopted was to use two Linux host machines, and run the Linux DRDB network replicator software. DRBD allows a partition on one Linux machine to be continually replicated to a second machine. Every disk write that takes place on the primary server is also written to the disk on the secondary server.

Blueberry’s approach was to setup a VM partition on the primary server, which is then replicated by DRDB to the secondary server, which has exactly the same configuration. In the event of a hardware failure on the primary server, the VM images could simply be restarted on the secondary. The virtual machines would lose any data in memory, and they would effectively behave as if they had suffered a hard reboot.

In addition, we configured a night-time backup of the complete VM. This was achieved by suspending the VM on the primary and flushing the disk. DRDB was then disconnected temporarily and the backup image on the secondary server was mounted and copied to a backup file. Then DRDB was reconnected. The effect of all this is to make a nightly copy of a suspended VM image (so data in memory is also included) on the secondary server with only 2 minutes downtime of each VM.

This system has now been deployed for many years, and has run well during this time. At one point, a hardware failure on the primary server required the VMs to be started on the secondary, and the system worked as designed. The customer was pleased.

However, we did find some limitations of this approach. DRDB’s requirement for a dedicated partition made disk space management very time consuming, as changing the partition size meant copying 100GB of VMs to a different place. On our systems, which were using standard SATA drives to reduce cost, DRDB did have a significant performance impact.

In addition, technology does not stand still, and VMWare themselves have solutions that deliver high availability for VM images. However, VMWare’s approach seems to require that all VM images be hosted on a common SAN – which is both expensive and introduces a new single point of failure (the SAN).