Tuesday, February 07, 2006

Pacífica vs. Vanderpool

Virtualization is becoming a very important issue in the server market. One of the reasons is that to ensure availability of services, every computer has to be configured exactly as the service application recommends. If you have an Oracle database for critical information, you can't be fooling around installing every other nice feature on your server, everything in that server has to be conformant to Oracle's specification. Further than that, is actually desirable to have every computer running a critical service with a minimum of extra features, so that in the case of a problem, the process of locating/correcting it is greatly simplified. But at the same time, it is impractical to install three physical machines in a data center, each to do web serving, databases, and fileserving for a corporation. Virtualization is what enables the same physical infrastructure to be shared in a safe way among different services that each think have the whole computer for themselves. Try to understand this extending the concept of multitasking to computers: In multitasking, every task is not aware of the existence of other tasks, they just do their job, and the operating system is the one who switches tasks back and forth in the machine's processors for execution. With multitasking the computer seems to do many things at once. Equally, in vitualization, every operating system/service is unaware of the existence of others, they just do their job; but from the outside, the same machine behaves as if it were many different computers.

Virtualization can be done 100% in software, provided a little assistance from the operating systems that will run in that computer. An example of this is VMWare's line of virtualization software, and the excellent open source Xen. But of course, there are limitations, and help from hardware is so crucial that can determine whether a particular server machine can be partitioned into as many virtual servers as needed successfully or not.

Intel offers Virtual Technologies, VT, codenamed "Vanderpool", and AMD offers "Pacífica". Since this is an issue of relevance for the medium/long term x86 server markets, it is good to make a comparison among them.

There is an excellent set of articles published in "Theinquierer.net" about this subject, in three parts: 1, 2, and 3.

In part 1, we find the following:
"VT['s memory management] is a software solution [...]. As with most software virtualisation techniques, it is quite costly compared to doing the same thing in hardware. [...] VT manages memory in software."
Whereas Pacífica is a hardware solution that allows interesting advanced features without equivalent in Vanderpool such as the modes of "Shadow Page Tables and Nested Page Tables".

Second part explains that Pacifica's Shadow Page Tables (SPT) mean that all accesses to the microprocessor's page directory address (where the actual translation between addresses the tasks require and the physical adresses), or CR3, are shadowed by Pacifica, which activates the Virtual Machine Manager (VMM) to put the right value into the CR3 that each virtual machine sees.

In the third part, it is explained that Nested Page Tables (NPT) adds another level of indirection, but in hardware, to account for the Virtual Machine Management. Therefore, there are three levels: Application, that manage their own space, the Operating System, which manages many application's memory, and the VMM. Since this is done in hardware, the performance hit can be made negligible, for instance, through caching.

It also mentions the Device Exclusion Vector, a table that allows/forbids devices to put data through DMA into memory when the wrong virtual machine is running. This is another result of Hypertransport/Direct Connect Architecture.

The article summarizes:
"Looking back over the Pacifica spec, it is clear that it is indeed a bigger body of water than a Vanderpool. The basic architecture of the K8 gives AMD more toys to play with, the memory controller and directly connected devices. AMD can virtualise both of these items directly while Intel has to do so indirectly if it can do so at all."


Anonymous said...

Awesome explanation. AMD's implementation remdinds me of how IBM did it for the old 370 mainframe evolution from its 360 architecture. Great blog - please keep it up.

Eddie said...

My honor!

Anonymous said...

Many an IT manager would make sure that any investment in hardware needs to take into account the Hardware advantages of the "Pacifica" aware AMD opterons in any Virtulization system that many will be implementing after a blitz from Intel with their vapour ware VP "Demoed with Vmware"
Thanks Intel.

knapp said...

One question I've yet to resolve about hardware virtualization: Caches and memory allocations, instruction units, and other resources are designed to optomize execution speed. E.g. Caches are designed to minimize misses and maximize speed. With a virtualized system, is there a loss of optimal loading? I.e. does virtualization cause a significant performance hit by fundamentally dividing the machines attention?

Eddie said...

Knapp: the answer to your question:

Virtualization effectively divides the same hardware among the multiple virtual computers, and introduces an overhead. In the case of Pacifica, the overhead is miminal.

In specific:

Caches: The "working set" is as large as the sum of all the working sets of the many different virtual computers, roughly speaking, if a computer is divided in two virtual computers with the same kind of tasks, you may say that each virtual computer will behave as if the cache was half the size.

This is not as scary as it seems at first sight: Many virtual computers reside in the same hardware because they shouldn't have high loads simultaneously, otherwise, it would make sense to have the physical multiple computers.

If the multiple virtual computers have substantially different usage patterns, then an effective sharing of the physical resources is achieved and it is more practical and cheaper to go the route of virtualization.

George said...

You canot talk about virtualazation without VMS