Inchoate Curmudgeon

The availability of larger multi-core and multi-processor x86 systems brings powerful new tools into our architectural arsenals for reducing space and power in our data centers, and simplifying and upgrading our data center footprints. For virtualization, larger systems are attractive since they provide a larger pool of resources for a group of guest OS images to share, thus potentially increasing aggregate utilization by coalescing underutilized resources.

But what are the pitfalls that await those that leverage this new potential that we should be aware of? We need only look at the history of SMP on modern operating systems to find some clues.

Early SMP operating systems took some time to deal with scaling issues that prevented the efficient use of all available processing resources. These issues were overcome by threading the kernel itself. This was extremely difficult work and took some time to mature. The key problem spaces were kernel thread synchronization and resource scheduling.

Virtualization technologies will not have to deal with the full set of services and resources provided by a modern operating system, but two areas are likely to be interesting: CPU scheduling and virtualized device drivers.

Early OS versions ran virtually all kernel operations on a single CPU. This had the benefit of effectively of protecting and synchronizing all kernel resources, protecting these resources from corruption by the concurrent operation of multiple kernel threads; if only one thread runs at a time, a lot of problems are avoided. This approach doesn’t scale. Developing efficient and resilient multi-threaded CPU resource schedulers is probably one of the more delightful exercises kernel engineers have had to deal with over the years.

Scheduling guest OS instances onto the CPU is a key task for virtualization products. This is an area where we will need to understand the performance and scaling limits of these schedulers on larger multi-core machines.

One particularly interesting area remains an obstacle to scaling for operating systems: socket connection handling. The issue is scaling and threading the kernel code that synchronizes socket operations. The most troublesome area is around building up and tearing down these connections at extremely high rates. The net effect of this situation is a cap on socket connection rates that is completely independent of the number of processors that are available to do work. This cap varies from OS to OS. The net effect of the situation on architectural solutions is the use of web-farms and stand-alone TCP or HTTP and HTTPS offload engine appliances to get the workload spread out across multiple kernels and socket IO stacks.

Since many virtualization products supply virtual devices, it will be interesting to see to what extent these technologies are affected by scaling issues in IO processing.

None of these issues are likely to materially affect the most likely early beneficiaries of large-scale virtualization: development and functional test environments. Another likely unaffected target is low-volume internal and departmental web services.

From an architectural viewpoint, its important that we learn how to effectively leverage virtualization technologies on larger scale machines as quickly as possible. Data centers filling up, power is becoming more expensive, and processors are becoming more powerful and numerous, continuously driving CPU utilization rates down. Effectively leveraging large-scale, multi-processor, multi-core machines is becoming more a core-case than an edge-case. (Please excuse the pun. There might be two there.) It won’t be that long before we have 1U machines with dozens of cores.