Inchoate Curmudgeon

Red Hat CTO elaborates on lofty ‘cloud’ vision discusses the effort Red Hat is making to engage its customer base on Wall Street:

“In terms of engaging its customer base, for example, Red Hat has worked with Wall Street’s financial institutions to simplify data center operations by capturing a single operating system image, including hardware and software. The payoff: IT has only one image to update and manage, which can be deployed across the network, he said. Currently, Red Hat is testing this technology by burning an OS image onto a USB key and using it to boot up servers, desktops and laptops, he said.”

The thing to note that an effective virtualization strategy does not virtualize the existing, diverse landscape, it redeploys into a highly standardized infrastructure with uniform configurations. Furthermore, the effective process involves discovering the 80% solution, the small set of standard virtualization strategies and standards, that meet the vast majority of workload requirements for the target workloads.

Vladis’s Blog has just posted an entry on good and bad reasons to virtualize titled “Virtualisation, comfortable or tight trousers“. I’d like to elaborate on the subject a bit.

Vladis mentions several important drivers supported by measurable, direct IT operational cost reductions, typically enabled by:

• More efficient use of the existing server footprint through higher utilization,
• More efficient operations through reduction in the number of physical servers,
• Peak utilization of a large number of servers is low, and
• Reduction in power and space consumption.

The blog rightly recommends sticking to a “by the numbers” approach to determining suitability of your situation for virtualization, and also mentions the tools we have for helping customers with consolidation and virtualization planning. In fact, I happen to work for a customer facing group in Sun that focuses on this topic. We use these tools for customers on a daily basis.

Although “below the line” kinds of IT cost initiatives are typical, they are not the only good reason for what I prefer to call optimization. Why optimization? Because optimizing for direct IT cost metrics is a subset of the overall discipline of optimization, and a business may actually have other things it considers in the optimization equation.

Agility

Business concerns outside of IT may have strong impacts on optimization initiatives. The most prevalent drivers in this area revolve around needs to increase IT agility and capability to quickly meet business demands. The typical virtualization scenario here is the ability to deploy applications with configuration versus procurement; deploying a new application does not necessarily cause IT to go procure hardware, generally at least lowering the time to deploy new applications.

Strategic Advantage

Businesses that are able to lower barriers and costs to automate will create greater internal demand for new IT services. Businesses that operate with higher business capabilities derived from automation disadvantage their competition. This is a Red Shift thinking: IT is a competitive weapon.

Above the Line Thinking

The real ability of IT to accomplish these sorts of optimizations has a lot more to do with IT understanding their customers than technology, and some of the business process related issues can be somewhat stickier. IT departments that are structured entirely as cost centers are particularly disadvantaged: how does an IT department, constructed as a cost center, go about developing their department as a “business within a business” to better serve customer demand?

Being wildly successful with this sort of optimization requires more than bright ideas, it requires reconstructing the IT organization, IT thinking, and the IT financial model to support the effort and the revised thinking.

Nonetheless, success on the revenue side of the business equation starts simply. We, as IT professionals, need to stop looking at our feet. A lot of good flows from a little opportunity exploration, and in many cases, a simple by the numbers approach from the revenue side of the equation can quickly move mountains. Take a walk on the wild side, the Red Shift side, the making money side of the equation. Making the company money is a good and righteous lever.

The OMG Model Driven Architecture (MDA) explores the formal concept of transforming and instantiating models. “In the MDA, models are first-class artifacts, integrated into the development process through the chain of transformations from Platform Independent Model (PIM) through Platform Specific Model (PSM) to coded application.”

The “coded application” terminology is bit unfortunate, since the MDA concepts, due to the meta-model approach, could translate to infrastructure implementations: “realization artifacts” might be a better phrase. However, from the viewpoint that the implementation is realized entirely by configuration in emerging infrastructure implementations, such as VMware and Sun Dynamic Infrastructure (DI), the “coded application” wording may still apply.

This whole topic of modeling infrastructure architecture is somewhat messy, probably due in large part to the overwhelming preponderance of Java coders and architects in the architecture modeling domain. The concepts of architecture usually start and stop with software architectures, for Java what I call “inside the container thinking.”

I do know that I personally think the way that we enterprise and infrastructure architects “draw architectures” is completely unconstrained and rife with opportunities to misunderstand what we are “saying”. There has been more than you might think written about this, mostly concluding that the UML is the “best we have.”

However, very few people bother with specifying infrastructure architectures in UML, since using the best available diagram for viewing and manipulating the model, the Implementation Diagram, results in a very cluttered artifact for all but the most trivial cases. The other thing is that the none of the tooling I’ve used does a good job of constraining the implementation model to conform with the Class Diagram, which represents the meta-model for the implementation. As far as I know, this is the same for all modeling tools.

As a plain English example, I might model in a Class Diagram that specifies a server has a connection to two or more load balancers. This is fine for general constraints and specification, but when I draw an implementation model, there are no checks that my implementation actually specifies at least two load balancers. Not good.

The difference for us versus Java developers and architects is that they generally leap from Class Diagram to code. The Implementation Diagram is a representation of the state of a running application, modeling instances of the classes and their relationships. It’s not where they spend their cycles since they can just write the freaking code.

For infrastructure architects and infrastructure implementers, there is no such thing as an “infrastructure compiler”, that is a formal, unified, centralized human readable code that is transformed and realized as a gob of functioning infrastructure. Although some modern mechanisms go a long way, we remain mostly at the infrastructure corollary of wire-wrapping to implement code. We currently need implementation diagrams and other more concrete artifacts than Class Diagrams to tell various disciplines how to wrap their wires.

Going forward, we need a Platform Independent Model, a Platform Specific Model, and the “codes” that instantiate the model, all tied together. Modern provisioning systems centralize the control mechanisms.

The typical interest in CMDB with respect to provisioning is good, since at least has the capability to model the desired end-state. However, CMDB probably isn’t the right place to model the PIM, any more than it is the right place store Java Class Diagrams, unless we expand the notion of CMDB as it is currently used.

The current limitations with respect to implementing architectures with highly distributed configurations shift quite a bit with a combination of the “virtualization of everything” and the capabilties of modern provisioning systems, but we still lack the tools, and frankly the aggregate wherewithal I’m afraid, to make that final leap to “compile infrastructure specifications” into implementations. We have to stop looking at our feet. I imagine all the emerging provisioning and orchestration tools are intended to address this, but I can’t quite see it all tied together yet.

Some of the available MDA tooling might be worth a survey to see how they could be adapted to help realize infrastructure implementations. In particular, one tool I’m aware of, AndroMDA, is capable of accepting user-coded cartridges that can generate code fairly flexibly for different implementations. It’s used for Java, but I think may be agnostic enough for the task.

The vision is that we could generate XML and other text based configurations and specifications for an infrastructure implementation from UML models and diagrams.

Another possible approach is just modeling infrastructure meta-concepts and implementation concepts in a database and figuring out ways to transform them back and forth from PIM to PSM and XML. Some of the things that I and a colleague of mine are considering, that involves using RoR for in a provisioning scenario with DI and Solaris Provisioning System (SPS), would be a subset of this effort. The shortfall of course is visualization, or “diagramming”. We’d have models and transformations but still would lack a pretty “view”, and ways to manipulate the model visually, other than reports or web pages with text. Kinda boring. Perhaps this is just GUI sugar?

An intriguing aspect of ITIL change expected in a virtualization project is CMDB. I missed this on my recent post on ITIL and virtualization. It’s a funny slip, in the sense I’ve been pondering this subject quite a bit lately.

Virtualization concepts are increasingly realized in all the classic components of infrastructure: compute, storage, and networking. What impacts might you expect to CMDB for enterprises deploying virtualization technologies? What are the particular challenges with respect to CMDB and virtualization?

The pondering for me started with a combination training trip and workshop sort of exercise to a CMDB vendor recently. The workshop was more focused on the use of the CMDB as an embedded component of another facility, not a classic CMDB implementation in an enterprise. I was probably more interested in the general and typical utility of the product myself, not having much direct experience with CMDB implementations.

What struck me were the exchanges we had with the vendor with respect to modeling virtualized infrastructure and components. The issue is that this CMDB implementation, like most if not all others I imagine, really had no existing constructs to deal with many emerging virtualization concepts. Furthermore, we were having some trouble describing these technologies in a way that the vendor could discern how they should be modeled. A lot of this could be attributed to discussing abstract concepts that were not realized as physical components in the infrastructure. Grasping abstractions is just naturally harder than grasping boxes and wires.

There was no particular difficulty with the concept of a VLAN, or a service IP or VIP, this is relatively mature technology. There was much more trouble with more nuanced differences between technologies like Solaris Zones versus VMware or Xen type virtualization technologies, and some of the networking concepts like virtual NICs and the impact of real or virtual VLAN tagging on interfaces as they relate to things like link aggregation. I feel like we only scratched the surface insofar as discovering all the modeling challenges.

The problem with this as it is related to CMDB is that many of the core use cases involve understanding the impacts of change and the impacts of component outages on dependent services. If you can’t model relationships correctly, the utility of the CMDB is significantly eroded. Jamming a virtual thing into a CMDB as though it were a real thing probably won’t cut it.

There’s also an irony. The need for CMDB in IT operations is far greater when using virtualization than when you aren’t. As an example of the kinds of issues we face, in my little own three machine lab, I have an explosion of DNS entries related to dozens of virtual machines, and I can no longer remember if these things still exist and where they are or were realized. I have iSCSI LUNs that are used both as direct attach block devices to real machines and others that are used by VMware to provide storage to dozens of machines, and others that are direct attach to virtual machines *within* the VMware infrastructure. The mind boggles.

Discounting the completely unconstrained lab as a good example of our modern, discipled and constrained enterprise architectures (*cough*, *cough*), it seems there is still a vast gapping chasm between what we need in a CMDB and what can currently be readily realized, both in terms of our collective experience and understanding as well as the readily available tools.

The availability of larger multi-core and multi-processor x86 systems brings powerful new tools into our architectural arsenals for reducing space and power in our data centers, and simplifying and upgrading our data center footprints. For virtualization, larger systems are attractive since they provide a larger pool of resources for a group of guest OS images to share, thus potentially increasing aggregate utilization by coalescing underutilized resources.

But what are the pitfalls that await those that leverage this new potential that we should be aware of? We need only look at the history of SMP on modern operating systems to find some clues.

Early SMP operating systems took some time to deal with scaling issues that prevented the efficient use of all available processing resources. These issues were overcome by threading the kernel itself. This was extremely difficult work and took some time to mature. The key problem spaces were kernel thread synchronization and resource scheduling.

Virtualization technologies will not have to deal with the full set of services and resources provided by a modern operating system, but two areas are likely to be interesting: CPU scheduling and virtualized device drivers.

Early OS versions ran virtually all kernel operations on a single CPU. This had the benefit of effectively of protecting and synchronizing all kernel resources, protecting these resources from corruption by the concurrent operation of multiple kernel threads; if only one thread runs at a time, a lot of problems are avoided. This approach doesn’t scale. Developing efficient and resilient multi-threaded CPU resource schedulers is probably one of the more delightful exercises kernel engineers have had to deal with over the years.

Scheduling guest OS instances onto the CPU is a key task for virtualization products. This is an area where we will need to understand the performance and scaling limits of these schedulers on larger multi-core machines.

One particularly interesting area remains an obstacle to scaling for operating systems: socket connection handling. The issue is scaling and threading the kernel code that synchronizes socket operations. The most troublesome area is around building up and tearing down these connections at extremely high rates. The net effect of this situation is a cap on socket connection rates that is completely independent of the number of processors that are available to do work. This cap varies from OS to OS. The net effect of the situation on architectural solutions is the use of web-farms and stand-alone TCP or HTTP and HTTPS offload engine appliances to get the workload spread out across multiple kernels and socket IO stacks.

Since many virtualization products supply virtual devices, it will be interesting to see to what extent these technologies are affected by scaling issues in IO processing.

None of these issues are likely to materially affect the most likely early beneficiaries of large-scale virtualization: development and functional test environments. Another likely unaffected target is low-volume internal and departmental web services.

From an architectural viewpoint, its important that we learn how to effectively leverage virtualization technologies on larger scale machines as quickly as possible. Data centers filling up, power is becoming more expensive, and processors are becoming more powerful and numerous, continuously driving CPU utilization rates down. Effectively leveraging large-scale, multi-processor, multi-core machines is becoming more a core-case than an edge-case. (Please excuse the pun. There might be two there.) It won’t be that long before we have 1U machines with dozens of cores.