Inchoate Curmudgeon

At some level, going to the Rails Conference has brought out the absolute worst in me. This is mostly because people ask me what I do, and I’m with an audience where I can really tell them.

In the keynote this morning, Cyndi Mitchell described the current state in IT as being bloated and corrupt. I live that life, and although I share the perspective of “bloated”, I think my spin on “corrupt” is closer to the meaning of the word as devolved and degenerate, as opposed to the one I think she meant.

My group at Sun is tasked with helping customers find ways of reducing space, power and cooling costs in data centers, which sounds helpful and nice, but it doesn’t really work out that way in real life. Increasingly, the situations we deal with are quite desperate. There is no more space or power. Data center power and cooling costs are bankrupting the companies they serve. By the time we get involved in these situations, and by the time my group shows up, we are facing a pretty tense crowd (mob?).

But here’s the problem. There is no silver bullet for these situations. The typical troubled data center is a sprawl of different architectures, configurations and nuances of the software and IT services that make them up, belied by even in the most seemingly homogeneous row-after-row of machine-after-machine, all just the same, data centers. If you could visually see how the applications and services in these operations are frequently constructed, it would look more like something from a Mad Maxx movie. The shiny machines are a thin veneer over the underlying rust and corruption.

The other thing you don’t see, when you look at row after row of shiny machine, is that all those machines, in aggregate, run at less than 10% utilization. Its like a national political convention without an air conditioner. What you are really looking at is an expensive bank of fans blowing hot air on stuff doing very little work. What a waste.

We actually can frequently replace large power hungry machines with smaller, more efficient equivalents, but these “tech refresh” approaches are likely to further reduce utilization; new machines are typically more powerful than the machines they replace.

Now, to be fair, not all operations run this way, but then my group never sees them. These operations never get in this mess. They have architectural governance and discipline. They have actual management of the technologies that their business operations require, at a much deeper level than simple vendor selection of software and hardware products. The other thing to be fair about is that most operations are at some future generation of the architectures they inherited: that is, the current operators can’t be fully accountable for the messes they’ve inherited.

What we see at a macro level is not a design, but a corrupt evolution. If what we see were a design, and if it really was actually designed, I like to I think it would have been designed at a happening. If the applications and services we see were designed, they would have been designed by a group of people that got together, dropped some acid, and drew it up on a bar napkin.

The result is, with apologies to Hunter S. Thompson, god rest his soul, is fear and loathing in the data center.

The point is, you don’t just consolidate this stuff without doing something about the chaos. There is no silver bullet. That fact doesn’t actually help our group at Sun that much, our customers and account teams don’t really want to face this, and both will persist in whatever they need to do to avoid “fixing” the situation. Its just too costly and painful.

Legitimately, many of the things that must be done to architectures to clean up the mess don’t add a lot of immediately visible business value. If you were call the business unit to talk about “fixing” his service in these situations, they really do not understand. Functionally, the Mad Maxx service is working fine for him. Don’t touch it. They remember, all too well, how hard it was to get the service up to begin with.

It’s refreshing to be at the Rails Conference, talking to folks about building brand spanking new applications. I pray that these new Rails applications, in some huge flood of data center Darwinism, will wash away a lot of the nastier things we see now.

This will happen only if we do a good job of deploying and managing Ruby in the data center at a macro architectural and integration level. This was the key message to me from Cyndi and Tim Bray in the keynote this morning.

So when I go on and on about deployment, security, backup and recovery, performance, scaling, management and monitoring, and as you fall gently into a restful slumber from my oh-so-boring diatribe, please humor me. I must retain some shred of hope for salvation from what we see in the data center now. Deploying new things like shiny new Ruby applications and services present huge opportunities to deal elegantly and effectively with this boring stuff.

Trackback URI | Comments RSS