Inchoate Curmudgeon

One of the advantages of a VMware image over a real server is that you can securely access the Solaris virtual machine filesystem, from your Mac, with NFS. You don’t even need to plead with your system administrator, that would be you! Also, when you have a server vm, you can carry it with you, on your laptop. Bits are a bunch lighter than real servers. There are a couple of tricks to doing this in a way that is secure and manageable.

I’m doing this with Solaris Nevada build 64a, but most of this should work for Solaris 10 as well. Most of the differences will be with sharemgr. Consult Mr. Google for the proper methods for managing NFS on Solaris 10. But, this is development, use Solaris 11! Also, you may need to use Fusion for this, I can’t find an option to add a NIC to a Parallels vm. If there is one, this should work on Parallels too.

You might ask why would you do this? Well, if you are using the Solaris vm as a server test platform, you will want to be able to edit files on the virtual machine, even if you are using svn, Capistrano or some other mechanism to deploy files to your virtual Solaris server for testing. You may even want to edit source in-place on the virtual server, but who wants to use vi or gedit when you have TextMate? Who wants to grovel around the filesystem with cd all the time when you have Finder?

The first part of the exercise is to add a NIC to your Solaris virtual machine. We need this because we want the NFS communication on a private, closed network between your laptop and the Solaris VM on your laptop. This is less important for a home lab, which should be on a secure network behind a firewall. But if you have your Solaris vm on your laptop, and are carrying that around to other networks, you will want to protect the NFS communication.

There are alternatives to using a secure, private network for securing NFS, such as using an SNFS or even NFSv4, but these are fairly involved. The NFSv4 approach allows you to tunnel NFS simply though ssh, but on OS X it requires downloading OS X kernel source, then patching and compiling your kernel. Yeech!. Maybe some other time. (I’d really like to try the NFSv4 stuff, but the “Hail Mary” before you reboot your hacked OS X kernel is just too scary for me. I need my lappy!)

Another alternative is to use macfuse, but it has too many bugs yet, I think. I’m not real fond of kernel panics while I’m editing. SAMBA would be another way to implement this, and the only reasonable setup if you are doing something like this with a Windows machine and VMware Workstation, but that’s another blog entry.

Anyway, the option to add a NIC to your VM is fairly easy to find the Fusion machine settings dialog. You need to add a NIC and configure it as a NAT or Host Only adapter. I used NAT. All of this is assuming your first network adapter is configured as Bridged, which allows inbound communication, from other machines on the network, to your virtual Solaris server.

If you are adding the adapter to a 64bit vm, Fusion gives you an e1000 adapter by default. If you are using a 32 bit vm, stop that and change it to 64 bit! You have a big fancy Apple dual-core Intel 64bit laptop. Open the vmx file for your vm and change the line that says guestOS = "solaris10" to guestOS = "solaris10-64". I’m fairly certain you get a “pcn” adapter if you are using a 32 bit vm.

After you add the adapter and boot the Solaris vm, get a root command prompt. I use ssh from my laptop to connect. iTerm is also good. I pretty much don’t use the vm console. In fact, the image I’m using now is stripped down and doesn’t even include the X11 stuff. If you want to ssh as root, you need to edit /etc/ssh/sshd_config and change the line that says “PermitRootLogin no” to “PermitRootLogin yes”. No, DO NOT go around and do this to all your production machines and give your administrator a heart attack.

By the way, another good thing to do is to change your TERM, on your Mac from “xterm-color” to “dtterm” or “xterm”. Solaris doesn’t have “xterm-color” in its termcap database, and these other TERM settings work just fine on your Mac. If you don’t, vi is wonky on the the Solaris vm.

The reason you change it on your Mac instead of in the Solaris profile is that ssh is smart enough to inherit the TERM setting from your Mac environment to the Solaris box when you connect, and if you change it on your Mac, you never need change it again anywhere you connect to a Solaris machine. Very nice and DRY!

You can add a line like “export TERM=xterm” to the /etc/profile on your Mac. (Actually, /etc/bash_profile might be a better place for this, especially if you ever use sh or some other shell on your Mac.)

There is one more important point for setting up users on the Solaris vm. If you want the NFS home directory mounting to work as seamlessly as possible, use the same uid for Solaris users as your user on your Mac laptop. This is usually uid 501, but you should check. (In my case, its 503.) When you create users on your Solaris vm, you can specify the uid for the user you create, with something like:

useradd -u 503 -d /export/home/lou -s /usr/bin/bash lou

replacing 503 with your Mac userid, output from id and replacing lou with your user name. User names do not need to match between the Solaris vm and your laptop, but its less confusing I think. The uid, in this case 503, should match for the simplest NFS configuration. The root uid always matches, it is always 0 on every *NIX flavor I know of. All of this greatly simplifies your NFS configuration, since by default, your userids all map on all your systems.

Enough on user accounts. In order to configure the NIC and NFS, you need a root shell. Once you have a root shell on the vm, you can set up the new NIC.

[root@atlas /]# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=201004843 mtu 1500 index 4
inet 192.168.1.129 netmask ffff0000 broadcast 192.168.255.255
ether 0:c:29:9:cc:ba
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
[root@atlas /]# ifconfig e1000g1 plumb
[root@atlas /]# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=201004843 mtu 1500 index 4
inet 192.168.1.129 netmask ffff0000 broadcast 192.168.255.255
ether 0:c:29:9:cc:ba
e1000g1: flags=201000842 mtu 1500 index 5
inet 0.0.0.0 netmask 0
ether 0:c:29:9:cc:c4
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
[root@atlas /]# ifconfig e1000g1 DHCPdhcp start
[root@atlas /]# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=201004843 mtu 1500 index 4
inet 192.168.1.129 netmask ffff0000 broadcast 192.168.255.255
ether 0:c:29:9:cc:ba
e1000g1: flags=201004843 mtu 1500 index 5
inet 192.168.200.129 netmask ffffff00 broadcast 192.168.200.255
ether 0:c:29:9:cc:c4
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
[root@atlas /]#

Most of this is output from ifconfig -a, but the steps are:

1. Plumb the new interface: ifconfig e1000g1 plumb
2. Start DHCP on the interface: ifconfig e1000g1 dhcp start

Notice the IP address for the new interface 192.168.200.129. You should not be able to ping this from any other machine on your network. If you can, you haven’t set it in the Fusion configuration as NAT or Host Only.

Also note we are using DHCP, and Fusion is providing an IP address to the vm.

For now, we will put this IP in the hosts table in /etc/hosts on the Mac, so we can refer to the Solaris vm by name. This name should not be the same name you gave the machine on the primary interface. That would confuse name resolution. You can enable name resolution on the secondary interface by editing /etc/hosts on the Mac and adding a line like 192.168.200.129 carpo at the end of the file. carpo is the name I gave this vm on the private network. Note this name will only resolve on the Mac, but that’s ok, only the Mac can see the interface anyway!

Later, in another blog post, we will set up a caching DNS server for all of our zones and virtual machines, but for now, the entry in /etc/hosts works fine. If we keep going with this stuff, we’ll have an entire data center full of virtual servers in our Mac, right were we have absolute power to play with them. Oh, the geeky joy of it! But I digress.

Next we need to share our filesystem. If we don’t have NFS enabled, we need to start that service with svcadm -v enable -r network/nfs/server

If we have a home directory or other filesystem that we’ve established with zfs, such as described in “Amping up Zones and ZFS: Setting up ZFS“, we only need tell zfs to share the zfs filesystem of interest. We are going to add a parameter to make sure the NFS filesystem is only shared on the private network we just set up.

So, first I run a route -v get 192.168.1.129. This will tell me the IP address on the Mac interface that is bound to the private network. You should get something like:

[lou@rapture lou]$ route -v get 192.168.200.129
u: inet 192.168.200.129; u: link ; RTM_GET: Report Metrics: len 128, pid: 0, seq 1, errno 0, flags:
locks:  inits:
sockaddrs:
carpo
route to: carpo
destination: carpo
interface: vmnet8
flags:
recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
0         0         0         0         0         0      1500      1160
locks:  inits:
sockaddrs:
carpo 0.c.29.9.cc.c4 vmnet8:0.50.56.c0.0.8 192.168.200.1

Showing at the end that 192.168.200.1 is the IP address of my Mac on the private network. This also proves that I’m getting to that magic IP from a route on the private interface. This is good. You can ping the interfaces from both sides for good measure.

Next, zfs list shows my zfs filesystems.

[root@atlas /]# zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
zpool01           5.39M  4.89G    18K  /zpool01
zpool01/home      55.5K  4.89G  26.5K  /export/home
zpool01/home/lou    29K  4.89G    29K  /export/home/lou
zpool01/opt       5.17M  4.89G    21K  /opt
zpool01/opt/csw   2.62M  4.89G  2.62M  /opt/csw
zpool01/opt/misc  2.53M  4.89G  2.53M  /opt/misc

So if I want to share my /export/home/lou directory:

zfs set sharenfs=rw=192.168.200.1,root=192.168.200.1 zpool01/home/lou

where 192.168.200.1 is the IP for my Mac on the private network.

We are now ready to mount our home directory to the Mac! Open Finder, choose connect to server, and put in a url like nfs://carpo/export/home/lou, where “carpo” is the name we gave our private interface on the the Solaris vm in our /etc/hosts file and /export/home/lou is the zfs filesystem we are exporting. If everything is copecetic, we should get a Finder window with the contents of the home directory. W00t!

You should be able to edit files on your Solaris vm with TextMate, or your favorite text editor.

If you want to share your whole vm filesystem, again not recommended for production machines, simply share your root directory on the Solaris vm:

share -o rw=192.168.200.1,root=192.168.200.1 /

The url to mount this for my vm named carpo is nfs://carpo/. Now you can use TextMate on the whole Solaris vm filesystem. No more vi groveling. No X11 gnome for browsing the filesystem.

Several final nits, now that we have all this working, wen need to make a couple of things persist between reboots (or maybe not, depending on your needs).

We need to make the new interface we brought up come up every time we boot. There are two files to deal with:

1. Touch /etc/dhcp.e1000g1 to create an empty file.
2. Create and edit /etc/hostname.e1000g1, adding a single line with the hostname for the second interface. Remeber, this should not be the same as the entry in /etc/hostname.e1000g0

If you reboot now, you should have the secondary interface enabled automagiically after the reboot, with the same DHCP address it got before.

Lastly, we need to configure NFS permanently. First we create a share group: sharemgr create nfsprivate where nfsprivate is the name of a new share group. You can use any name you like. Next we set properties on this group. All NFS shares in this group share these properties. How DRY is that! What we want is all the shares in this group to connect only to the Mac laptop: sharemgr set -P nfs -S sys -p rw=192.168.200.1,root=192.168.200.1 nfsprivate.

Finally, we add our root share to this group: sharemgr add-share -s / nfsprivate.

That’s it, we can reboot and see if all this works!

Woo-hoo! Now I’m off to try out vnics in Solaris 11!

I’ve encountered one minor issue, so far. Since the Solaris installer is 32 bit, I needed to create a 32 bit vm for the install. VMware was blowing up trying to get Solaris to install inside a 64 bit vm. I’ll need to futz with the Fusion configuration to see if it can stand being 64 bit now that the installation is complete.

This popped up on the SunRay Users mailing list today. David R. Partington is running a VMware installation of 5 x4600’s hosting 450 simultaneous XP VM’s. SunRay provides the desktops for these machines through the Windows connector.

… We have close to 3000 Sun Ray’s in production at Ft Huachuca. We have some Solaris Desktops, but the majority is Terminal Server Desktops. The problem with terminal servers is that the individual students dont have individual Windows IP’s, and a few apps don’t work well terminal services. To solve this issue we conducted a couple of pilots running Windows XP VM’s with VMWARE ESX. The pilot was very successful and we were able to establish a baseline for scalablity. What we determined was that we could run 80 users on a single 4600. Our (5)4600’s have 64GB of Ram and 8 dual core CPU’s. Each 4600 has 8 4GB FCAL ports attached to a 27TB file Server. For each VM session, we have allocate 2GB of RAM and 70GB of Disk. On the five 4600’s we are provided 450 simultaneous XP VM’s. The XP VM are served out to the Sun Ray’s via the uttsc windows connector. The cost savings over a FAT client is 70%…

I’ll be watching this thread closely.

A virtual appliance is a network service that is preconfigured and prepackaged to do useful work.

We don’t buy dryer parts to construct a dryer, and we shouldn’t be constructing, in most cases, many components of our business systems. The only reason this practice persists is the nature of the business of buying and selling useful service components that involves the competing interests of hardware, operating system and software vendors. Customers have traditionally only had the option of either buying pieces of a solution and constructing it, or buying or using a service managed elsewhere, like SalesForce.com or Google Docs and Spreadsheets and JotDot.

But the success of VMware has exposed another approach that may change market forces. The ability to buy and install prepackaged stacks that include the operating system components and application components that do useful work is possible and practical with VMware. The value proposition of this is independent of VMware itself, but its more difficult to perform this sort of operation with a raw piece of hardware and a CD or ISO image. There are too many variables. It’s possible this same preconfigured service stack is what Oracle will attempt by branding it’s own Linux support.

The complexity of interactions between operating system and application configurations drives a proliferation of underutilized hosts, or complex hosts with a fragile mix of applications and services. Neither of these alternatives is desirable. Dealing with these issues is expensive, difficult and complicated. Underutilization is costly, and overly complex machines with many services on a single host are difficult to manage.

But the impediments to the virtual appliance approach are akin to the problems Apple faced with launching iTunes. These problems aren’t technical. They are business issues and risks. Suppose someone were to build a service appliance for content management that could be installed as a VMware instance. It could include the operating system, the application server and the database server, ready to go. The impediments and risks presented by Grady are important, but not the real problem. How would you deal with the business issues presented by supporting the varied underlying commercial components?

The typical licensing and support arrangements that exist now between software, operating system and hardware vendors will not suffice. The OEM arrangements that are formed between hardware vendors and Microsoft for delivering and selling bundled desktop machines are probably the closest thing that exists today to what is required. Might Microsoft or even Apple be interested in offering competitively priced VMware virtual machine desktop or server bundles?

The movement to appliance oriented delivery and installation of network services is probably inevitable. The question is who will make a business of it and how. The Google and SalesForce approach is currently to own and manage the whole stack and provide an IP address for the customer to connect to. Will this approach really succeed, or is there something to this virtual appliance approach to service integration and installation that will force the market in another direction?

One thing Apple figured out is that once they dealt with DRM and mirco-charging from a business perspective, the CD was doomed as a mechanism for delivering media. They didn’t need to own the record companies, or the artist labels or the distribution channels to succeed. The value proposition of electronic distribution couldn’t be stopped. Whoever figures out how to solve the business issues with delivering a useful, preconfigured service, built from multiple commercial vendor components, as a prepackaged ISO or VMware image, could cause some serious disruption to current hardware and software delivery models.

The cheap iSCSI target I’ve put together may have better performance than I expected. Vdbench tests with 128K sequential writes are between 5 and 10 MB/sec. The performance is more variable than it should be. There is likely some aliasing between various parts of the stack: network, VMware block-size, iSCSI target configuration and so forth.

I’ve taken an old Dell machine and made an iSCSI storage target from it. I added a firewire card so I could easily add drives. It’s using Ubuntu Linux 6.10 and the iSCSI Enterprise Target iSCSI target implementation. Instructions from the Ubuntu forum were good, I needed to add “sudo” in front of some of the commands since I wasn’t root. In particular “make install” almost always must be done are root, and I didn’t try to do this as a regular user. I didn’t need to install “dapper” and I’m not sure what that is, nor could I find it. I’m not a Ubuntu expert.

I got one strange looking error starting and manipulating the service, “Starting iSCSI enterprise target service: [: 142: ==: unexpected operator”. I changed the shebang on “/etc/init.d/scsi-target” from #!/bin/sh to #!/bin/bash and cleared that up I think.

In order to get the storage to work with VMware I needed to create a VMkernel network connection.

The whole thing isn’t going to beat any speed records, but I’ve got VMware iSCSI storage now.

Most of the lab work I’ve been doing on the x4100 and x4600 has been relatively straightforward. Looking ahead, most of the things that require thought and consideration are the same issues that plague data centers now.

• VMWare doesn’t by itself give you a lot of information about individual vm’s misbehaving except through reports and alarms. The Virtual Center charts are all by disk and by cpu and by network. That is, they are by resource not service.
• Keeping track of virtual machines as they proliferate is likely to be more difficult than keeping track of physical machines. How do I make the virtual light blink on the front of the virtual box in my virtual data center? How will I see it?
• Cloning is cool, and snapshots are very helpful, but the best way to build from scratch is scripted. I need a virtual jumpstart server.
• The behavior of Cicso VPN client in an RDP session is quite annoying. Basically, its a system service that needs attention at the console when you start it, so if you are remote, it doesn’t work. I hope there aren’t a lot of things out there like this, but I’m certain the next time I run into something like this in the “real world” it will take a lot of wasted time and effort to mitigate.

The x4600 is an 8 processor, 16 core AMD Opteron machine. If you’ve ever wondered what 70 vm’s on a single box would look like, wonder no more.

You’ll probably want to download it to get a closer look.

Vince Hill and I have been doing some lab work with the x4600 and ESX 3.01. One of the tests we have been running has ~50 Linux vm’s doing a kernel compile in a loop to stress CPU and ~10 Red Hat vm’s running the Medrec, a J2EE test application to stress some of the 32Gb of memory.

Not surprisingly, the box is pegged under this load, but what is surprising is how responsive it is despite the load. Medrec response is snappy and the compile times for the kernel compile tests degrade linearly and gracefully.

This picture is of the Virtual Center console graphing CPU utilization. We’ve fenced off CPU 0 for the ESX console and CPU 15 few a few “utility” machines for things like a Linux virtual firewall/router, a Solaris 10 instance with some disk load generation tools (vdbench) and mediawiki site for our notes.

We have a number of customers preparing to kick the tires on this configuration with vmware. We think we’ll see a LOT of these x4600 servers installed where customers are looking to reduce space and power consumption in their data centers with an excellent data center solution.

I’ve put together a lab with an x4100, VMware ESX, a SunRay and an OSX laptop. This provides the infrastructure in my home office, and a super platform for experimenting with various software and architectural components.

Key components are:

• Sun x4100 w/ 4 cores and 8Gb Memory
• VMWare ESX 3.01
• Solaris 10 x86 virtual machine (vm) running SunRay server
• SunRay 1G appliance
• Several Windows XP and Windows Server vm’s
• cAos Linux vm running DHCP, bastion SSH and caching DNS
• PowerBook G4 client
• Synergy keyboard/mouse virtualization
• Apple 23″ 1920×1200 Cinema Display
• SyncMaster 1024×768 Display

Key features:

• SunRay client to access Solaris 10 via X and Windows via RDP (uttsc)
• X on the Mac for access to Linux and Solaris desktops with xnest and rdesktop
• Synergy to share keyboard/mouse between SunRay and OSX

There are a lot of nits to go through, most of which I have workarounds for:

• Sadly, there is some incompatibility between the SunRay and my Apple Cinema Display. (Update, this now works!)
• Synergy software looses connection between OSX and the SunRay session when the screensaver activates.
• Cisco VPN can’t be launched from an RDP client.

The setup shows the technical feasibility for virtualizing Windows, Linux and Solaris desktops with VMware, using SunRay as a thin-client to access displays.

The typical method for virtualizing Windows instances uses Terminal Server or Citrix. This method deploys individual Windows virtual machines, typically Windows XP Professional, allowing users greater control over their “machine”.

I’m thinking about setting up the SunRay with a Windows session in kiosk mode for one of my daughters. If she can’t break it, I think its a good initial indication of usability.

The cloning capabilities of VMware make keeping “clean” installs of various base types a breeze, except for one small niggle: I’m running out of disk. I think my next project will be a white-box iSCSI or NAS server that VMware can use for additional storage.

I recently purchased an X4100 from my employer, Sun. I have a certification from VMware and wanted to hone my skills in the area. I’ve had quite a bit of success and (dare I say it) fun hacking away on the installation, creating virtual machines with various operating systems and so forth.

The fun ended today when I tried to install XP Professional from an upgrade distribution. So for those that might try this, I have several cautionary notes:

• buy the full version of XP, the aggravation isn’t worth it
• if you MUST use the upgrade, make sure you have an ISO of the installation CD of whatever Windows OS you are attempting to upgrade

So here is what I’ve found. To start with, the installation process either needs to be on top of an existing, upgradable installation, or you need the original installation CD/DVD for the OS you are upgrading from. No big surprise here, but I had hoped my old keys would suffice.

I seriously didn’t think this would be a problem. Despite who I work for, I’ve purchased many PCs over the years, as has my employer, so I didn’t think it sensible to pay full boat for XP Professional, currently running $300. So I bought the upgrade for $200.

So here’s some of the problems:

• The XP installer didn’t recognize any of the various OS CDs I put in my X4100 when it insisted during the install.
• The installer wouldn’t recognize any existing installation of these OSs.

That is, when the installer apparently didn’t recognize install CDs, I attempted to install these old OSes to see if XP installer would recognize any of them. It didn’t.

To be clear, some trouble was unavoidable. I was installing an OS on a VM, not a real machine.

At any rate, the procedure that finally worked was to create an ISO of one of my old Windows CDs, Windows NT Workstation specifically, and mount that virtual CD in the VMware tools when asked by the installer, rather than mounting the real CD on my X4100 when the installer asked for it.

Go figure.