Original MRI on Linux

  def build
    pid = fork { run_command }
    Process.detach(pid)
  end

  def run_command
    `sudo /opt/script/deployserver/setupnewserver.sh -p #{poolserver} -i #{ip} -s #{@size} -v #{@vlan} -a "#{@owner}" -n #{@name} -e "#{@email}"`
  end

IronRuby
We tried IronRuby and the same bit of the script broke as on win32 MRI (though I was pleased and surprised that Sinatra worked)

  def build
    WindowsProcess.start "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
"-PSConsoleFile \"C:\\Program Files (x86)\\VMware\\Infrastructure\\vSphere PowerCLI\\vim.psc1\" \"& C:\\script\\DataStoreUsage.ps1\""
  end

Using the following DotNet code

class WindowsProcess
  def self.start(file, arguments)
    process = System::Diagnostics::Process.new
    process.StartInfo.FileName = file
    process.StartInfo.CreateNoWindow = true
    process.StartInfo.Arguments = arguments
    process.Start
  end
end

Workaround using Windows “start” command
I had hoped the module at win32utils would let me just use the original script but fork did not work properly still.

def build
  commandstr = "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe -PSConsoleFile \"C:\\Program Files (x86)\\VMware\\Infrastructure\\vSphere PowerCLI\\vim.psc1\" \"& C:\\Sites\\vmdeploy\\PrepNewMachine.ps1 -type #{@type} -machinename #{@name} -size #{@size} -vlan #{@vlan} -creator #{@owner} -creatoremail #{@email} -ipaddress #{ip}"

  system ("start #{commandstr} > ./log/#{@name}.log 2>&1")
end

This uses the windows “start” command and works pretty well.

Share on Facebook

You only need two things – the built-in “sc” command and an executable from Windows Server 2003 Resource Kit Tools called srvany (works with 2008 too). Get just that exe here (if you trust me of course )

Creating the service

Check it exists

Set Parameters In The Registry
Configure it at HKLM/SYSTEM/CurrentControlSet/Services/APPNAME/Parameters

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VMdeploy\Parameters]
"Application"="C:\\Ruby192\\bin\\ruby"
"AppParameters"="C:\\Sites\\vmdeploy\\server.rb -p 80"
"AppDirectory"="C:\\Sites\\vmdeploy"
"AppEnvironment"=hex(7):65,00,78,00,61,00,6d,00,70,00,6c,00,65,00,3d,00,32,00,\
  37,00,00,00,62,00,6c,00,61,00,68,00,3d,00,63,00,3a,00,5c,00,74,00,65,00,6d,\
  00,70,00,66,00,69,00,6c,00,65,00,73,00,00,00,00,00

Note the AppEnvironment is a multiline string, the rest are strings

This lets you run any executable file, change the directory you run it from and pass any arguments or environment variables so should cover most use cases.

I will be sharing the code for both the Sinatra app and the PowerShell deploy script in later posts.

Share on Facebook

Will #VMware Load-Based Teaming remove the need for #Cisco EtherChannel? Discuss….

I long ago investigated NIC Teaming algorithms and settled on IP Hash with Cisco Etherchannels for most environments, only really using something else if the client happened not have stacked switches. Thanks to Scott Lowe for this superb article on the matter.

When vSphere 4.1 came out with Load Based Teaming, I was pleased that at last we had an algorithm that would have a go at proper load balancing and not just load distribution but had not got round to investigating much more.

At Forward we have just updated to 4.1, Enterprise Plus and have bought some shiny new Extreme Summit X650 Series 10G switches; so Simon’s comment was particularly apropos.

I had decided I wanted to try and use LBT but was unsure if I should port-channel the uplink ports. It turns out you can’t. I thought maybe you should to be honest, it does not mention in the dvSwitch guide as far as I can see but the ESX host requirements for link aggregation KB (updated today) is very clear

• The switch must be set to perform 802.3ad link aggregation in static mode ON and the virtual switch must have its load balancing method set to Route based on IP hash.
• Enabling either Route based on IP hash without 802.3ad aggregation or vice-versa disrupts networking

ie you need both IP Hash and EtherChannel and neither will work without the other.

In answer to Simon’s question, my feeling is you may still get better performance from EtherChannel and IP based hash for some workloads but would guess “usually” LBT wins. I think the case where you may get better utilisation is when certain VMs have very high bandwidth requirements to different IPs. As described here IP Hash is the only way to allow traffic from one vNIC to leave over different pNICs at the same time.

It is interesting that even with LBT bandwidth is still limited to the maximum bandwidth a single pNIC can provide for individual VMs / vmkernels, also IP hash will not get higher than a single pNIC for a vMotion or other point to point connections. So 10G is going to perform better for these operations than 10x1G, however you team them.

Share on Facebook

In the 3par storage I could see that IO was unusually high of late.

Then I looked at the top-n datastores by IOPS and graphed them

A huge jump for sharedstorage8, so I looked at its VMs

and found the culprit VM.

Here it is against our big “Superhero” database and the vCenter server with the DB.

A lot of IO from a machine the owner thought was doing nothing!

Hyper9 is a pretty good tool for reporting, alerting and troubleshooting your VMware infrastructure, the query language is lucene based and this gives you lots of options in creating custom views and alerts.

Share on Facebook

$reportArray = @()

foreach($cluster in get-cluster){
    foreach($vm in get-vm -Location $cluster){
            foreach($ds in get-DataStore -VM $vm.name){
                $reportArray += New-Object PSObject -Property @{
                     ClusterName = $cluster.Name
                     VMName = $vm.Name
                     DatastoreName = $ds.name
                     FreeSpaceMB = $ds.FreeSpaceMB
                     CapacityMB = $ds.CapacityMB
                      }
                echo $cluster.name $vm.name $ds.name $ds.FreeSpaceMB $ds.CapacityMB
                        }
                   }
            }        

$reportArray | Sort-Object -Property ClusterName,VMName,DatastoreName | Export-Csv "DataStoreUsage.csv" -NoTypeInformation -UseCulture

The important bit is

$reportArray += New-Object PSObject -Property @{
                     ClusterName = $cluster.Name
                     VMName = $vm.Name
                     DatastoreName = $ds.name
                     FreeSpaceMB = $ds.FreeSpaceMB
                     CapacityMB = $ds.CapacityMB
                      }

I cant remember where I saw it but it creates an object with properties with the information I want to use that I can then sort, filter and send places.

I used it again today:

$reportArray = @()

Get-VM | % {$vmguest = Get-VMGuest -VM $_ ;
			$reportArray += New-Object PSObject -Property @{
				VMName = $_.name
				GuestOS = $vmguest.OSFullName
				IPAddress = [string]::Join(" ", $vmguest.IPAddress)
				HOSTNAME = $_.guest.hostname
     			}
			}

$reportArray | Out-GridView

Using the lovely Out-GridView commandlet

***UPDATE***
Turns out you need this update for the New-Object PSObject -Property bit to work.

Share on Facebook

Simple things should be simple, complex things should be possible.

The simple:
I want to go and set all the LUNs in VMware to use “RoundRobin” as the MultipathPolicy

My first attempt was,

Get-VMHost | Get-ScsiLun |
    Set-ScsiLun -MultipathPolicy "RoundRobin"

Loop through the hosts, loop though each of its LUNs,
set MultipathPolicy to “RoundRobin”

Which works, but was taking ages. I was about to make separate scripts for each server when I realised I was not quite doing what I wanted with that script. I really wanted to:

Loop through the hosts, loop though each of its LUNs,
*if it was not already RoundRobin*,
set MultipathPolicy to “RoundRobin”

Get-VMHost | Get-ScsiLun |
    where {$_.MultipathPolicy -ne "RoundRobin"} |
    Set-ScsiLun -MultipathPolicy "RoundRobin"

Which ran loads faster as the “set” command still ran and took a while even if you were setting it to what it already was.

The Complex:

I took the wonderful vcheck script and made it run every night. Great email overview of the virtual infrastructure each morning.

Share on Facebook

I have just read an article on roughly how many IOPs you can expect from a single disk and encountered what I consider to be a frequently repeated mistake in the calculation.

Before I begin I want to point out that it is only an approximation anyway and caching in enterprise storage systems makes it perhaps a moot point anyway.

The article is here if you want to go and see it.

He says there are 3 factors that influence the number of random IOPS you can do with one disk (the assumption throughout is that these are random)

and I agree, but would like to point out that 1 and 2 are related:

Average latency = how long 1/2 a turn takes

So, a 15000 RPM disk:

1/2 a turn takes 2ms (your average latency)
And lets say for arguments sake your seek time is 5ms

How do you work out IOPS (IO operations per second)?
There are 1000ms in a second so

where t is the average time taken to do an IO operation.

For example:
If the average IO takes 5ms, you would have 200 IOPS as you could do 200 of them in 1 sec (200×5=1000)

EVERYONE AGREES UP TO HERE

What is more important, latency or seek time?
Well you have to do both, wait for the disk to spin into position and move the head. These two things are in parallel and both need to happen so I would say your average IO takes the worst of the two, ie
t = max{seek time,average latency}
I have seen people say that
t = seek time
Which matches my thoughts with all disks I have ever seen

Or that
t = latency
as a max, as you can get heads to match any latency you like, but it is expensive. (Jeff Bonwick says so here )

The thing that I hate seeing is:
t = latency + seek time
By adding those two times together you are saying that your average IO time is the time it takes to do both bits added together. The only physical interpretation of this is that they happen one after the other with no overlap, which is clearly not true.

This fallacy is made explicit here on ZDnet

Since the overall access time is determined by the sum of the average rotational latency (2ms) and the average seek time (3.7ms), this high-end 15000 RPM hard drive has an average access time of 5.7 milliseconds.

Why the sum?

It comes up unchallenged in Duncans otherwise excellent article about the RAID IOPS penalty (interestingly he references the same article, saying

In short; It is based on “average seek time” and the half of the time a single rotation takes. These two values added up result in the time an average IO takes.

Again, why the sum?

I think it does not really make all that much difference, so long as you have a rule of thumb for roughly how many IOPS a disk has and understand how the way you combine them in RAID impacts it.

Share on Facebook

]]> http://www.thattommyhall.com/2010/02/15/iops/feed/ 0 http://www.thattommyhall.com/2009/09/28/getting-vmware-certified-professional-vcp-on-vsphere-4/ http://www.thattommyhall.com/2009/09/28/getting-vmware-certified-professional-vcp-on-vsphere-4/#comments Mon, 28 Sep 2009 16:17:27 +0000 tom http://www.thattommyhall.com/?p=88 On Saturday I took and passed the VCP410 exam to get VCP4.

It was not really that hard, though I have been reading about vSphere since before it shipped, follow loads of blogs on VMware, installed it as soon as it came into beta and migrated my companies clusters to it relatively early. I would say if you have VCP3 and have used vSphere you should be OK.

The frustrating thing about the exam was the questions on the config max document, in my view if you are approaching the maximums you could just look it up and memorisation is a pointless exercise. A lot of the maximums are just decisions someone in vmware made, how many NFS stores by default ? (8), max? (64). What it the tree-depth per resource pool? (12… unless you use DRS, then it’s 10). This kind of memorisation is stupid, pointless, hoop-jumping and will be the difference between passing and failing for lots of people.

The exam (like most IT certs) is multiple choice so the questions are fairly mundane and of course there is only 1 correct answer. When interviewing candidates, I always prefer questions that start “what is your” rather than “what is” as anything that is so unsubtle as to only have one answer is probably too uninteresting to spend time discussing.

I did do a nights worth of revision however, using:

• Vmware’s mock exam
• Simon Longs practice exams
• The config max doc – YAWN
• My own mind map (using FreeMind, or use the pdf version) of config max and a few of Simons questions
• vReference’s card

Also worth considering are

• Trainsignals videos (I may still get myself a copy)
• The Exam Blueprint
• The VMware Doc’s
• vReference’s vSphere4 notes (these look excellent but I did not have time to read them)
• Mike Laverick’s book VMware vSphere 4 Implementation (I have it pre-ordered)
• Scott Lowes book Mastering VMware VSphere 4 (seems to be the most popular one released so far)

You may like to see the things I have added to delicious on vmware over the last few years.

Good luck if you take it too!

Share on Facebook

]]> http://www.thattommyhall.com/2009/09/28/getting-vmware-certified-professional-vcp-on-vsphere-4/feed/ 2