ESXi 5.1 PowerCLI Host Install Script

I’ve been working on vCenter a lot this week. Here’s the script I use to configure hosts. It’s not the most awesome thing ever, but it might get you pretty far if you’re just starting out with PowerCLI, vSphereCLI, or esxcli.

It’s named Host-Install-Script.ps1 and it’s on my github repo: John Puskar’s Github Repo.

I’m pasting the content here for reference. I hope this helps you out.

# ESXi-Install-Script
# johnpuskar@gmail.com
# 02/02/2013

#PowerCLI
# Download -
# http://communities.vmware.com/community/vmtn/server/vsphere/automationtools/powercli
# Referece -
# http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.scripting.doc/GUID-7F7C5D15-9599-4423-821D-7B1FE87B3A96.html

#vSphere CLI (for snmp)
# Download -
# https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285

#VDSPowerCLI (no longer used)
# cmdlets download - http://labs.vmware.com/flings/vdspowercli
# http://blogs.vmware.com/vipowershell/2011/11/vsphere-distributed-switch-powercli-cmdlets.html
# not compatible with powercli 5.1!

#== Getting Started! ==

#== Variables ==
# Generic
$vCenterServer = "vcenter.domain.com"
$vmHostName = "vm1.domain.com"
$vSwitchName = "SAN-Switch"
$ntpHostname = "ntp.domain.com"
$snmpTrapReceiver = "opsmgr.domain.com"
$snmpTrapCommunity = "public"
$omsaPath = "/vmfs/volumes/san-esx0-lun0/VIBs/OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00/metadata.zip"
# Port Groups
$arrPGsToCreate = @()
$arrPGsToCreate += New-Object –TypeName PSObject –Prop (@{"Name" = "100_san1";"VLAN" = "100"})
$arrPGsToCreate += New-Object –TypeName PSObject –Prop (@{"Name" = "200_san2";"VLAN" = "200"})
$arrPGsToCreate += New-Object –TypeName PSObject –Prop (@{"Name" = "100_san1-vmk1";"VLAN" = "100"})
$arrPGsToCreate += New-Object –TypeName PSObject –Prop (@{"Name" = "200_san2-vmk1";"VLAN" = "200"})
$arrPGsToCreate += New-Object –TypeName PSObject –Prop (@{"Name" = "100_san1-vmk2";"VLAN" = "100"})
$arrPGsToCreate += New-Object –TypeName PSObject –Prop (@{"Name" = "200_san2-vmk2";"VLAN" = "200"})
$arrPGsToCreate += New-Object –TypeName PSObject –Prop (@{"Name" = "300_nfs";"VLAN" = "300"})
$arrPGsToCreate += New-Object –TypeName PSObject –Prop (@{"Name" = "400_vmotion";"VLAN" = "400"})
$arrPGsToCreate += New-Object –TypeName PSObject –Prop (@{"Name" = "401_ft";"VLAN" = "401"})
# VMKernels
$arrVMKsToCreate = @()
$arrVMKsToCreate += New-Object –TypeName PSObject –Prop (@{"PGName" = "100_san1-vmk1";"IP" = "x.x.x.x";"subnet" = "255.255.255.0"})
$arrVMKsToCreate += New-Object –TypeName PSObject –Prop (@{"PGName" = "200_san2-vmk1";"IP" = "x.x.x.x";"subnet" = "255.255.255.0"})
$arrVMKsToCreate += New-Object –TypeName PSObject –Prop (@{"PGName" = "300_pan";"IP" = "x.x.x.x";"subnet" = "255.255.255.0"})
$arrVMKsToCreate += New-Object –TypeName PSObject –Prop (@{"PGName" = "400_vmotion";"IP" = "x.xx.x";"subnet" = "255.255.255.0"})
$arrVMKsToCreate += New-Object –TypeName PSObject –Prop (@{"PGName" = "401_ft";"IP" = "x.x.x.x";"subnet" = "255.255.255.0"})
# iSCSI Targets
$arrIScsiTargetsInfo = @()
$arrIScsiTargetsInfo += New-Object –TypeName PSObject –Prop (@{"Address" = "x.x.x.x";"Type" = "send"})
$arrIScsiTargetsInfo += New-Object –TypeName PSObject –Prop (@{"Address" = "x.x.x.x";"Type" = "send"})
$arrIScsiTargetsInfo += New-Object –TypeName PSObject –Prop (@{"Address" = "x.x.x.x";"Type" = "send"})
$arrIScsiTargetsInfo += New-Object –TypeName PSObject –Prop (@{"Address" = "x.x.x.x";"Type" = "send"})
#NFS Targets
$arrNfsDatastores = @()
$arrNfsDatastores += New-Object -TypeName PSObject -Prop (@{"Name" = "vdr-backups"; "Path" = "/mnt/dataon1/vdrbackups/vdrbackups/"; "Host" = "10.146.232.113"})

#==== Do the Work ====
#Get the host password (for SNMP)
$rootPass = Read-Host -Prompt "Enter host root password" -AsSecureString

#Connect to vCenter Server
$VCUserCredentials = Get-Credential
Connect-VIServer -Server vCenterServer -Protocol "https" -Credential $VCUserCredentials

$vmHost = Get-VMHost -Name $vmHostName
$oCLI = Get-ESXCli -vmhost $vmHost

#Put the host in maintenance mode
Set-VMHost -VMhost $vmHost -State "Maintenance"

#Create the SAN virtual switch
$vs = New-VirtualSwitch -VMHost $vmHost -Name $vSwitchName

#Create the Port Groups
$arrPGsToCreate | % {New-VirtualPortGroup -VirtualSwitch $vs -Name $_.Name -VLanId $_.VLAN}

#Create SAN, vMotion, FT, and NFS vmkernels
$arrVMKsToCreate | % {New-VMHostNetworkAdapter -VMHost $vmHost -PortGroup $_.PGName -VirtualSwitch $vs -IP $_.IP -SubnetMask $_.subnet}

#Enable SSH
$vmHost | Get-VMHostService | where {$_.Key -eq "TSM-SSH"} | Set-VMHostService -Policy "On"
$vmHost | Get-VMHostFirewallException | where {$_.Name -eq "SSH Server"} | Set-VMHostFirewallException -Enabled:$true
$vmHost | Get-VMHostService | where {$_.Key -eq "TSM-SSH"} | Start-VMHostService

#Enable ESXi Service Console
$vmHost | Get-VMHostService | where {$_.Key -eq "TSM"} | Set-VMHostService -Policy "On"
$vmHost | Get-VMHostService | where {$_.Key -eq "TSM"} | Start-VMHostService

#Disable SSH Warnings
Set-VmHostAdvancedConfiguration -vmhost $vmhost -Name UserVars.SuppressShellWarning -Value ( [system.int32] 1 )

#Set NTP Server and Enable
Add-VmHostNtpServer -NtpServer $ntpHostname -VMHost $vmHost
$vmHost | Get-VMHostService | where {$_.Key -eq "ntpd"} | Set-VMHostService -Policy "On"
$vmHost | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Set-VMHostFirewallException -Enabled:$true
$vmHost | Get-VMHostService | where {$_.Key -eq "ntpd"} | Start-VMHostService

# Enable software iSCSI HBA
$oCLI.iscsi.software.set($true)
Sleep -s 10

# Add iSCSI Targets
$IScsiHba = Get-VMHostHba -vmhost $vmHost -Type "iscsi"
$arrIScsiTargetsInfo | % {$IScsiHba | New-IScsiHbaTarget -Address $_.Address -type $_.Type}

#Add NFS Datastore
$nfsDatastores | % {New-Datastore -Nfs -VMHost $vmHost -Name $_.Name -Path $_.Path -NfsHost $_.Host}

#Install Dell OMSA
Install-VMHostPatch -vmhost $vmHost -HostPath $omsaPath

#Configure SNMP
$expression = "perl ""C:\Program Files (x86)\VMware\VMware vSphere CLI\bin\vicfg-snmp.pl"" --server " + $vmHost.Name + " --username root --password " + $rootPass + " -t " + $snmpTrapReceiver + "@162/" + $snmpTrapCommunity
Invoke-Expression $expression
$expression = "perl ""C:\Program Files (x86)\VMware\VMware vSphere CLI\bin\vicfg-snmp.pl"" --server " + $vmHost.Name + " --username root --password " + $rootPass + " --enable"
Invoke-Expression $expression
$expression = "perl ""C:\Program Files (x86)\VMware\VMware vSphere CLI\bin\vicfg-snmp.pl"" --server " + $vmHost.Name + " --username root --password " + $rootPass + " --test"
Invoke-Expression $expression

#warn user of manual steps needed next
$msgs = @()
$msgs += "MANUAL STEPS REQUIRED:"
$msgs += " * Add vmnics to the vSwitches and Port Groups, and then test with vmkping."
$msgs += " * Bind vmk's to software iSCSI HBA."
$msgs += " * Give host's initiator access to LUNs on necessary iSCSI targets."
$msgs += " * Add host to VDS and configure dvUplinks"
$msgs += " * Migrate appropriate vmkernels to the VDS"
$msgs += " * Assign FT to the ft vmkernel"
$msgs += " * Assign mgmt traffic to PAN vmkernel"
$msgs += " * Assign vmotion to vmotion vmkernel"
$msgs | % {write-host -f yellow $_}

$go = $false
While ($go -eq $false)
	{$text = Read-Host "Type 'continue' when the steps are complete."; If($text -eq "continue"){$go = $true}}

# Configure round-robin multipathing policy on all iscsi paths
$oCLI.storage.nmp.path.list() | group-Object –Property Device | Where {$_.Name –like "naa*"} | %{$oCLI.storage.nmp.device.set($null, $_.Name, "VMW_PSP_RR")}

#Reboot host
Restart-VMHost -vmhost $vmHost -confirm:$false

#Exit maintenance mode
Set-VMHost -VMhost $vmHost -State "Connected"

# MANUAL STEP
# Attach update baselines
# Scan for updates
# Remediate updates

ESXi Errors – Failed write command to write-quiesced partition

I’ve been getting the following emails from all of my ESXi hosts since I’ve upgraded to 4.1 about 9 months back. I’d get 3-8 emails a day, and see large latency spikes on the corresponding datastore when the email was sent.


Target: vm6.chemistry.ohio-state.edu
Stateless event alarm
Alarm Definition:
([Event alarm expression: Host error] OR [Event alarm expression: Host warning])
Event details:
Issue detected on vm6.chemistry.ohio-state.edu in Chemistry Datacenter: ScsiDeviceIO: 2352: Failed write command to write-quiesced partition naa.6002219000a17f3d00003dcb4e0ccad3:1
(5:03:26:49.543 cpu1:5362)

I had engaged support from both VMWare for ESXi and Dell for my MD3000i’s. I’ve tried Jumbo Frames, Flow Control, different VLAN trunk configurations, etc. After many support calls and sessions, we found the problem on page 40 of the iSCSI SAN Configuration guide. In any situation where an iSCSI VMKernel can send data down a group of NICs, either because of a Virtual Distributed Switch, or multiple NIC’s assigned to an iSCSI port group, it’s mandatory to lock things down so that each VMKernel is assigned to send data through only 1 port group. Essentially, this forces multipathing from the network level up to the protocol level.

Resolution:

My VMKernels were on a VDS, so I had to perform the following operations:

  1. Open vSphere client, then navigate to Inventory -> Networking.
  2. Right click your first SAN\VMKernel Port Group -> Edit Settings.
  3. Click “Teaming and Failover”, and limit your active dvUplinks to only a single uplink. The rest should be placed under ‘unused’.
  4. Repeat this for every SAN\VMKernel port group.
Next, you must “bind” the iSCSI Software Adapter to the VMKernels:
  1. In vSphere client, find the name of your first host’s iSCSI adapter by choosing a host then clicking Configuration -> Storage Adapters. It’s typically vmhba34.
  2. Enable remote tech support mode and SSH to your first ESXi host.
  3. Run the following commands. After the first command, write down any vmk#’s that correspond with your iSCSI VMKernels.
    esxcfg-vmknic -l
    esxcli swiscsi nic list -d vmhba34
  4. If the ‘nic list’ command didn’t show any vmkernels, then you need to bind each iSCSI VMKernel with the following command:
    esxcli swiscsi nic add -n vmk# -d vmhba34
  5. When finished, run the following command to verify the work:
    esxcli swiscsi nic list -d vmhba34
  6. Repeat this for all hosts in your inventory.

Other Notes

After running these commands, it’s recommended that any unused dynamic and static iSCSI targets ne removed. However, the add\remove delay is faster with iSCSI bindings in place. For more info, see page 40 of the iSCSI SAN Configuration Guide 4.1 .

References:

ESXi 4.0 Standalone to 4.1

It’s update time! I updated my ESXi Enterprise Plus hosts with vCenter painlessly, but we have a single standalone ESXi host. So, time to fire up the vSphere host update utility and quickly update, right? Sorry buddy, that doesn’t work anymore: Cannot patch or upgrade ESX 4.0 hosts with vSphere Host Update Utility. The KB basically said that the host update utility was made to help people from 3.5 to 4.0, and that it doesn’t do much for 4.0 users. I swear I used it to patch my host to 4.0 U1, but maybe I’m crazy.

In any case, the new VMWare blessed upgrade process is relatively painless.

Process

  1. Download the “ESXi 4.1 Update 1 (upgrade ZIP from ESXi 4.0)” from the VMWare support site.
  2. Download the installer for “VMWare vCenter CLI“. Note: this is different than PowerCLI.
  3. Install VMWare vCenter CLI
  4. Run VMWare CLI, then the following commands:
    cd bin
    vihostupdate.pl --server <servername or IP> --install --bundle <pathToZIPfile>
  5. Enter your username\pass, then reboot when asked.
Good luck!

OpsMgr 2007 vCenter Management Pack b253

Hey all. I’ve updated my homebrew vCenter management pack. It’s quite beastly, and has the following features:

  • Diagrams your datacenters, clusters, hosts, and datastores.
  • Pulls a bunch host performance data.
  • Receives syslog events from the hosts (if set-up).
  • Pulls vCenter events.

I’m using this to track down some performance issues and warnings on my vCenter cluster. Getting this to work in your environment will be difficult, so please leave comments if you have a problem. I’ll answer as best I can. Future versions will generate alerts on discovery of misconfiguration. For now, check your vcenter server and opsmgr server ‘Operations Manager’ event logs. But, to get started check out the instructions on my previous post here: Operations Manager 2007 vCenter Management Pack (pre-alpha) .

Download: JPPacks.VMWare.vCenter (b253)

Operations Manager 2007 vCenter Management Pack (pre-alpha)

I’ve been working on a vCenter Management Pack for Operations Manager for a little over 4 weeks. I was driven to create this MP as an open alternative to nWorks since it’s so incredibly pricey! It doesn’t work nearly as well as nWorks, and it’s -really-, REALLY rough. I also haven’t done much documentation or testing. I’m posting it here in case anyone wants to poke around and test it out, or even help with the development! (wishful thinking, I’m sure).

Features left until 1.0:

  • Collect some Host and Datastore performance counters
  • Collect vSphere events

Features planned for 2.0:

  • Standardized, defensively programmed, optimized scripts
  • Documented MP Elements
  • Rules and Alerts on MP misconfiguration

To get it going:

  1. Install OpsMgr Command Shell and VMWare PowerCLI on your vCenter server.
  2. Install a VM for syslog collection. This will need OpsMgr Command Shell and VMWare PowerCLI as well. Configure the syslog receiver per this post: Monitoring ESXi Syslogs with OpsMgr 2007 R2 (Part 1).
  3. Import the MP.
  4. Add your vCenter Server and Syslog Collection server as ‘MOM Operators’.
  5. Change the discoveries to a shorter time if you’d like. Be careful though, they can cause significant load!

Download Here: JPPacks.VMWare.vCenter

For the download, see my latest post (sorry, you might have to search around a bit).

Good luck, let me know if it works for you. I’ll keep this post updated when I release new versions. It likely won’t be useful for at least a month.