» Publishers, Monetize your RSS feeds with FeedShow: More infos (Show/Hide Ads)
Date: Tuesday, 09 Feb 2010 22:52
Author: "Hugo Phan (hugo@vmwire.com)"
Date: Tuesday, 29 Sep 2009 23:53
The scenario:
A customer has a non-critical HP server that they would like to P2V. It is installed with 2 x SATA disks without a SATA RAID controller, runs Windows Server 2003 and uses software RAID 1 mirror over the two disks which are set as dynamic. On top of this, the mirrored disks are split into two logical partitions, C: and D:.
Breaking the mirror and performing a hot P2V using VMware Converter 4.0 Standalone, with the two volumes being P2V'd into two separate .VMDK files fails at 95% during the reconfiguration phase.
If you receive a failure at 95%, it just means that the reconfiguration has failed due to VMware Converter not being able to find the system partition, that actual data copy has successfully completed and the data is intact. Obviously the virtual machine will not boot so how can we fix this?
The solution:
- Boot the virtual machine, select F2 to go into the virtual machine's BIOS and make sure that the VM is booting from the correct virtual disk.
- Boot the machine into a disk management software like Acronis Disk Director Suite or similar and convert the partitions from logical to primary partitions and then select the C: partition as the active partition.
The virtual machine will now boot successfully.
Author: "Hugo Phan (hugo@vmwire.com)" Tags: "primary partition, dynamic disks, 95, er..."
Date: Saturday, 13 Jun 2009 17:37
I'm working on a paper, document, anything, (probably just this post now since my schedule is so busy) on something that's been in the back of my mind for a while now, and every time I speak to a new opportunity or a customer I always wished that I had something substantial to leave behind to show that yes, it is possible to achieve the desired future state without pain.
What I'm talking about is how to get from A to Z without pain, fear, risk, or increased cost and time.
'A to Z' is an expression that we all use, but in Lehmann's terms it is getting to the desired future state from the current state.
What is the future state? For example a server migration project of 1000 Wintel servers into VMware infrastructure in 6 months.
So if A is the origin and Z is the destination, then the journey of getting from A to Z is the experience. It is the experience that is all too important. In a project's lifecycle, the primary purpose of a project is to bring benefit to something (an organisation for example). But the experience can vary dramatically. Z can be achieved but at what cost? Z can be achieved but it could take a long time. Z can also be achieved but after how many mistakes, issues and actions that were required to achieve Z?
Is there a way to define the experience? To reduce the amount of risk and unplanned change, to limit the exposure to mistakes and unknowns. To cap the amount of time and cost to achieving Z. 'The Way' then is called a methodology. A methodology is a collection of processes and frameworks which are used to control the execution of change within a project.
So while I’m in a pessimistic mood, let’s go over why there are difficulties from having a comfortable journey:
• Lack of planning
• Lack of clear objectives
• Lack of support and acceptance (See Steve Chamber's Barriers to Virtualisation)
• Lack of risk management
• Lack of a business case or project justification
• Lack of change control
Why is change so feared?
Let's assume that your project justification and initiation are all good and that your project plans, objectives, business case and RAID are all up to scratch and now you are ready to embark on a project that changes your IT infrastructure. Have you considered how you will manage change? Are there push backs from the business or application owners who don't really need or want anything to happen to their precious server due to changing the way a workload is run?
How can you alleviate their fears and introduce controlled change?
So let's take the classic CIO/IT Director from a few years ago at a time when x86 consolidation using virtualisation was still in its infancy (there are those that still think transitioning to a virtual infrastructure is a risk too far). These CIOs had fears around change - change of management, change of skills, change of processes and changes with operations. These fears were prevalent then and are still prevalent now. In my view the main enablers for change are the frameworks that can be used to get from A to Z.
Without change, an IT organisation will never be able to evolve into an IT organisation that has more reliable infrastructure, more efficient processes and more streamlined operations. Those companies that do embrace change and evolve are considered to be the most high performing IT Organisations.
The consensus is basically this: change causes fear, therefore projects such as P2V take forever to do, and without the correct methodology your P2V project could fail before it has actually begun. But by introducing controlled change and then putting the processes and governance in place; the strategy controls, manages change and provides a framework for effective management and delivery of the project.
The barrier to evolution is due to a fear of change, we alleviate this fear by controlling change. Change then becomes the enabler for evolution: please welcome Change Evolution.
So what is Change Evolution?
Change Evolution is a framework that uses ITIL/Visible Ops methodologies to control migration to virtualisation projects. It expedites ROI due to enablement of change management as part of BAU/Operations.
Change Evolution is a framework for delivering projects with
What I'm talking about is how to get from A to Z without pain, fear, risk, or increased cost and time.
'A to Z' is an expression that we all use, but in Lehmann's terms it is getting to the desired future state from the current state.
What is the future state? For example a server migration project of 1000 Wintel servers into VMware infrastructure in 6 months.
So if A is the origin and Z is the destination, then the journey of getting from A to Z is the experience. It is the experience that is all too important. In a project's lifecycle, the primary purpose of a project is to bring benefit to something (an organisation for example). But the experience can vary dramatically. Z can be achieved but at what cost? Z can be achieved but it could take a long time. Z can also be achieved but after how many mistakes, issues and actions that were required to achieve Z?
Is there a way to define the experience? To reduce the amount of risk and unplanned change, to limit the exposure to mistakes and unknowns. To cap the amount of time and cost to achieving Z. 'The Way' then is called a methodology. A methodology is a collection of processes and frameworks which are used to control the execution of change within a project.
So while I’m in a pessimistic mood, let’s go over why there are difficulties from having a comfortable journey:
• Lack of planning
• Lack of clear objectives
• Lack of support and acceptance (See Steve Chamber's Barriers to Virtualisation)
• Lack of risk management
• Lack of a business case or project justification
• Lack of change control
Why is change so feared?
Let's assume that your project justification and initiation are all good and that your project plans, objectives, business case and RAID are all up to scratch and now you are ready to embark on a project that changes your IT infrastructure. Have you considered how you will manage change? Are there push backs from the business or application owners who don't really need or want anything to happen to their precious server due to changing the way a workload is run?
How can you alleviate their fears and introduce controlled change?
So let's take the classic CIO/IT Director from a few years ago at a time when x86 consolidation using virtualisation was still in its infancy (there are those that still think transitioning to a virtual infrastructure is a risk too far). These CIOs had fears around change - change of management, change of skills, change of processes and changes with operations. These fears were prevalent then and are still prevalent now. In my view the main enablers for change are the frameworks that can be used to get from A to Z.
Without change, an IT organisation will never be able to evolve into an IT organisation that has more reliable infrastructure, more efficient processes and more streamlined operations. Those companies that do embrace change and evolve are considered to be the most high performing IT Organisations.
The consensus is basically this: change causes fear, therefore projects such as P2V take forever to do, and without the correct methodology your P2V project could fail before it has actually begun. But by introducing controlled change and then putting the processes and governance in place; the strategy controls, manages change and provides a framework for effective management and delivery of the project.
The barrier to evolution is due to a fear of change, we alleviate this fear by controlling change. Change then becomes the enabler for evolution: please welcome Change Evolution.
So what is Change Evolution?
Change Evolution is a framework that uses ITIL/Visible Ops methodologies to control migration to virtualisation projects. It expedites ROI due to enablement of change management as part of BAU/Operations.
Change Evolution is a framework for delivering projects with
- less Risk
- less Time
- less Cost
Author: "Hugo Phan (hugo@vmwire.com)" Tags: "visible ops, migration, p2v, methodology..."
Date: Monday, 18 May 2009 11:46
1. INTRODUCTION
1.1 Overview
The VI Toolkit (for Windows) provides a powerful yet simple command line interface for task based management of the VMware Infrastructure platform. Windows Administrators can easily manage and deploy the VMware Infrastructure with a familiar, simple to use command line interface.
The VI Toolkit (for Windows) is a tool that system administrators and developers can use to automate the management of VMware Virtual Infrastructure. With the VI Toolkit (for Windows), many tedious and time-consuming tasks can be completely automated in as little as one line of code.
The VI Toolkit (for Windows) takes advantage of Windows PowerShell and .NET to bring unprecedented ease of management and automation to the Virtual Infrastructure platform. The VI Toolkit (for Windows) provides 125 PowerShell cmdlets that cover all aspects of Virtual Infrastructure management.
Some common tasks that the VI Toolkit (for Windows) can be used to perform include:
The following platforms are supported by the VI Toolkit (for Windows):
The following platform combinations are supported by the VI Toolkit (for Windows):
The following tables lists the software pre-requisites and the location to each installer. This guide focuses on the most recent releases as dated 05/02/2009, which are Windows PowerShell V2 CTP3, VI Toolkit (for Windows) version 1.5 and the VI Toolkit Community Extensions build 46896.
Windows PowerShell
VI Toolkit (for Windows)
VI Toolkit Community Extensions
Another pre-requisite that is also recommended for general administration is Notepad++. This is used to create and edit scripts that can be run with the VI Toolkit.
Notepad++ can be downloaded from here.
2. INSTALLATION
There are three installation tasks that need to be performed before you can start using the VI-Toolkit to manage a VMware Infrastructure.
Windows PowerShell. The VI Toolkit 1.5 (for Windows) requires Microsoft PowerShell V2 CTP 3.
Please download it from here.
VI Toolkit (for Windows). Can be downloaded from here.
VI Toolkit Community Extensions. Can be downloaded from here.
3. SETTING UP THE VI TOOLKIT
The procedures below go through in detail how to get the VI-Toolkit up and running after installation. Once installed the icon below will be available on the Windows Desktop.
DO NOT LAUNCH IT YET!
Before launching the VMware VI Toolkit application, you must first set up your PowerShell profile. The new desktop shortcut does two things for you: it starts powershell with the VI Toolkit snapin loaded and it runs a script which modifies the look of the Powershell window and adds some cool extra functions. If you want to have the same functionality in your normal Powershell window and your scripts, you have to copy some stuff to your Powershell profile.
3.1 First, set up your profile:
1. Start a normal PowerShell Window by navigating to Start | All Programs | Windows PowerShell V2 (CTP3) | Windows PowerShell V2 (CTP3), the following will be launched:
2. Run the following command:
Test-Path $profile
3. If it returned True then you already have a profile file. If it returned False, then proceed to the next step.
4. Create a profile file by running:
New-Item $profile –ItemType File
5. If an error is returned then create a WindowsPowerShell directory under your My Documents folder and then repeat step 4.
3.2 Adding the snap-in:
1. Open your profile by running:
Invoke-Item $profile
2. Add the following line to the profile file to load the snap-in:
Add-PSSnapIn VMware.VimAutomation.Core -ErrorAction SilentlyContinue
3.3 Adding undocumented functions
1. Open the file C:\Program Files\VMware\Infrastructure\VIToolkitForWindows\Scripts\Initialize-VIToolkitEnvironment.ps1
2. Copy the following Function Blocks to your profile file:
Get-VICommand, New-DatastoreDrive, New-VIInventoryDrive, Get-VIToolkitDocumentation, Get-VIToolkitCommunity
If the steps were performed successfully, then your profile will be present in the folder structure C:\Documents and Settings\Hugo Phan\My Documents\WindowsPowerShell/ Microsoft.PowerShell_profile.ps1
And its contents will look something like this:
3.4 Enabling the execution of scripts
The Set-ExecutionPolicy changes the user preference for the execution policy of the shell. The execution policy is part of the security strategy of Windows PowerShell. It determines whether you can load configuration files (including your Windows PowerShell profile) and run scripts, and it determines which scripts, if any, must be digitally signed before they will run.
You need to set the execution policy to unrestricted using the below cmdlet
set-executionpolicy unrestricted
get-executionpolicy will return the current execution policy.
The default ExecutionPolicy is Restricted. Unrestricted is unnecessarily risky.
Set-ExecutionPolicy RemoteSigned is more secure and works for VI Toolkit 1.5.
3.5 Loading the Community Extensions
The VI Toolkit for Windows Community Extensions is a PowerShell module designed to work with the VI Toolkit for Windows.
1. Download and extract the package and then copy the coreModule folder to the root of C:
2. Open up a Windows PowerShell session and then type in the following command
Import-Module “c:\coreModule\viToolkitExtensions.psm1”
Now you are ready to start using the VI Toolit by either logging into a vCenter environment or by launching scripts.
1.1 Overview
The VI Toolkit (for Windows) provides a powerful yet simple command line interface for task based management of the VMware Infrastructure platform. Windows Administrators can easily manage and deploy the VMware Infrastructure with a familiar, simple to use command line interface.
The VI Toolkit (for Windows) is a tool that system administrators and developers can use to automate the management of VMware Virtual Infrastructure. With the VI Toolkit (for Windows), many tedious and time-consuming tasks can be completely automated in as little as one line of code.
The VI Toolkit (for Windows) takes advantage of Windows PowerShell and .NET to bring unprecedented ease of management and automation to the Virtual Infrastructure platform. The VI Toolkit (for Windows) provides 125 PowerShell cmdlets that cover all aspects of Virtual Infrastructure management.
Some common tasks that the VI Toolkit (for Windows) can be used to perform include:
- Snapshoting all virtual machines.
- Disconnecting or removing all Floppy or CD-ROM drives from all Virtual Machines.
- Large-scale cloning of templates.
- Moving large numbers of Virtual Machines from one virtual switch to another.
- Migrating large numbers of Virtual Machines between ESX hosts.
- Reports and monitoring across the entire Virtual Infrastructure.
The following platforms are supported by the VI Toolkit (for Windows):
- Microsoft Windows Server 2003 R2 (32 or 64 bit)
- Microsoft Windows Server 2003 with Service Pack 2 (SP2) (32 or 64 bit)
- Microsoft Windows Server 2003 with Service Pack 1 (SP1) (32 or 64 bit)
- Microsoft Windows XP with Service Pack 2 (SP2) (32 or 64 bit)
- Microsoft Windows Vista (32 or 64 bit)
The following platform combinations are supported by the VI Toolkit (for Windows):
- Management of ESX 3.0.2 using Virtual Center 2.5
- Management of ESX 3.5 using Virtual Center 2.5
- Management of ESXi 3.5 using Virtual Center 2.5
- Direct management of ESX 3.0.2
- Direct management of ESX 3.5
- Direct management of ESXi 3.5
The following tables lists the software pre-requisites and the location to each installer. This guide focuses on the most recent releases as dated 05/02/2009, which are Windows PowerShell V2 CTP3, VI Toolkit (for Windows) version 1.5 and the VI Toolkit Community Extensions build 46896.
Windows PowerShell
VI Toolkit (for Windows)
VI Toolkit Community Extensions
Another pre-requisite that is also recommended for general administration is Notepad++. This is used to create and edit scripts that can be run with the VI Toolkit.
Notepad++ can be downloaded from here.
2. INSTALLATION
There are three installation tasks that need to be performed before you can start using the VI-Toolkit to manage a VMware Infrastructure.
Windows PowerShell. The VI Toolkit 1.5 (for Windows) requires Microsoft PowerShell V2 CTP 3.
Please download it from here.
VI Toolkit (for Windows). Can be downloaded from here.
VI Toolkit Community Extensions. Can be downloaded from here.
3. SETTING UP THE VI TOOLKIT
The procedures below go through in detail how to get the VI-Toolkit up and running after installation. Once installed the icon below will be available on the Windows Desktop.
DO NOT LAUNCH IT YET!
Before launching the VMware VI Toolkit application, you must first set up your PowerShell profile. The new desktop shortcut does two things for you: it starts powershell with the VI Toolkit snapin loaded and it runs a script which modifies the look of the Powershell window and adds some cool extra functions. If you want to have the same functionality in your normal Powershell window and your scripts, you have to copy some stuff to your Powershell profile.
3.1 First, set up your profile:
1. Start a normal PowerShell Window by navigating to Start | All Programs | Windows PowerShell V2 (CTP3) | Windows PowerShell V2 (CTP3), the following will be launched:
2. Run the following command:
Test-Path $profile
3. If it returned True then you already have a profile file. If it returned False, then proceed to the next step.
4. Create a profile file by running:
New-Item $profile –ItemType File
5. If an error is returned then create a WindowsPowerShell directory under your My Documents folder and then repeat step 4.
3.2 Adding the snap-in:
1. Open your profile by running:
Invoke-Item $profile
2. Add the following line to the profile file to load the snap-in:
Add-PSSnapIn VMware.VimAutomation.Core -ErrorAction SilentlyContinue
3.3 Adding undocumented functions
1. Open the file C:\Program Files\VMware\Infrastructure\VIToolkitForWindows\Scripts\Initialize-VIToolkitEnvironment.ps1
2. Copy the following Function Blocks to your profile file:
Get-VICommand, New-DatastoreDrive, New-VIInventoryDrive, Get-VIToolkitDocumentation, Get-VIToolkitCommunity
If the steps were performed successfully, then your profile will be present in the folder structure C:\Documents and Settings\Hugo Phan\My Documents\WindowsPowerShell/ Microsoft.PowerShell_profile.ps1
And its contents will look something like this:
3.4 Enabling the execution of scripts
The Set-ExecutionPolicy changes the user preference for the execution policy of the shell. The execution policy is part of the security strategy of Windows PowerShell. It determines whether you can load configuration files (including your Windows PowerShell profile) and run scripts, and it determines which scripts, if any, must be digitally signed before they will run.
You need to set the execution policy to unrestricted using the below cmdlet
set-executionpolicy unrestricted
get-executionpolicy will return the current execution policy.
The default ExecutionPolicy is Restricted. Unrestricted is unnecessarily risky.
Set-ExecutionPolicy RemoteSigned is more secure and works for VI Toolkit 1.5.
3.5 Loading the Community Extensions
The VI Toolkit for Windows Community Extensions is a PowerShell module designed to work with the VI Toolkit for Windows.
1. Download and extract the package and then copy the coreModule folder to the root of C:
2. Open up a Windows PowerShell session and then type in the following command
Import-Module “c:\coreModule\viToolkitExtensions.psm1”
Now you are ready to start using the VI Toolit by either logging into a vCenter environment or by launching scripts.
Author: "Hugo Phan (hugo@vmwire.com)" Tags: "vcenter, powershell, power cli, vi toolk..."
Date: Friday, 08 May 2009 14:58
There are three ways in which to upgrade to VMware vSphere, these are
- VMware Update Manager
- vSphere Host Update Utility 4.0, and
- a clean install of vSphere
This post goes through the upgrade process using the vSphere Host Update Utility 4.0. A 10 minute video is available here:
The vSphere Host Update Utility 4.0 is an application that is installed as part of the vSphere vCenter installation package.
- To start the upgrade process, launch the vSphere Host Update Utility.
- The vSphere Host Update Utility will request confirmation to connect to the VMware patch repository.
- Add the host to the update utility by clicking on Host | Add Host.
- Type in the FQDN or IP address of the host you wish to upgrade then click on Add.
- Now click on the Upgrade button to start the upgrade wizard.
- Next browse to the location of your vSphere ISO file then click on Next.
- Read and accept the license agreement to continue.
- Enter the root credentials then press Next.
- The Host compatibility check will perform some checks and will allow the upgrade to continue if the host meets the criteria.
- Next select a local datastore (recommended) to store the disk file for the Console OS and also select the disk size.
- Leave all other settings on default and finish the Wizard
- Once complete, reconnect the host in vCenter to install the new vCenter Agent.
Author: "Hugo Phan (hugo@vmwire.com)"
Date: Wednesday, 06 May 2009 18:42
Following on from vinternals post about what a terrabyte of RAM looked like. Here is what 2Tb of RAM looks like installed in 16 BL685c servers.
Author: "Hugo Phan (hugo@vmwire.com)"
Date: Wednesday, 22 Apr 2009 00:00
Notes on using vRanger Pro & ESXi for Disaster Recovery
Just succesffully proved vRanger Pro to restore backups taken from Production (ESX 3.5, vRanger Pro on physical with VCB) to infrastructure in DR (ESXi 3.5, vRanger Pro on a VM, non VCB). All this from provisioning DR Infrastructure (ESXi Servers, Storage, vCenter VM) within 1 hour. Silver tier recovery just got "sESXi"!
Infrastructure at Production
- ESX 3.5 Update 2 on BL460C
- Storage on 400Gb LUNs presented by IBM SVC
- VC 2.5 Update 2 VM
- vRanger 3.8.2.1 & VCB 1.5 & vRanger Pro VCB Plugin 3.0 on Physical DL380 G5 Server
- VM backups on TSM and replicated to DR
Infrastructure at DR
- ESXi Update 3 USB on DL360 G5
- Local Storage
- VC 2.5 Update 4 VM
- vRanger 3.2.9.7 & VCB 1.5 & vRanger Pro VCB Plugin 3.0 on W2K3 SP2 VM + .Net Framework 2.0 SP1
If you are running vRanger in a virtual machine to restore workloads backed up by vRanger installed on a physical host, with either traditional LAN based backup or VCB based backup. It is important that the software is installed in the correct order and all the necessary software is installed to enable vRanger to restore both types of backup. If the physical vRanger server performed a backup of a workload using the VCB framework, then you will not be able to restore that workload using another vRanger server unless the VCB framework is also installed. For example, you wish to perform a restore at a DR site.
The correct installation order is
- Microsoft .Net Framework 2.0 SP1
- vRanger Pro
- vRanger Pro VCB Integration module
- vRanger Pro file-level plugin
- VMware VCB Framework
- Install software in the correct order
- Create the same directory structure for the VM at the DR site as it is at Production. E.g, if the vRanger working directory is D:\vRanger_Backups at Production, then keep the same directory structure for the vRanger server at DR.
- This will enable you to first restore the vRanger database (esxRanger.mdb), which then populates the Restore table saving valuable time and effort because you will no longer need to use "Restore from Info"
- If restoring a vRanger backup that was taken using the VCB framework, then the vRanger server at DR will also need to have the VCB framework installed.
Author: "Hugo Phan (hugo@vmwire.com)" Tags: "ESXi, vRanger, vRanger Pro, esx, TSM, di..."
Date: Tuesday, 21 Apr 2009 22:44
Heres a picture of the box...
Author: "Hugo Phan (hugo@vmwire.com)"
Date: Wednesday, 11 Feb 2009 23:25
Following on from a previous post on how to configure vSwitch0 with active/passive network interfaces using vmware-vim-cmd for the two port groups, I have included some diagrams explaining how it all works.
The reason that Link Aggregation will not give any performance benefit if both the Service Console and the VMkernel share two active uplinks on a virtual switch configured with Route based on IP hash load balancing policy can be summarised as follows.
802.3ad/LACP aggregates physical links, but the mechanisms used to determine whether a given flow of information follows one link or another are critical.
You’ll note several key things in this document that are useful in understanding 802.3ad/LACP.
So how does the VMkernel distribute traffic over the Link Aggregation Group (LAG)?
The VMkernel distributes the load across the Link Aggregation Group by selecting an uplink to the physical network based on the source and destination IP addresses together. Each source / destination IP conversation gets treated as a unique route, and is distributed across the LAG accordingly. Using an IP-based load balancing method allows for a single-NIC virtual machine to possibly utilise more than 1 physical NIC. Returning traffic may come in on a different NIC, so Link Aggregation must be supported on the physical switch.
The diagram below shows what you would want to achieve with vSwitch0 when using a single virtual switch with two network interfaces with two port groups configured for the Service Console networking and the VMkernel networking.
The Service Console uses vmnic0 as the active uplink and vmnic2 as the passive uplink. The VMkernel uses vmnic2 as the active uplink and vmnic0 as the passive uplink. If one of the network interfaces fail or the corresponding physical switch fails or a cable fails, then the portgroup will utilise the standby network interface.
The configuration settings seen within the VI Client are shown in the figures below.
The vSwitch0 configuration
The vSwitch uses Route based on IP hash with both adapters set as active.
The Service Conso
le portgroup configuration
The Service Console port group uses vmnic0 as the active adapter and vmnic2 as the standby adapter and the load balancing policy is set to Route based on virtual port ID. This overrides the configuration inherited by the vSwitch.
The VMkernel portgroup configuration
The VMkernel port group uses vmnic2 as the active adapter and vmnic0 as the standby adapter and the load balancing policy is set to Route based on virtual port ID. This overrides the configuration inherited by the vSwitch.
Network Utilisation
The graph below shows the current utilisation of the network interfaces assigned to vSwitch0, under normal operations only vmnic0 is used because the Service Console traffic uses vmnic0.
Let's see what happens when I initiate a VMotion to the same server.
The VMotion traffic uses the active network interface vmnic2 as expected.
It is obviously apparent that using override active/passive uplinks for the Service Console and VMkernel port groups has significant advantages. By doing this we restrict the VMotion traffic from flooding the Service Console uplink adapter that can be experienced when using an active/active configuration.
The reason that Link Aggregation will not give any performance benefit if both the Service Console and the VMkernel share two active uplinks on a virtual switch configured with Route based on IP hash load balancing policy can be summarised as follows.
802.3ad/LACP aggregates physical links, but the mechanisms used to determine whether a given flow of information follows one link or another are critical.
You’ll note several key things in this document that are useful in understanding 802.3ad/LACP.
- All frames associated with a given “conversation” are transmitted on the same link to prevent mis-ordering of frames. So what is a “conversation”? A “conversation” is the TCP connection.
- The link selection for a conversation is usually done by doing a hash on the MAC addresses (Route based on source MAC hash) or IP address (Route based on IP hash).
- Link Aggregation achieves high utilisation across multiple links when carrying multiple conversations, and is less efficient with a small number of conversations (and has no improved bandwith with just one).
So how does the VMkernel distribute traffic over the Link Aggregation Group (LAG)?
The VMkernel distributes the load across the Link Aggregation Group by selecting an uplink to the physical network based on the source and destination IP addresses together. Each source / destination IP conversation gets treated as a unique route, and is distributed across the LAG accordingly. Using an IP-based load balancing method allows for a single-NIC virtual machine to possibly utilise more than 1 physical NIC. Returning traffic may come in on a different NIC, so Link Aggregation must be supported on the physical switch.
The diagram below shows what you would want to achieve with vSwitch0 when using a single virtual switch with two network interfaces with two port groups configured for the Service Console networking and the VMkernel networking.
The Service Console uses vmnic0 as the active uplink and vmnic2 as the passive uplink. The VMkernel uses vmnic2 as the active uplink and vmnic0 as the passive uplink. If one of the network interfaces fail or the corresponding physical switch fails or a cable fails, then the portgroup will utilise the standby network interface.
The configuration settings seen within the VI Client are shown in the figures below.
The vSwitch0 configuration
The vSwitch uses Route based on IP hash with both adapters set as active.The Service Conso
le portgroup configurationThe Service Console port group uses vmnic0 as the active adapter and vmnic2 as the standby adapter and the load balancing policy is set to Route based on virtual port ID. This overrides the configuration inherited by the vSwitch.
The VMkernel portgroup configuration
The VMkernel port group uses vmnic2 as the active adapter and vmnic0 as the standby adapter and the load balancing policy is set to Route based on virtual port ID. This overrides the configuration inherited by the vSwitch.Network Utilisation
The graph below shows the current utilisation of the network interfaces assigned to vSwitch0, under normal operations only vmnic0 is used because the Service Console traffic uses vmnic0.
Let's see what happens when I initiate a VMotion to the same server.
The VMotion traffic uses the active network interface vmnic2 as expected.
It is obviously apparent that using override active/passive uplinks for the Service Console and VMkernel port groups has significant advantages. By doing this we restrict the VMotion traffic from flooding the Service Console uplink adapter that can be experienced when using an active/active configuration.
Author: "Hugo Phan (hugo@vmwire.com)" Tags: "802.3ad, lacp, portgroups, esx, link agg..."
Date: Monday, 02 Feb 2009 12:39
Steps in order to progress
1) Login in the affected ESX server using Putty
2) service mgmt-vmware restart
If this doesn't work then the vmware-hostd daemon has to be killed.
3) ps -e | grep vmware-hostd
Look for the process_id associated with vmware-hostd
4) kill process_id
i.e. if 3) returned:
32470 ? 00:01:12 vmware-hostd
the command would be:
kill 32470
5) service mgmt-vmware status
if the service is started use
service mgmt-vmware restart
if it's stopped use:
service mgmt-vmware start
1) Login in the affected ESX server using Putty
2) service mgmt-vmware restart
If this doesn't work then the vmware-hostd daemon has to be killed.
3) ps -e | grep vmware-hostd
Look for the process_id associated with vmware-hostd
4) kill process_id
i.e. if 3) returned:
32470 ? 00:01:12 vmware-hostd
the command would be:
kill 32470
5) service mgmt-vmware status
if the service is started use
service mgmt-vmware restart
if it's stopped use:
service mgmt-vmware start
Author: "Hugo Phan (hugo@vmwire.com)"
Date: Monday, 02 Feb 2009 12:37
Recently a customer had a few issues with having all VMs using the same path to the LUNs, this was down to putting too many workloads onto servers that were used as a proof of concept. Inadvertently, SAN problems arose so I was asked to checkover the storage.
First a little background on the infrastructure.... a number of rack servers plus a number of blade servers, hooked into two fabrics with IBM SVC as the backend. Each ESX server has two FC HBA, and each fabric switch had two connections to the SVC, therefore each ESX server has four possible paths to the LUNs. The paths were all active as shown on this pic:
As you can see the path policy is currently set to mru, most recently used path policy is best used in an active/passive configuration.
mru:
The script from Yellow Bricks is of particular use, as for each LUN it finds it uses a different path for each LUN. The script just sets each LUN up to use a preferred path, but obviously for default installations of ESX, you cannot use preferred path when you are using mru policy. So we must change all LUNs to use fixed path policy first.
By re-using the script form Yellow Bricks, I've come up with this:
# vmhbafixedpath.sh Script to rescan vmhbas on ESX 3.5 host
# Written by hugo@vmwire.com
# 21/05/2008 18:20
for PATHS in 2 4 6 8
do
STPATHS=${PATHS}
COUNTER="1"
for LUN in $(esxcfg-mpath -l | grep "has ${STPATHS} paths" | awk '{print $2}')
do
esxcfg-mpath --lun=${LUN} -p fixed
COUNT=`expr ${COUNTER} + 1`
COUNTER=${COUNT}
if [[ ${COUNTER} -gt ${STPATHS} ]]
then
COUNTER="1"
fi
done
done
Then use the script from Yellow Bricks, to set up the preferred paths. Now the changes do not take into effect until the HBAs are rescanned, and the Storage is refreshed. The following script rescans the HBAs
#!/bin/bash
# rescanhbas.sh Script to rescan vmhbas on ESX 3.5 host
# Written by hugo@vmwire.com & nkouts
# 21/05/2008 18:50
# Assumes there is no vmhba0 and max vmhba9
for HBAS in 2 4
do
STHBAS=${HBAS}
COUNTER="1"
for HBA in $(esxcfg-info -w | grep vmhba | awk '{print $3}' | grep -e 'vmhba\+[1-9]' -o)
do
esxcfg-rescan ${HBA}
COUNT=`expr ${COUNTER} + 1`
COUNTER=${COUNT}
if [[ ${COUNTER} -gt ${STHBAS} ]]
then
COUNTER="1"
fi
done
done
And there is no known console based method to refresh the storage subsystem (anyone?) apart from using VI-Client, rebooting the ESX host or restarting the vmware management service:
service mgmt-vmware restart
UPDATE: Use /usr/bin/vmware-vim-cmd to refresh the storage
/usr/bin/vmware-vim-cmd hostsvc/storage/refresh
So now we have the servers using different paths for each datastore.
It only took a couple of seconds to change the policies on each server using these scripts, obviously using these as part of a build script would be ideal for deployments where you know the SAN configuration.
First a little background on the infrastructure.... a number of rack servers plus a number of blade servers, hooked into two fabrics with IBM SVC as the backend. Each ESX server has two FC HBA, and each fabric switch had two connections to the SVC, therefore each ESX server has four possible paths to the LUNs. The paths were all active as shown on this pic:
As you can see the path policy is currently set to mru, most recently used path policy is best used in an active/passive configuration.mru:
- LUNs presented on single Storage Processor at any one time
- Failover on NOT_READY, ILLEGAL_REQUEST or NO_CONNECT
- No preferred path policy
- No failback to preferred path if it returns online after failover
- LUNs presented on multiple Storage Processors at same time
- Failover over on NO_CONNECT
- Preferred path policy
- Failback to preferred path if it returns online after failover
The script from Yellow Bricks is of particular use, as for each LUN it finds it uses a different path for each LUN. The script just sets each LUN up to use a preferred path, but obviously for default installations of ESX, you cannot use preferred path when you are using mru policy. So we must change all LUNs to use fixed path policy first.
By re-using the script form Yellow Bricks, I've come up with this:
# vmhbafixedpath.sh Script to rescan vmhbas on ESX 3.5 host
# Written by hugo@vmwire.com
# 21/05/2008 18:20
for PATHS in 2 4 6 8
do
STPATHS=${PATHS}
COUNTER="1"
for LUN in $(esxcfg-mpath -l | grep "has ${STPATHS} paths" | awk '{print $2}')
do
esxcfg-mpath --lun=${LUN} -p fixed
COUNT=`expr ${COUNTER} + 1`
COUNTER=${COUNT}
if [[ ${COUNTER} -gt ${STPATHS} ]]
then
COUNTER="1"
fi
done
done
Then use the script from Yellow Bricks, to set up the preferred paths. Now the changes do not take into effect until the HBAs are rescanned, and the Storage is refreshed. The following script rescans the HBAs
#!/bin/bash
# rescanhbas.sh Script to rescan vmhbas on ESX 3.5 host
# Written by hugo@vmwire.com & nkouts
# 21/05/2008 18:50
# Assumes there is no vmhba0 and max vmhba9
for HBAS in 2 4
do
STHBAS=${HBAS}
COUNTER="1"
for HBA in $(esxcfg-info -w | grep vmhba | awk '{print $3}' | grep -e 'vmhba\+[1-9]' -o)
do
esxcfg-rescan ${HBA}
COUNT=`expr ${COUNTER} + 1`
COUNTER=${COUNT}
if [[ ${COUNTER} -gt ${STHBAS} ]]
then
COUNTER="1"
fi
done
done
And there is no known console based method to refresh the storage subsystem (anyone?) apart from using VI-Client, rebooting the ESX host or restarting the vmware management service:
service mgmt-vmware restart
UPDATE: Use /usr/bin/vmware-vim-cmd to refresh the storage
/usr/bin/vmware-vim-cmd hostsvc/storage/refresh
So now we have the servers using different paths for each datastore.
It only took a couple of seconds to change the policies on each server using these scripts, obviously using these as part of a build script would be ideal for deployments where you know the SAN configuration.
Author: "Hugo Phan (hugo@vmwire.com)" Tags: "lun, round robin, luns, mru, load balanc..."
Date: Monday, 02 Feb 2009 12:29
vmware-vim-cmd
For those of you familiar with vimsh and used it to configure a scripted install of ESX 3.5, have you noticed that the following error would occur when launching commands using /usr/bin/vimsh ?
/usr/bin/vimsh -n -e "hostsvc/maintenance_mode_enter
Alternatively, by using the wrapper developed for ESX 3.5, vmware-vim-cmd, you would get the following:
/usr/bin/vmware-vim-cmd hostsvc/maintenance_mode_enter
The two commands are detailed in the Xtravirt whitepapers, vimsh and vimsh for ESX 3.5. I would recommend at least having a quick browse to see what can be achieved with these commands. Using vmware-vim-cmd in conjunction with esxcfg- can achieve some very interesting results, especially if you love to create the perfect KickStart build script.
If only it is possible to launch vmware-vim-cmd commands using the RCLI just as esxcfg- can be launched using vicfg-. Anyone have an idea?
A few more examples
Refreshing the network settings
/usr/bin/vmware-vim-cmd hostsvc/net/refresh
Refreshing the storage
/usr/bin/vmware-vim-cmd hostsvc/storage/refresh
The all important enabling VMotion
/usr/bin/vmware-vim-cmd hostsvc/vmotion/vnic_set vmk0
And how about setting vSwitch1 to use Route Based on IP Hash?
/usr/bin/vmware-vim-cmd hostsvc/net/vswitch_setpolicy --nicteaming-policy=loadbalance_ip vSwitch1
And setting vSwitch0 to use Route Based on the Originating Virtual PortID. (vSwitch0 has two portgroups using VLAN tagging, 1 for Service Console and 1 for VMotion, we wish to use active-passive nic teaming policy)
Set active vmnic0 and standby vmnic2 for Service Console
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-active=vmnic0 vSwitch0 'Service Console'
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-standby=vmnic2 vSwitch0 'Service Console'
Set active vmnic2 and standby vmnic0 for VMkernel network
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-active=vmnic2 vSwitch0 VMkernel
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-standby=vmnic0 vSwitch0 VMkernel
Set vSwitch overide load balancing policy
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicteaming-policy=loadbalance_srcid vSwitch0 'Service Console'
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicteaming-policy=loadbalance_srcid vSwitch0 VMkernel
Let's not forget to refresh our network settings
/usr/bin/vmware-vim-cmd hostsvc/net/refresh
/usr/bin/vmware-vim-cmd internalsvc/refresh_network
For those of you familiar with vimsh and used it to configure a scripted install of ESX 3.5, have you noticed that the following error would occur when launching commands using /usr/bin/vimsh ?
/usr/bin/vimsh -n -e "hostsvc/maintenance_mode_enter
Alternatively, by using the wrapper developed for ESX 3.5, vmware-vim-cmd, you would get the following:
/usr/bin/vmware-vim-cmd hostsvc/maintenance_mode_enter
The two commands are detailed in the Xtravirt whitepapers, vimsh and vimsh for ESX 3.5. I would recommend at least having a quick browse to see what can be achieved with these commands. Using vmware-vim-cmd in conjunction with esxcfg- can achieve some very interesting results, especially if you love to create the perfect KickStart build script.
If only it is possible to launch vmware-vim-cmd commands using the RCLI just as esxcfg- can be launched using vicfg-. Anyone have an idea?
A few more examples
Refreshing the network settings
/usr/bin/vmware-vim-cmd hostsvc/net/refresh
Refreshing the storage
/usr/bin/vmware-vim-cmd hostsvc/storage/refresh
The all important enabling VMotion
/usr/bin/vmware-vim-cmd hostsvc/vmotion/vnic_set vmk0
And how about setting vSwitch1 to use Route Based on IP Hash?
/usr/bin/vmware-vim-cmd hostsvc/net/vswitch_setpolicy --nicteaming-policy=loadbalance_ip vSwitch1
And setting vSwitch0 to use Route Based on the Originating Virtual PortID. (vSwitch0 has two portgroups using VLAN tagging, 1 for Service Console and 1 for VMotion, we wish to use active-passive nic teaming policy)
Set active vmnic0 and standby vmnic2 for Service Console
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-active=vmnic0 vSwitch0 'Service Console'
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-standby=vmnic2 vSwitch0 'Service Console'
Set active vmnic2 and standby vmnic0 for VMkernel network
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-active=vmnic2 vSwitch0 VMkernel
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicorderpolicy-standby=vmnic0 vSwitch0 VMkernel
Set vSwitch overide load balancing policy
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicteaming-policy=loadbalance_srcid vSwitch0 'Service Console'
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set --nicteaming-policy=loadbalance_srcid vSwitch0 VMkernel
Let's not forget to refresh our network settings
/usr/bin/vmware-vim-cmd hostsvc/net/refresh
/usr/bin/vmware-vim-cmd internalsvc/refresh_network
Author: "Hugo Phan (hugo@vmwire.com)" Tags: "vimsh, vmware-vim-cmd, ESX 3.5"
Date: Thursday, 08 Jan 2009 23:41
You can download the zip file from here.
Author: "Hugo Phan (hugo@vmwire.com)"
Date: Thursday, 11 Dec 2008 11:37
Well here I am, starting a new project for a new customer at a new datacenter again. This time, its a large retail organisation looking to do the usual, consolidate, virtualise, go green etc etc. They have selected IBM System X and BladeCenter H as the platforms of choice for the new VMware ESX 3 environment. So here we go with the planning....
The BladeCenter H has eight switch bays and two Advanced Management Module (AMM) bays. The two AMM act in much the same way as the Onboard Administrator on HP C Class. There are two for redundancy. Two of the eight switch bays are used for FC Switches, for this project we are using Brocade 4Gb SAN switches.
The other bays are occupied by Cisco GbE Switch Modules.
HS21s are used for the initial phase of the project. These blades can accommodate upto 6 NICs and 2 HBAs, with 2 onboard and the other 4 provided by daughtercards. The customer has elected to use 4 NICs as opposed to the 6 that I normally recommend for ESX implementations. The two extra NICs are provided by the CFFh daughtercard, this daughtercard houses 2 network adapters AND 2 Fibre Channel HBAs.
The table below (from IBM) show the interface to bay mapping.
Since only 4 interfaces are available, teaming and VLANs will have to be used to provide resilience and to separate the SC and VMKernel networks.
I will be teaming Interface 0 (eth0) with Interface 3 (eth3) as opposed to the IBM table (dedicating an adapter to a service), as this will team one onboard port with one daughtercard port. Likewise eth1 will then be teamed with eth2.
* The location of the two Fibre Channel Adapters should be Daughter Card CFF-h, not v as shown in the IBM table.
The following diagram shows the correct mapping.
The table below details the network interconnects.
Interface is the network adapter inside a blade, Location is where the interface is, Chassis Bay is where the interface terminates at the rear of the BladeCenter chassis, pSwitch is the external core switch that the Chassis Bay uplinks to, vSwitch is the ESX virtual switch that the Interface provides an uplink for, vLAN is the ID that is assigned to each Port Group and Service is the type of port group assigned to a vSwitch.
The BladeCenter H has eight switch bays and two Advanced Management Module (AMM) bays. The two AMM act in much the same way as the Onboard Administrator on HP C Class. There are two for redundancy. Two of the eight switch bays are used for FC Switches, for this project we are using Brocade 4Gb SAN switches.
The other bays are occupied by Cisco GbE Switch Modules.
HS21s are used for the initial phase of the project. These blades can accommodate upto 6 NICs and 2 HBAs, with 2 onboard and the other 4 provided by daughtercards. The customer has elected to use 4 NICs as opposed to the 6 that I normally recommend for ESX implementations. The two extra NICs are provided by the CFFh daughtercard, this daughtercard houses 2 network adapters AND 2 Fibre Channel HBAs.
The table below (from IBM) show the interface to bay mapping.
Since only 4 interfaces are available, teaming and VLANs will have to be used to provide resilience and to separate the SC and VMKernel networks.I will be teaming Interface 0 (eth0) with Interface 3 (eth3) as opposed to the IBM table (dedicating an adapter to a service), as this will team one onboard port with one daughtercard port. Likewise eth1 will then be teamed with eth2.
* The location of the two Fibre Channel Adapters should be Daughter Card CFF-h, not v as shown in the IBM table.
The following diagram shows the correct mapping.
The table below details the network interconnects.
Interface is the network adapter inside a blade, Location is where the interface is, Chassis Bay is where the interface terminates at the rear of the BladeCenter chassis, pSwitch is the external core switch that the Chassis Bay uplinks to, vSwitch is the ESX virtual switch that the Interface provides an uplink for, vLAN is the ID that is assigned to each Port Group and Service is the type of port group assigned to a vSwitch.Author: "Hugo Phan (hugo@vmwire.com)" Tags: "bladecenter h, cisco gbe, hs21, vmnic, c..."
Date: Thursday, 11 Dec 2008 11:37
In the previous post I covered the network design for a HS21 with 4 network interfaces. This post will continue with a diagrammatic representation of the interface table.
As described previously, this configuration provides full network fault tolerance on all levels: adapter, port, CAT5, switch bay and core switch.
Put your finger over any individual constituent part, i.e., pNic, interface, bay switch or core switch, to simulate a failure and there will always be an alternative path.
I'm waiting for the customer to decide on whether to include the CFFv daughtercard in this phase of the project, and will update this post with the new design if required.
Next up, environmentals...
Those of you familiar with the HP c-Class blades will probably know that there is a superb tool called the HP BladeSystem PowerSizer 2.9, I've been trying to find an equivalent from IBM, but as yet have not found anything that comes as close. (Any pointers will be appreciated)
Instead I've had to resort to using data obtained from The Edison Group study titled Blade Server Power Study - IBM BladeCenter and HP BladeSystem, Nov 7 2007, document titled "BLL03002USEN.pdf".
The results show, in summary a BladeCenter H chassis with 14 blades on full load will need 14,352.51 BTU/Hr with a peak power consumption of 4,208.80 Watts. Most modern datacenters with good power feeds will be able to accommodate that kind of load. Cooling requirements will be left to the customer to calculate.
Additionally, this single chassis will require 9 rack units and 4 power feeds due to the additional 2900W power supply modules.
Part 2.... Continued..
Thank you Aaron for your help with the power sizer.
Here is the output from the tool (not as nice as HP's offerring by the way)
In the next part... network design for the x3650.
As described previously, this configuration provides full network fault tolerance on all levels: adapter, port, CAT5, switch bay and core switch.
Put your finger over any individual constituent part, i.e., pNic, interface, bay switch or core switch, to simulate a failure and there will always be an alternative path.I'm waiting for the customer to decide on whether to include the CFFv daughtercard in this phase of the project, and will update this post with the new design if required.
Next up, environmentals...
Those of you familiar with the HP c-Class blades will probably know that there is a superb tool called the HP BladeSystem PowerSizer 2.9, I've been trying to find an equivalent from IBM, but as yet have not found anything that comes as close. (Any pointers will be appreciated)
Instead I've had to resort to using data obtained from The Edison Group study titled Blade Server Power Study - IBM BladeCenter and HP BladeSystem, Nov 7 2007, document titled "BLL03002USEN.pdf".
The results show, in summary a BladeCenter H chassis with 14 blades on full load will need 14,352.51 BTU/Hr with a peak power consumption of 4,208.80 Watts. Most modern datacenters with good power feeds will be able to accommodate that kind of load. Cooling requirements will be left to the customer to calculate.
Additionally, this single chassis will require 9 rack units and 4 power feeds due to the additional 2900W power supply modules.
Part 2.... Continued..
Thank you Aaron for your help with the power sizer.
Here is the output from the tool (not as nice as HP's offerring by the way)
In the next part... network design for the x3650.
Author: "Hugo Phan (hugo@vmwire.com)" Tags: "vnic, pnic, hs21, vmnic, bladecenter, cf..."
Date: Thursday, 11 Dec 2008 11:37
It was used in "Fundamentals of Disaster Recovery in Virtualized Environments" VMware World Partner Day 2007.
Any help to indentify would be great!
Any help to indentify would be great!Author: "Hugo Phan (hugo@vmwire.com)"
Date: Thursday, 11 Dec 2008 11:37
Following on from optimising the storage for a customer, I decided to change the queue depths for the Emulex HBAs. The ESX hosts, each have two dual-port Emulex HBAs, the diagram below shows the setup..
Only two ports are in use at the moment, vmhba2 and vmhba5. To determine the instance numbers that are in use by the Emulex ESX driver - lpfc (use qla2300 or similar for QLogic), the output of the ls command includes a number for each active HBA in the system. We can then use the instance numbers to find the active adapters.
Emulex example
# ls /proc/scsi/lpfc
You should get an output similar to
Because of the way that the host is connected and from the picture above, I already know that 2 and 5 are the active adapters. Running the following command will confirm
# cat /proc/scsi/lpfc/2
this shows that vmhba2 is currently active and has 4-paths to the SAN
Running the same command on vmhba3 gives the following as expected
Running the command on vmhba5 is also as expected.
Now that we've found out which vmhbas are active, we can use the output to find out which lpfc# options we should add to the lpfc_740.o module to configure the queue length.
The outputs of # cat /proc/scsi/lpfc/2 and # cat /proc/scsi/lpfc/5, give us lpfc numbers of 0 and 3 respectively. So to configure a queue depth of 64 for lpfc2 and lpfc5 we run the following command
# esxcfg-module -s "lpfc0_lun_queue_depth=64 lpfc3_lun_queue_depth=64" lpfc_740
and
# esxcfg-boot -b
The -q option shows configured options for a module.
Now we reboot for the changes to take effect.
In this case, both HBAs lpfc0 (vmhba2) and lpfc3 (vmhba5) will have their queue depths set to 64.
With this post and the previous one, we have set manual load balancing for the LUNs over eight different paths and also changed the queue depth to 64, this should keep the ESX optimised for now, maybe I'll change the VMFS3.MaxHeapSizeMB to 64 for good measure!
Only two ports are in use at the moment, vmhba2 and vmhba5. To determine the instance numbers that are in use by the Emulex ESX driver - lpfc (use qla2300 or similar for QLogic), the output of the ls command includes a number for each active HBA in the system. We can then use the instance numbers to find the active adapters.
Emulex example
# ls /proc/scsi/lpfc
You should get an output similar to
Because of the way that the host is connected and from the picture above, I already know that 2 and 5 are the active adapters. Running the following command will confirm
# cat /proc/scsi/lpfc/2
this shows that vmhba2 is currently active and has 4-paths to the SANRunning the same command on vmhba3 gives the following as expected
Running the command on vmhba5 is also as expected.
Now that we've found out which vmhbas are active, we can use the output to find out which lpfc# options we should add to the lpfc_740.o module to configure the queue length.
The outputs of # cat /proc/scsi/lpfc/2 and # cat /proc/scsi/lpfc/5, give us lpfc numbers of 0 and 3 respectively. So to configure a queue depth of 64 for lpfc2 and lpfc5 we run the following command
# esxcfg-module -s "lpfc0_lun_queue_depth=64 lpfc3_lun_queue_depth=64" lpfc_740
and
# esxcfg-boot -b
The -q option shows configured options for a module.Now we reboot for the changes to take effect.
In this case, both HBAs lpfc0 (vmhba2) and lpfc3 (vmhba5) will have their queue depths set to 64.
With this post and the previous one, we have set manual load balancing for the LUNs over eight different paths and also changed the queue depth to 64, this should keep the ESX optimised for now, maybe I'll change the VMFS3.MaxHeapSizeMB to 64 for good measure!
Author: "Hugo Phan (hugo@vmwire.com)"
Date: Thursday, 09 Oct 2008 16:14
Author: "Hugo Phan (hugo@vmwire.com)" Tags: "ESXi, hyper-v"
Date: Wednesday, 28 May 2008 00:53
So I deal with a lot of customers on my travels, and most have multiple ESX servers and occasionally I receive the odd request for a change here and there. As you all know, with VI3 and the latest releases of ESX3.5/VC2.5, almost all configuration and most advanced configuration can be achieved by using the VI Client connected to VirtualCenter.
But how long would it take to add another portgroup to a vSwitch with a VLAN ID for 20 ESX servers? Quite long, if you have the time or the patience then thats fine, but I'd rather script something like that.
By using the VMware RCLI (Remote Client) you can send vicfg- (esxcfg) commands to both ESX 3.5 and ESXi hosts. Originally it was intended for use with ESXi due to it having limited service console but the functionality is also provided for ESX 3.5 hosts.
I use the RCLI with SSH access enabled, so now my RCLI acts as a service console proxy server. To send an esxcfg- command to an ESX 3.5 host, I would now log into the RCLI using SSH and then send the commands from the RCLI's command line, or execute a .sh script on the RCLI.
So let's use our example above.... to add another portgroup to vSwitch1, with a VLAN ID of 123 onto 20 ESX 3.5 hosts.
#Script to add portgroup with vlad id of 123 to vSwitch1 onto all ESX 3.5 hosts
# Assign port groups to vSwitch1
vicfg-vswitch --add-pg=VLAN123 vSwitch1 --server= --username=root --password=
vicfg-vswitch --add-pg=VLAN123 vSwitch1 --server= --username=root --password=
vicfg-vswitch --add-pg=VLAN123 vSwitch1 --server= --username=root --password=
#Assign vlan ids to port groups
vicfg-vswitch -v 123 -p VLAN123 vSwitch1 --server= --username=root --password=
vicfg-vswitch -v 123 -p VLAN123 vSwitch1 --server= --username=root --password=
vicfg-vswitch -v 123 -p VLAN123 vSwitch1 --server= --username=root --password=
Now save, make the script executable and then launch it, and the script will create the new portgroups on all the servers in a couple of seconds.
But how long would it take to add another portgroup to a vSwitch with a VLAN ID for 20 ESX servers? Quite long, if you have the time or the patience then thats fine, but I'd rather script something like that.
By using the VMware RCLI (Remote Client) you can send vicfg- (esxcfg) commands to both ESX 3.5 and ESXi hosts. Originally it was intended for use with ESXi due to it having limited service console but the functionality is also provided for ESX 3.5 hosts.
The VMware Infrastructure Remote CLI provides a command-line interface for datacenter management from a remote server. This interface is fully supported on VMware ESXi 3.5 and experimental for VMware ESX 3.5. Storage VMotion is a feature that lets you migrate a virtual machine from one datastore to another. It is used by executing the svmotion command from the Remote CLI. The svmotion command, unlike other RCLI commands, is fully supported for VMware ESX 3.5.
I use the RCLI with SSH access enabled, so now my RCLI acts as a service console proxy server. To send an esxcfg- command to an ESX 3.5 host, I would now log into the RCLI using SSH and then send the commands from the RCLI's command line, or execute a .sh script on the RCLI.
So let's use our example above.... to add another portgroup to vSwitch1, with a VLAN ID of 123 onto 20 ESX 3.5 hosts.
- Log into the RCLI using SSH
- the command line command is very similar to esxcfg- but we use vicfg- instead
- vicfg-vswitch --add-pg=VLAN123 vSwitch1 --server=
--username=root --password= - Now you can either repeat the above for all 20 servers or script it into a shell script..
- Create a new script on the RCLI called addportgroup.sh
#Script to add portgroup with vlad id of 123 to vSwitch1 onto all ESX 3.5 hosts
# Assign port groups to vSwitch1
vicfg-vswitch --add-pg=VLAN123 vSwitch1 --server=
vicfg-vswitch --add-pg=VLAN123 vSwitch1 --server=
vicfg-vswitch --add-pg=VLAN123 vSwitch1 --server=
#Assign vlan ids to port groups
vicfg-vswitch -v 123 -p VLAN123 vSwitch1 --server=
vicfg-vswitch -v 123 -p VLAN123 vSwitch1 --server=
vicfg-vswitch -v 123 -p VLAN123 vSwitch1 --server=
Now save, make the script executable and then launch it, and the script will create the new portgroups on all the servers in a couple of seconds.
Author: "Hugo Phan (hugo@vmwire.com)"
» © All content and copyrights belong to their respective authors.«
» © FeedShow - Online RSS Feeds Reader