You find yourself sitting at your desk on a Saturday morning waiting for the cloning of a VM to finish. It seems to take forever and I just wished I could run VSAN in my homelab to get some more speed out of my kit. I have tried to run VSAN on my home mode ESXi hosts, but my RAID controller is not compatible (yet) with VSAN, so I had to remove it again and now I’m waiting for the official release.
While waiting I browsed through some VMworld posts in my twitter feed and there was a link to William Lam’s post about his Barcelona session: “VMworld Barcelona #NotSupported Tips/Tricks for vSphere 5.5 Slides Posted”. Browsing through his slides, I got a really stupid idea. What if I would run Virtual ESXi with VSAN included in my home lab, would it be faster than my current NFS NAS that keeps me waiting forever? The virtual ESXi would only be used for offering the VSAN datastore to the physical ESXi hosts in the cluster there is no need to run VMs in the memory of these virtual ESXi hosts. Hmmm, sounds like a plan to test this config, let’s do it.
Disclaimer: Don’t use the data I got from my tests as a reference for anything. My kit is not good enough to produce serious numbers that you can relate to business environments. This is testing just for fun and for me personally to see if I can get some more speed out of my current environment even it means setting up this idiotic configuration.
To start with, this is what I have in my home lab:
- 2x ESXi host with 32GB each and Intel i5-3470 CPU Quad core
- Iomega IX4 200D NAS configured with NFS offering 2.7TB of storage.
- Disks on each host:
- SATA 640GB 16MB Cache, 5400 RPM, WD6400AACS-00G8B1, CAVIAR GREENPOWER 640GB
- SATA 400GB 16MB Cache, 7200 RPM,
- Samsung SpinPoint T166 HD403LJ SATA 256GB SAMSUNG SSD 830 Series
On one of the physical ESXi hosts I have a Windows 2012 Server which gets its storage from the IX4. This is my test VM on which I run IOmeter on the F-drive with a 4GB test file.
Windows 2012 Server (Test VM):
- 1 vCPU
- 1500MB RAM
- C-drive and D-drive of 40GB
- F-drive of 10GB on separate paravirtual SCSI controller
- Using iometer
- Each test is 300 seconds
- Size of the test file is 4GB
- Test specifications:
- Max throughput 100% read
- RealLife 60% random and 65% read
- Max throughput 50% read
- Random 8k 70% read
I did a number of test which I describe below:
- The test VM that is running IOmeter is fully running on the IX4-200D NFS volume.
- The test VM that is running IOmeter is fully running on the physical SSD of the physical ESXi host.
- The VSAN ESXi host has the virtual SSD disk on the local SSD disk of the physical ESXi host and has the virtual SATA disk also on the local SSD disk of the physical ESXi host.
- The VSAN ESXi host has the virtual SSD disk on the local SSD disk of the physical ESXi host and has the virtual SATA disk on the IX4-200D NFS volume.
You see that in step 3 and 4 I’m moving the data disk that offers the real storage to the VSAN. With VSAN this is the disk that should be relieved from heavy reads and writes by using the VSAN technology, the SSD read and write cache.
In the above image you see the situation of Test04 and how the pESXi (Physical ESXi) host has a pSSD (Physical SSD) and an NFS datastore to offer to the VMs running in the pESXi host. Inside that pESXi host I’m running a virtual ESXi (vESXi). That vESXi host is offered a virtual SSD (vSSD) which is running on the pSSD and a virtual SATA disk (vSATA) running on the NFS datastore. These vSSD and vSATA disks are then used by VSAN in the vESXi host and offered as the vSAN Datastore to all the hosts in the cluster of which the vESXi host is a member. And finaly, in memory of the pESXi host, there is my test VM getting running its VMDK on the vSAN datastore. Inception to the max…..
For my little home lab the most important thing is the comparison between the first test (all on the IX4) and the fourth test where I have the VSAN sitting between the IX4 and the VM. Looking at the results of the “Reallife 60% random and 65% read test”, I can see an improvement of 71 to 3865 IOPS and a throughput gain from 1 MBps to 30 MBps. That is a big improvement for a cheap home lab like this.
Even though the VSAN will cost me some overhead in RAM on the physical ESXi host since it has to run the virtual ESXi of 4GB, this configuration will bring me 2.7TB of pretty fast storage if I make the VSAN SATA disk as big as the storage available on the IX4. Of course there is the added risk of losing all VMs I put on the VSAN, if that one VSAN virtual SATA disk would fail, but hey we’ve got to be living on the edge a little don’t we.
Another option I have is to run the VSAN on the SATA disks of the physical host, which are the results shown in test number 3. That will give me a little better performance than when the SVAN SATA is on the IX4, but the difference is very small. The small difference can be explained since probably everything is written to the cache on the virtual SSD. In this configuration I don’t have the full 2.7TB available.
The fastest solution is of course to run everything on my physical SSDs on the physical host, but that will give me only 2x 256GB of capacity.
For me, putting the VSAN in front of the IX4, even by using virtual ESXi hosts that don’t have to run a VM load themselves, will greatly improve the performance of my IX4 and it will give me the opportunity to get more experience with the VSAN product even though I don’t have the proper RAID controller that is supported with VSAN.
Remember: Don’t use them for any performance reference, I just use them to see the difference in performance in my setup.
Max throughput test with 100% Read
|Max Throughput-100%Read||IOPS||MBps||Average Response Time|
Real Life 60% Random and 65% Read
|RealLife-60%Rand-65%Read||IOPS||MBps||Average Response Time|
Max Throughput with 50% Read
|Max Throughput-50%Read||IOPS||MBps||Average Response Time|
Random test with 8K – 70% Read
|Random-8k-70%Read||IOPS||MBps||Average Response Time|
Can’t wait for the official release of VMware’s VSAN….
See full post at: My completely ridiculous VSAN test
When trying to open the console of a VM using the vSphere Webclient with the VMware vCenter Server Appliance 5.5, I received the following error: “can’t establish a connection to the server at vcsa.vanzanten.local: 7331″.
It took me some searching through the VMware KB and finally found it. I’m posting it here merely for my self to find it quickly when I need it again. The solution is described in KB2060604 “HTML5 virtual machine console fails to open after restarting vCenter Server Appliance 5.5″. The problem is that an environmental variable is not set correctly. To solve the issue:
- Setup an SSH session into the vCSA
- Edit the file: /usr/lib/vmware-vsphere-client/server/wrapper/conf/wrapper.conf
- Add this line: set.default.VMWARE_JAVA_HOME=/usr/java/jre-vmware
- Save the file
- restart the vSphere Web Client: /etc/init.d/vsphere-client restart
Now you can login to the vSphere Web Client again and test the console connection. This solved it for me.
See full post at: Can’t establish connection to the server at 7331
Recently I was installing an additional VMware vCenter Server in an environment in which I had replaced all default certificates with official CA certificates and ran into an issue where installation of the VMware vCenter Inventory Service kept hanging when communicating with the VMware SSO server. In the installation wizard of VMware vCenter Inventory Service you’ll be asked for the vCenter Single Sign On information. You enter the admin@system-domain account, password and the URL to the Lookup Service.
Normally, in the next step you’ll be presented with a default certificate and you continue with installation of the certificate.
However in my configuration, installation hung before this step and would never present the certificate. After some playing around I discovered that killing the OpenSSL process in taksmanager would continue the install. After installation I continued with installation of the vCenter Server on the same Windows Server, but didn’t ran into any issues. After vCenter Server installation had finished all seemed to work nicely.
I’m not sure why this happens, but my guess is that openssl is waiting for a response, which you can’t see because it is running in a different context.
My environment has four different VMs:
- vCenter SSO 5.1 update 1a on Windows 2012 Server running with Enterprise signed certificate
- vCenter Inventory Service 5.1 update 1a and vCenter Server 5.1 update 1a on Windows Server 2012 running with Enterprise signed certificate
- vCenter Web Client 5.1 update 1a on Windows Server 2012 running with Enterprise signed certificate
- now adding another vCenter Inventory Service and vCenter Server 5.1 update 1 on Windows 2012, which didn’t have the Enterprise signed certificate during installation yet. And on this server I got the above mentioned issue.
Will also report this with VMware and see if there are any consequences. Use at your own risk
See full post at: Installation vCenter Inventory Service hangs forever
This week I installed a fresh vSphere 5.1 Update 1 environment and I wanted to configure it will real world certificates to get rid of all those “Do you really really reeeeeally accept this insecure website” messages. Using the VMware SSL Certificate Automation Tool I generated all the new certificates and then started changing the certificate on the VMware SSO server. When doing this, you’ll be asked for the Master password. Since I learned a while ago in a very painful way that the Admin@System-domain password is not equal to the Master password, I had written down the Master password and was 100% sure I had the correct Master password. But updating the certificate failed with the error: Incorrect master password. Tried it a few times but it kept failing. Logged in with admin@system-domain in the vSphere Web Client and this was the correct password.
I switched to command line and tried to run some SSO Util commands to make sure my password worked and then everything became very clear. I have a bad character in the password. In the password I set during install, there is an “&” (ampersand) and in many console languages this has a special meaning. When running some rsautil commands using the master password VMware&77 I get messages like: “77″ is not recognised as a command.
In my homelab I installed a fresh new SSO just for this test. During installation I set the master password to: VMware@55. Then I tested my rsautil command: rsautil manage-secrets -m VMware@55 -a list. This worked, I got a list of … well things.
I then changed the master password using this command: rsautil manage-secrets -m VMware@55 -a change -N VMware&77. This should set the old password to the new password “VMware&77″. Check the output below and notice that the rsautil did perform the change, but also reports an error. Trying the list command with the ‘new’ password, didn’t work.
What had happened is that the master password was changed to “VMware” and everything behind the & was lost. Proof would be if the rsautil command would work with the “VMware” password and it did: rsautil manage-secrets -m VMware -a list.
I did a new test. I removed SSO and the SQL Express database and again installed SSO using the master password “VMware&123″ to see what would happen. Login through the Web Client works using the user admin@system-domain and password VMware&123 but the command line tools don’t work.
As long as you don’t have to use the command line tools to change anything in SSO or to recover a password, you’re fine and using the & ampersand in your password won’t hurt you. But if you ever need to change anything with the help of the command line tools, for example when you lost your admin@system-domain password, then you’re lost.
My advice is to use a master password that doesn’t have the & in it. I tested with @ and that works fine. Using the exlamtion point ! also has some issues sometimes, so I would stay away from that too. And the release notes already mention that a space in the password will also get you into trouble.
Seems my friend Christian Mohn is better in using Google than me. He found a mention of this issue in the following KB: “vSphere 5.1 Single Sign On (SSO) installation fails with error: Error 29133. Administrator login error. (2035820)“. The KB article mentions that this issue is resolved now, but as you can see, it isn’t.
In some situations it can be fixed with: rsautil manage-secrets -a change I’ve also tested this in my home lab and somehow it didn’t work, later on I tried it in production because everything else had failed and it worked !!!
See full post at: Be carefull with VMware SSO Master password bug
Recently I had a chat with Saradhi Sreegiriraju and Ed Lee from Tintri about their new version of their Tintri Storage appliance, Tintri VMstore. This new version is an update of their first model with some really nice new features. New features that (I think) make the Tintri VMstore a real Enterprise solution.
Let me first explain what the Tintri appliance is. It is a box with a fixed set of SSDs for performance combined with a set of SATA disks for capacity. The utilization of the SSDs has been designed in such a way that by using compression and dedupe, between 98% and 100% of all read and write actions can be handled from the SSDs directly. Customer experience (Tintri currently has over 180 customers) has shown that only during backups a request is sometimes NOT serviced from flash. The Tintri T540 model offers 13.5TB of usable data storage of which 2.4TB is SSD storage (before dedupe and compression).
The Tintri appliance is able to keep “hot” blocks in flash if needed and if needed, the admin can move a whole VMDK into the SSD cache. “Cold” blocks of data are periodically moved to the SATA disks, continually freeing up space in flash for active workloads. The beauty in management of the storage is that you no longer need to worry about LUNs and RAID configurations, since there is only one big datastore managed by Tintri. Where other storage mostly reports IOPS and latency based on LUNs or RAID groups, Tintri is very VM oriented and will work with you on a VM level. You can see per VM and VMDK latency and how many IOPS each VM is generating.
The Tintri appliance is placed in your data center as one shelf and cannot be extended. If you need more storage you buy an additional appliance. Each appliance will present itself as NFS storage to your ESXi hosts. Each Tintri appliance can handle up to 1,000 VMs, depending of course on type and size of each one. Unlike other storage, Tintri enables both server and desktop VMs to coexist on the same datastore, allowing for both VDI and server apps to be on the same system.
On the hardware side, not much has changed. Where the very early model had just one storage adapter, this new release now has two redundant controllers and each controller has a 2x10Gbit connection. Nothing spectacular but something a true Enterprise ready storage should have.
At the software level we can find the most changes. Tintri now introduces the Tintri Clone feature. As stated in the first part of this article, Tintri is VM orientated storage and this new cloning feature is also fully VM orientated, which means you clone at the VM level, not at the LUN level.
This new cloning technique works mostly in the meta data and therefore has minimal performance overhead. When using the clone feature for VDI solutions, Tintri is able to deploy 1000 desktops including customization in little over an hour.
Another interesting observation and improvement I noted from this software release was the similarity of the software-defined storage features that Tintri delivers today to the future functionality that vVols will deliver in a couple of years. This will make it extremely easy for Tintri to support and their customers to adopt vVols when it does arrive on the market.
Tintri can now attach storage attributes and policies to a VM and set a QoS on a VM. It allows for VM and VMDK pinning to flash and you can isolate VDI IOPS from Server IOPS. To be clear, this is already present without VMware vVols.
Snapshots and Replication
The biggest changes in this new product from Tintri are the snapshot and the replication features. The snapshot feature allows you to create VM snapshots on a regular schedule and use them for fast recovery, shortening your RTO. A snapshot can be created hourly, daily or weekly or a combination of all. Per snapshot you can configure if it should be replicated and how long to keep the snapshot on the local storage and how long to keep it on the remote system. You can also choose for a crash-consistent or VM-consistent snapshot. The crash consistent is just a “dirty” snapshot of the VM, where the VM-consistent snapshot will quiesce the guest OS using the VMware Tools.
The new replication feature, called ReplicateVM, will replicate on a per VM basis to its partner using the snapshot schedule for each VM, which can be as often as every 15 minutes. Replication can be done in a many to one relation, meaning that you can have multiple Tintri systems replicate to one destination system.
To replicate as efficient as possible, Tintri has come up with this intelligent algorithm. First of all, you can pre-seed the target side with existing VMs or backups. Secondly there is a very intelligent dedupe and compression over WAN mechanism where the source will send a fingerprint of all the changed blocks. The target compares this fingerprint with data the target already has and then reports the missing parts of the fingerprint to the source. The source will now send the missing data compressed and the target will write this to SSD. After a few iterations the snapshot is available on the destination.
These snapshots on the target can also be used to create clone VMs on that same target that then can be offered to the ESXi hosts on the target side.
One of the big plusses of Tintri is the in-depth monitor that can tell you latency, IOPS, replication bandwith, etc. on a per VM basis. In most vSphere environments you need to combine vCenter Server performance graphs and your storage monitoring to get a clear view on how your VMs are performing storage wise. The storage will often give you a good view at what is happening at the LUN and storage level, but not at the VM level. Tintri offers a view on your VM performance that goes right from vCenter to the storage level. This is a very big plus when searching for possible performance issues.
When writing this post, Tintri gave me access to their lab and I was impressed by the speed at which VMs could be deployed and how easy it is to setup snapshotting and replication. All functions available in the Tintri appliance are easily accessible and it is very simple to learn how to use them. In my opinion you will get a storage appliance that is very easy to use and maintain. No more worries on LUN sizing, pool sizing, RAID sets, etc, etc. This will save the storage admin a lot of time.
See full post at: The new Tintri VMstore
A customer of mine, who was already running a vSphere environment on Cisco UCS blades, asked to expand his environment with a number of ESXi hosts that could run a VMware View environment. To be able to run as much View desktops on each host as possible, we offered them Cisco UCS Blades equipped with a Fusion IO card, the UCS 785GB MLC Fusion-io ioDrive2 to be exact.
For this customer I had already deployed a vSphere environment a year ago, fully based on VMware Auto Deploy and it now was time to try and add a driver to this Auto Deploy environment, which I had never done before. It wasn’t an easy road, but I wrote down all the steps and hope it helps you should you ever have to do this too.
Download the drivers
The company Fusion IO has a very tight policy as it comes to driver downloads. You can only get access to the support site and download the drivers after registering the serial number of the Fusion IO card. This will grant you access to the support site, but only to the drivers for that specific card. Very inconvenient when you’re only the guy implementing the stuff and especially if the cards are already built-in the UCS Blades located in a datacenter far far away.
After some time on the phone with their support department, I was able to get a link to the Cisco Support site and download a big ISO file containing all drivers for the Cisco UCS Blade. After unpacking the ISO and browsing through the folders, the Fusion IO driver only contains a shortcut to…….. the VMware Website. And there it is, free for download, the Fusion IO driver: VMware website – Fusion IO driver
After you downloaded the driver zip, place it in a separate directory and unpack it. In the directory in which you unpacked the zip, you will now find the following:
- iomemory-vsl-5X-220.127.116.119-offline_bundle-895853.zip <– this one we want !!!
- Subdir: doc
Prepare the Auto Deploy image
For the deployment we need three images. One is of course the ESXi image that can be downloaded from the VMware website, second is the Fusion IO driver (iomemory-vsl-5X-18.104.22.1689-offline_bundle-895853.zip ) and third very often is the vmware-fdm VIB for VMware HA.
I’m using vc-mgmt.vanzanten.local as the vCenter Server since that is the one running in my homelab, replace it by your own vCenter Server. Let’s get started:
Connect-VIServer -Server vc-mgmt.vanzanten.local Add-EsxSoftwareDepot -DepotUrl https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml Add-EsxSoftwareDepot f:\AutoDeploy\iomemory-vsl\iomemory-vsl-5X-22.214.171.1249-offline_bundle-895853.zip Add-EsxSoftwareDepot http://vc-mgmt.vanzanten.local/vSphere-HA-depot
Make sure that with both above actions you get a response similar to this:
Depot Url: zip:F:\AutoDeploy\iomemory-vsl\iomemory-vsl-5X-126.96.36.1999-offline_bundle-895853.zip?index.xml
After these three depots have been added, we can now combine packages from them into a rule, we just need to find out which ESXi image we would like to use. Let’s take a look:
Get-EsxImageProfile | Sort-Object -Property name | ft Name
You now get a list of all images available, for this case I pick the image: “ESXi-5.1.0-20130504001-standard” and use it to create a new ImageProfile called “ESXi-5.1.0-20130504001-std-FusionIO” and give it a Vendor ID.
New-EsxImageProfile -CloneProfile "ESXi-5.1.0-20130504001-standard" -Name "ESXi-5.1.0-20130504001-std-FusionIO" -Vendor "Fusion-IO"
You should get similar output like this:
Name, Vendor, Last Modified, Acceptance Level ESXi-5.1.0-20130504001-std.., Fusion-IO, 5/1/2013 07,PartnerSupported
Now we want to add the drivers to this new profile but for this we need to know the name of the software package that holds the Fusion IO driver. Run this to get a list of all software packages available:
When you browser through this list, you’ll see one line like this:
block-iomemory-vsl 188.8.131.529-1OEM.500.0.0.472560 Fusion-io 10/28/2012 01
We now know that the software package is called “block-iomemory-vsl”. Let’s add it:
Add-EsxSoftwarePackage -ImageProfile "ESXi-5.1.0-20130504001-std-FusionIO" -SoftwarePackage "block-iomemory-vsl"
And last but not least, let’s add the VMware HA package, which is called “vmware-fdm”:
Add-EsxSoftwarePackage -ImageProfile "ESXi-5.1.0-20130504001-std-FusionIO" -SoftwarePackage "vmware-fdm"
What we’ve done until now is create a new ESXi image that can be deployed to an ESXi host, but we didn’t create any rules for deployment yet. I have already explained this in earlier posts, so I’ll keep it short. Deploy this image to the ESXi host with IPv4 address 192.168.0.146 and assign the image “ESXi-5.1.0-20130504001-std-FusionIO”. Put the host in the cluster CL-MGMT and use the host profile named “ViewFusion-Hosts”.
New-DeployRule -Name "ViewHosts-FusionIO" -Item "ESXi-5.1.0-20130504001-std-FusionIO", "CL-MGMT", "ViewFusion-Hosts" -Pattern "ipv4=192.168.0.146"
You might get the following error when doing this:
New-DeployRule : 6/14/2013 19:56:58 New-DeployRule Could not find a trusted signer
Try this command to fix it and then run it again:
This command will take some time to complete. After this we need to push the Deploy rule to the “Active-Set”:
Add-DeployRule -DeployRule “ViewHosts-FusionIO”
We’re almost done after this, just a little fix is needed:
Get-VMHost * | Test-DeployRuleSetCompliance | Repair-DeployRuleSetCompliance
When you now look at the screen of the ESXi hosts booting, you should see the line with “block-iomemory” passing by.
See full post at: Add Fusion IO driver to VMware Auto Deploy
A few days ago I blogged about my new idea in this post: “The Doctor is: IN” and I received a lot of positive comments and even my first request for assistance. So… here we go !!!
Tonight ( June 7th ) at 19:00 CEST the first session will go live. That is, if Google Hangouts works because last night during my test run there were some issues with Google Hangouts.
This first session has been requested by Ryan Conley. Ryan is originally from Nashville area, moved to Greenville for two years which is where he got his feet wet with ESX/ESXi 4.x and 5.0. He took a position in San Francisco over a year ago and is now the primary person for managing two vSphere 5.0 clusters for two separate companies. He passed VCP5 in April and as a result of the prep time he is now hooked on learning more and more. He has his IaaS exam scheduled for the 17th of June and plans to attend VMworld in SF and sit the VCAP-DCA exam since it will be 75% off. Work wise he is about to standup a DR cluster in SF and begin implementing SRM to tie the two together.
Ryan e-mailed me at Gabrie (at) GabesVirtualWorld (dot) com and asked me for help on a few subjects. In tonights Hangout On Air we will be discussing the following:
- Some help, tips and tricks on VMware SRM and his Nimble Array.
- Upgrading vSphere 5.0 to 5.1
- Using the vCenter Server Appliance (vCSA)
- Study tips for his VCAP-DCA exam
The session has ended. See the video below. Click here for the PowerPoint slides I used.
See full post at: The Doctor is IN on SRM, Upgrade to vSphere 5.1, vCSA and VCAP-DCA
Many of us in IT read a lot of whitepapers, blogposts, how-to articles and view numerous Podcasts or training video’s to learn all the details about new products or features. Still, I don’t always get some of the details or can’t find the info I need. Meeting people at VMUGs or VMworld gives me the changes to ask for those last missing piece of information. But what if you don’t have that chance?
When I blog about stuff I’m sometimes surprised about the comments I get and about how more people than I thought were struggling with the same questions. In responds to the comments I have been able to help quite a number of readers of my blog by e-mail and lately I did a few Google Plus Hangout sessions to help in an even better way. And that is when I came up with the following idea: Why not do a video-chat help session?
I’m still thinking about what would be the best way to do this and I may change everything from session to session but this is what I came up with until now:
- Send me an e-mail ( Gabrie AT GabesVirtualWorld dot com) in which you write about a specific subject you would like to know more about or maybe an issue you have with a design or installation.
- I’ll then plan a meeting with you for a Google Hangout session. This might be an one-to-one session or maybe we’ll have a session with more people and collect a number of questions.
- The session will be live broadcasted and published on YouTube afterwards.
Some examples on what I think a session could be about:
- Think together with you on design questions
- Explain a specific vSphere or vCloud concept to you
- Help you study for VCP / VCAP DCA or DCD.
What will this NOT be?
- A complete training on vSphere or vCloud or other subject. If you need good training contact Eric Sloof or David Davis at TrainSignal.
- A support desk you can call when your server is down or a plane landed on your datacenter. Call VMware Support.
And as a disclaimer: I don’t know everything and I don’t have all the answers, but I do want to make the effort to help you and do some research myself when needed to be able to help you.
See full post at: The Doctor is: IN
Customer had all his VMware databases for vCenter Server, Update Manager, Single Sign On (SSO), vCloud Director and vCenter Chargeback running on one big SQL Server where they shared resources with other databases. They asked me to move the databases to a new MS SQL Server because the load of the VMware databases was more than expected. I prepared the move by reading a few VMware KB’s that described how to move the databases, but still experienced some issues, that is why I wrote this blogpost to have a good manual for the next time I have to move these databases.
This first post shows how to move the SSO database.
Move the SSO database
Before the actual move, make sure that you’ve read the following KB: Updating the vCenter Single Sign On server database configuration (2045528). Since we also have to move the database, which is not described in the mentioned KB article, there are some extra steps to perform. This post only works for MS SQL Server.
- Make a connection to the old database server
- On the SSO server, stop the Single Sign On service.
- On the old database server make a backup of the SSO database (default name is RSA).
- After a successful backup, set the SSO database to Offline
- Copy the backup to the new database server
- On the new database server, import the SSO database
- On the security section of the SSO database, check if the user RSA_USER is present.
After this, the SSO database is available, but there is no login user connected. Usually when you create a SQL user, you also make a mapping to a database which then automatically creates the database user. In our case the database user is already present but is not yet mapped to a SQL user. That’s what we’ll do now. First let’s make sure the RSA_USER is not yet mapped:
- Run the following sql query against the SSO database to show all unmapped users of the database: sp_change_users_login report
- Now create a new SQL User (SQL Authentication) at the SQL Server level not at database level. Name this user RSA_USER and use the same password the database RSA_USER has. Set the default database to RSA (the SSO database).
- Run the following sql query against the SSO database to map the user RSA_USER (server level) to the RSA_USER (database level): sp_change_users_login ‘update_one’, ‘RSA_USER’, ‘RSA_USER’
- To check if things worked out, rerun the query against the SSO database. The RSA_USER should not show up: sp_change_users_login report
When running the queries, make sure you run them against the correct database. See image.
Next step is to create the RSA_DBA user which is only a SQL Server user and not a database user. But this user should become the owner of the SSO database.
- At SQL Server level create the SQL user RSA_DBA and be sure to use the same password you previously used. (Well, you can always reset it later on).
- After the RSA_DBA user has been created, open the properties of the user and now set this user as the owner of the SSO database
Your database is now ready for use. We just have to tell the SSO Server that it should look somewhere else from now on. To do this run the following on the SSO Server:
- Go to the ssoserver\utils folder (usually: C:\Program Files\VMware\Infrastructure\SSOServer) . Run the command: ssocli configure-riat -a configure-db –database-host new_host_name
- You will be prompted for a password, this is the master password you also used to login to SSO with the user: admin@system-domain or root@system-domain.
- Check the jndi.properties file for the correct database reference. You can find it in C:\Program Files\VMware\Infrastructure\SSOServer\webapps\ims\web-inf\classes\
- Edit the file C:\Program Files\VMware\Infrastructure\SSOServer\webapps\lookupservice\WEB-INF\classes\config.properties and modify all values that need to be updated.
- To update the SSO DB RSA_USER password, run the command if, for example, the RSA_USER password has expired or the Database has been moved to another SQL instance: ssocli.cmd configure-riat -a configure-db –rsa-user-password new_db_password –rsa-user New_RSA_USER
Now cross your fingers and start the SSO Service. Check if you can logon to the SSO Server: https://<your SSO server>:9443/vsphere-client/ Login with: admin@system-domain
See full post at: How to move VMware Single Sign On (SSO) database
When working on a design in which there would be two physical datacenters, I started to rethink about why I would create multiple datacenters inside vCenter Server. I noticed that almost all example designs I look at that cover two physical datacenters, also have two datacenters in the vCenter Server design. But why?
When searching for an answer in the vSphere 5.1 documentation, the only reason mentioned to create a datacenter is “You can create multiple datacenters to organize sets of environments.“. But it also states that creating datacenters creates limits: “Inventory objects can interact within a datacenter, but interaction across datacenters is limited. For example, you can hot migrate virtual machines from one host to another host in the same datacenter, but not from a host in one datacenter to a host in a different datacenter.“
For organising objects in vCenter Server, I can also use folders. I can put a datacenter in a folder, I can put a cluster in a folder and I can set permissions at folder level. In other words, creating the logical design in vCenter Server to match the physical design, can also be done using folders, but without the limitations.
My basic statement is: “When I have two physical datacenters, I should not by default create two logical datacenters in vCenter Server”.
Yes, there are reasons to do create two datacenters, but those are very limited. When I asked the question on twitter I got quite a few responses and only one really demanded two logical datacenters, when creating a DMZ and you want to make sure VMs don’t travel between the two datacenters.
Most other cases in which people created two datacenters could also be matched by using folders.
I’m not saying that you shouldn’t be using datacenters, but I just want to see if my statement is correct and that you shouldn’t just blindly create multiple datacenters and thereby limiting functionality like Shared-Nothing Storage VMotion.
Would love to see your comments on that.
Update: Forbes Guthrie wrote a blogpost in response to my question. In his post he shows the do’s and don’ts for the vCenter Server datacenter: Why use the Datacenter object in vCenter?
See full post at: Design question: Why vCenter Server Datacenter?
Hope you enjoyed the previous series on VMware vCloud Director networking. In this post I will show you how to make the vCloud Cells available through a Load Balancer using the vShield Edge. By publishing two or more vCloud Cells behind a load balancer, it is easier to separate management network from user network and you make the vCloud more redundant because you can shutdown a cell without bringing down the whole vCloud. Do keep in mind that a user session will get disconnected when you shutdown a cell, but the user can immediately reconnect and will be redirected.
To better understand the configuration we need a design. Below you’ll see the design in which the ESXi hosts and the vCenter Server are in the management network (172.10.0.x range). The user network is in the 192.168.0.x range and all traffic goes over the user network to the internet. The real situation this design is based upon is a little different, since the management network is routed to the internet over another network, but that would only complicate the example.
Preparing the vCloud cells
Before we start configuring the load balancer, something have to be prepared on the vCloud cells.
- The first step is to make sure the certificates are the SAME on both cells. If you generated a self-signed certificate on each cell during install, you will have to change this.
- The second step is to make sure that the vCloud cells are properly syncing time.
I have been fighting an issue where the console proxy session (VMware Remote Console) would disconnect when two vCloud cells where behind the load balancer. If only one vCloud cells was running, then everything worked fine. Starting the second vCloud cell behind the load balancer would work well for the HTTP sessions, but the VMRC (VMware Remote Console) session would always disconnect. Didn’t matter which vCloud cell I would shut down, as long as just one was running the VMRC would work fine. This turned out to be an issue with the cells running a 2min time difference.
Installing same certificate on both vCloud cells
During installation of the vCloud cells you had to generate a certificate. In my case I put my certificates in a file named /etc/certificates.ks. Assuming the certificate on vCloud cell is ok, we’re now going to copy the certificate to the second vCloud cell. Let’s first check the certificates currently on both the vCloud cells. Open a browser and go to both the vCloud cells websites. You’ll see the security certificate icon telling you that this website uses a certificate. In my case I’m using firefox and when opening the certificate information, I see the following. Notice how the thumbprints are different.
- Logon through SSH on the first vCloud cell, make sure you are root or if you want to do it really safe use sudo
- Copy the certificates from this vCloud cell to a location where you can access it from the second vCloud cell. In my case I copy it to the data transfer directory. I have a special directory created there that holds the installation bin files of vCloud Director, the responses file and the certificates. Copy:
cp /etc/certificates.ks /opt/vmware/vcloud-director/data/transfer/install/
- Make sure you have the responses file from the first installation in a location that you can access from the second vCloud cell, but actually I would have expected that you already have this since you already installed a second cell. Just to be sure:
cp /opt/vmware/vcloud-director/etc/responses.properties /opt/vmware/vcloud-director/data/transfer/install/
- Logon to the second vCloud cell and switch to root (leave the session on the first open because we’ll return here later on for the time sync)
- Stop the vCloud services on the second vCloud cell:
service vmware-vcd stop
- Copy the certificates file from the first vCloud cell to the location of the certificates file of the second vCloud cell. In my case that is /etc/certificates.ks. Copy:
cp /opt/vmware/vcloud-director/data/transfer/install/certificates.ks /etc/certificates.ks (yes to overwrite).
- To active the certificate, you need to rerun the configuration of vCloud Director. By using the responses file, you don’t have to answer all the database questions. In my case I need to run this to reconfigure:
/opt/vmware/vcloud-director/bin/configure -r /opt/vmware/vcloud-director/data/transfer/install/responses.properties
- After configuration is complete answer YES to have the configuration script start the vCloud service or start it yourself using:
service vmware-vcd start
- Monitor the startup of the vCloud cell and wait for “Complete. Server is ready”. Monitor:
tail -f /opt/vmware/vcloud-director/logs/cell.log
Let’s check if the certificates on the second vCloud cell indeed has changed by browsing to the second vCloud cell. Open the information of the certificate in your browser and see how the thumbprints match. Fase1 completed.
Time sync of the vCloud cells
It seems that having a time drift between the two vCloud cells of more than 2 minutes will cause disconnects of the console proxy session. Since the ntpd is NOT running in a standard RedHat install, we need to make sure it is, it will at start up and that the time is synced. The ntpd depends on the /etc/ntp.conf file which is already configured correctly by default if your vCloud cell is allowed to connect to the internet. In the next steps we’re going to start ntp, make sure it will run at startup and force a time sync:
service ntpd start
chkconfig --level 345 ntpd on
ntpdate -u clock.redhat.com
- watch date
Run the above on both cells and to see if time is in sync, it will run the date command and refresh every 2sec. Stop by hitting ctrl+c. Fase 2 completed. Now we can finally start with configuring the load balancer.
vShield Edge Load Balancer
To create the vShield Edge Load Balancer, we need to go to the vShield Manager. Logon to the vShield Manager and go on the lefthand side to the datacenters section and select your datacenter, not the cluster. On the righthand side click “Network virtualisation” and “Edges”.
Before we start configuring the vShield Edge Load Balancer, make sure you have all the IPs noted down that you need. The vShield Edge Load Balancer will get two IP addresses in the user network, one for HTTP, one for ConsoleProxy. On the management network only one IP is needed which can connect to the vCloud cells.
Now press the plus sign to create a new Edge device. The wizard will pop up and will ask for the name of the Edge device. For the name I entered: “vCloud-LoadBalance“. The hostname is “vcloud.domain.com“, this is the outside FQDN. You can leave the tenant field empty and I didn’t enable HA. Press next and you’ll be asked to enter the credentials for this edge device. If you want to use the same credentials you use for the vShield Manager, just leave it like it is. Click next. In this screen you can select the Appliance size, choose Compact and you can specify where the appliance should be place. Click on the little plus to configure placement. Select the Cluster/Resource pool and the datastore. I choose the “System vDC” resource pool because I feel that it belongs there. If you have better suggestions let me know. Click SAVE and then next.
In this next step we will be configuring the interfaces of the Edge device. Click the plus sign to add a new interface.
The first interface will be the external interface. On this interface we will connect two IP addresses, the HTTP (192.168.0.1) and ConsoleProxy address (192.168.0.2). Click the plus sign to add a new interface. Name it: “external“, select the PortGroup you want it to connect to. Make sure the connectivity status is “connected” and then click the plus sign to add IP addresses. Enter the IP addresses and the subnet mask. In our case: 192.168.0.1 (primary) and 192.168.0.2. Subnet mask 255.255.255.0. Save the IP settings and you will return to the first wizard. Leave Mac addresses empty, adjust MTU if needed, leave Enable Proxy ARP and Send ICMP Redirect empty. Click ADD. Now add the second interface which will be connected to the internal address 184.108.40.206 in the same way you did for the external interface.
Click Next in the wizard and you will get the “Default Gateway” screen. Enable “Configure default gateway”, select the external nic and enter the correct gateway address. In my case that would be 192.168.0.254. Change the MTU if needed. Again click Next.
Now the firewall & HA screen will ask to enable the firewall. Since this is a device that could be connected to the big bad world, enable the firewall policy and set the default to “Deny”. Click next to continue and finish to close the wizard. When you look at your vCenter Server you should be able to see the deployment of an OVF file.
When done you should see the following in the vShield Manager.
The next step is to configure the real load balancing. First we need a “Pool” of servers that offer the same functionality. In our case the Pool of servers will hold all vCloud cells. Second we then need a virtual server. The virtual server is what is published to the outside world.
Click the “vCloud-LoadBalance” we’ve just created and click “Actions” -> “Manage”. Here you’ll see an overview of the configured settings. Click “Load Balancer” and be sure that “Pools” is selected. Click the plus sign to start the “Add Pool” wizard. The name will be “vCloud_HTTP” (no spaces allowed). Next we configure the services. Enable HTTP and HTTPS and choose the balancing method “Least connections“. Leave the ports unchanged at 80 and 443.Click next. Now the Health Check screen appears. Only select HTTP (although I think HTTPS should work too) and for the URI for HTTP Service use: “/cloud/server_status“. Be sure to start with a forwardslash. Click Next.
Now we need to enter each member of the pool. Be aware that we are only defining the HTTP part of the vCloud cell, so we add only the HTTP addresses of the vCloud cells for this pool. In our case add two IP addresses: 220.127.116.11 and 18.104.22.168. Add them both with a weight of 1. Leave the defaults of HTTP and HTTPS.
Next we’ll add the Console Proxy servers to the pool. Run the same wizard again and name it “vCloud_Console“. In the Services screen you have to pay a little attention, you only need TCP over port 443, not HTTPS !!! This is because the console proxy traffic travels over port 443 but it is no HTTPS traffic. In other words, on the services screen select TCP and change the port to 443 and select the “Least Connections” balancing method. In the Health Check screen select TCP and monitor port 443, leave the URI for HTTP Service empty. In the next screen we again need to add some IP addresses. Enter the IP addresses of the console proxy of the vCloud Cells ( 22.214.171.124 / 126.96.36.199). You now return to the vShield Manager screen. A little above the list of pools you see the “ENABLE” button, click this to enable the pool and push the “Publish Changes” button.
We now have a pool of servers defined that can accept the traffic from the load balancer. Now we need to define the services that the load balancer will allow from the outside.
Click “Virtual Servers” and then the plug sign to start the “Add Virtual Server” wizard. You are now prompted to enter a name for this virtual server, use “vSrv_vCloud_HTTP“. Now enter the IP address of the outside HTTP interface 192.168.0.1. Select the pool “vCloud_HTTP” and enable HTTP and HTTPS. Leave the defaults of persistence method cookie for HTTP and SSL_SESSION_ID for HTTPS. Click Add.
Again click plus to add a new virtual server. Name this server “vSrv_vCloud_Console“. Now enter the IP address of the outside HTTP interface 192.168.0.2. Select the pool “vCloud_Console” and enable TCP and change the port to 443. Click Add. Click Publish Changes.
That’s it. Your vCloud Edge Load Balancer is ready.
More posts on vCloud Director Networking can be found here:
- VMware vCloud 5.1 Networking for dummies part 1 – External access for a vCloud VM
- VMware vCloud 5.1 Networking for dummies part 2 - Connecting to vApp networks and giving external access
- VMware vCloud 5.1 Networking for dummies part 3 – Publish a vCloud deployed VM to the outside world
See full post at: VMware vCloud 5.1 Networking part 4 Cell Load balancing
Today Mike Laverick drew my attention to his blogpost about BareMetalCloud. He writes about the possibilities to run your virtual ESXi host on their cloud offering. And best of all, by using a promocode, something like MikeLaverickIsAnAwesomeDudeAndThisWillGiveMeCloudForFree or maybe a shorter version, you can get free access.
Read all about it on Mike’s blogpost: Bare Metal Cloud for your Auto-Lab.
See full post at: Bare Metal Cloud for your Auto-Lab by Mike
I don’t often publish news bulletins and especially not about products that haven’t been released yet, but today I made an exception. In my mailbox I found Veeam’s announcement of Veeam Backup & Replication v7. The announcement is about the ability to back-up vCloud Director VMs including the vApp, metadata and attributes.
This is great news and something I’ve been wanting since my first vCloud Director project. When designing a vCloud Director environment for my customer I was surprised that there was no way to backup the intelligence of the vCloud which is the vApp including all the NAT & Firewall rules and networking that have been created. The only way for backups of vCloud VMs was to just backup VMs through vCenter and if you needed to restore them, you would just restore the VM in a different location and then bring it into the vCloud through an import.
Veeam also announced the possibility to restore your vApps and VMs. Because Veeam also makes a backup of the metadata the integration is much further than just a simple backup of a group of VMs. Nice !!!
So MY assumption is that it will now be able to restore vApps including NAT & Firewall rules, networking and maybe even restore into a different vCloud. Unfortunately I can’t get any confirmation on this, but knowing Veeam in the past years, I don’t think they will deliver only half a solution.
Of course…. the product hasn’t been released yet, but I do keep my fingers crossed!!! Follow this link to see all new features published in the next few months: Go.veeam.com/v7
Below the content of the e-mail:
Veeam is extending its support for VMware vCloud Director (vCD).
NEW integration with vCD will extend Veeam support for vCD, which currently includes the ability to backup and restore virtual machines (VMs) managed by vCD. Using the vCD API, Veeam Backup & Replication v7 will also:
- Display the vCD infrastructure directly in Veeam Backup & Replication
- Backup all vApp metadata and attributes
- Restore vApps and VMs directly to vCD
- Support restore of fast-provisioned VMs—and more!
Sound exciting? Learn more about upcoming updates here!
See full post at: Veeam first to offer real vCloud Backup?
It’s that time of year again, Eric Siebert launched the new vLaunchpad top VMware/virtualization blogs survey and is asking you to vote for the best blogs on virtualization in 2013. Go to: http://www.surveygizmo.com/s3/1165270/Top-vBlog-2013 and select your top 10 favourite blogs.
This year my blogging was mostly about getting to know vSphere 5.1 and my first steps in to the world of VMware vCloud Director. With my vCloud Directory post I’ve written a lot about how vCloud Director networking isn’t always as straight forward as you think. Well at first it doesn’t look like it is easy but once you “get it”, you’ll learn to love it. These posts on VMware vCloud networking for dummies got a lot of attention this year:
- VMware vCloud 5.1 Networking for dummies part 1 - External access for a vCloud VM
- VMware vCloud 5.1 Networking for dummies part 2 - Connecting to vApp networks and giving external access
- VMware vCloud 5.1 Networking for dummies part 3 - Publish a vCloud deployed VM to the outside world
Ive also stumbled a few times on bigger and smaller issues with vCloud. Have a look at these posts:
- VMware vCloud error: Batch update returned unexpected row - About very vague error messages
- VMware vCloud 5.1 network troubleshooting - Tips on checking your firewall and NAT rules
- VMware vCloud Transfer spooling area is not writable - Strange issue I encountered when installing a second vCloud Director cell
- Change IP address of VMware vCloud Cell - When you need to change the IP of your vCloud Director cell.
Now please vote at: http://www.surveygizmo.com/s3/1165270/Top-vBlog-2013
Thank you for reading my blog.
See full post at: Voting for the top virtualisation blogs of 2013
Oh those little things in life that sometimes cost you a whole day before you figured out that you we’re doing it all wrong. This was one of those days where I got alerted by some colleagues about vCloud Director reporting “Disk space red threshold crossed.”
When looking at the “Datastores” in the “Manage & Monitor” section, I noticed that there was still quite some free space and I had only added more space yesterday. When checking the properties of each datastore, I noticed the “threshold section” and I remember changing them on the newly added storage and just to be sure, yesterday I also set them for the existing data stores because I had left them at default before.
*light bulb moment* When looking at those thresholds, I noticed that you can read them in two ways.
The way I read this, was that the red threshold would give an alarm when xx GB was in use and a yellow threshold when yy GB was in use. So I set them to be very close to the Total capacity. But that turns out to be quite the other way around. These thresholds are talking about how much free space should be at least left. In other words I should have set them as low as possible in my case. When entering the values there is no check that tells you that red should always be smaller than yellow, in fact you set switch these values around as you like.
Well, I’ll never forget again, hope you will neither.
See full post at: vCloud disk space thresholds crossed
(Shamelessly copied from Yellow-Bricks.com)
Would like to hear more about Software Defined Datacenters from experts like Frank Denneman, Mike Laverick, Cormac Hogan, Kamau Wanguhu and many others? VMware and IBM are organizing an awesome event in the Benelux. Yes this is a full day event, and it is free for everyone, if you just want to sign up… go here. If you need to be convinced keep reading as there are some awesome sessions scheduled.
09.00 - 09.30 Registration
09.30 - 09.45 Welcome
09.45 - 10.30 Keynote VMware: Software-Defined Data Center
10.30 - 11.15 Keynote IBM: Converged Systems: beyond NextGen DC’s
11.15 - 11.30 Break and split into parallel sessions
11.30 - 12.15 Parallel track 1 or meet the expert
12.15 - 13.00 Lunch
13.00 - 13.45 Parallel track 2 or meet the expert
14.00 - 14.45 Parallel track 3 or meet the expert
15.00 - 15.45 Parallel track 4 or meet the expert
16.00 - 16.45 Parallel track 5 or meet the expert
16.45 - 17.30 Networking drink
The awesome part is that at this event you will also have the ability to sit down with one of the experts for a 1:1 discussion and get your questions answered. Below is the list of people you can sit down with, make sure to register for that!
Frank Denneman – Resource Management Expert
Cormac Hogan – Storage Expert
Kamau Wanguhu – Software Defined Networking Expert
Mike Laverick – Cloud Infrastructure Expert
Ton Hermes – End User Computing Expert
Tikiri Wanduragala – IBM PureSystems Expert
Dennis Lauwers – Converged Systems Expert
Geordy Korte – Software Defined Networking Expert
Andreas Groth – End User Computing Expert
So if you live in The Netherlands, Belgium or Luxemburg… make sure to sign up. As mentioned, it is a free event. And with people like Cormac Hogan, Frank Denneman, Mike Laverick and Kamau Wanguhu you know it is going to get deep technical.
- 5th March – Amsterdam
- 7th March – Brussels
- 8th March – Luxemburg
See full post at: Software Defined Datacenter Roadshow – Benelux – Free Event!
I’m still getting used to the important part vCenter SSO (Single Sign-On) is playing in vSphere 5.1. In my home lab I was switching domain controllers from W2k8 to Windows 2012. Transferred FSMO roles, integrated DNS, changed IP addresses for DNS on all servers and all seemed fine. My w2k8-dom01 server was demoted and removed. Few days later when trying to make a vCenter connection, I couldn’t logon anymore. As a good Windows Admin I of course first rebooted the vCenter VM but (as all real admins know) that seldom fixes the issue. Diving in the vCenter log files at “C:\ProgramData\VMware\VMware VirtualCenter\Logs” I found the following error:
2013-01-27T10:46:20.302+01:00 [04304 info '[SSO][SsoAdminFacadeImpl]‘] [FindPersonUser] 2013-01-27T10:46:20.427+01:00 [04304 warning 'Default'] Warning, existence of user “VANZANTEN\Administrator” unknown, permission may not be effective until it is resolved. 2013-01-27T10:46:20.427+01:00 [04304 error 'Default'] The user account “VANZANTEN\Administrator” could not be successfully resolved. Check network connectivity to domain controllers and domain membership. Users may not be able to log in until connectivity is restored. 2013-01-27T10:46:20.427+01:00 [04304 error 'Default'] [ACL] Adding unresolved permission for user “VANZANTEN\Administrator” 2013-01-27T10:46:20.427+01:00 [04304 info '[SSO]‘] [UserDirectorySso] GetUserInfo(VANZANTEN\vSphereAdminGroup, true) 2013-01-27T10:46:20.427+01:00 [04304 info '[SSO][SsoAdminFacadeImpl]‘] [FindGroup] 2013-01-27T10:46:20.536+01:00 [04304 warning 'Default'] Warning, existence of group “VANZANTEN\vSphereAdminGroup” unknown, permission may not be effective until it is resolved. 2013-01-27T10:46:20.536+01:00 [04304 error 'Default'] The group account “VANZANTEN\vSphereAdminGroup” could not be successfully resolved. Check network connectivity to domain controllers and domain membership. Users may not be able to log in until connectivity is restored.
That is when the lightbulb in my head went on (hey it was Sunday morning and I didn’t have my coffee yet). I figured the vCenter SSO (Single Sign-On) service was still trying to talk to the old domain controller for authentication, so I opened the administration page of SSO: https://<your SSO server>:9443/vsphere-client/ Login with: admin@system-domain (or root@system-domain for the vCenter Server Appliance). You did write down that password, did you? Go to: “Sign-On and Discovery” -> Configuration and look at the identity sources. There it is, the reference to the old domain controller.
Change vCenter SSO domain controller
Changing it is easy, click “Edit identity source” and change the name of the new domain controller. Test your settings and you would expect to be done then.
Unfortunately I received the following error: ”The edit identity source operation failed for the entity with the following error message. Cannot connect to one or more of the provided external server URLs: w2k8-dom01.vanzanten.local:389″.
Strangely enough it is still pointing at the old domain controller. The easiest way to solve this was to just delete the entry and create a new one with the new domain controller in. Just to be sure I restarted the vCenter SSO (Single Sign-On) service first and then vCenter Server would start without any issues.
In the comments on this post, Jad (JM) wrote that you can also use the domain name when entering the primary server URL. In my case I would NOT enter: “ldap://w12-dc01.vanzanten.local” as my Primary URL but just “ldap://vanzanten.local”. vCenter SSO will then query the domain for the special domain controller DNS record and use this to find the domain controller to talk to. Btw, this doesn’t work if you already have entered two URLs and want to change them and make the second one empty. Think it is purely a GUI issue. When I empty the second URL, save it and then edit to check again, it keeps the old entry. When you delete the whole entry and create a new one that has an empty secondary URL, you’re fine. Thanks Jad!
See full post at: vCenter SSO changes when demoting domain controller
In my last VMware vCloud Director project I ran into an issue that I want to share with you. Initially the design was to create two OrgVDC to use different allocation models, but the resources would be almost the same because it was only a fairly small environment. I created one Edge Gateway and two OrgVDCs connected to this Edge Gateway, with a shared Org network. The first OrgVDC was meant to be for real performance testing and would get FC Storage, while the second orgVDC would host most VMs and was only for Development and testing.
Because of some changes in the final design, I didn’t need the first OrgVDC anymore and I tried to delete this, but this was impossible. When trying to delete it, the well known error: ‘The vCloud Org cannot be deleted because it is in use”. Checking all vApps, vApp Templates, catalogs etc, but there was nothing still connected to this OrgVDC except of course for the shared Organizational network and the shared Edge.
This is when I first noticed that the second OrgVDC didn’t show that the Edge was connected to it. See the screenshots.
The only way to delete the OrgVDC is to first delete the OrgNetwork the OrgVDC is connected to. It seems that somewhere inside vCloud the direct connection to the first OrgVDC with the OrgNetwork and the Edge remains the primary connection that can never be deleted without deleting the OrgNetwork. Creating a new OrgVDC that uses the shared OrgNet, doesn’t “transfer” the OrgNet to the second OrgVDC.
Seems logical, but I expected that somehow you could transfer this shared OrgNetwork and connect it to a new OrgNetwork. This does mean that any extra OrgVDCs you build on a shared OrgNet always stay dependent on the first OrgVDC that “created” the shared OrgNet. Good thing to remember in your next design.
See full post at: vCloud can’t delete Organizational VDC
When creating NAT and firewall rules in a vCloud environment, not everything works out directly the way you had planned. This post will show you how to use Splunk to collect sys logs from the vCloud environment and make it an easy to use tool when performing some network troubleshooting.
Splunk comes in a number of flavours like Windows, OSX, Linux. In my environment I have a RedHat based NFS server that is doing almost nothing and will now be promoted to be my Splunk syslog server. Start by downloading the free Splunk version from www.splunk.com. You’ll need to create an account and then go to the Linux RPM (because it is RedHat). My RedHat server is without GUI, so I first tried the download link in a wget command, but that didn’t work since there are some scripts behind the download link. I was very pleasantly surprised though that the Splunk website offers a special wget link. Excellent! After downloading the RPM you can simply install it with “rpm -i <download rpm>”. Open the webpage of your Splunk server at port 8000 and walk through the brief configuration wizard.
To add sys logging as a service (called an App in Splunk), go to: App -> Home (upper right corner). Click “Add Data” and choose “syslog” from the list, then select “consume syslog over UDP”. In the next screen enter UDP port 514, leave “Source name override” blank and select source type “syslog” from the list, click SAVE. Your syslog server is ready to receive data.
Enabling syslogging in vCloud Director
In the vCloud interface make sure the loggings are sent to the syslog server. To set the IP address of the syslog server go to the system tab in vCloud Director, select Administration, General and in the networking section you can now enter the IP address of the syslog server. Press apply to save.
When building firewall rules you can choose to log traffic per rule. This is very handy to exclude a lot of traffic of rules that already work fine. When creating a new rule I would advise to enable logging for that rule and (for a short time) also enable logging for the bottom rule, which is usually the rule that denies everything. By enabling logging on this last rule you can see that packets that don’t meet any of the previous rules are dropped. This often helps you understand why those packets are dropped.
After sys logging is enabled in vCloud Director and you have enabled logging at the firewall level, it is now time to show how Splunk can help you troubleshoot.
Find the syslog data
The first time it took me quite some time to find the correct view / report and I still have to try a few times before I get the right report, but luckily you can save a search once you found the correct one and access it very quickly. Open the Splunk web interface again and select “Manager” in the upper right corner. Here you’ll see a page with lots of options, choose “Searches and reports”. At the top of this page, select “All” for the App Context. In the list below find the item “Messages by minute last 3 hours” and select “Run” at the end of the item. A new view will open and you can now see a lot of matching events in a graph. At the top of the page select “Dashboards & Views” and “Summary”. And finally, there is the “UDP:514″ option we need, Click it. A new page will open and show you the latest events coming from your Edge firewall. To make it easier the next time you want to access this report, press the “SAVE” button and give this report a name. Now you can easily access this report from “Manager” – “Searches and Reports”.
After setting up Splunk, configuring vCloud Director and finding were the reports are hidden, we can now finally do some troubleshooting. I have created a number of firewall rules which are working nicely and I’m just about to add a new rule that I want to test: RDP to desktop1 with a Source address that is external and a destination of 192.168.0.45. See how I enabled logging for this rule. Also notice the last rule in the background on which I also enabled logging.
After the rule is created, go to the Splunk interface and open the syslog search page and generate some traffic that should be handled by the newly created rule. For this example I created a rule that I know isn’t going to work, just for demonstration purpose. Hit refresh and you can now see on the search page that traffic has been captured, showing you the packets were dropped. Notice the text: “DROP_131073″. This is a clear clue on WHY the packets were dropped.
To understand the 131073, you will have to open the vShield Manager in your vCloud Environment. After logon to the vShield manager web interface select the “Edges” view, then select “Edge Gateways”, in the middle pane select your Edge device and click “Actions” – “Manage” – “Firewall”. In this view you can see the firewall rules that are generated automatically by the Edge device and your manually created rules. Note that the “Rule Tag” number is equal to the number you see in the vCloud interface. In this case the DROP_131073 tells us that the packets were dropped because of the default any/any rule which denies all traffic. (Click on the image to enlarge)
Not only the DROP message is important. To understand why your traffic is blocked and try to create a working rule, you’ll also need to check:
|SRC||Source IP address|
|DST||Destination IP address|
|PROTO||Protocol (TCP, UDP, IGMP)|
If you look at the syslog and now understand why your rule failed, you can easily edit the rule and check if it is working now. What you’re hoping for is of course the “ACCEPT” message in the logs. In my case it shows “ACCEPT_6″ because custom rule number 6 was the match to accept these packets:
When you’re done troubleshooting, don’t forget to disable logging on the rules where it is no longer needed. Have fun !!!
See full post at: VMware vCloud 5.1 network troubleshooting
In November 2009 I already did a review on StorMagic’s SvSAN and was then very pleased with the product. Now, three years later I’m again reviewing SvSAN and am anxious to see if in those three years SvSAN has improved.
For those not familiar with StorMagic SvSAN I will shortly explain what SvSAN does. With SvSAN you will be able to use the local disks of one or two ESXi hosts and present that local storage as shared storage to your ESXi environment. By mirroring the storage between two ESXi hosts, SvSAN is protected against a single host failure. The following diagram shows how on each ESXi host a SVA (Storage Virtual Appliance) is installed and how the local disks are connected to the SVA and then mirror and presented to the whole vSphere cluster over iSCSI. Should the ESXi host holding the SVA with the primary mirror fail, then the second SVA will present the datastore to the ESXi hosts in the cluster.
What is a neutral storage host?
|Counter||Test 1||Test 2|
|Total IOs per second:||143.08||119.70|
|Total MBs per second:||1.17||0.98|
|Average IO Response time:||417.6953 ms||508.2921 ms|
|Maximum IO Response time:||883.2917 ms||39038.1664ms|
Comparison with VMware VSA
See full post at: [Review] Stormagic SvSAN 5