Planning the upgrade to vSphere 6

Upgrades bring unease to the minds of lots of vAdmins and a major version upgrade can particularly cause trepidation. With the right planning, testing and implementation, it can be an uneventful experience – an upgrade from any older version of vSphere to 6.0 is no different. In this post, I’ll run through planning the upgrade to vSphere 6.0. Keep in mind though, I cannot possibly talk about every feasible scenario and you should do what works for your environment.

Requirements and near-future goals?

Single Sign On

With vSphere 6.0, SSO is one the those things you need to decide to nail the design for first simply because it is not possible to change the layout later or very hard to do so. With 6.0 comes the concept of an embedded PSC or an external PSC. PSC stands for Platform Services Controller in which SSO + VMCA (certificate authority) + lookup service + licensing have been combined into a single machine. The vCenter VM will now contain all the other remaining services, the vCenter Server service itself, the Inventory Service, the Web Client Service, AutoDeploy, Syslog and Dump Collector Services. I think this is great – you don’t have to worry about multiple VMs anymore. The ‘problem’ with this is you’ve got to decide whether you go with an external PSC or an embedded one. Questions I’d ask you:

    • Do you have more than 1 vCenter Server in the same SSO site/location? Yes – external PSC. No – maybe an embedded PSC
    • Do you have multiple services, such as vRA/vRO/SRM that require SSO? Yes – external PSC. No – maybe an embedded PSC
    • Do you have multiple sites/locations with their own vCenter Server and would like linked mode (more on linked mode in a moment)? Yes – external PSC. No – maybe an embedded PSC
    • Do you require High Availability for PSC (not just VMware HA)? Yes – external PSC with HA rules. No – still an external PSC without HA rules
    • Do you require Linked Mode across your vCenter Servers? Yes – external PSC. No – maybe an embedded PSC
    • Do you generally or by policy separate licensing from management? Yes – external PSC. No – maybe an embedded PSC
    • Do you need to minimize the number of machines you are deploying? Yes – maybe an embedded PSC.

Why do I have the word ‘maybe’ here? Any combination of the above points could apply to you. If they do – you get yourself an external PSC. Future proof yourself with an external PSC.

Why do I have “near-future” in the heading? Too short-sighted? Not really – we are in a once in a lifetime stage of virtualization or even technology in general where things are moving forward with breakneck speed making it hard to predict the future and plan too many years in advance.

Note a Windows PSC and a Linux PSC can’t be put behind a load balancer as this isn’t supported (so one type of PSC only at this stage).

The Deployment Guide contains all the supported and unsupported scenarios. Your point of truth should be the Deployment Guide. For PSC HA, check this nice article to get PSC to work with the F5 load balancer.

Joseph Griffiths (@Gortees) put it quite succinctly during a techtalk at VMworld 2015:

Separate your PSC from your vCenter Server for:

      • Single management point (using Enhanced Linked Mode) for all vCenters via the web client only (note with a 6.0 PSC you’ll see the 5.5 vCenter instances, but not the other way around)
      • Common central location for tags and categories (be aware though that tags aren’t stored in the vCenter Server DB so you gotta find a way to move them from 5.x to 6 manually)
      • Permissions applied at a single location (apply once and you are set, possible issue could be too many unintended permissions on a vCenter so check the permissions)
      • Central authentication for all VMware services (future release)
      • Single storage and generation point for SSL certificates (mostly true though host SSL certs are located on the hosts themselves, not in VECS)
      • Cross vCenter vMotion (requires Enhanced Linked Mode which is provided by an external PSC)

Linked Mode

VMware now call it Enhanced Linked Mode. Why ‘Enhanced’? You can now have Windows and/or Linux based vCenter Servers in linked mode. This is especially great for Linux shops that’d like to save a Windows license or for folks wishing to move away from Windows in general (given its greater patching requirements). If you need Linked Mode going, you need an external PSC. An embedded PSC isn’t supported to work with multiple vCenter Servers (though it works).

How does it work? When a user logs on, they are issued with a SAML 2.0 token by SSO or PSC (in vSphere 6). This SAML token allows them access to all vCenter Servers they have permissions to. As long as the SAML token is valid, users are allowed access to all vCenter Servers they have permissions to (even if the local PSC was down).

Upgrade1

Windows vCenter and want to move to Linux based vCenter

Now’s your chance to do it. VMware have a fling (check if it’s supported) for this, check it here. Slight catch is you need to have the Windows vCenter talking to an external SQL database (so embedded SQL wont do) and you need to migrate Linked Mode instances one after the other meaning the Linked Mode configuration wont be migrated too. For all other requirements and limitations check the article I linked to.

You can also have Windows vCenter working in tandem with the VCSA in the enhanced linked mode I referred to above, here’s the VMware documentation.

To set up a VCSA from scratch, apart from the usual VMware kb, here‘s a link.

Current setup and other VMware solutions?

Current SSO layout?

Your current SSO layout will dictate how you setup your PSC instance(s). If you already have your SSO instance(s) separate from vCenter you’ll need to go with the external PSC because the installation wizard will detect an external SSO and ask you to upgrade to an external PSC. You should upgrade one SSO instance at one time, confirm successful operation, proceed to next SSO machine, upgrade it to a PSC and so on. At no stage should you do concurrent upgrades.

If you have SSO and vCenter on the same machine, you will be upgraded to an embedded PSC.

As discussed above, if you require PSC to be highly available, you need to put it behind a supported load balancer. NetScaler and F5 are the only two supported load balancers at this stage.

As part of the upgrade, you can define a different vCenter SSO domain name to be used instead of vsphere.local. In addition, you may notice local OS accounts have stopped working. Check this kb article for information on how sign in with local users is affected with this upgrade. Local OS account become relatively irrelevant or unnecessary with SSO though some folks have traditionally kept them as a backdoor into SSO if domain authentication has broken or the SSO administrator password has been forgotten.

Collapse number of SSO machines?

If you need to decrease the number of SSO machines, you should do so before the upgrade to 6.0. This can happen particularly when past policies have stated you need to separate management services from sign-in services or vCenter sprawl that is common in larger organizations. Repoint vCenter, the Web Client and the Inventory Service to a SSO machine of your choice, confirm successful login and operation, backup the old SSO machine and decommission it. Leave the environment running for a few days, plan your upgrade and proceed.

5.0 and earlier instances?

If you have 5.0 or older versions which did not have SSO, you have the option to go with an embedded or external PSC. Again, future proof yourself and go with an external PSC to begin with. If you are a tiny shop or this is a segregated environment such as a DMZ where you don’t forecast any growth, an embedded PSC may work for you. Check with your organization’s security policies and go accordingly.

If you have any 4.x or older instances you need to go to 5.x first and then onto 6.0. Take note that hosts must have greater than 350MB in their boot partition (if they dont, you can upgrade them interactively or script your way through).

Need to upgrade/migrate OS or VM?

If you are running vCenter on a Windows 2008 R2 machine and need to upgrade/migrate to Windows 2012 R2, you should do so before the upgrade. VMware don’t support migration to a new VM or OS as part of the vCenter upgrade.

AutoDeploy?

If you run AutoDeploy on a different machine than vCenter (which some large organizations do, since it is typically the larger ones that use this feature), note that AutoDeploy is moved to the same machine as vCenter. You need to repoint your hosts and/or fix your rules so your hosts point to the right location. You then use AutoDeploy to dish out 6.0 images to your hosts.

View/vRO/vROps/vRA/SRM?

All of these VMware solutions require SSO and if any or all of these exist in your setup, you need to go with the external PSC. No options. Ideally, you’d separate the vCenter Server for say your VDI environment from the one managing your server virtualization – this would require an external PSC.

SRM is even more tightly integrated with the Single Sign On service in vSphere 6.0 making it critical to ensure you deploy a PSC instance in both your protected and recovery sites. If you had a single PSC working for both your protected and recovery sites and a DR procedure was initiated, you’ve just got an impossible to recover from scenario if the PSC were offline. SRM requires the PSC to be online to function and without one working instance, your sites are dead in the water. Deploy an external PSC at protected and recovery sites, each PSC will talk to vCenter and SRM at its respective site. Check the various deployment models here and best practices here.

You must ensure you point SRM to its local instance of PSC so when the need to recover a site arises, your SRM machine is working with a live and online instance of the PSC.

Upgrade or Migrate?

Now this is a dilemma, ain’t it?! Short answer, it depends. VMware have generally recommended a fresh install but this is not always possible. The questions I’d ask you are:

    • Is your hardware EOL or close to it or you require more functionality? Yes – migrate. How? I’d upgrade SSO, vCenter and VUM first (assuming no other solutions). I’d either add the new hosts to the same cluster (with the right EVC mode), vMotion VMs over to the new hosts and decommission the old ones or I’d create a new cluster and move the VMs over. CPU instruction sets may likely change so test before you forge ahead. Don’t mix hosts containing AMD and Intel chips in the same cluster, EVC requires the same chipset vendor for hosts in a cluster. If you are looking to move from Intel to AMD or the other way around you should create separate clusters and cold-migrate your machines over.
    • Do you need to maintain historical data? Yes – definitely upgrade. vCenter data is of particular importance in any virtual infrastructure and historical data is necessary for trending, forecasting and just comparing old with new. An implementation plan may look like this without going into much detail:
      • Take a full backup of the vCenter database and the vCenter VM (include the SSL certs)
      • If needed, get your SQL team involved in the migrating of the database from say SQL 2008 to SQL 2012. Check interoperability matrix.
      • Snapshot the vCenter machine
      • Follow upgrade sequence
      • Replace SSL certs
      • Confirm successful operation and commit snapshots
    • Is your current hardware compatible with vSphere 6? Yes – upgrade. No – migrate to supported hardware first. You do not want to be in the situation where things don’t work and a call to VMware GSS says your hardware isn’t supported or compatible. Again, bookmark the compatibility guide and use it as your point of reference to check for compatibility queries.
    • Strict policies? Yes – upgrade. If you have strict regulatory and auditing policies, you need to upgrade. Banks and other financial organizations usually have such requirements which require you to maintain old data rendering you unable to make a fresh start.

Other bits

Backups

Backups are critical and you cannot chance an unsupported or untested version in a production environment. Backups software vendors have been notorious for lagging behind in upgrading their code to support backups of VMs running on newer versions of vSphere. You can upgrade your backups software to a version that works with vSphere 6 or hold off upgrading to vSphere 6 or perhaps move to a vendor that supports vSphere 6. Some of the popular vendors such CommVault announced their support for vSphere 6, so did Symantec here. Leverage the VADP functionality if you aren’t already and move away from in-guest agent based backups.

Andrea Mauro has this nice table of versions of the popular backups software that are compatible with vSphere 6.0. I suggest you check with your vendor for updates. Andrea also has another article about an issue with Veeam backups, see it here.

Licensing?

Yep, make sure your licensing carries over to vSphere 6.0 since 4.x and/or 5.x licenses won’t work. In most cases all you need to do is go the VMware licensing portal and upgrade your licensing to 6.0. Here’s a kb on how to do this. Just make sure you have your licenses upgraded before you upgrade. You will have 60 days to run in evaluation mode but that’s usually not recommended nor the norm in production environments though probably okay when transitioning over.

For a how-to on installing licenses, check this kb. Visit the Licensing section of the upgrade center for procurement and eligibility information.

Software inter-operability

Check the interoperability matrix for guidance on whether certain VMware products work with others. It’s paramount to maintain and ensure compatibility for a smooth upgrade, you don’t what to be in situation where you need to rollback (which by the way isn’t supported for vCenter for example). Another blogger has this interesting post on how a customer wanted to upgrade, they confirmed all was well in the matrix, but when they looked deeper there were a few incompatibilities. Check and double-check!

AV

AV is an interesting topic when it comes to virtualization. Lots of non-virtualization folks think “oh, it’s a hypervisor, it doesn’t need AV”. Well, no, and you are going to run guest OSs on top of it, that’s the purpose of virtualization, ain’t it? Anyway, I digressed there a bit. My point is you need some kind of AV running in there – be it guest OS based (not generally a good idea – think storage bloat in large environments) or off-loaded to the host (a good idea but you need to plan for it and cough up money). VMware have their own product called vShield Endpoint which is installed as a secure appliance on the host to which AV and malware processing is offloaded to take the load off guests. If you already run it or intend doing so, make sure you upgrade in right sequence. Great read here in a blog post by Josh Townsend on vShield Endpoint and vCenter. As always, consult the interoperability matrix for vCenter versions that go with vShield Endpoint:

vShield Interoperability Matrix1

Certificates

Last but not the least, don’t forget SSL certificates! With vSphere 6, VMware have moved the certificate services into the VMCA or the VMware Certificate Authority (this is part of the PSC). The VMCA will now handle all certificates issued to your vCenter Servers, solution users and hosts. VMware have this term called “solution users” which essentially means collections of vSphere services. Certificates are stored in the VMware Endpoint Certificate Store or VECS for all endpoints (vCenter and PSC for example), for ESXi hosts they are stored on the hosts themselves.

Certificates are provisioned to hosts either explicitly or when they are added to vCenter Server. If you need to use your PKI authority issued SSL certificates, it’s best to install the PSC, replace the root VMCA certificate with one issued by the PKI authority and then create/upgrade vCenter and hosts. This way all hosts and vCenter Servers will get your custom SSL certificates. Here’s a post I wrote about replacing the VMCA signed Root Certificate with one issued by your organization’s PKI authority. It is also important to set the certificate mode to ‘custom’ in vCenter Server’s advanced settings so every host gets the custom SSL certificate. If you decide to replace SSL certificates after creating/upgrading your environment, you are going to have reissue certificates for all your endpoints. Plan in advance!

For an overview on the VMCA, check this VMware blog. Here’s another VMware blog post with screenshots and detailed steps on requesting and replacing certifcates. For design decisions on which kind of VMCA mode to chose, check this blog post. Finally, for VMware documentation on certificates for 6.0 check this link.

Leave a Comment

Your email address will not be published.

1 Trackback

  1. Newsletter: September 13, 2017 | Notes from MWhite (Pingback)