vCenter root certificate expiry using Sectigo-AddTrust-External-CA-Root-Expired on May-30-2020.

Are you facing an issue with the root certificate expiry issue and your using the certificates provided SECTIGO.

This would cause the services to fail on the vCenter server we can see the below line in the log:

  • Vpxd-svcs.log:

Caused by: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint verification is not configured at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:224)]Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sat May 30 10:48:38 UTC 2020

  This root is due to expire at the end of May, 2020.  Any applications or installations that depend on this cross-signed root must be updated by May, 2020 or run the risk of outage or displayed error message.

What to do?

Validate the logs in the vCenter server.

  • Generate a CSR (Certificate Signing Request) on the PSC/vCenter for the certificate replacement using certificate manager.
    • VCSA – /usr/lib/vmware-vmca/bin/certificate-manager
    • Windows vCenter – C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager)

Refer below information to enter values for CSR generation

Country : Two uppercase letters only (Eg. US), the country where your company is located.
Name : FQDN of the vCenter Server
Organization : Company Name
OrgUnit : The name of your department within the organization. Example: “IT”
State : The state/province where your company is located
Locality : The city where your company is located.
IPAddress : IP Address of vCenter Server, this field is Optional
Email : Email Address
Hostname : FQDN of vCenter Server
VMCA Name : FQDN of vCenter Server with VMCA (Usually External PSC or VC with Embedded PSC FQDN)

  • This task replaces the Machine SSL Certificate with a Custom CA Signed Certificate.
  • This certificate is not issued by VMCA. It is issued by an external Certificate Authority.
  • If you are running an external Platform Services Controller, you will need to restart the services on the external vCenter Server 6.x and then proceed with replacing the Machine SSL of the vCenter Server 6.x.

Useful links:

https://kb.vmware.com/s/article/2112277

https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020

vSphere SSL Certificates

What is a certificate?

A Certificate or digital certificate is a unique, digitally Signed document that authoritatively identifies the identity of an individual or organization. Using public-key cryptography, its authenticity can be verified to ensure that the software or website you are using is legitimate. On the Internet, a certificate is signed by a trusted CA (certificate authority), and verified with the authority’s public key. The decrypted certificate contains a verified public key of the certificate holder (website operator), with which encrypted HTTPS communications can be established.

An operating system or web browser may alert the user when loading software or a website whose digital certificate is not verified by a trusted CA.

A certificate, contains information about the owner of the certificate, like e-mail address, owner’s name, certificate usage, duration of validity, resource location or Distinguished Name (DN) which includes the Common Name (CN) (web site address or e-mail address depending of the usage) and the certificate ID of the person who certifies (signs) this information.

It contains also the public key and finally a hash to ensure that the certificate has not been tampered with. As you made the choice to trust the person who signs this certificate, therefore you also trust this certificate. This is a certificate trust tree or certificate path.
Usually your browser or application has already loaded the root certificate of well known Certification Authorities (CA) or root CA Certificates.

The CA maintains a list of all signed certificates as well as a list of revoked certificates.

A certificate is insecure until it is signed, as only a signed certificate cannot be modified.

You can sign a certificate using itself, it is called a self signed certificate.

All root CA certificates are self signed.

We have seen what is a certificate, Now let’s see how are certificates used in the vSphere Environment.

vSphere Certificate:

The below is error that we get when we try to login to the vCenter server using the browser, because the certificate is not trusted by the computer in your organization by default.

In day to day scenario’s most of us see the below web browser certificate warnings when accessing the vSphere Web Client? Those are caused by an untrusted (and perhaps self-signed) Machine SSL certificate.

We will be able to bypass warning by different methods, let see how it works and what are the different ways to use the certificates in the vSphere environment.

vSphere Certificate Management:

Certificates ensure that communication between services, solutions, and users are secure and that systems are who we think they are. By default, VMCA acts as a root certificate authority. Certificates are issued that chain to VMCA where the root certificate of VMCA is self-signed as it is the end of the chain.

The certificate Lifecycle can be defined in two ways,

VMware vSphere 6.x Solution for Complete Certificate Lifecycle Management

VMware Certificate Authority (VMCA):

Located on: Embedded Deployment and Platform Services Controller (vCSA with external PSC).

VMware Endpoint Certificate Store (VECS):

Located on: Embedded Deployment, and vCenter Management Node (vCSA with external PSC).

VMware Certificate Authority (VMCA):

  • Dual Operational modes:
    • Root CA
    • Issuer CA
  • Root CA:

Automated one which is created during the installation, this has the capability of issuing other certs, all solutions and endpoint certificates are created and trusted to this root cert

  • Issuer CA:
  1. Can replace all default root CA certificate created during installation.
  2. This requires a CSR from VMCA to be used by an enterprise or 3rd party CA to generate a new issuing certificate.
  3. This will replace all the default certificates issued during the installation.
  4. Managed using the certificate manager utility
    • /usr/lib/vmware-vmca/bin/certificate-manager
  5. VMCA then issues certificates to any vCenter Servers and associated ESXi hosts that are registered to it.
  6. The real value of the VMCA is in the automation of replacing and renewing certificates without having to manually generate CSRs, mint certificates, then manually install those certificates.

VMware Endpoint Certificate Store (VECS):

  • Repository for certificates and private keys
  • Key stores:
    • Machine SSL certs
    • Trusted roots
    • CRLs
    • Solution users (certificates issued by the VMCA are for internal service-to-service communication within vCenter Server)
    • others (e.g. VVOLS, VASA etc.).
  • We can manage VECS using the vecs-cli
  • Does not manage SSO certificates

Types of Certificates in VMware vSphere 6.x:

  • ESXi Certificates
  • Machine SSL Certificate
  • Solution User Certificates
  • Single Sign-On Certificates

ESXi Certificates:

Machine SSL certificates:

Solution User Certificates:

  • Solution User Certificate Certificate stores are located in VECS on each management node (PSC) and embedded deployment.
  • machine – Used by component manager, license server, and the logging service.
  • vpxd – vCenter service daemon (vpxd) store on management nodes and embedded deployments. vpxd uses the solution user certificate to authenticate to vCenter Single Sign-On.
  • vpxd-extensions – Includes the Auto Deploy service, inventory service, and other services that are not part of other solution users.
  • vsphere-webclient – Includes the vSphere Web Client and some additional services such as the performance chart service.

Single Sign-On Certificates:

  • Not stored in VECS.
  • Not managed with certificate management tools.
  • Security Token Service (STS) – an identity provider that issues, validates, and renews SAML tokens that are used for authentication throughout vSphere
  • By default, the STS signing certificate is generated by VMCA
  • Manually refresh STS certificate via vSphere Web Client when the certificate expires.

We will see in detail about the different types of certificates and how to implement them. Please follow my page for regular updates on virtualization.