This time I want to introduce the ability to request the famous Let’s Encrypt certificates on your NSX Advanced Load Balancer (also known as Avi Vantage) using comprehensive on-board functionality it offers.

I was improving, adding new features and testing the script for quite some time now, which got now reviewed and merged by Engineering on 31st of August 2021 on GitHub. One major change was to support the "Virtual Hosting" feature with Let’s Encrypt and also allow specifying multiple domains for the certificate (SANs).

Support for the script: If you have any troubles or questions regarding the script, I’d recommend raising a GitHub Issue. It’s easier to keep track of them and more people might be able to assist.

Why?

In the Controller you can manage all your SSL/TLS certificates at one central point. All those certificates can be used across your virtual services, running on different Service Engines. Obviously, before a certificate expires it should be renewed and replaced.

In the case of free, well-known public certificate authority Let’s Encrypt certificates are only valid 90 days, where the best-practise is to renew it every 30 days. Performing this step manually each month, is probably not the most exciting work.

This is where Certificate Management and the "ControlScript" in NSX ALB joins the party. This feature – by default – initiates a renewal 7 days before the certificate expiry. Or, in other words: Just right before the penultimate certificate expiry notification as configured. (For more information see Avi’s documentation for "Customizing Notification of Certificate Expiration" here)

Controller: Certificate Management

With this feature issuing Let’s Encrypt certificates for Virtual Services can be completely automated.

Takeaway: To debug the certificate renewal it’s really handy to manually trigger the renewal via CLI. For more details, you can give this QuickTip a read.

Setup

To set up everything you need, follow below steps from your Avi Controller’s webinterface.
You need to do this once per tenant.

To note: Starting 21.1 (Release Notes), what I use, the webinterface has been rebuilt using VMware’s Clarity website framework going forward, hence previous versions do look differently. Beside the visual difference, the steps however should be pretty much the same.

Step 1: Check the requirements/disclaimer

1. You need a public domain which is also publicly reachable.
2. You need a Virtual Service with an Application Profile of type HTTP (so Layer 7). Issuing certificates to virtual services of type L4 do not work, as it requires HTTP Policy Sets to complete the validation.

Some more notes:

1. You should be aware that Let’s Encrypt does do Rate-limitating based on IP address. If issuing certificates fails, you might want to enable dryrun, investigate the cause and keep retries to a minimum to not reach the limit.
2. Please read carefully through this guide. When done a few times, it’s way less complicated than it looks like.
3. This is a community-supported script. So no official support is provided. If there’re issues, please consider raising a GitHub issue.

Step 2: Create ControlScript

1. Go to Templates - Scripts - ControlScripts and hit Create. A dialog will open.
2. Open the LetsEncrypt renewal script in a separate tab: raw.githubusercontent.com/avinetworks/devops/master/cert_mgmt/letsencrypt_mgmt_profile.py. Copy the content in your clipboard.
3. Pick a name like request_letsencrypt_certificate.
4. Paste the script, we copied right before, in the large textarea below.
5. It will now look like this:
   
   

   LetsEncrypt: ControlScript

6. Click Save to return to the previous dialog.

Step 3: Create "Certificate Management"

1. Go to Templates - Security - Certificate Management and hit Create. A dialog will open.
2. As Name, pick something like use_letsencrypt (Only alphanumeric, undersscore, period or hyphen characters are allowed)
3. At Control Script, select the script named request_letsencrypt_certificate we created in the previous step.
4. Enable Enable Custom Script Parameters. At a minimum, you need to add at least following two values: (Check additional parameters in the next step!)
   1. user for the username used for the API calls. Can be a custom user (recommended), or some admin account.
   2. password for the password, marked as Sensitive.
   3. This account needs permissions to manage and change SSL/TLS Certificates via API/WebUI.
5. Additionally to above parameters, there are more options you can and might need to define. I recommend to set most of these as Dynamic, so it can be changed on a certificate-basis individually.
   1. tenant contains the name of the tenant to be used. If not specified, admin will be used.
   2. dryrun defaults to False and production servers are used. If True set, the staging/test servers of LetsEncrypt will be used which have different ratelimiting settings.
   3. disable_check determines if the token will be validated from the Avi Controller. See more details in Appendeix-section down below. Usually no change is needed.
   4. debug defaults to False. If set to True, more debug messages are printed.
   5. contact can be a e-mail address provided to Let’s Encrypt. Certificate expiry warnings will be sent to this email address.
6. In the end, it will look like this:
   
   

   LetsEncrypt: CertificateManagement

Step 4: Request a certificate

1. Go to Templates - Security - SSL/TLS Certificates, click on Create - Application Certificate.
2. As a Name you can pick something like patrik.kernstock.net RSA.
3. As the Type pick CSR.
4. As the Certificate Management Profile we pick use-letsencrypt, what we created in the previous step. (Make changes to Dynamic Parameters, if defined and required.)
5. As Common Name pick the FQDN to what the certificate should be issued to. For example: patrik.kernstock.net.
6. Chose Algorithm and Key Size as required. We can go for RSA and 3072 Bits.
7. Add any Subject Alternate Name (SAN), if required. This are additional domain names which should be included in the certificate.
8. Click Save.
9. When saved, the script will be run in the background and used to issue your certificate accordingly. This might take a few seconds. If it fails, you will see the output of the script with more details. (Note: Script output will only displayed starting 20.1.6)

If the script suceeds, you will see the recent issued certificate in the list:

SSL/TLS Certificates List in Controller

Now your certificate is ready to use. Happy certificiating… or so…

Final notes

Notes

I don’t see an error!

If you don’t see any additional errors (e.g. when using older versions than 20.1.6), you can see more logs in the log file /var/lib/avi/log/portal_exception.log on your Avi Controller. To check this log file, login to your controller via SSH and check the log file using less -i /var/lib/avi/log/portal_exception.log or tail -f /var/lib/avi/log/portal_exception.log.

Additionally you might want to set the custom parameter debug to True.

How to use RSA and ECDSA?

To issue both, a RSA and ECDSA certificate, you simply create two SSL/TLS Certificates entries and chosing the Algorithm down below accordingly. You can then define both certificates on your Virtual Service:

ECDSA and RSA

Errors

Error: "All 5 internal token verifications failed."

(This also provides more details about the parameter disable_check)

As described earlier in Step 3, point 5, the token verification can be disabled by setting the parameter disable_check to True.

To understand this parameter further, I’ll need to briefly explain the token validation of Let’s Encrypt:

1. The ACME standard, what Let’s Encrypt invented, is used to automatically issue certificates and proof ownership. Of course noone wants to have other/evil people issue valid certificates for their domains.
2. At first, we’re going to tell Let’s Encrypt – or the ACME server in general – for which domain we want a certificate issued. It gives us back a token Let’s Encrypt expects to see at a certain URL to proof ownership to them.
3. The script then sets a HTTP Policy on the corresponding Virtual Service to return a specific token string at the URL http://our-domain-on-avi.tld/.well-known/acme-challenge/<TOKEN>.
4. Before the script tells Let’s Encrypt to verify the token, the Avi Controller (the script to be precise) makes a HTTP call to above URL and validate the token locally. This keeps us from getting rate-limited to quick in case validation fails.
5. If the validation suceeds, we inform Let’s Encrypt that the token can be verified. On success, the certificate will be issued and handed over to the Avi Controller to process. Issuing complete.

In some setups you might use split-horizon DNS:

1. our-domain-on-avi.tld inside your network points to an internal server, directly on the webserver.
2. our-domain-on-avi.tld from outside your network points to the NSX ALB/Avi Load Balancer.

As the Avi Controller validates the token within the local network, it will never go through the Load Balancer and therefore never hit the HTTP Policy set through the script. Essentially causing the local validation to fail. By setting disable_check to True we simply bypass this check.

End

I hope this was useful for some Avi/NSX ALB fans out there!

Changelog

• 2021-12-27: Added note regarding support on GitHub repo.