--- title: "Automate certificate renewal via Let's Encrypt on Avi/NSX ALB" url: 'https://patrik.kernstock.net/blog/automate-certificate-renewal-via-lets-encrypt-on-avi-nsx-alb' markdown: 'https://patrik.kernstock.net/blog/automate-certificate-renewal-via-lets-encrypt-on-avi-nsx-alb.md' date: '2021-09-03' description: 'This time I want to introduce the ability to request the famous Let’s Encrypt certificates on your NSX Advanced Load Balancer (also known as Avi Vantage) using comprehensive on-board functionality it offers. I was improving, adding new features and testing the script for quite some time now, which …' taxonomy: category: - VMware - 'NSX & Avi' tag: - 'Avi Load Balancer' - "Let's Encrypt" --- # [Automate certificate renewal via Let's Encrypt on Avi/NSX ALB](https://patrik.kernstock.net/blog/automate-certificate-renewal-via-lets-encrypt-on-avi-nsx-alb) September 3, 2021 (updated December 27, 2021) 7 min [VMware](https://patrik.kernstock.net/category:VMware) [NSX & Avi](https://patrik.kernstock.net/category:NSX%20%26%20Avi) [Avi Load Balancer](https://patrik.kernstock.net/tag:Avi%20Load%20Balancer) [Let's Encrypt](https://patrik.kernstock.net/tag:Let%27s%20Encrypt) Note This article is more than 5 year(s) old – information may be outdated. ![Automate certificate renewal via Let's Encrypt on Avi/NSX ALB](https://patrik.kernstock.net/images/2/0/2/1/-/2021-08-3122-48-44-uygsplba-f449c76d.png?g-b78a5bcf "Automate certificate renewal via Let's Encrypt on Avi/NSX ALB") This time I want to introduce the ability to request the famous Let’s Encrypt certificates on your NSX Advanced Load Balancer (also known as Avi Vantage) using comprehensive on-board functionality it offers. I was improving, adding new features and testing the script for quite some time now, which got now reviewed and merged by Engineering on 31st of August 2021 on [GitHub](https://github.com/avinetworks/devops/pull/237). One [major change](https://github.com/avinetworks/devops/pull/237/files) was to support the "Virtual Hosting" feature with Let’s Encrypt and also allow specifying multiple domains for the certificate (SANs). **Support for the script**: If you have any troubles or questions regarding the script, I’d recommend raising a [GitHub Issue](https://github.com/avinetworks/devops/issues). It’s easier to keep track of them and more people might be able to assist. ## Why? In the Controller you can manage all your SSL/TLS certificates at one central point. All those certificates can be used across your virtual services, running on different Service Engines. Obviously, before a certificate expires it should be renewed and replaced. In the case of [free, well-known public certificate authority Let’s Encrypt](https://letsencrypt.org/) certificates are only valid 90 days, where the best-practise is to renew it every 30 days. Performing this step manually each month, is probably not the most exciting work. This is where [Certificate Management](https://avinetworks.com/docs/latest/certificate-management-integration-for-csr-automation/) and the "ControlScript" in NSX ALB joins the party. This feature – by default – initiates a renewal 7 days before the certificate expiry. Or, in other words: Just right before the penultimate certificate expiry notification as configured. (For more information see [Avi’s documentation for "Customizing Notification of Certificate Expiration" here](https://avinetworks.com/docs/latest/customizing-notification-of-certificate-expiration/)) [![Controller: Certificate Management](https://patrik.kernstock.net/user/pages/01.blog/2021-09-03-automate-certificate-renewal-via-lets-encrypt-on-avi-nsx-alb/2021-09-01_00-43-27-E1gWlVF0.png?g-b78a5bcf "Controller: Certificate Management")](https://patrik.kernstock.net/user/pages/01.blog/2021-09-03-automate-certificate-renewal-via-lets-encrypt-on-avi-nsx-alb/2021-09-01_00-43-27-E1gWlVF0.png?g-b78a5bcf) Controller: Certificate Management With this feature issuing Let’s Encrypt certificates for Virtual Services can be completely automated. **Takeaway**: To debug the certificate renewal it’s really handy to manually trigger the renewal via CLI. For more details, you can [give this QuickTip a read](https://patrik.kernstock.net/blog/quicktip-manually-trigger-certificate-renewal-on-avi-controller-nsx-alb). ## Setup To set up everything you need, follow below steps from your Avi Controller’s webinterface. You need to do this once per tenant. **To note**: Starting 21.1 ([Release Notes](https://avinetworks.com/docs/21.1/release-notes/)), what I use, the webinterface has been rebuilt using VMware’s Clarity website framework going forward, hence previous versions do look differently. Beside the visual difference, the steps however should be pretty much the same. ### Step 1: Check the requirements/disclaimer 1. You need a public domain which is also publicly reachable. 2. You need a `Virtual Service` with an `Application Profile` of type `HTTP` (so Layer 7). Issuing certificates to virtual services of type L4 do not work, as it requires `HTTP Policy Sets` to complete the validation. Some more notes: 1. You should be aware that Let’s Encrypt does do [Rate-limitating](https://letsencrypt.org/docs/rate-limits/) based on IP address. If issuing certificates fails, you might want to enable `dryrun`, investigate the cause and keep retries to a minimum to not reach the limit. 2. Please read carefully through this guide. When done a few times, it’s way less complicated than it looks like. 3. This is a community-supported script. So no official support is provided. If there’re issues, please consider raising a [GitHub issue](https://github.com/avinetworks/devops/issues). ### Step 2: Create ControlScript 1. Go to `Templates - Scripts - ControlScripts` and hit `Create`. A dialog will open. 2. Open the LetsEncrypt renewal script in a separate tab: [raw.githubusercontent.com/avinetworks/devops/master/cert\_mgmt/letsencrypt\_mgmt\_profile.py](https://raw.githubusercontent.com/avinetworks/devops/master/cert_mgmt/letsencrypt_mgmt_profile.py). Copy the content in your clipboard. 3. Pick a name like `request_letsencrypt_certificate`. 4. Paste the script, we copied right before, in the large textarea below. 5. It will now look like this: [![LetsEncrypt: ControlScript](https://patrik.kernstock.net/images/2/0/2/1/-/2021-08-3123-32-10-opbm15ez-1728fa59.png?g-b78a5bcf "LetsEncrypt: ControlScript")](https://patrik.kernstock.net/user/pages/01.blog/2021-09-03-automate-certificate-renewal-via-lets-encrypt-on-avi-nsx-alb/2021-08-31_23-32-10-OPbm15EZ.png?g-b78a5bcf) LetsEncrypt: ControlScript 6. Click `Save` to return to the previous dialog. ### Step 3: Create "Certificate Management" 1. Go to `Templates - Security - Certificate Management` and hit `Create`. A dialog will open. 2. As `Name`, pick something like `use_letsencrypt` (Only alphanumeric, undersscore, period or hyphen characters are allowed) 3. At `Control Script`, select the script named `request_letsencrypt_certificate` we created in the previous step. 4. Enable `Enable Custom Script Parameters`. At a minimum, you need to add at least following two values: (Check additional parameters in the next step!) 1. `user` for the username used for the API calls. Can be a custom user (recommended), or some admin account. 2. `password` for the password, marked as `Sensitive`. 3. This account needs permissions to manage and change `SSL/TLS Certificates` via API/WebUI. 5. Additionally to above parameters, there are more options you can and might need to define. I recommend to set most of these as `Dynamic`, so it can be changed on a certificate-basis individually. 1. `tenant` contains the name of the tenant to be used. If not specified, `admin` will be used. 2. `dryrun` defaults to `False` and production servers are used. If `True` set, the staging/test servers of LetsEncrypt will be used which have [different ratelimiting settings](https://letsencrypt.org/docs/rate-limits/). 3. `disable_check` determines if the token will be validated from the Avi Controller. See more details in Appendeix-section down below. Usually no change is needed. 4. `debug` defaults to `False`. If set to `True`, more debug messages are printed. 5. `contact` can be a e-mail address provided to Let’s Encrypt. Certificate expiry warnings will be sent to this email address. 6. In the end, it will look like this: [![LetsEncrypt: CertificateManagement](https://patrik.kernstock.net/images/2/0/2/1/-/2021-08-3123-46-48-pwymwwtd-c301d12e.png?g-b78a5bcf "LetsEncrypt: CertificateManagement")](https://patrik.kernstock.net/user/pages/01.blog/2021-09-03-automate-certificate-renewal-via-lets-encrypt-on-avi-nsx-alb/2021-08-31_23-46-48-PwyMwwtd.png?g-b78a5bcf) LetsEncrypt: CertificateManagement ## Step 4: Request a certificate 1. Go to `Templates - Security - SSL/TLS Certificates`, click on `Create - Application Certificate`. 2. As a `Name` you can pick something like `patrik.kernstock.net RSA`. 3. As the `Type` pick CSR. 4. As the `Certificate Management Profile` we pick `use-letsencrypt`, what we created in the previous step. (Make changes to `Dynamic Parameters`, if defined and required.) 5. As `Common Name` pick the FQDN to what the certificate should be issued to. For example: `patrik.kernstock.net`. 6. Chose `Algorithm` and `Key Size` as required. We can go for `RSA` and `3072 Bits`. 7. Add any `Subject Alternate Name (SAN)`, if required. This are additional domain names which should be included in the certificate. 8. Click `Save`. 9. When saved, the script will be run in the background and used to issue your certificate accordingly. This might take a few seconds. If it fails, you will see the output of the script with more details. (Note: Script output will only displayed starting 20.1.6) If the script suceeds, you will see the recent issued certificate in the list: [![SSL/TLS Certificates List in Controller](https://patrik.kernstock.net/images/2/0/2/1/-/2021-08-3122-48-44-uygsplba-726eb0a7.png?g-b78a5bcf "SSL/TLS Certificates List in Controller")](https://patrik.kernstock.net/images/2/0/2/1/-/2021-08-3122-48-44-uygsplba-7b4c23d9.png?g-b78a5bcf) SSL/TLS Certificates List in Controller Now your certificate is ready to use. Happy certificiating… or so… # Final notes ## Notes ### I don’t see an error! If you don’t see any additional errors (e.g. when using older versions than 20.1.6), you can see more logs in the log file `/var/lib/avi/log/portal_exception.log` on your Avi Controller. To check this log file, login to your controller via SSH and check the log file using `less -i /var/lib/avi/log/portal_exception.log` or `tail -f /var/lib/avi/log/portal_exception.log`. Additionally you might want to set the custom parameter `debug` to `True`. ### How to use RSA and ECDSA? To issue both, a RSA and ECDSA certificate, you simply create two `SSL/TLS Certificates` entries and chosing the `Algorithm` down below accordingly. You can then define both certificates on your `Virtual Service`: [![ECDSA and RSA](https://patrik.kernstock.net/user/pages/01.blog/2021-09-03-automate-certificate-renewal-via-lets-encrypt-on-avi-nsx-alb/2021-09-01_00-41-45-8lTCLJxL.png?g-b78a5bcf "ECDSA and RSA")](https://patrik.kernstock.net/user/pages/01.blog/2021-09-03-automate-certificate-renewal-via-lets-encrypt-on-avi-nsx-alb/2021-09-01_00-41-45-8lTCLJxL.png?g-b78a5bcf) ECDSA and RSA ## Errors ### Error: "All 5 internal token verifications failed." (This also provides more details about the parameter `disable_check`) As described earlier in Step 3, point 5, the token verification can be disabled by setting the parameter `disable_check` to `True`. To understand this parameter further, I’ll need to briefly explain the token validation of Let’s Encrypt: 1. The ACME standard, what Let’s Encrypt invented, is used to automatically issue certificates and proof ownership. Of course noone wants to have other/evil people issue valid certificates for their domains. 2. At first, we’re going to tell Let’s Encrypt – or the ACME server in general – for which domain we want a certificate issued. It gives us back a token Let’s Encrypt expects to see at a certain URL to proof ownership to them. 3. The script then sets a `HTTP Policy` on the corresponding Virtual Service to return a specific token string at the URL `http://our-domain-on-avi.tld/.well-known/acme-challenge/`. 4. Before the script tells Let’s Encrypt to verify the token, the Avi Controller (the script to be precise) makes a HTTP call to above URL and validate the token locally. This keeps us from getting rate-limited to quick in case validation fails. 5. If the validation suceeds, we inform Let’s Encrypt that the token can be verified. On success, the certificate will be issued and handed over to the Avi Controller to process. Issuing complete. In some setups you might use split-horizon DNS: 1. `our-domain-on-avi.tld` inside your network points to an internal server, directly on the webserver. 2. `our-domain-on-avi.tld` from outside your network points to the NSX ALB/Avi Load Balancer. As the Avi Controller validates the token within the local network, it will never go through the Load Balancer and therefore never hit the `HTTP Policy` set through the script. Essentially causing the local validation to fail. By setting `disable_check` to `True` we simply bypass this check. # End I hope this was useful for some Avi/NSX ALB fans out there! ## Changelog - 2021-12-27: Added note regarding support on GitHub repo. ## Comments Sort by: Loading comments... --- ## Navigation - Parent: [Blog](https://patrik.kernstock.net/index.md) - Previous: [QuickTip: Utilize TPS (Transparent Page Sharing) on ESXi to noticeable save memory](https://patrik.kernstock.net/blog/quicktip-utilize-tps-transparent-page-sharing-on-esxi-to-noticeable-save-memory.md) - Next: [QuickTip: Manually trigger certificate renewal on Avi Controller/NSX ALB](https://patrik.kernstock.net/blog/quicktip-manually-trigger-certificate-renewal-on-avi-controller-nsx-alb.md)