Quick Tip: NSX-T 3.0: Removing VIBs manually from ESXi host
This post was published 1 year 5 months 26 days ago, so the post may be outdated.
Due to some experiments with NSX-T 3.0 in my lab with ESXi 7.0 hosts I was in need removing the VIB files manually from the host, as the NSX Manager failed doing so.
Just to be very clear: This is NOT officially supported by VMware! Do that at your own risk. If unsure, reach out to the support first.
Should you ever be in similar situation, here a few takeaways and lessons learned so far:
- If you think this is going to be a good idea… it’s not. As there are various resources configured during NSX-T installation and its configuration (like the VDS being configured/migrated in 7.0+ to a NSX Switch), there are dependencies of installed NSX components so that the ESXi network stack can load properly.
- Without proper removal of NSX configuration from the host, you will end up in a ESXi installation without a working netstack – in other words: No network at all.
- For example in
/var/log/bootyou can seefailed to deserialize config data:
- If it’s already late too and you ended up in a half-broken ESXi, you might want to consider clean installation. Otherwise there might be still one chance left: Get the NSX VIB package as the previous installed NSX-T version on that specific host, install the VIB and reboot.
- You can either load the ESXi VIB before removal on a local datastore,
- or craft a custom ISO file with the NSX-T module ZIP to mount over iDRAC/iLO/IPMI (to mount ISO on ESXi this might be helpful)
Still not scared enough?
Removing NSX-T configuration
This essentially consists out of two commands:
[root@ieesxi01:~] nsxcli
ieesxi01.netkern.local> del nsx
[...]
Are you sure you want to remove NSX-T on this host? (yes/no) yes
vsipioctl clearallfilters -OverrideExample output:
[root@ieesxi01:~] nsxcli
ieesxi01.netkern.local> del nsx
*************** STOP STOP STOP STOP STOP ***************
Carefully read the requirements and limitations of this command:
1. Read NSX-T documentation for 'Remove a Host from NSX-T Data Center or Uninstall NSX-T Data Center Completely'.
2. Deletion of this Transport Node from the NSX-T UI or API failed, and this is the last resort.
3. If this is an ESXi host:
a. The host must be in maintenance mode.
b. All resources attached to NSXPGs must be moved out.
If the above conditions for ESXi hosts are not met, the command WILL fail.
4. If this is a Linux host:
a. If KVM is managing VM tenants then shut them down before running this command.
b. This command should be run from the host console and may fail if run from an SSH client
or any other network based shell client.
c. The 'nsxcli -c del nsx' form of this command is not supported
5. For command progress check /scratch/log/nsxcli.log on ESXi host or /var/log/nsxcli.log on non-ESXi host.
Are you sure you want to remove NSX-T on this host? (yes/no) yes
Terminated
[root@ieesxi01:~]
[root@ieesxi01:~] vsipioctl clearallfilters -Override
Removing all vmware-sfw filters...
Cleared dvfilter include table.
Updated all VMs to remove filters.
Destroyed all disconnected filters (please ignore `Function not implemented' error if there is).Remove NSX-T VIBs from host
After we’ve removed the NSX-T configuration, we go ahead removing the NSX-T VIBs from the host. Removing requires you to remove in a specific order, as those VIBs have dependencies on each other. The order is following:
esxcli software vib remove -n=nsx-adf
esxcli software vib remove -n=nsx-context-mux
esxcli software vib remove -n=nsx-exporter
esxcli software vib remove -n=nsx-host
esxcli software vib remove -n=nsx-monitoring
esxcli software vib remove -n=nsx-netopa
esxcli software vib remove -n=nsx-opsagent
esxcli software vib remove -n=nsx-proxy
esxcli software vib remove -n=nsx-python-logging
esxcli software vib remove -n=nsx-python-utils
esxcli software vib remove -n=nsxcli
esxcli software vib remove -n=nsx-sfhc
esxcli software vib remove -n=nsx-platform-client
esxcli software vib remove -n=nsx-cfgagent
esxcli software vib remove -n=nsx-mpa
esxcli software vib remove -n=nsx-nestdb
esxcli software vib remove -n=nsx-python-gevent
esxcli software vib remove -n=nsx-python-greenlet
esxcli software vib remove -n=nsx-python-protobuf
esxcli software vib remove -n=nsx-vdpi
esxcli software vib remove -n=nsx-ids
esxcli software vib remove -n=nsx-esx-datapath --no-live-install # <- required parameter
esxcli software vib remove -n=vsipfwlib
esxcli software vib remove -n=nsx-cpp-libs
esxcli software vib remove -n=nsx-proto2-libs
esxcli software vib remove -n=nsx-shared-libsAlternatively as one-liner: (which is noticeably faster than individual commands above)
esxcli software vib remove -n=nsx-adf -n=nsx-context-mux -n=nsx-exporter -n=nsx-host -n=nsx-monitoring -n=nsx-netopa -n=nsx-opsagent -n=nsx-proxy -n=nsx-python-logging -n=nsx-python-utils -n=nsxcli -n=nsx-sfhc -n=nsx-platform-client -n=nsx-cfgagent -n=nsx-mpa -n=nsx-nestdb -n=nsx-python-gevent -n=nsx-python-greenlet -n=nsx-python-protobuf -n=nsx-vdpi -n=nsx-ids; esxcli software vib remove -n=nsx-esx-datapath --no-live-install; esxcli software vib remove -n=vsipfwlib -n=nsx-cpp-libs -n=nsx-proto2-libs -n=nsx-shared-libsAfter removal it should look like: (some VIBs requires reboot, therefore still present)
[root@esxi04:~] esxcli software vib list | grep -Ei "nsx|vsipfwlib"
nsx-cpp-libs 3.0.1.1.0-7.0.16556498 VMware VMwareCertified 2020-07-22
nsx-esx-datapath 3.0.1.1.0-7.0.16556498 VMware VMwareCertified 2020-07-22
nsx-proto2-libs 3.0.1.1.0-7.0.16556498 VMware VMwareCertified 2020-07-22
nsx-shared-libs 3.0.1.1.0-7.0.16556498 VMware VMwareCertified 2020-07-22Then simply reboot the host and your NSX-T VIBs are gone. If something’s broken… You’ve been warned.
Updates
- Update 2021-03-05: Added "Remove NSX-T configuration" part, fixed nsx-ids VIB-name typo, added one-liner to remove all NSX-T packages. Tested this procedure successfully with NSX-T 3.1.0 on ESXi 7.0 U1.
Thanks !!
also.. the following line has a typo:
should be “esxcli software VIBs remove -n=nsx-ids”
Yes, indeed. Thanks for bringing this up! Fixed.
BE WARNED!!!!! This order of removal works.. but I will caution you that you need need to do a “reset system configuration” … on my hosts, it didn’t take by using F11… instead I used: /sbin/firmwareConfig.py –reset. (<- found here: https://www.vm-help.com/esx40i/esxi-reset-system-configuration) When I ran the vin removal above (BEFORE RESETTING CONFIGURATION), it hosed everything… From VMware’s site: If you don’t get the physical pics uncoupled with your NVDS switch, it will black screen… because there were physical nics tied up in an “opaque” switch on my ESXI node, it hosed the system and it black-screened. I did a “Shift-R” recovery… Read more »
If you only remove the VIBs and still have some NSX-T configuration leftovers in ESXi config files, you end up in a broken vmkernel network stack indeed – as warned in the article. I’ve added steps to get rid of the configuration as well and was able to clear a ESXi host with NSX-T just fine this way, without resetting system configuration.