Quick Tip: NSX-T 3.0: Removing VIBs manually from ESXi host

Due to some experiments with NSX-T 3.0 in my lab with ESXi 7.0 hosts I was in need removing the VIB files manually from the host, as the NSX Manager failed doing so.

Just to be very clear: This is NOT officially supported by VMware! Do that at your own risk. If unsure, reach out to the support first.

Should you ever be in similar situation, here a few takeaways and lessons learned so far:

1. If you think this is going to be a good idea… it’s not. As there are various resources configured during NSX-T installation and its configuration (like the VDS being configured/migrated in 7.0+ to a NSX Switch), there are dependencies of installed NSX components so that the ESXi network stack can load properly.
   • Without proper removal of NSX configuration from the host, you will end up in a ESXi installation without a working netstack – in other words: No network at all.
   • For example in /var/log/boot you can see failed to deserialize config data:
     
2. If it’s already late too and you ended up in a half-broken ESXi, you might want to consider clean installation. Otherwise there might be still one chance left: Get the NSX VIB package as the previous installed NSX-T version on that specific host, install the VIB and reboot.
   • You can either load the ESXi VIB before removal on a local datastore,
   • or craft a custom ISO file with the NSX-T module ZIP to mount over iDRAC/iLO/IPMI (to mount ISO on ESXi this might be helpful)

Still not scared enough?

Removing NSX-T configuration

This essentially consists out of two commands:

[root@ieesxi01:~] nsxcli
ieesxi01.netkern.local> del nsx
[...]
Are you sure you want to remove NSX-T on this host? (yes/no) yes

vsipioctl clearallfilters -Override

Example output:

[root@ieesxi01:~] nsxcli
ieesxi01.netkern.local> del nsx

*************** STOP STOP STOP STOP STOP ***************

Carefully read the requirements and limitations of this command:

1. Read NSX-T documentation for 'Remove a Host from NSX-T Data Center or Uninstall NSX-T Data Center Completely'.

2. Deletion of this Transport Node from the NSX-T UI or API failed, and this is the last resort.

3. If this is an ESXi host:

        a. The host must be in maintenance mode.

        b. All resources attached to NSXPGs must be moved out.

If the above conditions for ESXi hosts are not met, the command WILL fail.

4. If this is a Linux host:

        a. If KVM is managing VM tenants then shut them down before running this command.

        b. This command should be run from the host console and may fail if run from an SSH client
           or any other network based shell client.
        c. The 'nsxcli -c del nsx' form of this command is not supported

5. For command progress check /scratch/log/nsxcli.log on ESXi host or /var/log/nsxcli.log on non-ESXi host.

Are you sure you want to remove NSX-T on this host? (yes/no) yes
Terminated
[root@ieesxi01:~]

[root@ieesxi01:~] vsipioctl clearallfilters -Override
Removing all vmware-sfw filters...
Cleared dvfilter include table.
Updated all VMs to remove filters.
Destroyed all disconnected filters (please ignore `Function not implemented' error if there is).

Remove NSX-T VIBs from host

After we’ve removed the NSX-T configuration, we go ahead removing the NSX-T VIBs from the host. Removing requires you to remove in a specific order, as those VIBs have dependencies on each other. The order is following:

esxcli software vib remove -n=nsx-adf
esxcli software vib remove -n=nsx-context-mux
esxcli software vib remove -n=nsx-exporter
esxcli software vib remove -n=nsx-host
esxcli software vib remove -n=nsx-monitoring
esxcli software vib remove -n=nsx-netopa
esxcli software vib remove -n=nsx-opsagent
esxcli software vib remove -n=nsx-proxy
esxcli software vib remove -n=nsx-python-logging
esxcli software vib remove -n=nsx-python-utils
esxcli software vib remove -n=nsxcli
esxcli software vib remove -n=nsx-sfhc
esxcli software vib remove -n=nsx-platform-client
esxcli software vib remove -n=nsx-cfgagent
esxcli software vib remove -n=nsx-mpa
esxcli software vib remove -n=nsx-nestdb
esxcli software vib remove -n=nsx-python-gevent
esxcli software vib remove -n=nsx-python-greenlet
esxcli software vib remove -n=nsx-python-protobuf
esxcli software vib remove -n=nsx-vdpi
esxcli software vib remove -n=nsx-ids
esxcli software vib remove -n=nsx-esx-datapath --no-live-install # <- required parameter
esxcli software vib remove -n=vsipfwlib
esxcli software vib remove -n=nsx-cpp-libs
esxcli software vib remove -n=nsx-proto2-libs
esxcli software vib remove -n=nsx-shared-libs

Alternatively as one-liner: (which is noticeably faster than individual commands above)

esxcli software vib remove -n=nsx-adf -n=nsx-context-mux -n=nsx-exporter -n=nsx-host -n=nsx-monitoring -n=nsx-netopa -n=nsx-opsagent -n=nsx-proxy -n=nsx-python-logging -n=nsx-python-utils -n=nsxcli -n=nsx-sfhc -n=nsx-platform-client -n=nsx-cfgagent -n=nsx-mpa -n=nsx-nestdb -n=nsx-python-gevent -n=nsx-python-greenlet -n=nsx-python-protobuf -n=nsx-vdpi -n=nsx-ids; esxcli software vib remove -n=nsx-esx-datapath --no-live-install; esxcli software vib remove -n=vsipfwlib -n=nsx-cpp-libs -n=nsx-proto2-libs -n=nsx-shared-libs

After removal it should look like: (some VIBs requires reboot, therefore still present)

[root@esxi04:~] esxcli software vib list | grep -Ei "nsx|vsipfwlib"
nsx-cpp-libs                   3.0.1.1.0-7.0.16556498               VMware   VMwareCertified   2020-07-22
nsx-esx-datapath               3.0.1.1.0-7.0.16556498               VMware   VMwareCertified   2020-07-22
nsx-proto2-libs                3.0.1.1.0-7.0.16556498               VMware   VMwareCertified   2020-07-22
nsx-shared-libs                3.0.1.1.0-7.0.16556498               VMware   VMwareCertified   2020-07-22

Then simply reboot the host and your NSX-T VIBs are gone. If something’s broken… You’ve been warned.

Updates

• Update 2021-03-05: Added "Remove NSX-T configuration" part, fixed nsx-ids VIB-name typo, added one-liner to remove all NSX-T packages. Tested this procedure successfully with NSX-T 3.1.0 on ESXi 7.0 U1.

Sources

• https://blog.zuthof.nl/2020/01/29/manually-removing-faulty-nsx-t-transport-nodes/
• Self-experiments