QuickTip: Utilize TPS (Transparent Page Sharing) on ESXi to noticeable save memory

What is it?

TPS (Transparent Page Sharing) is a proprietary functionality in VMware ESX(i) which essentially does deduplication of memory pages used for virtual machines. To put it in other words: Identical memory content across multiple machines only consume memory one time.

When you have multiple virtual machines with the same guest operating system running, there is a high chance of identical content which can be easily deduplicated to save memory. This however comes with costs in terms of performance decrease and security.

Disclaimer: This should not be used for production environments – especially not in environments where security is critical. It is usually more suitable in HomeLab and lab environments.

Due to security concerns this functionality is disabled in VMware ESXi versions released after about end of 2015. Published papers showed that specific in-memory operations could leak encryption keys to other VMs (Source, VMware KB 2080735)

The current default – as per VMware ESXi 7.0 GA – is that TPS is active in the same VM and only across VMs with a specific salt. See more information about the specific settings and its meaning in the KB article 2097593 here.

How it works?

Simply explained:

1. The memory of virtual machines are split up in so called pages.
2. Each memory page gets a specific hash value based on its content.
3. ESXi scans the memory of the virtual machines to find sharing opportunities. If a hash matches, the content of the page can be re-used and does not need to be put into memory twice.

Here’s a nice illustration showing how this works (Source is the whitepaper below)

See more details in the whitepaper here. While it still refers to the legacy VMware ESX, it’s core principle should be quite the same.

You might want to also give the vSphere 7.0 Resource Management Guide a read, if you’re interested in more details (e.g. Large Page Sizes).

The knobs

If you want to efficiently do a memory deduplication using TPS, we need to adjust two knobs on VMware ESXi. Please bear in mind, that TPS does bring a bit of a negative performance impact. But on systems where RAM is limited, it might be worth turning it on.

1. We are disabling the use of large pages to be allocated (AllocGuestLargePage). Understandbly when having large pages with 2 MB or even 1 GB, it is hard to find identical pages to duplicate. Therefore we disable and fallback to 4 KB pages. This makes TPS way more efficient, while reducing performance a bit.

2. We enable TPS on Inter-VM (TPS between different VMs) and Intra-VM (TPS within the same VM) to get the most out of TPS (ShareForceSalting).

You can see more details of these two advanced settings via esxcli on the ESXi host directly:

[root@esxi04:~] esxcli system settings advanced list -o /Mem/ShareForceSalting
   Path: /Mem/ShareForceSalting
   Type: integer
   Int Value: 0
   Default Int Value: 2
   Min Value: 0
   Max Value: 2
   String Value:
   Default String Value:
   Valid Characters:
   Description: PShare salting allows for sharing isolation between multiple VMs and is controlled by the VMX option sched.mem.pshare.salt. Two VMs sharing the same salt will be able to share memory through PShare. This option controls how this VMX option will be used: 0: no salting or isolation between VMs, 1: the value of sched.mem.pshare.salt is honored, 2: each VM will get its own salt unless the sched.mem.pshare.salt is option is non-empty.

[root@esxi04:~] esxcli system settings advanced list -o /Mem/AllocGuestLargePage
   Path: /Mem/AllocGuestLargePage
   Type: integer
   Int Value: 0
   Default Int Value: 1
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Enable large page backing of guest memory

Go, TPS!

After all the theory, here’s probably the part you’re most interested in. As described above, we’re going to disable large pages and enable TPS between different and within the same VM for most efficiency.

1. Enable the settings

Via esxcli:

esxcli system settings advanced set -o /Mem/ShareForceSalting -i 0
esxcli system settings advanced set -o /Mem/AllocGuestLargePage -i 0

Via PowerShell (PowerCLI):

Get-VMHost -Name "esxi04" | Get-AdvancedSetting -Name Mem.ShareForceSalting | Set-AdvancedSetting -Value 0 -Confirm:$false
Get-VMHost -Name "esxi04" | Get-AdvancedSetting -Name Mem.AllocGuestLargePage | Set-AdvancedSetting -Value 0 -Confirm:$false

(You can apply this on all hosts within same cluster by using Get-Cluster "CL1" | Get-VMHost instead.)

2. To force settings to take effect

While the settings can be changed live, TPS does not kick in immediately. To trigger TPS you either need to restart the virtual machine or vMotion it to a different host.

This way, I was able to save between 10 and 20 GB on one host with 96 GB RAM. Hope this helps!