Let’s say you want to create a lot of snapshots for a single VM inside VMware ESX. How many snapshots can you create before your ESX server’s performance goes to crap? Furthermore, is this maximum number relative, depending on how the snapshots are organized in the tree? (For example, can I have more snapshots whose of depth is 1 versus less snapshots of depth 3?)
Those were just some of the questions I’ve been dealing with, in porting the HoneyClient code to ESX.
Here’s the short answer: If your snapshots are getting stored as regular files (which is the default), then UNDER NO CIRCUMSTANCES should you exceed 32 snapshots per VM.
The longer answer: Regardless of how the snapshots are organized, VMware ESX appears to seek through ALL snapshots (in chronological order) when performing ANY snapshot-related operation (e.g., renaming a given snapshot). So, if you create 128 snapshots and just want to rename/revert/alter/delete the 3rd snapshot, then guess what? The ESX server will still iterate through all 128 snapshots before performing the requested operation.
You’d think ESX would be smart enough to perform some sort of depth-first (or even breadth-first search), but no, it doesn’t. This limitation comes from the fact that the snapshot metadata is stored as a FLAT FILE in chronological order.
Wait, it gets better. So it appears that as the number of snapshots increases linearly, the amount of time ESX needs to seek through these snapshots increases in a geometric rate (not quite exponential, but bad enough). And so, this delay becomes noticeable when the number of snapshots is greater than 32.
So, here are my questions to VMware and other ESX users:
1) I’m guessing that this 32 snapshot limit still exists, even if the disks and snapshot data were mapped to a physical iSCSI volume via RDM. Can anyone confirm?
2) VMware: Are you ever planning on improving this snapshotting implementation, for those who want more than 32 snapshots?
3) Anyone aware of similar limitations in other virualization products? (VirtualBox, Xen, etc.?)
Darien Kindlund's Blog 
3 Comments
Why would someone want more then 32 snapshots? That makes no sense? Snapshots are not made to be abackup solution for VM’s thats what VCB (among others) are for.
I’m using ESX’s snapshotting capability for the next version of Honeyclients.If you check out that link, I think you’ll begin to understand how I could generate more than 32 snapshots per VM.
Nice post.
I’m suspecting that snapshot usage is really hammering my vSphere4 ESX server. I really wish that there would be official publicly available information about snapshot performance. Especially that my storage is a low-end RAID-5 disk array.
BTW, here are my experience with ESX tuning so far: http://www.javatuning.com/esx-server-tuning-quick-tour/