The current thought is that hypervisors like VMware, Proxmox and XCP-NG should be a separate device from the storage. There are many good reasons to do this. You can grow your hypervisor farm without increasing your storage costs if need be. Conversely, you can grow your storage without impacting your hypervisors.
The most common way to connect your hypervisors to the storage is by using a network connection. Ideally, this should be a high speed connection physically isolated from the other connections to the VMs (Linux, Windows, etc.) to the users.
All of the following decisions will require planning, thought, experience and possibly some testing. This is not a complete treatise on the subject.
The two most common ways to connect hypervisors to storage is by using block storage (iSCSI for instance) or file storage (NFS or Samba/SMB).
- Think of block storage as a hard disk over a network connection, typically a SAN (storage attached network). Block storage by itself has to be formatted and partitioned. Block storage on a SAN is just a bunch of blocks of data. The OS (VSphere, Linux or Windows) has to manage the block storage and how to make use of the block storage. To the SAN, block storage is just space on it’s storage with no file or folder structure.
- File storage is storage that the NAS (network attached storage) manages. The NAS provides a format that files and folders can be created on. The OS (VSphere, Linux or Windows) can “mount” this storage so it is accessible.
Neither block nor file storage is “better” in features or performance. Each have their pros and cons.
There are certain features that are available for either block or file storage:
- Compression - Files or blocks can be compressed in such a way that is transparent to the OS. Example: a file or a block may have long sections of blank or repetitive areas which can be replaced with a placeholder that takes much less space.
- Deduplication - Files or blocks can be deduplicated in such a way that is transparent to the OS. Example: a file or a block may have long sections of duplicated blocks which can be replaced with a placeholder that takes much less space.
- Thin or Thick provision - Thick provisioning assigns the entirety of the space all at once. Thin provisioning “lies” to the operating OS (VSphere, Linux or Windows) that it has access to all of the space all at once, when it reality it once allocates the space as it is needed by the OS.
There are also several points in the data flow that these features can be applied to. Consider the following:
- The SAN/NAS unit can compress or deduplicate that space.
- The hypervisor itself can compress or deduplicate that space.
- When a VM is created and space is assigned to that VM, the hypervisor can thin or thick provision that space.
- An OS like Linux or Windows can compress the disk and even deduplicate what ever is on the disk.
As you may have noticed, there is a lot of area for “duplication of efforts” to happen. Compression of blocks or file storage in more than one place will not provide more compression. The same for deduplication.
Thin provisioning has it’s own perils. Say you have a 1 TB disk that is thin provisioned and it the OS and applications are only using 32 GBs of space. Suddenly you need to create a 512 GB file. Allocation of that new space takes a little time so the disk write will be delayed. Some OSes do not take kindly to this situation. Suppose you thin provisioned at the SAN/NAS level and at the OS level. Now at two points disk space has to be allocated.
Generally, thin provisioned disk space is not given back automatically when it is no longer needed.
So why would you ever use thin provisioning?
- An engineer comes to you claiming his new VM has to have 10 TB of disk space but you know he is just guessing. You as the storage administrator manage how disk space is allocated, not the engineer.
- Snapshots and backups take less time to perform since it is only processing space that is actually used, not empty space.
Snapshots ARE NOT backups. Snapshots should only be kept around for 2-3 days.
General guidelines:
- Understand how compression and deduplication works in your environment. Video/audio file will not compress or dedup well so don’t make you SAN/NAS work harder that it needs to.
- Deduplication rarely works as well as the adverts say. Make a deduped and a non-deduped share and put the same representative files on each and see how things work out. Deduplication also requires extra CPU horsepower and memory.
- Most SAN/NAS units allow you to create shares or dataset with different settings. Use this to you advantage.
- Consider making different datasets or shares for each VM. Since you can assign a fixed size or a quota on each, you can stop one VM from chewing through all of you storage and crashing other VMs.