VMotion over L3? And what about iSCSI?
Many of us have still doubts how to handle traffic like VMotion or iSCSI. Should I use L2 and connection or can I route such traffic over L3?
In the past VMotion over L3 was not officially supported, but it worked as well. From vSphere 6.0 new feature called custom TCP/IP stack was introduced. With custom TCP/IP stack you can select what services are running on this stack and you can assign different default routes to the stack. This simplifies the deployment because you do not need to configure static routes using ssh / PowerCli.
So why would you even want to route such traffic like VMotion or even iSCSI? L3 connectivity adds latency and throughput might be lower compared to L2 connection no?
Modern switches do both L2 and L3 lookups almost the same speed using merchant-silicon ASICS as Broadcom Trident / Tomahawk, so you get the same performance for L2 and L3 traffic.
So why would I consider routing my traffic? Well, it depends on the topology. In the past traditional tree topology was used, usually switches ware deployed in MC-LAG configuration and there was L2 adjacency all the way up to core.
But such topology might not provide necessary performance and scaling in today’s datacenter, so new topology – leaf/spine aka clos networks was introduced.
There are several tunneling technologies like NVGRE, VXLAN and others to provide L2 adjacency over L3 network and those are being deployed for virtual machine workloads that require mobility and other advanced software defined network features, but DON’T use it together with VMotion for example. Such configuration is NOT supported at all.
Vmkernel custom TCP/IP stack
Multiple default gateways for different TCP/IP stacks
If you are building new infrastructures and you do use leaf/spine design, you can safely route your VMotion or iSCSI traffic over the physical network with no negative performance impact and such configuration will be fully supported.