VMotion over L3? And what about iSCSI?

 In Architecture, Education, vSphere basics

Many of us have still doubts how to handle traffic like VMotion or iSCSI. Should I use L2 and connection or can I route such traffic over L3?

In the past VMotion over L3 was not officially supported, but it worked as well. From vSphere 6.0 new feature called custom TCP/IP stack was introduced. With custom TCP/IP stack you can select what services are running on this stack and you can assign different default routes to the stack. This simplifies the deployment because you do not need to configure static routes using ssh / PowerCli.

Host networking

So why would you even want to route such traffic like VMotion or even iSCSI? L3 connectivity adds latency and throughput might be lower compared to L2 connection no?

No!

Modern switches do both L2 and L3 lookups almost the same speed using merchant-silicon ASICS as Broadcom Trident / Tomahawk, so you get the same performance for L2 and L3 traffic.

So why would I consider routing my traffic? Well, it depends on the topology. In the past traditional tree topology was used, usually switches ware deployed in MC-LAG configuration and there was L2 adjacency all the way up to core.

But such topology might not provide necessary performance and scaling in today’s datacenter, so new topology – leaf/spine aka clos networks was introduced.

Leaf - Spine network

There is a big change in L2 and L3 adjacency in such topologies and the L2 boundary typically ends with top of rack switches, or even on individual server level!

There are several tunneling technologies like NVGRE, VXLAN and others to provide L2 adjacency over L3 network and those are being deployed for virtual machine workloads that require mobility and other advanced software defined network features, but DON’T use it together with VMotion for example. Such configuration is NOT supported at all.

 

Vmkernel custom TCP/IP stack

New TCPIP stack

 

Multiple default gateways for different TCP/IP stacks

Multiple TCPIP stacks

Conclusion?

If you are building new infrastructures and you do use leaf/spine design, you can safely route your VMotion or iSCSI traffic over the physical network with no negative performance impact and such configuration will be fully supported.

Leave a Comment

Contact Us

We're not around right now. But you can send us an email and we'll get back to you, asap.

Not readable? Change text. captcha txt