Just as a quick update:

I have now also acquired a Mellanox 36-port MSB7890 36-port 4x EDR IB switch as well with a total switching capacity of 7.2 Tbps (36 ports * 0.1 Tbps per port * 2 for full duplex) = 7.2 Tbps. (or 72,000 Gbps. Just think about that. Some home installations might go up to like a 48-port gigabit switch. This single 36-port switch would, in theory, be able to support like 72000 gigabit ethernet ports. Cable management would be a colossal PITA, but yeah.)

So now, all four nodes are able to talk to each other.


Here is what the switch looks like (front view, albeit stock image):


Here is what the rear looks like (upside-down):


And here is what the insides look like:


(Sidebar: the managed version of this switch comes with an Intel Celeron 1047UE processor (2-core, 1.4 GHz) and 4 GB of DDR3-1600 ECC Unregistered RAM in a single SODIMM.)

So now, all four of my compute nodes can talk to each other and they're no longer bound/restricted to a 2+2 configuration, so any one node can talk to any other node directly, which is super exciting for me.

And perhaps interestingly enough, the most expensive element for the update was actually the switch. The cards and the cables themselves weren't all that terribly expensive, consider that you can pay less, but also get SIGNIFICANTLY slower speeds as well.

What's also interesting is that the rest of my home network is only wired gigabit ethernet (total combined 32 Gbps) which means that I can fit all of that in a single port or in a breakout cable that supports up to two 50 Gbps connections.

In otherwords, I now have massive excessive network switching capacity - more than I LITERALLY know what to do with. (I thought about making the new IB switch, the backbone switch for my entire house, but the problem with that is for distances greater than 5 metres, I would have to start using active optical cables, and those are EXTREMELY, PROHIBITIVELY expensive.)

And each port is capable of delivering between 96-97% of the theorectical/designed capacity, which is still RIDICULOUSLY fast.