When I was putting this together I was running some tests on a bench with the following layout:
I moved the cluster and router (acting as an 8 port managed switch) to an area where I have a UPS so that the power would be filtered. The first item on the list is to get a reasonable IPSec VPN running, hopefully using StrongSwan with some decent encryption, say AES256. Since the router has a WAN port, I can use that any time that I need to update the RPis in the cluster - just add a wire and then take it away. I did discover one issue though, I am going to have to remove the Mac Mini from the NML vlan because I noticed that all of the ports that were open on my personal vlan were open on the NML vlan. I don't know how to correct that so for now, after I get the IPSec VPN setup, I will remove the Mac Mini so I can reduce the probability that someone will hack into my network.