Changeover of Ubuntu Server to Proxmox Box

 I have been wanting to try my hand at a Type 1 hypervisor for a while but it was too expensive.  I looked at ESXi, but decided that each change of the software meant a world of hurt for the VMs that would be running.  I do like to use KVM and have wanted to learn LXC containers, so I decided to change out the Ubuntu Server and go with Proxmox.

I changed the HW for the Proxmox server to have a 1TB SSD, repurposed the 4TB spinner from the Ubuntu Server to use, and reconnected the DVD drive.  I did a fresh install from the Proxmox 6.3-1 iso that I downloaded from the site and came up running very quickly.  I then added a number of OS isos, including CentOS, Ubuntu, and Fedora.  I was able to quickly spin up Ubuntu and start working on the challenge of other VMs and LXCs.

I connected the GUI/SSH port of the Proxmox to my admin vlan and ran the first Ubuntu VM on my Pers vlan.  So far so good.

The Network is Starting to Show Signs of Life

After much pushing and proding, I can now see the network coming back to life.  I was very quickly able to get the Media portion of the network back online.  But I was having trouble getting the other things like the printer and the WiFi working again.  The main culprit was the Netgear switch that controlled the printer, HA Server, and UniFi WAP.  When that went down I couldn’t control anything because of the multiple vlan nature of my network.  The switch had all the appearance of being fried, and since I had not succeeded in gettin extra port connections in one of the patch panels upstairs I was limited in what I could do.  Fortunately, I found out that the switch I used on the Ubuntu server was still operational.  Since the Ubuntu server is currently down I decided to use the switch as a replacement for the one upstairs.

So I obtained the text configuration of the switch that failed, changed the IP, and pushed it onto the replacement.  By replacing the previous switch, I now have a working system - at least in part.  This has also given me time to think about how to get my network back and running should I have such a disaster happen again. First, I need some things to make sure that a power outage doesn’t decimate things again.  I ordered some coax surge protectors to put on the TiVo Edge and Mini. Those are installed and I should not have a problem from the coax side of things.  I also purchased a couple of Ethernet surge protectors.  Lastly I have a whole house surge protector on order that should protect that part as well.

OVS Changes #2 - getting Docker containers to attach to the OVS bridges

Now that I have been successful with KVM VMs and their connections to the OVS bridges, I need to figure out how to do much of the same thing with Docker containers.  One of the reasons that I run ubuntuServer is because it is very easy (so they say) to get Docker containers running.  So now, what I am going to attempt is to startup a Docker container attached to one of the OVS bridges, used on a pfSense KVM VM.

So according to the site, http://containertutorials.com/network/ovs_docker.html, I have to do the following steps:

Install OVS (already done on my system) -

Install OVS-docker utility (already done on my system) -

Create an OVS bridge (already done on my system, except the bridge doesn't have an IP address) -

Create the docker container

Connect the container to the OVS bridge, such as

$ ovs-docker add-port ovs-br1 eth1 <container_name> --ipaddress=<ip_address_for_container>

OVS Changes #1 - Updating the ubuntuServer to use OVS bridges

One of the things that I wanted to do with my network was to be able to access multiple vlans while I was away on trips or from the office.  I hit upon the use of OpenVSwitch (OVS) as a means of connecting an Ubuntu VM running under VMWare on my Windows 10 laptop.  In addition, I wanted to be able to run a set of Docker containers from my Mac Mini server and have them connected to the ubuntuServer.  So, in effect I have a SDN within my house that can be connected to through my VPN server.

I had setup bridges to multiple Ethernet ports using the following template:

iface <ethernet-port-name> inet manual

auto <bridge-name>

iface <bridge-name> inet manual

        bridge_ports <ethernet-port-name>

The setup is in a number of bridge definition files located in /etc/network/interfaces.d/; one file per interface.  The only template that is in /etc/network/interfaces is the one that I have for my day-to-day network activities.  Again, I separate out vlans for different purposes making sure that the vlans do not talk to each other except in controlled instances.

What I did to use an OVS generated bridge was to remove the bridge definition file from /etc/network/interfaces.d/ and then delete the previous bridge using:

sudo ip link set <bridge-name> down

sudo brctl delbr <bridge-name>

sudo brctl show

and then perform the following command (note the ovs addition to differentiate it):

sudo ovs-vsctl add-br <bridge-name-ovs>

sudo ovs-vsctl add-port <bridge-name-ovs> <ethernet-port-name>
sudo ovs-vsctl show

Which is done for each port that I have defined, except for the day-to-day port.  It turns out that you can use the Virtual Machine Manager to pull up the VM, reset the bridge designations from the pull-down list, hit apply, then launch the VM.  Works very well with pfSense.  The interesting thing is that the <bridge-name-ovs> will now show up in an ifconfig command, where as before (when first created with OVS) would not.  Also, there are now tap devices that show up in ifconfig, like macvtap0 and macvtap1.  Those correspond to the tap device attached to the KVM VM; created from the use of Virtual Machine Manager.  I still need to figure out how to do this with Docker.

Anyway it works!  I am able to pull up the VMs on the correct vlan now.

Surprised at the changes in 18.04 LTS

So I went ahead and updated my Ubuntu Server from 16.04 LTS to 18.04 LTS.  I did this in order to stay current with the loads from ubuntu and look at the new stuff at the same time.  So I performed the upgrade over the weekend; I did have some issues with the vlan setups but didn't think anything of it.  I sort of kinda followed the instructions here.  Now I have discovered that the way that Ubuntu handles networks has completely changed in 18.04 LTS (see article here on NetPlan).

So now I need to look into NetPlan and figure out how it impacts with OpenVSwitch (OVS) since I am about to use that to do some remote vlan stuff.

Update: after requesting some help in the Reddit Ubuntu forum, I found out that NetPlan is not a typical Linux community project.  This means that it is somewhat of an Ubuntu.com project.  That being said, the responder showed me how to take NetPlan out of the equation.  Instructions are at UbuntuGeek, and are duplicated here:

Note:- This is not recommended and this is for advanced users only

Edit the /etc/default/grub file

sudo nano /etc/default/grub

Add the following line

GRUB_CMDLINE_LINUX="netcfg/do_not_use_netplan=true"

Save and exit the file

Now update the grub using the following command

sudo update-grub

You need to install ifupdown package

sudo apt install ifupdown

Now you can add all the interface details in /etc/network/interfaces file and reboot the ubuntu PC/server.

New Network Layout #1 - Layout of my experimental network vlan

I have been very fortunate to be able to put together pretty much any network layout that I want.  I have made some changes over the years culminating in what I have currently.  Through many trials, I have been able to get to the point that I am comfortable with the layout and how it works.  The experimental network is below:

As you can see in this diagram, I have a Gateway vlan which I connect most of my toys to.  The Gateway vlan in turn is connected, through various routers to the Internet hosted by my ISP.  For purposes of this discussion, how I connect is immaterial.  I have a pfSense router which isolates the Experimental vlan (Exp) from the Gateway vlan.  I also have a Personal vlan (Pers) which is used to monitor the Exp and is isolated from the Exp by another pfSense router.  I can update/command experiments in the Exp from anywhere in the house that is connected to the Pers.  The Exp is also connected to the outside world during Christmas (and other specific times) through a Ubiquiti WAP.  When Christmas rolls around the wifi connection is turned on to command some wifi-enabled switches and some wifi-enabled LED displays via experiments in the Exp vlan.  Since I am using a lot of managed switches, I am able to connect an Ubuntu server to the Exp, along with an RPi Cluster, and additional RPi experiments.  I also have a permanent RPi development terminal that I can use to ssh into various experiments as necessary.  The Ubuntu server supplies a number of VMs that are useful for development work including a subversion repository and some bug/issue tracking software like RedMine.