Tips #9 - The Sacrificial Port and Admin vlans

One thing I have discovered in my attempt to be secure is the use of Admin vlans.  An Admin vlan is a vlan that you use to limit changes to your network.  This is for internal network infrastructure protection. There are a couple of important points about this:

• Each router, Managed Switch, and Type 1 hypervisor that you have in your network is configured so that changes can only be made from the Admin vlan.
• All ACs/Rules setup in the network enforce the Admin vlan to be separate from all others and enforces the items in this list.
• You set up connections throughout the network so that you have to be physically connected to an Admin vlan port to make changes to any infrastructure elements in the network.
• We have a port on each device dedicated to the Admin vlan, but the Admin vlan is allowed to traverse between devices only on Trunk lines and those lines are physically protected as much as possible.
• All connections on the Admin vlan devices have to be encrypted; this is a zero trust approach.
• Encrypted Certs are controlled by a local Certificate Authority (CA) that is usually offline.
• The Admin vlan port is called the Sacrificial Port because that port is only used for the purpose of getting to the Admin vlan.
• The sacrificial port is protected with an 802.1x/Radius connection by MAC address; if you aren’t supposed to be there you shouldn’t be allowed to get in.
• The sacrificial Port is also there to make sure you can still control infrastructure devices if part of the network becomes unresponsive.
• The Sacrificial Port is also protected by a keyed Ethernet dust plug (suggest the color Red); key is necessary to take the plug out of the device.

Network General - Had major crash in the network

I suffered a major crash in my network last weekend.  What ever happened it borked my PWS Raspberry Pi setup.  That was when I discovered that the network, specifically the Verizon Quantum router, was no longer getting DHCP assigned IP numbers.  I did some tracing and found out that all of my TiVo equipment no longer had IP addresses.

When I went to troubleshoot the Quantum, I discovered that it no longer recognized the FIOS cable that was plugged into it.  I checked everywhere for potential hardware breakages that would explain it but it would come up for a few seconds and then disappear.  That is when I decided to take the Quantum out of the loop.  For one, it was no longer issuing IP addresses from its DHCP server and the TV cable thing.  I ordered a Motorola MOCA 2.0 adapter even though the TiVo Bolt was capable of setting up a MOCA network (I might still use this at a later date though).  The real problem was the fact that the TiVo equipment (Bolt and 2 Minis) were not responding at all and had taken on Bonjour addresses.  The Mini was able to communicate with the Bolt, but could not get back to the TiVo home page to receive updates.  It also appeared that this was causing some issues in using the TiVo equipment.

I then added a vlan behind my Cisco router and moved all of the TiVo equipment there.  When the MOCA adapter came in, I put it on the FIOS cable and connected it to the same vlan as the TiVo equipment.  I then saw that my TiVo equipment had IP addresses and could be updated from the TiVo site.

I determined that the weewx setup on my Raspberry Pi had failed.  It was causing kernel panics.  I then changed out the RPi hardware and setup a new weewx software build and got the PWS back online.

All of my traceback seemed to indicate that the Quantum had undergone a hard failure, no amount of reseting the Quantum seemed to revive it's capabilities.  So I made the decision to phase out the Quantum router.  I was able to get to the outside through the Quantum, but only because my Cisco Router has a manually assigned IP address in the Quantum LAN subnet.  I did some research and it seems that it is a fairly simple matter to take down the Quantum and bring up the Cisco in its place.

Will report back when I get this working.

Added some patch panels to the network over the weekend

Well, I broke down and finally added a couple of patch panels (PP) to the homelab over the weekend.  I was getting tired of having to reconstruct connections whenever I wanted to move equipment from one room to the next.  My homelab is distributed across the house and I like to think that I can move and update things at will.  The problem was that I could not conveniently switch things around because of having to move existing Ethernet cables from one location to the next.  What I needed was a set of Ethernet cables that didn't move but still allowed me to connect at will, hence the use of patch panels.

Right now I am concentrating on getting the minimal number of patch panels up along with keeping equipment and connections the same.  This is little more than routing an Ethernet cable to a patch panel and then using a patch cable to go from the PP to the device.  In the throes of putting the initial two PP together, I made the mistake of routing a piece of equipment to the patch panel as though it were part of the infrastructure.  I now know to keep lines primarily from PP to PP within the network.

I started out with a 16 port and one 24 port PP that I placed in two locations.  The first location was in the tool room where my main switch is located, that got a 16 port PP.  The main switch is like the hub for most of the other managed switches.  The second location was part of the computer room shelf where I am not concentrating most of my efforts.  I originally drew up a diagram of what I wanted to do first followed by a diagram of where I wanted to go.  I followed the first diagram pretty closely but ended up with a lot of differences due to the position of the ports and other equipment that was around it.

Checking Power for the Computer Room

So today I am checking the power coming to my new computer room.  I have cleared out most of the things that were in the room, enough to be able to get around and move my equipment into the room.  There were two lines that I uncovered coming from the main panel that went into the tool room.  One went to two outlets through a breakout box: power for the main managed switch and a 4 socket setup on the workbench.  Since I will not be using the workbench much after I move the 3D printer into the computer room, I should be okay with the power draw.  The line is on a 20 amp circuit and the switch does not draw that much.  Having the 3D printer in the computer room allows me to setup an exhaust outlet for when I start doing stinky plastics like ABS, nylon, and PETG.  The other line coming from the main panel goes to a breakout box that is tied to power going into the computer room.  I need to trace down the wires to make sure that I understand where they are going to.  I am hoping that there is not a lot of activity on this power line.

I have been looking for a power meter that I can use to determine what the current draw is on each of these lines so that I can make sure not to exceed 80%.  That should keep the lines cool enough for continuous usage.  I went to Loews last night and got some 12 gauge power wire to hook the first line I mentioned to some of the other outlets in the room.  I need to determine what the power requirements are for the following groupings of equipment:

1. Computer Rack

- 1500 watt APC

- Server One - generic PC with 7 Ethernet ports

- Server Two (future buy) - probably a NUC device

- Raspberry Pi Cluster: 4 RPis, plus 6 usb connector power supply

- 16 port 1GB managed switch

- 8 port 1GB managed switch

- 8 port 1GB switch

- Two WD myCloud devices (4TB and 3TB) plus One backup usb drive (4TB)

2. Hutch

- Smaller APC unit

- Mac Mini

- Two monitors: one with dual HDMI inputs

- powered speakers

- PC laptop docking station

- external 4TB drive

- external DVD burner

- old 3in1 printer for scanning

- Apple TV 3/Roku

- 8 port 1GB managed switch

3. DIY Electronics Table

- Smaller APC unit

- 8 port POE switch

- 8 port 1GB managed switch

- 4-unit HDMI switch

- PC power supply for bench

- One monitor w/attached RPi

- 6 usb connector power supply for RPis/Arduinos

- RPi with 7" LCD screen for monitoring

4. 3D Printer (in Lack table enclosure)

- 3D printer

- OctoPi setup with camera

- power supply for OctoPi

- Ethernet connection from OctoPi to the Hutch managed switch

Alterations, Alterations in the Network!

Well I'm back after a long hiatus.  I was consumed with some Security Engineering projects at work and it pretty much took all of my time.  Now I can come back to this network and continue on with my projects.  However, I have a wrinkle in the matrix - I now have to consolidate a lot of my equipment into a space that is much smaller than what I originally had available.  Fortunately, having the layout that I had through the judicious use of vlans and vlan switches, the logical movement is not going to be that complicated.

My big problem with the move is getting rid of things that were in the area that I will be moving into and consolidating the things in other areas that will be necessary to continue this HomeLab project of mine.  I am vacating a room that will shortly be made into a bedroom for the potential of having either my granddaughter or my grandson and wife stay with us.  I am all for that, and it won't happen for another year, giving me time to accomplish what I need.  I am just a hop skip and jump away from retirement anyway.  But now I have to figure on not only my servers, but my main computer in my Personal vlan, and my desk where I piddle with Raspberry Pis and Arduinos.  I also have a new hobby of 3D printing which I will be adding to the HomeLab through an OctoPi interface.  Now I have to deal with lack of power for all these devices in that one room.  I also have to think of possible future inclusions such as servers and whatnots.  I will have to run one or two Ethernet cables to support the move, but that is the least of my problems.  Fortunately, I also have enough APC devices to power up everything.

I have decided that three areas should be enough to do what I want.  I am moving my RPi Cluster and Server/NAS to a shelf unit which was already in place in the room.  I will deposit my main computer that is behind the second firewall into the hutch.  I will then bring the table that I have been using for the main computer into the room and place it where I can do experiments with electronics and small computers.  I am also thinking that I need to setup a terminal at the desk to connect to RPis and computers with VNC over SSH and RPN over SSH.  Seems like I will need to setup a localized certificate authority for everything.  I really have grown to the point that I do not trust anything anymore in terms of moving information across a network, even if it is my own network.

I don't really want to buy any more equipment so I am going to repurpose one or two switches to fit into the network.  I do need to clean up the switches to not reference vlans that are not being used.  In addition, I have to be careful to keep an administrative vlan going between major systems.

Bought a 16 port Netgear Switch

So I have bought a 16 port Netgear GS116E Switch to use in the place of the TP-Link switch that didn't give me what I wanted.  Since the user interface was similar, I had no difficulty in setting up the vlan ports on the GS116E the exact same way as the 8 port GS108T.  The footprint is slightly larger in width than the GS108T but seems to be able to fit in the space that I have.  I will need to move some mounting screws higher in order to have enough room over the power outlet to put cables.  There might be some additional things that I can do like mount it lengthwise to remove that limitation.  I need to further investigate.

I have not put the switch into position yet and I will report back this weekend as to how it went.  I am crossing my fingers that it comes off without a hitch since my media is flowing through it.

Update: the GS116E switch is in place of the GS108T switch and I couldn't be happier with its performance.  Install went without a hitch and by simply rebooting devices around it all connections are working flawlessly.  Good job Netgear!  This was way better than my experience with the TP-Link switch.

Switch Install Failure

As I said in the previous post, I took the time to install the TP-Link SG2424 24-port managed switch under the Verizon Quantum Gateway and setup a pass-through vlan so that I could check to see if the switch was going to work in the network.  I should have tried out some other things as well.

It turns out that the SG2424 and the Quantum don't like each other.  When I set up the SG2424 with the same settings as the GS108T, moved over the Ethernet lines and attempted to get everything running, the problems began.  Problem One: the SG2424 would not keep all of its settings between reboots.  I kept having to reset the IP address of the switch even though I made sure that I hit the confirm button.  It just would not keep the new IP address and kept reverting back to the initial IP address which conflicted with other elements in my network.  Problem Two: the SG2424 seemed to not like pass-throughs where the I came from a different switch (the GS1-08Ts) through the SG2424 to the Quantum Gateway.  Problem Three: it would take sometimes 5 to 10 minutes before the Ethernet activity would settle down between the SG2424 and the Quantum.  It appeared that the SG2424 kept resetting itself multiple times.  Problem Four: After multiple resets, the traffic for my media equipment which flowed through the SG2424 to the Quantum would refuse to come up.

I took the SG2424 back to Micro Center where I bought it.  I will review the Netgear GS724 and other switches like it to see if I can use it to replace the one GS108T.  Very frustrating!

Rearranging the Network

I bought a TP-Link SG2424 24-port managed switch over the weekend.  This thing is quite the beast and is made for a rack mount:

I am hoping to replace the Netgear GS108T 8-port switch in the center of my network; because I am running out of ports to use.  There is absolutely nothing wrong with the GS108T, but as you can see, I have a number of cables that are not connected because of lack of ports:

I am also thinking of other types of things that I can do, like using this switch placement as a main switch point for the network since it is right outside where I have my Fios Quantum router.  The biggest problem I am facing at this moment is how to mount the 2424.  The current mounting is a simple sheet of plywood with two screws that the GS108T mounts onto.  With a rack mount size device, I am going to have to raise it further to get over the heating duct, and potentially have issues with heat and cold on the device.

Manuevered Some connections around to make way for a special VLAN

Over the weekend I made some changes to my network in preparation for venturing out as an Anonymous user.  My real reason for doing this is to see if I can somehow escape Google's tracking.  This is an interesting twist since I am using Blogger to blog on and that in turn is owned by Google.

I have been reading up on the DarkNet and what that really means.  Since I am getting more involved with security I thought it might be a good thing to explore various anonymization networks, e.g. TOR, to see how they function and what to expect when I am out there.  This obviously involves assuming a pseudo id to mask who I really am - new territory, haven't ventured there before.

I was able to set up a VLAN on the ActionTec router from http://support.actiontec.com/doc_files/Creating_an_Ethernet_VLAN.pdf on the ActionTec site.  The instructions are a little old but after a bit of trying out different things, I was able to have a VLAN (tagged) on a specific port with a DHCP server and a localized subnet.  I then proceeded to change the wiring around a little, moving my MacMini ethernet connections from the tail-end switch to the one just before it.  I set up the first switch to have a tagged input port then connected an Ethernet cable from the ActionTec to it.  Next I set up the switch to send the new VLAN down a trunk line to the second switch and from there to the third and last switch.  In the process of doing this I consolidated a number of Ethernet connections to one switch which actually speeded up my access to the outside.  I was also able to reduce the usage of the last switch in the stream to make it more experimenter like, including adding the new VLAN.

I now have a line from the ActionTec router that has only one port (I will be putting a laptop on this port) isolated from my network that I will be able to use for Anonymity.

I now have a personal cloud device in the home network

I just purchased a Western Digital My Cloud drive (3TB) and added it to my network in the Pers vlan.  Right up front I can say that I am very pleased with this drive and how it operates.  The My Cloud drive has a 1GB Ethernet port and a USB 3.0 port on the back.  The 1GB Ethernet gives the drive plenty of flexibility to stream media content within the house, so I am planning on using it to host both video and audio files for use throughout.  I have also connected a 4TB backup drive to the USB 3.0 port on the back, giving a total of 7TB of storage.

I was able to get the drive up and running through the WD website for use with my iOS apps.  The 4TB drive shows up as a share on the drive, which is perfect.  Unfortunately, my work network's proxy server doesn't recognize the WD website, so I had to go through our guest wireless to test out general access via PC from work.  I was able to log into the WD website and select one of the shares to be opened in Windows Explorer, and it came right up.  So that is an interesting way to interface to the drive from the outside.  It appears that Western Digital did their homework.

The problem that I am currently facing is that the My Cloud drive is on my Pers vlan while the media components (that would make use of the video and audio file storage) are on the Media vlan.  At the moment I am in the process of setting up an Ubuntu VM to be the router between the Pers and Media vlans.  My constraints are as follows:

• Use Shorewall firewall on Ubuntu VM to do routing; propose using Media and Pers zones; connection is via the vlans that were created on the Thunderbolt interface.  These show up as different ethX devices in the Ubuntu VM
• Using 1 to 1 NAT, present the Mac Mini and the My Cloud drive on the Media vlan with Media vlan IPs
• Setup firewall to allow connection to iTunes protocol, AFS protocol, SMB protocol, and http protocol on the My Cloud drive from Media vlan
• Setup firewall to allow connection to iTunes protocol, AFS protocol on the Mac Mini from Media vlan

More later.

I Now Have a Movable TOR Proxy as Part of the Network

I was able to complete the TOR proxy that I wanted to put together.  I first followed the instructions at http://learn.adafruit.com/setting-up-a-raspberry-pi-as-a-wifi-access-point/ to get one of the RPis to work as an access point.  I then started following the instructions at http://learn.adafruit.com/onion-pi to get the TOR proxy working.  I reset my managed switch to have a port open to the outside (my psuedo-DMZ), plugged in the RPi, and restarted it.  I was able to check it out using http://www.ipchicken.com and found that my ip was coming from the output of a TOR relay.  Success ...

I have also opened an identical port to the outside in the third switch that I have in the house.  In that way, I am able to move the TOR proxy from one part of the house to the other.

FInally broke the code on the Vlan setup

Well, leave it to me to become confused about how to use my managed switches.  I own three Netgear GS108Ts and use them throughout my house in an effort to manage my home network.  These are a little overkill but I got them because I had this great idea about setting up multiple Vlans throughout the house and I wanted to learn a little more about configuring level 2 switches.  The problem was, up until this last weekend I really didn't have enough time to sit down and configure a "true" vlan using the boxes.  The main difficulty was figuring out how to mix untagged ports and tagged trunk lines throughout the house.  It turns out that I was reading the instructions wrong.  Yes, you heard that, I actually read instructions from time to time.

The GS108T instructions are a little vague on how to set up ports as tagged and untagged on isolated vlans.  I needed to set up ports between two managed switches with a trunk line, i.e. the packets were 802.1Q tagged, with packets going in opposite directions on the same physical ethernet line.  My problem centered around understanding what the instructions were saying, not that they are bad instructions, it's just that they left out some details or someone assumed that the user would automatically know what to do.  I am learning how to use the product correctly.  I found out that you needed to set up the ports on either side of the trunk line in the following manner:

1. you need to set the T on each of the vlans that you want to appear as tagged on the port; the vlan numbers should be the same on either switch.
2. on the PVID screen you need to set the acceptable frame types as "VLAN only" instead of "Admit All"; this forces the port to discard any untagged packets that appear
3. on the PVID screen ignore the PVID field for the trunk port; this was not clear in the documentation
4. on the PVID screen leave the ingress filtering to "Disable" as the opposite drops tagged packets that are not the id in the PVID field; since you can only have one number in the PVID field, this would not be a good choice (not clear in the documentation)

Ports that are not trunk ports, but are untagged members of the same vlan are configured as follows:

1. you need to have a U on each of the ports represented with the vlan that you want to recieve/transmit on; note that only one vlan should be present on the untagged ports to isolate the vlan from others; this is not true of the tagged ports
2. on the PVID screen change the PVID field to be the vlan number that you wish to have on the port
3. on the PVID screen leave the acceptable frame types to "Admit All" for the untagged port; this will ensure that the incoming packets will be destined for the specific vlan mentioned in the PVID field
4. on the PVID screen leave the ingress filtering to "Disable" on the untagged port; this will ensure that the rules in 802.1Q are followed for the port

Well, at least I feel better now that I can have isolated vlans running throughout the house.  If I need to move data between them, I will setup some routers to perform that function.

First Vlan Tryout Setup

After having borked the router last time when I was trying to set up a vlan, I am now going to be a little more cautious. To that end, I am going to list out all of the steps and the proper sequence so that I don't get into trouble. My first tryout is to set up a vlan from a VMware Stack through two managed switches, and then to the final router. The following steps will be necessary:

1. Setup the Stack to connect through vlan8.
2. Setup the Mac Mini to put vlan8 out as a tagged stream.
3. Setup the Mac Mini to be able to get to any system in vlan8, but not the other way around (iptables equiv.).
4. Setup the first switch to accept vlan8 packets on the same port as the Mac Mini.
5. Setup the first switch to pass the tagged vlan8 to the second switch on a trunk.
6. Setup the second switch to receive the tagged vlan8 packet on the trunk and pass it as a tagged packet to the port on which the router resides.
7. Setup the router to receive the tagged vlan8 packet on the trunk line and pass it on through the NAT firewall to the outside.
8. The router should be set up so that it does not allow cross communication between vlan8 and any other vlan on the router.
9. Setup the router with a port forward to the VMware stack.

That is basically what I need to accomplish, now on to figuring out the particulars of each.

Update: 12/3/2012 - I have performed steps 4, 5, and 6 with the two switches.  That was the easy part of this sequence.  I have decided to make the subnets reflective of the vlan number by using the vlan number as one of the 4 octets.  Not sure whether to use C, B, or A addressing scheme.  I have also performed step 2 to have the vlan8 output as a tagged stream.  When I went to perform step 1 I set the Stack to the vlan that I had created in step 2, I encountered no problems.  When I changed the DHCP address to the new vlan8 subnet that is when the problems began.  More later.


- LW

Tried Working with the New Switch

Ok, so I thought I understood the VLAN concept.  However, the description and setup on the new managed switch was different than what was described on the Buffalo router.  So in confusion, I have left the router and switch in the current state.  It is interesting to note that when I put the managed switch in place of the unmanaged 1GHz switch everything worked just fine.

I spent a good part of today figuring out what VLANs were going to what ports and where the trunking was going to take place.  It turns out that the easiest way to setup VLANs is to work with a port - trunk metaphor.  Trunks carry packets with VLAN tags and ports strip the tags off or put the tags on.  So the trunk carries more than one VLAN.  Anyway, that is how I learned it.

I need to go back and read the DD-WRT way of setting up VLANs to figure out how to set the switch up correctly.  Right now it doesn't appear too difficult as to how I will proceed once I understand the setups.

-- LW