Thinking of making the IOT network self contained

 It occurred to me that at some time in the future, I might have problems with my network again.  But this time I will probably be more dependent on the elements of the network since I am highly dependent on the router providing DHCP IP addresses and the WAP providing the wifi connection for other elements that require support.  I now think it would be wise to be able to isolate my HA-IOT network from the other part of my network, and still be able to have it work if things happen.  Since I use an Atom based computer for the Home-assistant and MQTT base, I might be able to use that to provide both a WAP and a DHCP/DNS server for the network.  If I do that the wifi port can provide connection for the wifi components, the z-wave adapter for the z-wave components, and if the DHCP/DNS server extends to the HA-IOT vlan as it's control, I don't have to depend upon the router to give me those elements.  If I go on a trip, I can cut off everything else and still have the HA-IOT network working as it needs to be.  Since it is on an UPS, it won't be going down anytime soon.  I just have to figure out how to accomplish this on the Atom processor.

Update: over the weekend 1/9 - 10/2021, I modified the HA-IOT vlan to be completely isolated from other vlans, but at the same time able to communicate within the vlan itself.  It is also isolated from modifying the router or any other switch component in the network.  I am also thinking of changing the bandwidth to the internet to something that is below video capability.  My cameras are on a separate vlan which is isolated from everything due to their nature of using an external server.

Tips #7 - My Network Philosophy in So Many Words

When considering how to think about, design, and develop a home network you must consider your specific situation.  If I were to pick one topic I would have to stress the word CHANGE.  Change will happen to your network over time and you must be able to react and plan for it.  Networks are never static over time.  Here are some items to consider:

1. Are you going to stay put for very long?
• Renters know this, you don't just punch a hole in the wall to do something; you will end up having to repair it before you leave
• The answer to this question might mean the difference between an all wifi or a partial wifi solution
2. What are you planning on doing with your network?
• You may have simple plans for what you are doing now, but what about the future uses?
• It's not always about gaming
• Consider things like setting up for IOT, cameras, and media in the future
• Write up a list of the order of importance for things you will use the network for
3. Cables will stay, devices will always be subject to moving
• If you have spent the time putting a cable into a wall or overhead, you will really not want to move it again
• I have moved my main router to 5 different locations in my house over the years - will you have to do the same
• Consider the fact that equipment may fail and have to be replaced
• You may, over time, have different ISPs that you deal with - I have dealt with 3 different ISPs over the years, each one had a different way of presenting their services to my house (dial up, cable, fiber optic)
4. Do you really want to drill that hole?
• Once you drill a hole, you are stuck with it
• If you ever move you will have to repair it
• Placement of a hole is important, can it be hidden - not everyone wants to see a cable sticking out of a wall, especially spouses
5. Always start with the cabling
• If your neighbors are "jaming" the wifi signal it will be nice to be able to fall back on cabling to supply your network needs
• Don't forget you have options: Ethernet, Coax Cable, fiber optic in addition to wifi
6. Patch panels, strategically placed, help in the movement of equipment
• If you use keystone patch panels with straight through couplers you will always have the ability to use already made cables
• Couplers allow you to move the positions of the Ethernet cable ends on a single patch panel
• Patch panels allow you to use small patch cables to make one long connection between patch panels when a direct connection was not available
7. Consider the judicious use of vlans in the Network to isolate equipment
• Vlans give you the ability to minimize the number of cables run to different parts of the house; you use a "trunk" line from one managed switch/router to another and keep other ports on the router/managed switch to one specific vlan
• Vlans give you the ability to isolate different segments of your network for different purposes
• If the router from your ISP doesn't have vlan capability, you can always put a router behind it that does, or alternatively use your own router for the connection to the ISP
8. Use the highest bandwidth switches and routers that your budget will allow
• As bandwidth gets cheaper, and/or the signal from your ISP changes over time you may encounter a situation where it is advantageous to have already installed equipment that can meet the bandwidth increase
• This means that you should always consider the cabling and what it's bandwidth capability provides; right now Cat5e should be sufficient, but you may want to consider Cat6 and above for growth
9. Always consider the unthinkable, like failures and how it will affect the network
• I have had older equipment fail on me when it was most inconvenient; being able to quickly replace the equipment because you thought out the problem beforehand will help
• I have lost equipment to lightning strikes; never assume that it will not happen to you
• Put lightning arrestors on Ethernet lines from the outside and on Coax cable connections to equipment
• If at all possible get a whole house surge protector for those times when a higher than usual voltage spike happens that gets into your house
• If it is within your budget consider the judicious use of UPS at different spots in your network; sometimes a hard fail of power will cause a failure in hard drives
• If some piece of equipment fails in the network, how fast can you come back up to a semi normal state, or at least most of your list of network importance from #1 above
10. Consider security when you are setting up the network
• It goes without saying that IOT equipment needs to be isolated, due to it's very nature of contacting outside servers; if at all possible, keep the information in your house
• Never assume that the router on the perimeter will always protect you from intruders; in fact, always assume that someone has already attacked and entered your network and design accordingly

Unusual problem connecting Node Red to Home-Assistant

I decided to update my IOT network, by updating all of my Docker containers.  In doing so, I encountered an unusual problem with connecting Node Red to my Home-Assistant container.  This was from several containers.  Normally, I would be using the simple username password authentication, but now that authentication mechanism is not working at all.  Interestingly enough, part of the problem shows up with the Node Red extension that connects to HA via websockets.  I can no longer set the Home Assistant server or any of the authentication mechanisms associated with it.  In fact, it now shows up as a simple text box to enter the server's name.  But that does no good because I cannot specify what IP, port, or authentication mechanism that I want to use.  I am not sure of how to fix this issue.

Update (2020-10-08): I put the Node Red container on a Raspberry Pi, completely away from the Home -Assistant container and I was able to get Node Red to connect.  Now I have to figure out what has happened to my Node Red container on the HA-IOT Server.

Project #22 - Heartbeat / Shutdown Container to shutdown computer equipment in a graceful manner

I was also thinking of a third container that might be useful.  A container, that interfaces to MQTT, that issues heartbeats, and can issue a shutdown command on equipment.  The master controller would then have the ability to issue an mqtt shutdown command, listen for heartbeats, then when the heartbeat ceases, could issue a command to the IP Power Strip to turn off the power to that computer and report back status on the computer.  This has the added thought that it would be universal, working on different operating systems.  The code is pretty much the same, but the implementation is different.

Trying to Resurrect the WeatherPi

Now that I have most of the previous parts of the network seemingly working again (have found replacements for the two Netgear GS108Tv2s that crapped out), I can concentrate on some of the other items that seemed to have gone south.  The first up is the WeatherPi that is connected to my Personal Weather Station.  It doesn't seem to be pushing data out anymore.

I am also wondering if this would be a good time to replace the WeatherPi with a USB server and move the weewx program to the HA Server.  The HA Server is also giving me problems right now, but I can deal with that later.  First off, I am going to try to get the WeatherPi to at least push data to weather underground again.

I did manage to remove the WeatherPi from underneath the TV stand and discovered that the SD Card was pretty much done with (probably a result of the power outage).  I put a new SD Card into the RPi that used to be the WeatherPi and discovered that the RPi was okay.  The new designation was PiHoleLDAP; apparently this was the one that was going to be the one that I was going to use for the PiHole, Radius, and LDAP setup using Docker.  Which reminds me, there is no reason that I cannot run weewx under Docker, I just have to research it.

There seems to be a bit of a problem with weather underground.  Since IBM took over, the API has been removed; so things like predictions and the like are no longer available to gather for an html display.  Weewx has not been updated for Python 3, so should I go to a different outlet to publish the weather station data, or should I just abandon the weather station all together?  Part of me is saying the latter, because I still have to maintain the station, look at the stats, and generally update portions of this as my time allows.  Going into retirement, it would be a scheduled activity that I would do say once every two weeks.  Not sure if it is worth the trouble.

Update: it is looking like weewx under Docker is going to be a pain to complete.  There are a number of builds on the Docker site, but all of them are rather complicated.  I am going to have to delay getting this accomplished.

The Network is Starting to Show Signs of Life

After much pushing and proding, I can now see the network coming back to life.  I was very quickly able to get the Media portion of the network back online.  But I was having trouble getting the other things like the printer and the WiFi working again.  The main culprit was the Netgear switch that controlled the printer, HA Server, and UniFi WAP.  When that went down I couldn’t control anything because of the multiple vlan nature of my network.  The switch had all the appearance of being fried, and since I had not succeeded in gettin extra port connections in one of the patch panels upstairs I was limited in what I could do.  Fortunately, I found out that the switch I used on the Ubuntu server was still operational.  Since the Ubuntu server is currently down I decided to use the switch as a replacement for the one upstairs.

So I obtained the text configuration of the switch that failed, changed the IP, and pushed it onto the replacement.  By replacing the previous switch, I now have a working system - at least in part.  This has also given me time to think about how to get my network back and running should I have such a disaster happen again. First, I need some things to make sure that a power outage doesn’t decimate things again.  I ordered some coax surge protectors to put on the TiVo Edge and Mini. Those are installed and I should not have a problem from the coax side of things.  I also purchased a couple of Ethernet surge protectors.  Lastly I have a whole house surge protector on order that should protect that part as well.

Ok People, Learn from my Mistakes

Updated: with new information on discovering electrical wiring mistakes.

This is a long post so bear with me.  I am putting this out there so that others can learn from my mistakes in networking.  If you read through to the end you will learn a lot of lessons including the ones that you will comment as “well everyone knows that.”

I have a distributed HomeLab that I have built up over the years that I had become quite proud of.  It allowed me to move equipment about the house and make changes as I needed as well as include new equipment that I purchase.  This included at least 8 different vlans for different purposes, including separating media and IOT equipment.  On the night of July 23rd we experienced a power outage.  It expressed itself as a loud bang, followed by the lights going out, and I witnessed a few sparks fly from the pole in front of my house (no transformer, just Verizon Fios gear).  The power came on about 30 seconds later but my internet, phone, and TV channels weren’t working (I have Verizon Fios).  SWMBO went to reading while I went off to explore what had happened.

I went downstairs to look at the ONT interface (I have the older version) and it was flashing as well as telling me that it was using battery power however, no problem lights.  Checked the electric panel and no breakers had been thrown.  Went back and reset the ONT interface, unplug and pull battery; reverse order to get it back.  Flashing stops for about 2 seconds, then goes back to the same flashing and battery lights.  So I get my meter and check the plug that the ONT interface was plugged into, power level is where it is expected.  I then went into my computer room and the first thing I notice is that my main router and accompanying managed switch have no lights on the front panel.  Sure enough, I try to access the router from my Mac Mini and no luck.  I pull the power from the router and the switch and I am getting power to both showing on the meter; also the UPS they are plugged into is showing no signs of problems.  The UPS has regulated power as well as surge protection so I was surprised.  I then pull out all of the patch cables from one of my extra Edgerouter-X, then I pull the ONT patch cord out from my main router and put it into the ER-X wan port, then I pull up a cable and connect the Mac Mini to the ER-X.  Looking into the configuration screen, I see that the ONT is alternating between connected and disconnected, and no IP.

Lesson 1: make sure you have a back up plan to troubleshoot in case you have a catastrophic network failure.

Lesson 2: a Patch Panel setup can save your bacon if you have to reconfigure your network in a hurry.

I tried a few different things, then concluded that the ONT wasn’t going to do its thing anytime soon.  I went off to bed knowing that SWMBO would not be happy in the morning.  The next morning I went searching for damage.  I found that I had lost the ONT (apparently), my main Cisco router (on UPS), my main 26 port Netgear managed switch (also on UPS), along with two 8 port managed switches (both on UPS) in my network.  I then set about reconnecting patches in a different way to connect my media equipment to the ER-X I had setup last night.  Fortunately, that was rather quick given that I had a diagram of the patch panel connections.  Once I did that, I called Verizon to get a tech out to fix the Fios connection issue, delay was a couple of days at least due to volume.  Having the media equipment on a flat network then allowed me to at least view the recordings SWMBO had made on our TiVo Bolt.  Unfortunately, it also showed me that I couldn’t see any of the TV channels on the Bolt, which was odd.

The Verizon tech appeared the next day and after some analysis replaced the ONT, and set me up.  Now, I was getting internet and my phone (on internet) was working.  Unfortunately, there were still no channels that I could receive on the Bolt, even after a cable card swap; I could however get to the internet.  So, I reasoned that the Bolt was bad so I contacted a family member that had recently converted their Bolt to a TiVo Edge.  I picked up the Bolt, got it back to the house, connected it and now I was receiving tv channels, verifying that I had lost my TiVo Bolt somewhere in the process.  I went downstairs and attempted to connect the TiVo Mini to the new Bolt, but was unsuccessful because these pieces of equipment were on two separate TiVo plans and the TiVo DRM kept me from connecting them together.  I then went out and purchased a couple of unmanaged switches that had decent throughput (2GB per port) to use with setting up the media on the network.

Lesson 3: Don’t forget that there may be DRM issues when you try to bring your equipment back up.

Lesson 4: Having equipment on UPS doesn’t guarantee that your equipment will be perfectly protected, there may be other issues.

I swapped some more patch cables so that I could get my UniFi AC-AP-Pro to at least give me wifi to the internet.  Now that I had at least a semblance of media working for SWMBO, it was time to determine what else had happened and how.  I went back and looked at my Cisco router and discovered that I could reset it and at least get the front lights working.  However, it was apparent that at least half of the 16 ports were fried, including the two WAN ports.  This led me to realize that there must have been a surge through the Ethernet lines; again odd.  In fact when I traced the path of the equipment that had failed, it was apparent that the surge came through the Ethernet cables connecting each device.  Since my TiVo Bolt was able to access internet but not the tv channels, it must have gotten the surge through the Coax cable.  Update: I later discovered that electricians that had wired my new kitchen appliances had used the main conduit that I had my longest Ethernet cabling and COAX cabling on.  Evidently I was the recipient of a power surge that had induced voltage in the Ethernet and COAX lines enough that it overcame the networking and media equipment.  I didn't first figure this out due to the odd failures I was seeing.

Lesson 5: it is important to invest in surge protectors on both Ethernet lines and Coax cables in your network because stuff happens.  Oh, and it wouldn't hurt to have a whole house surge protection system to limit collateral damage.

I have since purchased an Edgerouter 12 and the same Netgear M4100-26g switch that I had before (I actually like this switch).  I am still going through provisioning my network even two weeks after the initial power outage.  This morning I discovered another device that had bit the dust, my Ubuntu Desktop server.  It has all the appearance of a power supply problem; it boots, then powers off very quickly, even after getting to the screen where I enter credentials to login.  The server was also plugged into the UPS, go figure.  I am going to check out this one UPS to make sure that it is doing what it is supposed to do.  I also don’t have the UPS connected to a device to register error messages and do something with that information.  Update: I did indeed have to replace the power supply in the Ubuntu server.

Lesson 6: if you are using a UPS, make sure it is connected to a server to gather error messages, if nothing else but to determine if it is going bad.

Long story short, I am still finding things that have failed or partially failed due to this power outage.  I haven’t even looked at my home-assistant server and IOT equipment yet.  I am almost afraid to do so.

Lesson 7: have at least a minimal setup in mind to get your network back up after failures.  It may involve using older equipment or less capable equipment but those devices can come in handy.  Don’t just throw out old equipment, but be reasonable as well.  Don’t keep everything you have ever had in the network, just keep enough to get you back up and running.

Lesson 8: make a decision on what is important and what can wait should a catastrophe happen and you have to configure quickly.

Post log: I now have Ethernet surge protectors, Coax surge protectors, a whole house surge protector setup, have removed / rerouted the central Coax, and have two fiber optic setups running through the original central channel for network connection.

Hopefully, these thoughts will be important enough to you to think through your network and what may happen.  Good luck to everyone!!

LW