IP Power Strip #05 - Some Operational Considerations for the IP Power Strip Project

 I have discovered that most of what you want to do with a scene in IOT is actually a binary sequence, it either is or isn't performing it's function.  Lights on / off, doors locked / unlocked, security alarmed / not-alarmed, etc.  You have some secondary effects like dimming values or changing a lights color but that is after you turn the light on, likewise with setting the temperature on a furnace.  So in essence it is a two message sequence.  Likewise, if you are shutting down a plug and a computer is connected to it, you would want to send a message to start the shutdown, followed by a timer of so many seconds before you issue the turn off plug message.  

As I have been using Node Red, I realize that it doesn't necessarily scale well, and you are re-deploying the Node Red thread multiple times as you are making changes on different tabs (unless you purposefully select just the new flows before deploy). And if you need to run tests on a separate vlan as you are doing development it breaks the stuff that you already have working. Also, by partitioning the problem up into smaller bits, your Node Red applications can continue to run while you are developing something else. By having the applications in smaller chunks it supports the CI/CD development pipeline.

What do you do if you want to add additional plugs to power your HomeLab network? I need to bring stuff up and down all the time. How do you do this in such a way that it is a minor bump if you add additional power strip plugs to control and additional ways to control them with? What if you need to drop in a dimming controller to be controlled by a new set of Node Red controls without disrupting Node Red flows that already do something that works? What do I do if I need the host that my Node Red flows are running on for something else? Can I simply move it to a different host? What if the host is not an ARM but it's an i386 based processor?

This experiment has everything to do with getting very familiar with Node Red as a prototyping tool. But I also wanted to be able to take down portions of my network (servers, switches, etc.) when I would shift attention to some of my other hobbies like woodworking/woodcarving. I wanted to be able to take on multiple types of smart plugs, some based on RPi hats, some based on wifi power strips, some based on z-wave devices. I wanted to be able to add onto, or subtract from what I was doing without having to continuously update one server worth of Node Red. Doing it in Docker containers has allowed me to move things around easily. I can see this generic control set (Master and multiple Slaves controlling multiple Interfaces to the hardware) being used for all kinds of things IOT wise. I also wanted to have control over what is powered up (including lights) from both home-assistant, NanoMotes and WallMotes and garage controllers without changing either a scene def in home-assistant or changing up my single Node Red instance.

That being said, let me use my experience in other systems to explain why this setup might need multiple instances of Node Red and why that makes sense. Think of those automations that are temporary, with two examples: a test and a temporary power setup.

Say I have an already running automation with Node Red in a Docker container. It is using an MQTT broker as the go between for all the “production” automations. In this case I would like to add some functionality to my “production” automation. If I have my MQTT broker running in a container, I can just spin up another instance on a different host (note different IP). I pull my definition for the automation from a Configuration Management server, like Git, modify the MQTT IP to point to my test MQTT broker and spin up the container (this could be automated with a script). If I require other containers for development I can also pull those, change the MQTT IP, and spin them up before making changes (this could also be part of the same script), I might even need some test driver NR containers as well. I can now make changes as necessary for the new function and test it until I am satisfied it is what I want. I push the definition back to Git. The containers I needed for development and test can now be taken down. I can now deploy the change, that involves taking down the “production” automation container, pulling the updated automation container, setting the MQTT IP for production, and spinning up the new container which now should take the place of the old one. By doing it this way, I have minimized the down time for the “production” automation.

In the case of the temporary power setup, in my experiment, I might need a new automation for part of a Halloween or Christmas display where it changes from year to year as to what lights are lit; or alternatively I might have to set up the lights for a party which I just found out about the night before. In my power experiment I have at the bottom layer, an InterfaceProxy that controls individual power strips, switches, lights or what have you. Each plug is given a name representative of what it is powering (and can be changed on the fly). Each plug also has a semaphore count that dictates whether the plug is turned off or not. The next level up is a Slave Function, which controls a grouping of plugs. Each plug might have more than one grouping assigned to it. The Slave takes care of one grouping of plugs which might go across multiple devices. The Slave knows it’s list of plugs when it is given a list of names. The Slave issues the turn off, turn on when it is commanded. A Master Function controls the whole thing at the top. You only spin up a Slave container when you have a new list of plugs to control. You only spin up an InterfaceProxy container when you have a new set of hardware “plugs” to control.

So for my Christmas lights example, you would bring out the smart power strips (probably Tasmota), spin up one or more InterfaceProxy containers, set the plug names, spin up one or more Slave containers, give the list of plugs to control for each from the Master container after the Slave announces itself, and you can then control your Christmas lights through the Master container. After Christmas the new containers are spun down, smart power strips and lights are put back in storage until next year. For the party example, just feed a list of the plug names that need to be controlled, and the Master container spins up a Slave container and the cycle repeats. The new Slave container goes down after the party.

So development of the MQTT messages that control all this will be paramount.

IP Power Strip #04 - the Strikedown Container

I had suspected that the fastest way to build this was to use node red, at least I know that shutdown and reboot are easily implemented.  The issue is Mac and Ubuntu computers.  I don't think that  I will need to consider PCs (except for the Atom HA-IOT Server which runs Ubuntu).  I got the initial sequence done in node red, however I have not set it up yet to do an actual shutdown/restart - still figuring out the MQTT messages to use.  I decided on the following messages

• relay/strikedown along with relay/strikedown_answer
• reboot: 0/1, name: IP1, ping: 0/1
• if reboot is 0 then does a shutdown
• if reboot is 1 then does a restart
• if ping is 1 then starts ping message sequence (5 times with 2 sec in between)
• note when ping is 1, upon restart the ping sequence will start up
• relay/myping
• name: IP1, ping: 0/1
• if ping is 1 then starts ping message sequence (5 times with 2 sec in between)
• if ping is 0 then stops ping
• note when ping is 1, upon restart the ping sequence will start up
• relay/ping
• name: IP1

strikedown, the Docker container in node red, will always use port 1885 on each host.  That way, I will be able to get to the node red invocation to make changes.  For strikedown on Raspberry Pi, theer is a node red contrib library that does both functions.  For the one on Ubuntu servers it will have to be made specific to the server.  I am thinking of using Python since there is a Python function node in the contrib library.

The strikedown node red sequence is in two parts, the startup and the strikedown sequence.  The startup sequence is:

The strikedown node red sequence is:

Update: I have since discovered that the Shutdown and Restart Nodes in this sequence do not work under a Docker Container running on RPi.  So I will have to figure out how to accomplish the task in another way.

Tips #7 - My Network Philosophy in So Many Words

When considering how to think about, design, and develop a home network you must consider your specific situation.  If I were to pick one topic I would have to stress the word CHANGE.  Change will happen to your network over time and you must be able to react and plan for it.  Networks are never static over time.  Here are some items to consider:

1. Are you going to stay put for very long?
• Renters know this, you don't just punch a hole in the wall to do something; you will end up having to repair it before you leave
• The answer to this question might mean the difference between an all wifi or a partial wifi solution
2. What are you planning on doing with your network?
• You may have simple plans for what you are doing now, but what about the future uses?
• It's not always about gaming
• Consider things like setting up for IOT, cameras, and media in the future
• Write up a list of the order of importance for things you will use the network for
3. Cables will stay, devices will always be subject to moving
• If you have spent the time putting a cable into a wall or overhead, you will really not want to move it again
• I have moved my main router to 5 different locations in my house over the years - will you have to do the same
• Consider the fact that equipment may fail and have to be replaced
• You may, over time, have different ISPs that you deal with - I have dealt with 3 different ISPs over the years, each one had a different way of presenting their services to my house (dial up, cable, fiber optic)
4. Do you really want to drill that hole?
• Once you drill a hole, you are stuck with it
• If you ever move you will have to repair it
• Placement of a hole is important, can it be hidden - not everyone wants to see a cable sticking out of a wall, especially spouses
5. Always start with the cabling
• If your neighbors are "jaming" the wifi signal it will be nice to be able to fall back on cabling to supply your network needs
• Don't forget you have options: Ethernet, Coax Cable, fiber optic in addition to wifi
6. Patch panels, strategically placed, help in the movement of equipment
• If you use keystone patch panels with straight through couplers you will always have the ability to use already made cables
• Couplers allow you to move the positions of the Ethernet cable ends on a single patch panel
• Patch panels allow you to use small patch cables to make one long connection between patch panels when a direct connection was not available
7. Consider the judicious use of vlans in the Network to isolate equipment
• Vlans give you the ability to minimize the number of cables run to different parts of the house; you use a "trunk" line from one managed switch/router to another and keep other ports on the router/managed switch to one specific vlan
• Vlans give you the ability to isolate different segments of your network for different purposes
• If the router from your ISP doesn't have vlan capability, you can always put a router behind it that does, or alternatively use your own router for the connection to the ISP
8. Use the highest bandwidth switches and routers that your budget will allow
• As bandwidth gets cheaper, and/or the signal from your ISP changes over time you may encounter a situation where it is advantageous to have already installed equipment that can meet the bandwidth increase
• This means that you should always consider the cabling and what it's bandwidth capability provides; right now Cat5e should be sufficient, but you may want to consider Cat6 and above for growth
9. Always consider the unthinkable, like failures and how it will affect the network
• I have had older equipment fail on me when it was most inconvenient; being able to quickly replace the equipment because you thought out the problem beforehand will help
• I have lost equipment to lightning strikes; never assume that it will not happen to you
• Put lightning arrestors on Ethernet lines from the outside and on Coax cable connections to equipment
• If at all possible get a whole house surge protector for those times when a higher than usual voltage spike happens that gets into your house
• If it is within your budget consider the judicious use of UPS at different spots in your network; sometimes a hard fail of power will cause a failure in hard drives
• If some piece of equipment fails in the network, how fast can you come back up to a semi normal state, or at least most of your list of network importance from #1 above
10. Consider security when you are setting up the network
• It goes without saying that IOT equipment needs to be isolated, due to it's very nature of contacting outside servers; if at all possible, keep the information in your house
• Never assume that the router on the perimeter will always protect you from intruders; in fact, always assume that someone has already attacked and entered your network and design accordingly

Unusual problem connecting Node Red to Home-Assistant

I decided to update my IOT network, by updating all of my Docker containers.  In doing so, I encountered an unusual problem with connecting Node Red to my Home-Assistant container.  This was from several containers.  Normally, I would be using the simple username password authentication, but now that authentication mechanism is not working at all.  Interestingly enough, part of the problem shows up with the Node Red extension that connects to HA via websockets.  I can no longer set the Home Assistant server or any of the authentication mechanisms associated with it.  In fact, it now shows up as a simple text box to enter the server's name.  But that does no good because I cannot specify what IP, port, or authentication mechanism that I want to use.  I am not sure of how to fix this issue.

Update (2020-10-08): I put the Node Red container on a Raspberry Pi, completely away from the Home -Assistant container and I was able to get Node Red to connect.  Now I have to figure out what has happened to my Node Red container on the HA-IOT Server.

IP Power Strip #03 - the Python development environment setup for the IP Power Strip

I have been reviewing some blog material concerning Python docker development, here, here, and here.  In addition I have been reviewing some Python libraries developed to interface with MQTT.  I intend to use docker-compose to separate out the different elements so that I can individually change the code.  I also need to think closely about how I will be developing containers that might run on different platforms other than the Raspberry Pi, for instance on the Ubuntu server which is of a different architecture.  The environment should look like this:

• IP_Power_Strip_Project
• docker-compose.yaml
• Master
• Dockerfile
• requirements.txt
• Src
• globals.py
• Master.py
• Slave
• Dockerfile
• requirements.txt
• Src
• globals.py
• Slave.py
• InterfaceProxy
• Dockerfile
• requirements.txt
• Src
• globals.py
• InterfaceProxy.py
• Shutdown
• Dockerfile
• requirements.txt
• Src
• globals.py
• Shutdown.py

For right now that covers it.  I will update this post as I come into other aspects of the development.

IP Power Strip #02 - How the IP Power Strip will work in the network

I am looking to make the IP Power Strip as useful in my network as is possible.  I should be able to use the IP Power Strip outside of the normal turn on/off my HomeLab and extend it to things like control lights during Christmas and Halloween.  What I am doing is turning power on and off.  Throughout the year it makes sense that I would use the units in the HomeLab, but these units would also have a use elsewhere.  I am rather interested in integrating it with my HA-IOT network and use the MQTT server that already resides there.

For the Master container, I visualize the following:

• it should be able to be "programmed" with control sequences to a certain point through MQTT messages
• I should be able to issue different sets of controller sequences "programs" so that the IP Power Strip can be controlled independently depending on the context
• I should be able to issue a list of IP Power Strip containers (IP addresses) to the Master container so that it knows the locations
• I should be able to effect a controlled sequence when tasked by an MQTT message
• I should be able to simply indicate a vacation mode and have the Master container do the rest
• I should be able to poll the Master container to reveal the current status of the IP Power Strips and what they are controlling
• The Master container should be able to respond to heartbeat messages in context

For the Slave container, I visualize the following:

• I should be able to have the Slave container come up issuing heartbeat messages to the Master container to indicate the status
• The slave container should be able to be controlled via MQTT messages for a single change or multiple changes
• The slave container should be able to be controlled via MQTT messages to stop/start issuing heartbeat messages

For the InterfaceProxy container, I visualize the following:

• This container is how we interface to the actual equipment, it is a proxy by reason that there may be other already built IP Power Strips, or power like systems that we want to integrate into the IP Power Strip system
• The InterfaceProxy container should be controlled by a RESTful interface, with json coding
• The only container that would interface would be the Slave container, on the other side is hardware
• The InterfaceProxy should receive unique IDs and be able to respond to a request for number of "ports" as well as initial setup and current setup for the "ports".  Initial setup defines how the "ports" come up from a hot start
• Notice that the InterfaceProxy can also serve as a test container which can be metered and check for correct sequences at the low end

For the Shutdown container, I visualize the following:

• I should be able to have the Shutdown container come up issuing heartbeat messages to the Master container to indicate the status
• The Shutdown container should be able to be controlled via MQTT messages to start a shutdown on the computer where it resides
• The Shutdown container should be able to be controlled via MQTT messages to stop/start issuing heartbeat messages

Project #22 - Heartbeat / Shutdown Container to shutdown computer equipment in a graceful manner

I was also thinking of a third container that might be useful.  A container, that interfaces to MQTT, that issues heartbeats, and can issue a shutdown command on equipment.  The master controller would then have the ability to issue an mqtt shutdown command, listen for heartbeats, then when the heartbeat ceases, could issue a command to the IP Power Strip to turn off the power to that computer and report back status on the computer.  This has the added thought that it would be universal, working on different operating systems.  The code is pretty much the same, but the implementation is different.

IP Power Strip #01 - Some initial considerations

 I started thinking about an IP Power Strip almost 3 years ago in December 2017.  Since that time I have been inundated with work issues and never really came back to this project since I was always tired from work.  Now that I have been having more time at home due to the pandemic, and the fact that I had a power outage in my house that damaged a lot of equipment, I have returned to this project.  I have a definite need to turn a lot of my HomeLab equipment off when I am involved with other things.  I hope to travel and my interests have gone to other things like woodworking and wood carving.  I don't wish to give up the HomeLab completely, but at the same time I would not like to pay the whole bill if it just sits there idle.

First, I rearranged my network for power conservation and opened up a new project for an IP Power Strip.  I am hoping to be able to control the power strip from an MQTT based service.  I figure that if I setup Raspberry Pis with some relay hardware on a hat, then have the controller software running on the RPi interfacing to an MQTT server, I should be able to make this work.

I put in an order for three Keyestudio 5V DCAC 4-Channel Relay Shield Module Expansion Board for Raspberry Pi from Amazon.  I figure the easiest way to develop an IP Power Strip is to use a Raspberry Pi for the "brains" and a shield/hat that has relays on it.  Note that this one has a 10Amp limit on current through the relays.  But, and here is the good thing, individual items that will be controlled don't have anything near the 10A limit.

If I develop the software using a python controller, embedded in a docker container, then updates will become easier and my development environment can move around at will.  There will be two actual containers that do the work: (a) a master controller that does the real thinking concerning sequencing of on-off commands, looking for heartbeats from the slaves, etc. and (b) a slave controller that does the heavy lifting in that it actually turns the relays on and off as well as issues heartbeats, etc.  This should allow me to put as many IP Power Strips around the house that is necessary.  Also, I can issue commands via the home-assistant console.

I was also thinking of a third container that might be useful.  A container, that interfaces to MQTT, that issues heartbeats, and can issue a shutdown command on equipment.  The master controller would then have the ability to issue an mqtt shutdown command, listen for heartbeats, then when the heartbeat ceases, could issue a command to the IP Power Strip to turn off the power to that computer and report back status on the computer.  This has the added thought that it would be universal, working on different operating systems.  The code is pretty much the same, but the implementation is different.

Trying to Resurrect the WeatherPi

Now that I have most of the previous parts of the network seemingly working again (have found replacements for the two Netgear GS108Tv2s that crapped out), I can concentrate on some of the other items that seemed to have gone south.  The first up is the WeatherPi that is connected to my Personal Weather Station.  It doesn't seem to be pushing data out anymore.

I am also wondering if this would be a good time to replace the WeatherPi with a USB server and move the weewx program to the HA Server.  The HA Server is also giving me problems right now, but I can deal with that later.  First off, I am going to try to get the WeatherPi to at least push data to weather underground again.

I did manage to remove the WeatherPi from underneath the TV stand and discovered that the SD Card was pretty much done with (probably a result of the power outage).  I put a new SD Card into the RPi that used to be the WeatherPi and discovered that the RPi was okay.  The new designation was PiHoleLDAP; apparently this was the one that was going to be the one that I was going to use for the PiHole, Radius, and LDAP setup using Docker.  Which reminds me, there is no reason that I cannot run weewx under Docker, I just have to research it.

There seems to be a bit of a problem with weather underground.  Since IBM took over, the API has been removed; so things like predictions and the like are no longer available to gather for an html display.  Weewx has not been updated for Python 3, so should I go to a different outlet to publish the weather station data, or should I just abandon the weather station all together?  Part of me is saying the latter, because I still have to maintain the station, look at the stats, and generally update portions of this as my time allows.  Going into retirement, it would be a scheduled activity that I would do say once every two weeks.  Not sure if it is worth the trouble.

Update: it is looking like weewx under Docker is going to be a pain to complete.  There are a number of builds on the Docker site, but all of them are rather complicated.  I am going to have to delay getting this accomplished.

The Network is Starting to Show Signs of Life

After much pushing and proding, I can now see the network coming back to life.  I was very quickly able to get the Media portion of the network back online.  But I was having trouble getting the other things like the printer and the WiFi working again.  The main culprit was the Netgear switch that controlled the printer, HA Server, and UniFi WAP.  When that went down I couldn’t control anything because of the multiple vlan nature of my network.  The switch had all the appearance of being fried, and since I had not succeeded in gettin extra port connections in one of the patch panels upstairs I was limited in what I could do.  Fortunately, I found out that the switch I used on the Ubuntu server was still operational.  Since the Ubuntu server is currently down I decided to use the switch as a replacement for the one upstairs.

So I obtained the text configuration of the switch that failed, changed the IP, and pushed it onto the replacement.  By replacing the previous switch, I now have a working system - at least in part.  This has also given me time to think about how to get my network back and running should I have such a disaster happen again. First, I need some things to make sure that a power outage doesn’t decimate things again.  I ordered some coax surge protectors to put on the TiVo Edge and Mini. Those are installed and I should not have a problem from the coax side of things.  I also purchased a couple of Ethernet surge protectors.  Lastly I have a whole house surge protector on order that should protect that part as well.

Ok People, Learn from my Mistakes

Updated: with new information on discovering electrical wiring mistakes.

This is a long post so bear with me.  I am putting this out there so that others can learn from my mistakes in networking.  If you read through to the end you will learn a lot of lessons including the ones that you will comment as “well everyone knows that.”

I have a distributed HomeLab that I have built up over the years that I had become quite proud of.  It allowed me to move equipment about the house and make changes as I needed as well as include new equipment that I purchase.  This included at least 8 different vlans for different purposes, including separating media and IOT equipment.  On the night of July 23rd we experienced a power outage.  It expressed itself as a loud bang, followed by the lights going out, and I witnessed a few sparks fly from the pole in front of my house (no transformer, just Verizon Fios gear).  The power came on about 30 seconds later but my internet, phone, and TV channels weren’t working (I have Verizon Fios).  SWMBO went to reading while I went off to explore what had happened.

I went downstairs to look at the ONT interface (I have the older version) and it was flashing as well as telling me that it was using battery power however, no problem lights.  Checked the electric panel and no breakers had been thrown.  Went back and reset the ONT interface, unplug and pull battery; reverse order to get it back.  Flashing stops for about 2 seconds, then goes back to the same flashing and battery lights.  So I get my meter and check the plug that the ONT interface was plugged into, power level is where it is expected.  I then went into my computer room and the first thing I notice is that my main router and accompanying managed switch have no lights on the front panel.  Sure enough, I try to access the router from my Mac Mini and no luck.  I pull the power from the router and the switch and I am getting power to both showing on the meter; also the UPS they are plugged into is showing no signs of problems.  The UPS has regulated power as well as surge protection so I was surprised.  I then pull out all of the patch cables from one of my extra Edgerouter-X, then I pull the ONT patch cord out from my main router and put it into the ER-X wan port, then I pull up a cable and connect the Mac Mini to the ER-X.  Looking into the configuration screen, I see that the ONT is alternating between connected and disconnected, and no IP.

Lesson 1: make sure you have a back up plan to troubleshoot in case you have a catastrophic network failure.

Lesson 2: a Patch Panel setup can save your bacon if you have to reconfigure your network in a hurry.

I tried a few different things, then concluded that the ONT wasn’t going to do its thing anytime soon.  I went off to bed knowing that SWMBO would not be happy in the morning.  The next morning I went searching for damage.  I found that I had lost the ONT (apparently), my main Cisco router (on UPS), my main 26 port Netgear managed switch (also on UPS), along with two 8 port managed switches (both on UPS) in my network.  I then set about reconnecting patches in a different way to connect my media equipment to the ER-X I had setup last night.  Fortunately, that was rather quick given that I had a diagram of the patch panel connections.  Once I did that, I called Verizon to get a tech out to fix the Fios connection issue, delay was a couple of days at least due to volume.  Having the media equipment on a flat network then allowed me to at least view the recordings SWMBO had made on our TiVo Bolt.  Unfortunately, it also showed me that I couldn’t see any of the TV channels on the Bolt, which was odd.

The Verizon tech appeared the next day and after some analysis replaced the ONT, and set me up.  Now, I was getting internet and my phone (on internet) was working.  Unfortunately, there were still no channels that I could receive on the Bolt, even after a cable card swap; I could however get to the internet.  So, I reasoned that the Bolt was bad so I contacted a family member that had recently converted their Bolt to a TiVo Edge.  I picked up the Bolt, got it back to the house, connected it and now I was receiving tv channels, verifying that I had lost my TiVo Bolt somewhere in the process.  I went downstairs and attempted to connect the TiVo Mini to the new Bolt, but was unsuccessful because these pieces of equipment were on two separate TiVo plans and the TiVo DRM kept me from connecting them together.  I then went out and purchased a couple of unmanaged switches that had decent throughput (2GB per port) to use with setting up the media on the network.

Lesson 3: Don’t forget that there may be DRM issues when you try to bring your equipment back up.

Lesson 4: Having equipment on UPS doesn’t guarantee that your equipment will be perfectly protected, there may be other issues.

I swapped some more patch cables so that I could get my UniFi AC-AP-Pro to at least give me wifi to the internet.  Now that I had at least a semblance of media working for SWMBO, it was time to determine what else had happened and how.  I went back and looked at my Cisco router and discovered that I could reset it and at least get the front lights working.  However, it was apparent that at least half of the 16 ports were fried, including the two WAN ports.  This led me to realize that there must have been a surge through the Ethernet lines; again odd.  In fact when I traced the path of the equipment that had failed, it was apparent that the surge came through the Ethernet cables connecting each device.  Since my TiVo Bolt was able to access internet but not the tv channels, it must have gotten the surge through the Coax cable.  Update: I later discovered that electricians that had wired my new kitchen appliances had used the main conduit that I had my longest Ethernet cabling and COAX cabling on.  Evidently I was the recipient of a power surge that had induced voltage in the Ethernet and COAX lines enough that it overcame the networking and media equipment.  I didn't first figure this out due to the odd failures I was seeing.

Lesson 5: it is important to invest in surge protectors on both Ethernet lines and Coax cables in your network because stuff happens.  Oh, and it wouldn't hurt to have a whole house surge protection system to limit collateral damage.

I have since purchased an Edgerouter 12 and the same Netgear M4100-26g switch that I had before (I actually like this switch).  I am still going through provisioning my network even two weeks after the initial power outage.  This morning I discovered another device that had bit the dust, my Ubuntu Desktop server.  It has all the appearance of a power supply problem; it boots, then powers off very quickly, even after getting to the screen where I enter credentials to login.  The server was also plugged into the UPS, go figure.  I am going to check out this one UPS to make sure that it is doing what it is supposed to do.  I also don’t have the UPS connected to a device to register error messages and do something with that information.  Update: I did indeed have to replace the power supply in the Ubuntu server.

Lesson 6: if you are using a UPS, make sure it is connected to a server to gather error messages, if nothing else but to determine if it is going bad.

Long story short, I am still finding things that have failed or partially failed due to this power outage.  I haven’t even looked at my home-assistant server and IOT equipment yet.  I am almost afraid to do so.

Lesson 7: have at least a minimal setup in mind to get your network back up after failures.  It may involve using older equipment or less capable equipment but those devices can come in handy.  Don’t just throw out old equipment, but be reasonable as well.  Don’t keep everything you have ever had in the network, just keep enough to get you back up and running.

Lesson 8: make a decision on what is important and what can wait should a catastrophe happen and you have to configure quickly.

Post log: I now have Ethernet surge protectors, Coax surge protectors, a whole house surge protector setup, have removed / rerouted the central Coax, and have two fiber optic setups running through the original central channel for network connection.

Hopefully, these thoughts will be important enough to you to think through your network and what may happen.  Good luck to everyone!!

LW

Project #19 - Get the Network back to Normal after the Power Outage

After the power outage that took out the main part of my Network this project is to get the Network back to normal.

Catastrophic Surge Failure in the Network!

I am writing this for others to learn from my mistakes.  On July 23, 2020 at approximately 9:30 pm I experienced a power outage for about 30 seconds.  There was a loud bang followed by the power going out.  I looked up just in time to see some sparks fall from the pole outside my house.  This power outage produced a surge that took out:

1. my Verizon ONT
2. my Cisco RV325 router ($260 used)
3. my Netgear M4100-26G 26 port switch ($260 - $525 new)
4. two Netgear GS108Tv2 ProSafe 8 port switches ($66 - $80 new each)
5. my TiVo Bolt ($175 new)

Unfortunately, I didn't have any surge protectors in the circuit coming into the house.  This was not an electrical surge that took out the equipment, except for the ONT.  The surge came through both the Cable and the Ethernet coming out of the ONT.  Fortunately for me I had some spare router equipment (Edgerouter-X) and some unmanaged switches.  Since I had patch panels it became an easy rerouting of the cabling to get my Media up and running and get connected back to Verizon Fios after the ONT was changed out.

My plan going forward is:

1. put in a Cable surge protector between the ONT and the splitter outside my house
2. put in an Ethernet surge protector between the ONT and the main router
3. at each of the 5 points in my network, put a conditioned power UPS with both Ethernet and COAX surge protectors

Please learn from my mistake, this was expensive to undo.

Project #18 - Setup a RetroPie on the Mac Mini monitor downstairs

I have for a while been wanting to get into games.  Normally, I am quite the nerd, only wanting to do things that have some "merit".  However, as I approach retirement I want to be able to have some down time from my technical activities and woodworking.  I have been looking at RetroPie and have been very interested in getting a system up and running.  Since I already have an RPi4-4GB attached to the Mac Mini monitor and I already have a game controller, I thought it would be a prudent use of resources to put RetroPie on an SD card and use it there.

Project #17 - Add RPi 3D Printer viewer to Network

I find myself more and more wanting to be able to look at what is going on when I am doing a long print on my 3D printer.  I do have a Wyze camera that I installed downstairs, but I would like to be able to see how far along the print is, as well as temp of components, etc.  I can do this by installing any number of packages that are out there.

Update: with the advent of using Wyze Cameras in my network this is now OBE.  I now have a Wyze camera pointed to the 3D Printer when in use.

Tips #6 - Faking an Admin vlan on the Netgear GS108Ev3 and GS116Ev2

My network at home is distributed and I try to minimize the number of wires going throughout the house.  That becomes a problem if you are single switch dependent.  But with judicious use of vlans throughout the house, I can distribute these through the use of trunk lines.  One thing I am very conscious about is security in the network.  I make judicious use of Admin vlans in order to keep the network framework from being changed by someone who has hacked into my system.

I went out and bought a GS108Ev3 thinking it was a good and cheap alternative to the GS108Tv2.  Boy was I wrong.  My focus on security took a nose dive, or so I thought, when I started incorporating this switch into my network.  It was the same thing for the GS116Ev2 when I added this switch.  If you follow a simple set of rules, you can keep your Admin vlan:

1. a normal Netgear managed switch like the GS108Tv2 has an Admin vlan capability.  In fact, you can actually put the vlan number into a field in the web interface.  The switch will use this vlan number to accept DHCP IP addresses for the switch.  If you fix the IP address of the switch, you will need to be in the subnet of the Admin vlan for the switch to work.
2. there is a new class of Netgear switches, like the GS108Ev3 and the GS116Ev2 which are simplified but have much of the capabilities of the GS108Tv2.  One of the capabilities that is missing is the Admin vlan.  There is no field to set it and you are forced to use vlan 1 in some manner to interface to your network.  You do that on a trunk port
3. On GS108Ev3/GS116Ev2: either use DHCP or set the IP address/Gateway to be within the Admin vlan subnet
4. On GS108Ev3/GS116Ev2: set vlan 1 to be untagged on the trunk port; set the Admin vlan to be tagged on the trunk port; set the Admin vlan to be the PVID on the trunk port
5. On GS108Tv2/M4100: set the Admin vlan to be the PVID on the trunk port; set the Admin vlan to be tagged on the trunk port
6. On GS108Ev3/GS116Ev2: no other ports may have vlan 1 as untagged, except the trunk ports

Not a difficult configuration, but as long as you remember the rules you can get an Admin vlan on these GS108Ev3/GS116Ev2 switches.

DHCP-DNS on a PiHole

So now is the time to start putting in a PiHole to control DNS access.  All along I have been thinking of constraining the dns connections from the IOT equipment.  There are times when equipment will attempt to get it’s DNS ip addresses through a hard coded IP address.  What I want to do is force the DNS access through the PiHole and have the PiHole get its DNS addresses via DNSSEC to either 1.1.1.1 or 9.9.9.9.  That way I have the PiHole restricting the trackers and the DNSSEC servers to provide restriction from known bad addresses, whether web or ip.  Since I have moved the IOT equipment to one of the Edgerouter-Xs, I should be able to control the DNS access through some judicious ACL setups.  Should be interesting.

First of all the PiHole is put into the IOT lan subnet, and the ER-X then uses the PiHole as both DHCP and DNS services.  I will setup the ER-X to force any DNS service connections to go through the PiHole (https://community.ui.com/questions/Intercepting-and-Re-Directing-DNS-Queries/cd0a248d-ca54-4d16-84c6-a5ade3dc3272 and https://benninger.ca/posts/force-dns-go-through-pihole/ and https://www.myhelpfulguides.com/2018/07/30/redirect-hard-coded-dns-to-pi-hole-using-edgerouter-x/ ).  Since I know all of the equipment in my network, including the IOT network, I will force IP addresses where I want them through Static assignment (TBD).  In addition, I am going to restrict Bonjour access throughout the ER-X lan subnet (TBD).  I will of course restrict changes to the ER-X through my Admin vlan and not through the IOT vlan (TBD).  Anyway that is the idea at the moment.  Time will tell how well this works.

Tips #5 - Alignment 101 on a 3D Printer

After I purchased my Hictop Prusa i3 Clone four years ago, I went through a very trying time attempting to discover what things were going wrong with the printer.  It seemed like I got excellent prints, then there was a disaster, something out of kilter, or even an indication that squares weren't square anymore.  The more I fiddled, the more I discovered some unwritten rules of keeping a 3D printer up to date, at least in alignments.  So here are some tips for making better use of time between issues that arise (note that these items are for Reprap style printers, like the Prusa i3, but the principles are the same with most printers):

Assumptions:

1. Get used to the idea that you will have to re-align the printer every so many prints, even with an auto-leveler
2. Over time the correction adjusters will come loose because of shaking and general acceleration of parts
3. The same shaking will cause corners and cams to come loose
4. Belts do not stay perpetually tight
5. Mechanical stuff will wear out and will need replacing

Rules:

1. Make sure the printer is off, it makes it easier to move things around without having to disable the stepper motors.
2. Start by making sure that all of your 3D printer corners are exactly square.  You may think that you have a 90 degree corner but unless it is measured to be 90 degrees, it is not.  If the corners are not aligned, you will not see a rectangle on prints, you will see a rhomboid, and if it is really bad a trapezoid.
3. First and foremost after squaring the printer, make sure that all bolts are tightened on the printer
4. Make sure that all belts are tight and that the cams are on the stepper motors correctly (usually by having the set screw on the flat portion of the spindle); make sure all cam set screws are tight.
5. Check the movement of the belts as they are moved back and forth.  What you are looking for here is making sure that the belt moves easily without binding, or rubbing on anything; especially important for the X and Y axis belts.
6. Clamp the printer structure to its surface to prevent shaking that might cause the printer to move from its location.
7. Now we start aligning stuff.  Start by making sure the printer structure itself is level in the XY plane, use a level - I prefer a magnetic level and sometimes use a circular level.  You would be surprised to find that most surfaces are not level.  Sometimes some wooden shims are needed to get it level on a table or bench surface.
8. Once the structure is level, level the print bed (moves in the Y axis direction) using the adjustment screws on all four corners.  You need to have it level from corner to corner and diagonally.  I like to start in the middle of the bed checking diagonal level in both diagonals.
9. Apply blue tape if you use it to the bed.
10. Next, check the level of the X axis beam on the metal rods that the print head travels on.  If the metal rod is not level you will need to adjust the Z axis couplers on one side or the other to bring it into a level condition.  If the printer had been on you would need to unlock the stepper motors to do this.
11. Next, check the distance of the print head from the bed by adjusting the Z axis couplers on either side of the bed to bring it up or down as necessary.  I generally find that the width of a separation tool is enough, others like the width of a piece of paper.
12. Recheck the X axis level after changing the Z axis couplers.
13. Turn on the printer.
14. Adjust the auto-leveler so that it just comes on with the present settings.
15. Attempt an auto home. Recheck the distance of the print head from the bed.  If adjustment is needed, unlock the stepper motors prior to moving the Z axis couplers and readjust auto-leveler as per #11 and #14.  Repeat until the distance of the print head is acceptable.
16. At this point you are as aligned as you can be mechanically.
17. To help keep alignment, consider lowering the maximum acceleration level and the xy jerk value; this will also serve to lower the shaking and “walking” that most printers tend to have.
18. Higher temperatures will add to misalignment due to expansion of metal parts.  So recheck alignment of the bed after a high temperature print.

Tips #4 - vlan expressions for documentation

If you have a network running through your house like I do, and have vlans as part of it, then you probably have some so called "trunk" Ethernet cables.  So how do you explain what vlans are where?  This is what I do:

• A port on a router or a switch that has an untagged vlan is noted with brackets around the vlan number like [4] for an untagged port with vlan4 present.
• When designating a "trunk" port on a router or a switch you have tagged vlans and you have a PVID assigned.  I put the PVID first, to know what vlan is used when there is no tagging, a slash, and then a T followed by a list of each of the vlans that are tagged on that port.  So it looks like [6/T4,8,88,123-124,152] for a PVID of vlan6 with tagged vlans: 4, 8, 88, 123, 124, and 152.  I still put brackets around the values to designate that these are vlans I am dealing with.  I use the dashes as a shorthand noting that this includes all of the vlan numbers in between.
• I generally like to make the "trunk" lines between switches and routers have a standard grouping of vlans.  In that way, a shorthand would be [6/T] where the tagged vlans are common across the entire network.
• When I use "trunk" lines, I like to reserve a couple of vlans for specific purposes other than normal traffic.  
• I use vlan850 as a shorthand in a switch to denote that the ports that use vlan850 are all untagged and serve as a short unmanaged switch.  I never allow vlan850 to be included in any "trunk" configurations.
• I use vlan11, always untagged, as a dead port designation.  What this means is that vlan11 does not get assigned to any "trunk" configuration and serves to simply block a port from being useful when it is not used.
• I use several vlans, for example 66 through70, as temporary patch vlans.  What this means is that I can attach an untagged port to a switch on one end of the house and an untagged port to a switch on the other end of the house and effectively have a "patch cable" in between the ports.  One caution, normally you cannot assign one end to the same port as a monitor port on a switch, at least I haven't figured out how to yet.

Tips #3 - Documentation on Networks and Vlans

As your home network grows, so does the level of confusion about what was wired up where and when.  In general, you can document what happened through a journal about changes you have made to the network.  But filtering through that journal when you are in the midst of making lots of changes will be very time consuming.  It is better to keep the information in a separate list.  Normally, devices that you have in the network will retain certain Ethernet addresses, but sometimes the wiring will change and the placement of the equipment as well.  There are four specific documents that I use to document different views of the network and provide an easier way of changing the layout as my equipment, equipment placement, wiring, etc. change over time.  In my case, I keep the four documents as notes in my CellPhone under a classification of HomeNetwork.  The four documents are:

1. Current IPs in the Network
2. Patch Panels
3. Cable Connections
4. Home Vlans

These are described below:

-- Current IPs in the Network

• The "Current IPs in the Network" and "Home Vlans" rarely if ever change.  However, if I move a piece of equipment in my network, "Patch Panels" and "Cable Connections" will be altered.  If I move a piece of equipment from one vlan to another, by definition the "Current IPs in the Network" will change, along with notations about the connections in the other documents.
• The "Current IPs in the Network" note should contain all known IPs that you have in your home network.  I have even included IPs from equipment that are connected to multiple vlans.  That way the document is all inclusive.  I also make sure that all IPs that show up in my network are known.  If it isn't on the list then I know someone has attached themselves to my network that I don't know about and I go and hunt down what it is.  Rogue pieces of equipment or IOT equipment that doesn't behave I disconnect.

-- Patch Panels

• I have already described how I label patch panels here.
• The "Patch Panels" note documents the infrastructure of the network
• The "Patch Panels" note is broken down to each patch panel with what is connected by port number on that patch panel.
• The "Patch Panels" not does not contain any information about what equipment the port connects to, i.e. with a patch cable connection from the device to the patch panel, only documentation of what connections exist.  
• There is an exception where the patch panel port actually goes directly to a device and is designed to be there consistently.  For instance, I have a PiHole in my network and that is connected behind one of my patch panels.  So for the label on that patch panel, I label the port as "PiHole", e.g. PP#3-7 is labeled PiHole so I know where the port goes to.
• I use the term "patch to" when describing the use of a patch cable at a specific patch panel.  For instance one of my patch designations at patch panel 3 port 11 is "PP#3-11 - PP#1-10".  I can usually read where the port is going by looking at the actual patch panel.
• If I have a patch cable going between ports on a given patch panel, my "Patch Panels" note will have something like this: "PP#1-10 - PP#3-11 (patch to PP#1-9)" for the designation on patch panel one port 10, and I will have "PP#1-9 - PP#2-6 (patch to PP#1-10)" for patch panel one port 9.  Note that this shows where my patch cables are and how they connect.  I can always go to the "Patch Panels" note and trace from one point to another to understand where the cable ends.
• If I move/remove the patch cable from a patch panel, then I can simply modify a couple of lines to show how things are connected.  Again, the idea is to be able to trace to the end.
• Note that I do not list any patch cables from devices that are plugged into a port on one end or the other.

-- Cable Connections

• The "Cable Connections" note is where I list the patch cable connections to devices.  It also designates a logical connection to an end point.  Normally there would be a patch cable from a device, and on the other end of the trace there would be a connection to a different device.
• The "Cable Connections" note is grouped by device which allows me to immediately find out where the device is connected by port number.
• In the "Cable Connections" note I describe on a device port where the connection ends up.  For instance, I have a connection from my main switch which is a "trunk" connection to another switch.  The way that I write it is as follows, using the last portion of the switch IP addresses:  ".119-15 - to PP#3-23 -> .118-1 [6/T]" which is a shorthand way of listing where the connection ends up, even though the actual trace through my network goes through 3 patch panels.  I can trace the physical connections through the "Patch Panels" listing and I note the vlan trunk elements (a tip on vlan nomenclature is here).  Correspondingly, I look at the port for the receiving switch and it looks like this: ".118-1 - to PP#4-X -> .119-15 [6/T]" which indicates the reverse direction.  Again, I can look through the "Patch Panels" listing and discover the physical connections through my network.

-- Home Vlans

• The "Home Vlans" note is where I give a listing of each vlan used in my network, including why I have the vlan, what DHCP server address is used, and what DNS server address(s) are used.
• The "Home Vlans" note also contains a line which lists the trunk line vlans for the normal case.

Tips #2 - Network Journal

Having been an engineer for the last 45 years, I do keep track of what I do in my network.  This includes how I wire up things and why, what I do with equipment, things that I am thinking of doing, future projects, documenting current projects, and software changes.  I use a manually maintained engineering notebook format to do this.  In my case, I call it a Network Journal.  I use the following format (based on the principle of linked lists and indexes) when documenting stuff in the Network Journal (it just works for me):

1. I always use quad ruled composition books for my notebooks.  The quad rule gives something for the eye to use when drawing diagrams.
2. Each page in the notebook needs to have a number starting from 0.  The page numbers provide an easy reference point when indexing.
3. Page 0 is reserved for a Table of Contents.
4. The title of the notebook in our case is "Network Journal".  In general, I title each notebook with the subject that the notebook contains.  If this is the second notebook in the subject, it would be titled "Network Journal 2".  Write it in ink.
5. The notebook should be individually numbered, I like to use an abbreviation of the subject, in caps, with a dash and a number.  In this case the number of the notebook would be "NETJRN-1" with "NETJRN-2" for the second notebook in the subject.  It should be written on the front of the notebook and also written across the bottom of the notebook (there should be enough space to write this when holding the pages together.  The purpose is to be able to see the number if the notebook is laid down or is stored vertically in a page sorter used for storing multiple notebooks on different subjects.
6. The rule when referencing page numbers is that if it is in the same notebook you use the page number alone.  If it references a page in a different notebook you reference the notebook number a slash and the page number in that notebook.  If I reference page 53 in the same notebook, it is "->53".  However, if I reference page 87 in a different notebook, say "NETJRN-1" then I use "->87/NETJRN-1".  In this manner I can reference things in all of my notebooks from each other.
7. One specific rule is that I never erase a page in a notebook.  I keep it for posterity, including references.
8. When I put an index on a page, I title the index drawn on a line about 3/4 of an inch below the top.  I underline the title, and put the date that I started the index in the margin to the left.  If this index is a continuation of an index, I use the same title, but reference the page that has the previous index by a back arrow ("<-page_number") with a box around it.  If this index is a lower hierarchy from the previous index, I use an up arrow curving left ("<-|page_number") to reference the page.   Note that if I continue the index to a different page, I use an arrow pointing to the page and draw a box around it - "->page_number".
9. An item to me is an individual thing that I wish to keep track of by referencing it from an index.  This could be meeting minutes if I discuss something with someone, an idea that I want to expand upon, a specific change that I am making to cabling, a new vlan and a description of its purpose, etc.  This includes text, drawings, portions of pages that have been scanned.  Each item has a title and that title should be on the same page
10. When I add an item to an index page, I put the title that I have used on the page in the notebook, put the date in the left hand column, and on the same line I point to the page that the item occurs on "->page_number".  This way I can simply look down the page and go right to the page number of the item I am interested in reviewing.
11. Likewise when items go on a page, if this is the first thing on the page - I put the topic of the index that this page refers to at the very top of the page (if it is not there already).  Note, I limit it to one index topic per page to ensure that everything on the page is of the same index topic.  If there is already some items on the page for the same topic, I draw a line across and continue after the line on the same page.  I start with the title underlined, with the date in the left hand margin, then I put an up arrow curving left ("<-|page_number") with the page of the index that refers to this item.
12. If I need to continue the item on a different page, I just put an arrow pointing to the page I am continuing on and form a box around it ("->page_number").  Then I repeat on the new page.  Note that I don't have to use the very next page, I can even go backwards if necessary.

Tips #1 - Patch Panels and Markings

In my network, I have found that the best way to maintain my understanding of how my house is wired with Ethernet is how that I document my Patch Panels.  I have a few rules that maintain a cohesiveness to this:

• Patch Panels serve to be the ends of Ethernet wires.  In general, I will not have a stray Ethernet cable coming out of a wall.
• I use keystones in all patch panels.  This gives me the choice of terminating Ethernet wires either in a punch-down keystone, or a straight through keystone which has an Ethernet female on both sides.  
• The advantage with using keystones is that they are easy to move around in a patch panel, or in the case of the straight through keystone, Ethernet connections behind them.  Normally equipment moves and cables stay put; the keystones are a way of augmenting that "staying put".
• I refer to an individual keystone as a "port" in a patch panel.  Each "port" must be numbered from 1 to the total number of ports on the patch panel.
• All patch panels in my house are numbered.  If the patch panel serves only to connect to another patch panel on the other side of the wall, I may number it with the same number but I have to use the same "port" numbers on each side.  So if I use "port" 1 on one side, then I must use "port" 1 on the other side so there is no confusion. In this case each patch panel gets the same patch panel number.  I must also ensure that the Ethernet cable is straight through.
• The full designated "port" identifier is patch_panel_number-port_number, e.g. PP#3-12 would refer to keystone (or "port") 12 on patch panel number 3.
• I use a labeler to mark each "port" on each patch panel.  When I mark a specific "port" I label it with the "port" that it connects to on the other end of the Ethernet cable.  For instance, if PP#3-12 connects to PP#1-6 then on patch panel number 1, "port" 6 would be labeled PP#3-12, and on patch panel number 3, "port" 12 would be labeled PP#1-6.  That way I always know where the Ethernet cable is connected and to what patch panel.
• If a cable is moved from one keystone to another on a patch panel, the label goes with it and I make up a new label for the port that it connects to so I am consistent.  For instance, if PP#3-12 connects to PP#1-6, and I move the Ethernet cable from port 12 to port 10 on patch panel 3, then I move the PP#1-6 label from port 12 to port 10 on patch panel 3, then I make up a new label of PP#3-10, go to patch panel 1, remove the label on port 6, and replace it with the new label of PP#3-10.  I also make note of it on any note that I have.
• Patch panels can always be used as "patch panels", i.e. I have places on patch panels that have a patch Ethernet cable between "ports" on the same patch panel.  That way, it can form one long Ethernet connection.

HomeLink HA Interface #8 - Now have Working Interface

From the point of getting NodeRed to run, I ran several tests to figure out how to publish mqtt messages that could be sent to my home-assistant server.  Since there are three buttons on the clicker for testing, here are the flows that I used:

HomeLink HA Interface #7 - altered way to put NodeRed and setup interface

I was having difficulties with getting the docker version of node-red to do what I wanted to do with the Raspberry Pi.  It seems like I wasn't able to update Node.js and the docker repository wasn't helping.  So I decided to abandon the docker version of node-red, but keep docker, run node-red natively, and set it up to command the interface to send MQTT topics to my Home-Assistant server.

-- remove the docker-compose elements
sudo docker-compose down -rmi all

The preceding command not only stops all containers setup using docker-compose, but it removes the images after stopping the process.

Now that I have accomplished this, I move to install NodeRed on the Raspberry Pi natively.

-- install nodered, node.js
sudo apt update
sudo apt install build-essential git
bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

This did manage to take a long time due to this being an RPi Zero.