Ran Another Test by Connecting to a Different Room Power Outlet

 I ran a different test with the AV2 Powerlines.  This time I purposefully plugged a different AV2000 into my small bedroom outlet to see what the increase in bandwidth would be from the living room to the small bedroom.  Unfortunately, the increase was only about doubling the speed.  I was getting around 93 Mbit/sec throughput instead of 35 Mbit/sec.

The test was composed of putting a new Powerline adapter into the small bedroom, connecting the new Powerline adapter to my HomeLab switch, and running some iperf tests.  The small bedroom circuit breaker is on the opposite side of the breaker box from my large bedroom circuit breaker; and on the same side as the circuit breaker for the living room.  Again this is 25 year old wiring.  I guess in the grand scheme of things I could have an electrician move the wires between the two breakers to get the large bedroom circuit breaker on the same side of the breaker box as the living room circuit breaker.  But, is the increase in speed worth it?  It will cost me money but in the end I will have more bandwidth.

Update: I am now trying to see if there is a different way of putting my Travel Routers and arranging my network in the back bedroom that will increase my throughput to the HomeLab.

Added a new 802.11ax Router to the Apartment

So I have purchased a new router, the Gl-iNet FLINT (GL-AX1800).  The reason that I did this was to optimize the connection between my HomeLab equipment and the apartment WiFi end point.  The FLINT is located 17 and a half feet from the end point and has 4 external antenna.  In addition, the FLINT is based on OpenWRT, which the SLATE was also based on.  Right now, being a new product, the software load is not quite up to the current edition of OpenWRT, but the beta test versions are promising.  I have downloaded the latest Beta and will be updating this router prior to the full software release.  The reason is to be able to setup vlans so that I can also plug in my TV and DVD player without interfering with the HomeLab connection.

After setting up the router in the space in the living room, I am measuring a speed of 400Mbps/400Mbps to the end point when setup as a repeater.  This is the kind of speed gain that I am hoping for.  The WiFi connection is continuing to reset itself and I need to know why.  I have noticed that the main indication of failure is that the DNS is no longer able to work through the connection and that the Weather Station is no longer pushing out information.

Update: so now the FLINT router has crapped out, possibly because I did some setting I shouldn’t have.  I have updated to the latest v4.1.0 beta without any custom values and now it works.  Go figure.

Update2: the problem with the WiFi connection stopping appears to be related to a DNS issue.  I can still ping IPs but just can't use DNS names.  Is this some setting somewhere?

There appears to be an issue with the WiFi in my apartment

I have discovered that the WiFi is now stopping on a regular basis in my apartment.  I have no control over it but I would like to set things up so that I don’t have as many issues in what is available.  I have moved the Tempest Weather Station to the Portal SSID so that it has the most connectivity that it can get without being connected to the equipment in my rack.  I am also going to take down the UniFi AP AC Pro so that it doesn’t add to the interference in the apartment.  Perhaps removing this source will help with the overall speed of the WiFi in my apartment.  The problem with this is that I am using the UniFi to connect to the Tasmota switches.  I need something to do this with.  It occurred to me that I might use one of the RPi Zero Ws that I have to make up an isolated network within the rack.  If I keep the power level low enough, I should not cause interference outside of the rack.  Note, I do not have control over what the WiFi does or does not do.

I am also setting up an active/passive WiFi checker within the apartment so that I can get some data with which to analyze what might the problem be.

Update:  I have found 25 APs for the complex included in 832 devices just in 45 minutes.  There is ample amount of WiFi packets that could potentially cause issues, especially if they are on the cusp of the signal.  Further research is warranted.

There seems to be an issue with the Gateway input keeping the DNS server up

Now that I have the Tempest Weather Station online, albeit without pushing the data to an official source, it seems strange that I am constantly losing DNS resolving.  The retirement community network uses 1.1.1.1 and 1.0.0.1 as the DNS servers.  I seem to lose connection to the DNS servers after about a day and a half.  This could be due to several things like pushing too much information out over the resident vlan or not having sufficient TLS connection credentials.  I am not sure at this point.

The first item, pushing too much information out, might be the likely culprit since the network setup might think that this is a spamming engine.  I will likely have to call IT to find out.  This would involve some sort of QoS with the UDP output from the Tempest Hub.

The second item, not having sufficient TLS connection credentials, is quite possible since the 1.1.1.1 and 1.0.0.1 both require a DNSSec connection (I think).  I will have to investigate further.

Update: I have noticed that the TCL TV occasionally reconnects, so it appears that I have an issue with WiFi in general. I am now connecting the Tempest Weather Station to the apartment WiFi system without going through my rack.  I will rely on extracting information from it across the internet using API calls now.

Updating RPiCICD with Docker Containers for the CI-CD pipeline

I have started the update to RPiCICD with:

• Added a 2TB USB 3.0 drive; added it to fstab to make sure it comes back up each time there is a restart
• Added a number of docker-compose containers from https://github.com/vimagick/dockerfiles
• ansible
• confluence
• drone
• gitea - this is going to be my git repository
• jenkins
• mariadb - provides MySQL on the 2TB drive
• mosquitto - there as part of the overall sequencing
• nextcloud
• nexus3
• nfs - provides nfs access to the 2TB drive
• nginx
• owncast
• prometheus
• rsyslog
• samba - provides smb access to the 2TB drive
• telegraf
• wiki
• wireguard
• wireguard-ui
• wordpress
• zookeeper
• zoonavigator

Other items to complete the transform include:

• Setting up the file share portion with SMB and NFS
• I appear to have NFS connected to ../data on the 2TB drive
• The NFS connection appears to be tcp6 only; need to investigate
• Setting up the MySQL server (mariadb)
• Setting up the nextcloud service
• Setting up Jenkins scripts for the WWD project

Project #29 - Development of a CI-CD Pipeline on RPiCICD

In order to support some of the other projects that I am working on, I need to develop a Continuous Integration (CI) Continuous Deployment (CD) pipline.  To this end, I am pulling in a number of different docker containers that can be used for this purpose.  I want to make this as simple as possible but no simpler.

Experiments Show PowerLine Adapters Often Don’t Work Like You Think They Should

The IT department here in the Retirement Community keeps making changes to make things good.  So it occurred to me that maybe things might have changed in my back bedroom for the good.  So I set out to find out what kind of speeds that I was getting between my input setup and my back bedroom where my HomeLab rack resides.  So I set up some speed tests using iperf on opposite sides of the input flow.  Much to my surprise, I found out that I was only getting around 38 Mbps throughput.  I had thought I was getting closer to 1 Gbps, given the type of AV2 units that I was using.  I tried different things like getting the managed switches out of the way, but still was achieving only the 38 Mbps that I had first measured.  I also tried using Speedtest.net and found that this was the max speed that I could get.

I downloaded the TP-Link app so that I could tell what the system was saying that it was getting between units.  It was displaying around 100 Mbps speed between the units, so I started looking into what kind of speeds I could expect with different wall outlets.  I brought in a different unit and added it to the mix and started moving it about the apartment to see what I could find out.  When it was plugged into a circuit that was near one of the two AV2000 units, it would register a speed of 1300-1400 Mbps, but would still measure the 100 Mbps to the other AV2000 unit.  As I was swapping them around, I could see patterns that identified a good vs bad line.  The low bandwidth is probably due to the age of the wiring in this apartment and having to go through an old breaker box.  Nothing I can do about that right now.

I am still trying to find the “sweet” spot to have equipment in the living room, but so far have not found it.

Update: so I have discovered that there are three circuit breakers that the signal is probably passing through.  One for the living room, one for the small bedroom, and one for the large bedroom.  They also appear to be on opposite sides of the mains, so much for finding a "sweet" spot to use.  I am stuck with what I have then.

Project #28 - Development of a WireGuard-Hub and Breakout-Box

I would like to have a way of contacting my HomeLab without incurring a port on the internet, which in my case I have no control over.  The idea is to put a WireGuard-Hub (using a Raspberry Pi) into a friend's network, with a port open to the outside, subject to dynamic dns.  The dynamic dns would always point to the WireGuard-Hub's location.  The WireGuard-Hub would be on it's own private vlan or be the DMZ host in the friend's network.  The WireGuard-Hub would be the clearing house for WireGuard connections privately outside of anyone's network and connections would come to the WireGuard-Hub either through laptops/cellphones/tablet WireGuard encrypted connections or through a Breakout-Box in someone's network.  The Breakout-Box would push a WireGuard connection to the WireGuard-Hub through the dynamic dns location and at the same time would be the connection to whatever network it was connected to.  The idea is to have a cheap and quick connection between many networks and devices that is peer to peer and doesn't require an elaborate setup.

The New Input Architecture to my HomeLab

 I think I may have solved my problem with the WiFi system here at the Retirement Community.  As I said before, this is a WiFi only environment; I can run Ethernet but not through the walls and not connected to the WiFi equipment and I have no control over the WiFi system.  The WiFi equipment has two main SSIDs, SSID1 and SSID2 in the following diagram.

The Incoming WiFi Architecture

Notice that there are a number of additional elements to the architecture.  The main input to the MainRouter in my HomeLab is through the WiFi Bridge, a Gl-iNet SLATE router.  That in turn goes into a managed switch, RemoteSwitch, which has a trunk line to my MainSwitch through Two TP-Link AV2000s.  The MainSwitch in turn poses the flow as the [WAN] vlan to the WAN port of my MainRouter.  Having the trunk line between two managed switches gives me great flexibility in how I arrange items in my HomeLab.  In fact I can put part of my [SRV] vlan in the living room where the main input equipment goes.  The MainSwitch also connects to other equipment in my HomeLab as necessary.  Normally I have almost a 30 dBm signal drop between the WiFi Access Point in the living room and my large back bedroom.  This input architecture resolves that.

I have an RPiGateway which is being primed to host input ports into my HomeLab from the outside.  As it turns out I can access RPiGateway from anywhere on the Retirement community campus; I just can't access it from the outside (and I don't need to, I'm not working now).  

Over on the right of the diagram is a problem that was solved through the use of a simple switch.  I have a Sleep-Number bed and it requires a WPA/WPA2 shared key encryption setup on 2.6GHz.  It is also a bit fussy on the level of the signal that it connects to.  So in that respect I have a DIR-505L which is connected to the SSID1 and retransmits a WPA/WPA2 shared key setup to the Sleep-Number bed since WPA/WPA2 is not available from the main WiFi setup.

I have a Tempest Weather station on my balcony which has a hub that was not able to receive the WPA/WPA2 shared key setup from the small bedroom, so it is now connected to a NEWIOTSSID coming from my Unifi AP AC Pro access point.  I also have a SRVSSID broadcast from that access point that I can connect to with my personal laptop in order to perform admin functions on my HomeLab while on my balcony.

I am also using the NEWIOTSSID to connect to a series of Tasmota switches mounted in my HomeLab Rack for the purpose of controlling power to several devices, since I may only have need of a couple at a time.

Looking at newer ways of accessing HomeLab

I was looking at a Breakout-Box implementation at https://github.com/DCKcode/breakout-box this weekend and it seems that I would like to put together such an item, including a WireGuard hub; both on RPis.  If I could figure out how to implement a Lets Encrypt cert on the Hub using a dynamic dns service IP, then this could be a way of connecting the family to self hosted services.  

In addition there is an automated Lets Encrypt cert gatherer made for my case at https://github.com/adferrand/dnsrobocert .  Perhaps for the WireGuard Hub I might make use of https://github.com/ngoduykhanh/wireguard-ui .

Setting up the RPiGateway into my HomeLab

So now that I have a better handle on how the wifi works at GSV, I want to provide myself a way into the HomeLab through a portal.  I have a Raspberry Pi, RPiGateway, that is connected to the GSV portal wifi network that I can get to from my iPad.  I have the luxury of being able to get out that Raspberry Pi from anywhere on campus using my iPad through the wifi network. I want to be able to get to applications, update my HomeLab services, change things within my HomeLab, and do other types of things remotely.  I can do this by setting up the RPiGateway for access.  I will need to control access to the RPiGateway to ensure that only known entities should be able to access the information.  That means that my personal laptop or any iPhone/iPad device that I own should be able to get into the network.  Access control can be done via encryption and known certificates.  So the following items may need to be considered:

• Use certificates validated through Lets Encrypt and their process of 90 days
• Use synchronous certificates that are self certified including a local CA
• Provide both SSH and VPN access into the HomeLab through RPiGateway
• SSH should rely on the synchronous certificates
• VPN should rely on the Lets Encrypt certificates
• I have the following protocols that need to be addressed over each HomeLab service: RDP, HTTP/HTTPS, SSH
• May have to include some NGINX and Apache Guacamole interfaces to equipment
• Might need to decide if this is a service through the Proxmox server, or on the RPiGateway
• Security might need to be offloaded from RPiGateway
• What needs to be done to the HomeLab side of RPiGateway to decrease risk if RPiGateway is compromised

All of these elements need to be considered.  Right now I can start setting things up that are accessible via the SSH port on RPiGateway, even though this would mean some special Apps on the iPhone/iPad.

Update: it turns out that getting a Lets Encrypt cert is a pain if you don’t have internet access to your web server.  This is not my case, so I will have to rely on a local CA to do certificates.  I am also realizing that I may want to move RPiGateway further into the HomeLab in the future, so I need to adjust for that.

History - The Current Setup of the GSV HomeLab

Environment - I live in an apartment in a wifi only environment.  There are two SSIDs for the tenants which allow connection of most devices. One SSID is open but is MAC address enabled based on a list, the other is encrypted and requires authentication.  The two wifi services are keyed to the resident accounts so that communication can happen between them for a single resident; but not to other resident computers using the same network topology.  The biggest problem for me having a HomeLab is to be able to do things in this environment.  In addition, the construction of the apartments are done with a view to fire safety, not to the transmission of wifi signals.  I experience a drop of almost 30 dBm from the living room to the back bedroom in signal level.  But, my HomeLab equipment is in the back bedroom and the wifi end point is in the living room, hence my dilemma.

Desires - Within the HomeLab, I have things that I want to accomplish in the living room, as well as the back bedroom.  In addition, I need the opportunity to move around the apartment at will and still do things on my HomeLab vlans.  I have no need to leave the apartment on a regular basis since I am retired.  So having a VPN connecting into my HomeLab is not necessary, unless I go on a trip.  I have no control over the wifi environment, nor can I tunnel through walls to pull Ethernet cable.  So, my HomeLab setup has to be coordinated with the circumstances.

Setup - I have to have a way into my HomeLab which is connected to the wifi environment.  I do this through a GL-iNet SLATE router which acts as a wifi bridge to my rack of equipment.  Early on I had the SLATE on top of the rack of equipment connected into the wifi environment on the encrypted side and the LAN of the SLATE plugged into the WAN port of my Ubiquiti Edgerouter-12 (ER-12).  However, I discovered that the signal level was about -78 dBm with a noise floor of around -71 dBm.  So my signal was below the noise floor; not a good situation if you want to get maximum speed.  What I then decided to do was put the SLATE in the living room, with a connection through a couple of Powerline adapters to the back bedroom where my rack of equipment resided.  That worked very well since I was getting essentially the maximum level of wifi signal that I could get in the environment.  But, as I started getting my HomeLab organized, it was apparent that I needed to get some additional equipment into the living room.  

I decided to change out the single channel through the Powerline adapters into a trunk line between two managed switches.  I placed a Netgear GS108Tv2 switch in the living room with a port configured as a trunk line with all of my defined vlans (13 in all).  That was plugged into the living room Powerline adapter and in the back bedroom the Powerline adapter was plugged into a port on a Netgear M4100-26g switch port configured with the same trunk line.  The untagged vlan on each trunk port was a dummy that only exists between the two switches.  I then configured an untagged vlan port on the two switches that would be used to pass Ethernet signals in the direction of SLATE to ER-12.  I plugged the SLATE into the resultant unmanaged port on the GS108Tv2.  The other resultant unmanaged port on the M4100-26g was then plugged into the WAN port of my ER-12.  I placed a Raspberry Pi in the living room and plugged it into a different untagged port on the GS108Tv2 which led back to the M4100-26g via the trunk to other things I was doing.  This setup now allows me to bring other equipment into the living room and place them on different vlans as I see fit.

An additional need was presented when I discovered that my Sleep Number bed needed connection to the internet but would only work with a WPA/WPA2 enabled wifi setup.  Neither of the two wifi SSIDs allowed this capability. I had to add an additional router, a D-Link DIR-505L, setup as a wifi repeater connected into the open SSID on the WAN side and presenting a WPA/WPA2 wifi signal on the LAN side.  I was then able to connect my Sleep Number bed once I added it to the small bedroom near the bed itself.  Since the DIR-505L is a wall-wart it was easily concealed under a vanity.  The WPA/WPA2 signal was also used by my Tempest Weather Station hub for connection to the internet.  In addition, I have a Raspberry Pi which is connected into the WPA/WPA2 signal and also to one of the vlans in my HomeLab rack.

I had brought a couple of my wifi access ports with me to the retirement apartment.  I set one of them up with 4 SSIDs (two disabled for now), one of them for use with my personal laptop to connect within my HomeLab and the other connected into the vlan used for controlling my Tasmota switches.  The Tasmota switches are currently used to turn on/off equipment in the HomeLab rack remotely.

Thus the saga of my changes to my HomeLab as a result of the wifi environment ends.

Using an RTL-SDR usb dongle to get other weather stations signals

I was futzing around last night repurposing some Raspberry Pis (bought before they were $180 apiece due to chip shortage) in my network rack. I took one of the RPis and installed RTL_433, node red, and plugged an RTL-SDR dongle into it along with a small antenna. I am going to use node red to push data to an MQTT service for use with my weather data extraction setup. I mostly wanted to see if I could pick up an Accurite Temperature sensor (606TX) on my balcony at 433.925 MHz. What I wasn’t expecting was picking up a couple other temperature sensors, one Accurite and one Oregon Scientific models.

I live in a retirement community and for the most part the residents are not really that tech-savvy, or computer-savvy for that matter. So I wouldn’t expect them to have a lot of weather station gear. My 606TX only gives back temperature on the balcony, but I noticed that it was reading a temperature about 2 degrees C higher than the Oregon Scientific model. The Oregon Scientific model also gave back humidity. I am thinking of gathering data from each of the other models to extract trends. My balcony faces north and that might be the cause of the temperature difference in the summer time.

I haven’t played with the gain of the dongle or looked into other frequencies yet. I am pretty sure that my Tempest, also on my balcony, transmits on the 915 MHz band, but I have not been able to capture it’s data set yet with the dongle.

Alteration to the Network Input to/from the Rack

 I have been contemplating how to provide an interface into the rack that would allow me to provide an interface to the RPis and the Proxmox server on the rack.  So far I have had an interface to the Edgerouter-12 (ER-12) through a SLATE router.  However, I found out that the rule set that is being used here in the retirement community doesn’t allow me to access ports on the SLATE unless I am connected to their portal vlan.  I decided to start some experiments to find out what I can actually do with the network.  Be aware that all IP addresses within the retirement community are RFC1918 private addresses.

I had previously connected the SLATE router to the ER-12 through two AV2000 Ethernet over power interfaces, each with two Ethernet ports apiece.  So it was SLATE LAN -> AV2000 -> AV2000 -> ER-12 WAN.  The SLATE was connected into the resident vlan which does not allow conversation between hosts on the resident vlan.  However, I have been able to connect between a host on the resident vlan and a host on the portal vlan.  I decided to put an RPi connected to the portal vlan, but also connected to a vlan within the rack.  That way, I can connect my laptop, or iPhone/iPad to any port on the RPi.  To do this, I decided to implement a trunk line between the two AV2000s.

I resurrected one of the managed switches from the tubs that I brought to the community and configured a trunk port and an untagged port to a vlan (vlan32) specifically for traffic between the SLATE and the ER-12.  I configured the same vlan (vlan32) on my M4100-26G switch along with a trunk port that had the same setup as the trunk port on the managed switch from my tub stores.  By connecting these up I now have the following:  SLATE LAN (vlan32) -> managed switch -[T]-> AV2000 -> AV2000 -[T]-> M4100-26G -[vlan32]-> ER-12 WAN (vlan32).

Since I can get to the managed switch within the rack from the Admin vlan, I was able to set other vlans so that I can add an RPi Gateway (ServerNet vlan/portal vlan).  This allows me to setup a firewall with open ports to the ServerNet vlan from the portal vlan which is wifi facing.  Next up is the configuration of the RPi Gateway.

WWD #03 - Setting up data_gather for Tempest weather station

 I have been extracting information from my Tempest weather station (TWS) via the WeatherFlow Tempest REST api.  It involves signing up for a key from their website for your particular TWS, then sending a crafted URI with the key, and sometimes your weather station id.  There is a limitation on the number of samples you can get over time unless you sign up for a business account, but at least you can get to your own data.

To sign up for the key you log into your account at TempestWx.com.  Then select the Settings icon on the upper right hand side.  That brings up a Settings page.  Go to the bottom, there you will find a link, "Data Authorizations".  When you select that you will see a create Token button on the top right corner.  Selecting that button will create a token key that can be used with the Tempest api to get information that was sent to TempestWx.com from your TWS.

You use the token as follows:  assume that my token is 12345678-1234-1234-1234-123456789098.  You can get the station information, which comes back in a JSON format by using the URI, https://swd.weatherflow.com/swd/rest/stations?token=12345678-1234-1234-1234-123456789098.  This will send back a JSON message similar to the following (showing a station_id of 12345 from stations[0].station_id) - 

{

  "stations": [
    {
      "location_id": 12345,
      "station_id": 12345,
      "name": "My TWS Name",
      "public_name": "My TWS Name",
      "latitude": 99.99999,
      "longitude": -99.99999,
      "timezone": "America/New_York",
      "timezone_offset_minutes": -240,
      "station_meta": {
        "share_with_wf": true,
        "share_with_wu": true,
        "elevation": 78.30184936523438
      },
      "last_modified_epoch": 1656712447,
      "created_epoch": 1630250472,
      "devices": [ ...

To get a sample, probably once every 1.0 minutes max, you would use a URI, https://swd.weatherflow.com/swd/rest/observations/station/12345?token=12345678-1234-1234-1234-123456789098. This will send back a JSON message similar to the one above, with the latest observation from your TWS under obs[0].

I will be prototyping this using an http request node within the data_gather container written in node-red.  I am going to push this to the MQTT server as a /weather/TWS/observation topic.  I am however going to reduce the number of items sent.

WWD #02 - Initial Architecture for the Web Weather Display

I am currently trying to thrash out the initial architecture for the Web Weather Display.  I know that it will be split into multiple parts and use Microservice patterns as the starting point.  This will take some time as I explore what items to add into the mix.  I do know that I will need the following Microservices as a start:

• NGINX to separate out the web front end
• A web front end that can be displayed via NGINX
• A Service Registry Microservice with a pattern allowing switching between Microservices on the fly, across platforms and vlans
• A Database back end to record changes as they happen
• A Microservice to push data to the Database back end
• A Microservice for each of:
• OpenWeatherMap data extraction
• Tempest weather station data extraction
• NWS data extraction
• NOAA data extraction
• WeeWx data extraction
• A Microservice for extracting current weather in a format that can be easily displayed
• A Microservice for merging current weather information
• A Microservice for building graphs of weather over time
• A Microservice for interacting with other sensors
• WeeWx (optional) to extract data from sensors

Each of these Microservices can be done with stubs as the overall fixed return.

So I have the following to start with:

• Defined Microservices
• service_def
• service_reg
• data_logger
• data_assemble
• data_gather
• web_assembly_proxy
• web_presentation
• MQTT
• Services that define the front end
• NGINX
• wordpress
• phpmyadmin
• mysql (mariadb)
• Services that provide temporal DB and graphs
• grafana
• influxdb
• Services for prototyping
• node-red-1
• node-red-2
• node-red-3
• Service for Docker control
• portainer

WWD #01 - Sources for Web Weather Display

 I was trying to pick through my mind and my notes to come up with a list of sources for the Web Weather Display.  I wanted to concentrate on free sources; this effectively eliminates any access to Weather Underground.  This is the list that I came up with:

• OpenWeatherMap - using a key from OpenWeatherMap you can download current predicted values for your given lat and long (https://openweathermap.org/api/one-call-api)
• NWS - the National Weather Service has a web services connection that allows you to gather information on current conditions in your area (https://www.weather.gov/documentation/services-web-api) as well as warnings and alerts for the area
• NOAA - NOAA has a web services connection that can gather satellite images and loop movies for the WWD (https://www.ncdc.noaa.gov/cdo-web/webservices/v2, https://www.ncdc.noaa.gov/cdo-web/token) along with other weather related information
• DarkSky - DarkSky is another source similar to OpenWeatherMap that provides world wide data; however it will be eclipsed and merged into an Apple product in 2023, so it doesn’t appear to be a good place to get information at this time
• Accuweather - Accuweather is another source of weather information; this one allows a limited set of 50 hits/day for free (https://developer.accuweather.com/apis)
• WeeWx - WeeWx is a general purpose posting and data gathering platform.  It is extensible with a number of different plug-ins and thus could be merged into the WWD as a data source/sink via MQTT interactions without transferring information to either CWOP or Weather Underground.  It does provide a good general purpose interface to most existing weather stations out there as well as RTL-SDR interfaces and other sensors (http://www.weewx.com/).
• Air Quality API - TBD
• Hurricane Models API - TBD
• Weather Models API - TBD

Project #27 - Developing a multi-source Web Weather Display using Microservices and DevOps

 I have been wanting to get into DevOps for a while.  Now that I am retired I should be able to find the time to immerse myself in the overall designs and setup for such a system.  I have also been studying Microservices and how that impacts the pipeline for a DevOps CI/CD approach.  Since I have a number of computer resources available to me, both RPi and Proxmox servers, I should be able to stand up a complete system fairly easily.  I also need to make it easy to modify as needed.  I am just a one person shop using a HomeLab for development, so much of this will not make sense to most people.

So for the general idea for the Web Weather Display:

• Most programming to be done with Node-Red, but allow for JavaScript and Python
• The Weather Display will be a Kiosk style web display
• Since I don’t have a Weather Station hooked into CWOP or Weather Underground, I need different sources of information
• Should be able to be updated with a display every 5 minutes
• Should show a forecast for at least 5 days
• Should allow for a display of radar data
• Should allow for a display of satellite data
• Should allow merging of micro-weather sensors to show alteration from the normal source of updates
• I want it to be able to display different models from different sources
• I want it to be able to display other environmental information such as earthquakes, hurricanes, tornado damage from different parts of the world
• I would like it to have a map display capability 
• I would like it to display data over a range of time from a backend storage medium
• I would like it to be self contained in a single RPi, or pulled to an RPi display from a remote source

For the general idea for Microservices:

• I would like to use some Microservices patterns for the development; those are TBD at this point
• I want to make the Weather Display components split into Microservices that make sense
• The Microservices should be able to be moved to a different host without causing disruption, including to different operating systems and processors
• Individual Microservices should be able to be changed without having to shut the entire set of Microservices down

For the general idea for DevOps:

• I would like to do the development on a separate Development vlan which works in conjunction with a Production vlan for the Weather Display
• The introduction of a newly tested and developed Microservice should be scripted so that I do not have to worry about forcing a change
• The movement of portions of the Microservice architecture should be fluid within the CI/CD pipeline

Some constraints for the Weather Display development:

• I should not have to buy anything additional in order to be able to complete the first version of this project
• I should be able to add additional sensors and merge their output into the displays without issue
• The first iteration of this project should use MQTT as the adhesion point, but allow for use of web services later on - so the mechanism should be hidden

HomeLabUpdate #03 - The more things stay the same, the more they change

 So here I am, a month longer into the process of updating my HomeLab, changing what I am going to do with the Proxmox server.  I have been able to add an additional 1TB SSD to the Proxmox server chassis and am now focusing in on what kinds of programs that I will be doing.  I have discovered that ZFS does indeed take a lot of memory, something that I don’t have a lot of.  I have also discovered that decent backups alleviate the need to have redundant drives that don’t contain critical information.  I have enough backups of critical information on different media that in my estimation preclude the need for redundancy like Raid1.  My critical information now resides on laptops that I backup on a regular basis and have nothing to do with the HomeLab structure in the rack.  The new 1TB SSD will become an LVM-thin component to complement the LVM-thin on the original 1TB SSD that the system resides on.  I can watch the statistics on the drives to know if they are ready to be replaced or not.  The UPS will take the shock of brownouts, spikes, and blackouts that will arrive this summer.  I have also discovered that I will not be able to force the on-board GPU to  a specific VM on the Proxmox server, so that is now off the table.

What this means is that I will be extending the current Proxmox configuration and will probably not be updating the version to 7.2.

I have decided to center on one program that I can focus my attention on and that allows me to achieve many of the goals that I have for my HomeLab.  That program will be an implementation of a Weather data gatherer focused on Docker and Node-Red.  I have already determined that I can spin up a Node-Red instance in Docker and duplicate it multiple times on the same host.  That will be the starting point for both implementing the Weather Data Gatherer and playing around with DevOps.

I recently came across some information on Microservices which seems to fit in nicely with the idea of DevOps in that you break an application down into several different parts, all communicating with each other.  It allows for spinning up new versions of the Microservice without having to stop the rest of the microservices, as well as allowing the full CI/CD chain to be implemented in development.  This should give me a software means of learning new development techniques.

HomeLabUpdate #02 - What am I going to do with the Proxmox server?

After some starts and stops with the Proxmox server, I am determined to change it to allow some additional capabilities.  I have had heat problems that were solved when I put a second fan in the case and connected it to the power supply.  The problem is I only have 360 watts coming from the power supply and I have to work within that limit.  I think I have come to the end of what I can do with a 1TB SSD drive and two 4TB drives.  I may be able to add an additional 1TB SSD drive, but that remains to be seen.  I want to maintain the motherboard and the drives at a temperature value that will not cause early failure.  I am thinking of the following:

• Add an additional 1 TB SSD drive and use ZFS for the file system for redundancy
• Move the files off of one of the 4TB drives and use ZFS for the combination 2@4TB drives for redundancy
• Setup the 4TB ZFS redundant drive to allow spill over of the LVM-thin components as well as use for general storage from the NFS server
• Update Proxmox to version 7
• Update Proxmox to use OVS as the bridging mechanism (might require addition of ifupdown2)
• Install Mac OS and Windows 10 in two different VMs
• Add an additional display card to allow iGPU passthrough for the Windows 10 VM / Ubuntu VM
• Start integrating the outside RPis with the Proxmox server

This is the short list as it stands now.  First off I want to keep things below about 60 degrees C within the cabinet and work within the 360 watt budget.  Currently, the temperature of components within the Proxmox cabinet range between 35 to 45 degrees C.

Changed out the WiFi hookup in the Apartment

I did some measurements of the 802.11ac signal levels in my apartment.  When I am up close to the Cisco endpoint, I am measuring around -41 dBm, which is a pretty high level signal (-30 dBm being the theoretical maximum).  But when I measured the level in the back bedroom it came up as -78 dBm.  A level of -70 dBm is about minimum for not dropping packets.  So I am below that and that is why I have been having issues with the 802.11ac not having enough speed at 11 o'clock at night when everyone is streaming their stuff.  The reason for the drop in signal level is due to the construction of the apartment.  The apartment is basically a firebox, made to withstand fire from the outside, including the positive pressure in the hallways to keep fires at bay.  The walls inside the apartment have steel studs and the plasterboard is almost the consistency of concrete.  Both of these material types are bound to have an effect on RF transmission from the endpoint.

I decided to move the GL-iNet SLATE to the living room (around -44 dBm).  I am now connecting the SLATE to the rack by the use of some TP-Link AV2000 power adapters.  In that way I should be able to have an increase in speed of the network connection.  I discovered that the Signal and Noise levels, from the SLATE itself, are Signal = -35 dBm and Noise = -70 dBm.  That indicates that my -78 dBm measured in the back part of the apartment was way below the noise level, which means my packets were probably suffering from re-transmission.  No wonder I was seeing a slowdown of the connections to the outside.

Net Interface #02 - Spin up an NGINX Server with accompanying web server

I am starting with spinning up an NGINX server on my Proxmox server box in an LXC container using Docker.  The NGINX server will interface from the gateway vlan to the web vlan.  This will require me to spin up a web vlan and provide it with a dns/dhcp service and connect the web vlan to the Proxmox server.  Fortunately I already have a vlan setup on the Edgerouter-12 that I can use for this.  I just have to provide the path for the web vlan to pass directly to the Proxmox server.

Net Interface #01 - Defining an Initial Set of Interfaces that Can be Accessed by My Devices

In the future, as part of my ability to keep up doing my HomeLab stuff, I will need to be able to get to some software/VM/container things from the GSV-Network side of things using my laptop/iPhone/iPad.  These items will consist of experiments that are ongoing, documentation software, calendar/project items, etc.  So as part of this development, these are the initial components that I will put up to interface to:

• An NGINX server to provide reverse proxy control to web based items in my network such as NextCloud, Redmine, Blog, Journal, NodeRed, HomeAssistant, etc.
• A web services port for interfacing to my experiments
• An IPSec w/GRE port for interfacing directly to individual vlans within the HomeLab through a special interface built from a RPi Zero W
  

Project #26 - Making a Gateway interface for the Retirement Community WiFi Network

In addition to interfacing the rack equipment to the Retirement Community WiFi Network (GSV-Network), I need to have a couple of interfaces that can be connected to by my personal laptop and iPhone/iPad.  I have those devices connected to the GSV-Network directly and need to determine if I can port redirect from the rack equipment to provide some services to myself.  In addition, I need to have the ability to access any of the vlans that I have defined by using my laptop anywhere in the community for updates and HomeLab things.  This of course will require a number of tests.

UPDATE (2022-06-08):  Project is currently paused in order to fixate on one and only one project at a time.

HomeLabUpdate #01 - Initial Rack Setup

 As I came into the Apartment, I had setup the rack to have (from top to bottom):

• 24 port patch panel
• Ubiquiti Edgerouter-12 Router
• Short shelf for KVM and regular switch/ER-X
  
• Netgear M4100-26g Switch
  
• Panel with 4 Raspberry Pi 4s (RPiInject, RPiCICD, RPiLDAP, RPiPlex)
  
• Panel with 4 Raspberry Pi 3Bs (RPiSecure, RPiPower3, RPiPower2, RPiPower1)
• Shelf for containing WDMyCloud drives
• Open area (2U) TBD on contents
  
• Proxmox Server (2U case)
• APC UPS (1U)

I had left 6 Ethernet cables connected, numbered #8-1 through #8-6.  When I got to the apartment, I added the GL-iNet SLATE router (in Repeater mode, without second SSID) as the input to the apartment's wifi system.  My printer is connected to the wifi system and I have connected the HA-IOT server to the back of my Acer Monitor with connections through the 4-way KVM switch.

MOVED INTO NEW APARTMENT AT RETIREMENT COMMUNITY

Turned off the Home Assistant and IOT Server

There were a number of changes I made to the network this week in preparation for the move to Greenspring:

• Took down the RPiPWS and turned off the Tempest weather station.  This is now ready to be packed up.
• I moved the NodeRed docker containers from RPiKodi to an LXC container on my Proxmox server.  I then took down RPiKodi and it is now ready to be packed up.
• I copied the HA-IOT docker containers from the HA-IOT server to an LXC container on my Proxmox server.
• This morning I turned off the HA-IOT server that has been running commanding my lights in the house.  The HA-IOT server and the switch that I was using to send back the data to my router are now ready to be packed up.
• I then spun down the two LXC containers on the Proxmox server since they are no longer needed.
• I removed the hardware IOT equipment from the living room and side room.  These are now ready to be packed up.

All of this seems like loosing an old friend that I have been playing with over these past few years.  But, in hindsight I did learn a lot and now I am ready to take on other challenges.  It will be a few weeks until I am packed and ready to move to Greenspring Village, but at least I will not have these items nagging me in the back of my head.

Turned off the PWS from streaming data to Weather Underground and CWOP

So, while I was out on vacation, my contractor did some work on the house in preparation for sale.  One of those things involved taking down my PWS from the chimney and painting it.  Since I am going to be moving into a retirement community at the end of the month, I decided to stop transmitting data to WU and CWOP in order for the data to remain "pure".

When I got home from vacation, I simply shut down the Raspberry Pi that had been running my weather station for almost 8 years now.  I don't think that I will remove the WU and CWOP IDs just yet.  That is for another time if I can't get the station back up in a good spot.  Right now, with 8 inches of snow on my deck, I will have to wait until some thawing before I will be able to find the Tempest station in order to turn it off.

Update: I have also signed out of Weather Underground and CWOP since I am no longer sending data.