WWD #02 - Initial Architecture for the Web Weather Display

I am currently trying to thrash out the initial architecture for the Web Weather Display.  I know that it will be split into multiple parts and use Microservice patterns as the starting point.  This will take some time as I explore what items to add into the mix.  I do know that I will need the following Microservices as a start:

• NGINX to separate out the web front end
• A web front end that can be displayed via NGINX
• A Service Registry Microservice with a pattern allowing switching between Microservices on the fly, across platforms and vlans
• A Database back end to record changes as they happen
• A Microservice to push data to the Database back end
• A Microservice for each of:
• OpenWeatherMap data extraction
• Tempest weather station data extraction
• NWS data extraction
• NOAA data extraction
• WeeWx data extraction
• A Microservice for extracting current weather in a format that can be easily displayed
• A Microservice for merging current weather information
• A Microservice for building graphs of weather over time
• A Microservice for interacting with other sensors
• WeeWx (optional) to extract data from sensors

Each of these Microservices can be done with stubs as the overall fixed return.

So I have the following to start with:

• Defined Microservices
• service_def
• service_reg
• data_logger
• data_assemble
• data_gather
• web_assembly_proxy
• web_presentation
• MQTT
• Services that define the front end
• NGINX
• wordpress
• phpmyadmin
• mysql (mariadb)
• Services that provide temporal DB and graphs
• grafana
• influxdb
• Services for prototyping
• node-red-1
• node-red-2
• node-red-3
• Service for Docker control
• portainer

WWD #01 - Sources for Web Weather Display

 I was trying to pick through my mind and my notes to come up with a list of sources for the Web Weather Display.  I wanted to concentrate on free sources; this effectively eliminates any access to Weather Underground.  This is the list that I came up with:

• OpenWeatherMap - using a key from OpenWeatherMap you can download current predicted values for your given lat and long (https://openweathermap.org/api/one-call-api)
• NWS - the National Weather Service has a web services connection that allows you to gather information on current conditions in your area (https://www.weather.gov/documentation/services-web-api) as well as warnings and alerts for the area
• NOAA - NOAA has a web services connection that can gather satellite images and loop movies for the WWD (https://www.ncdc.noaa.gov/cdo-web/webservices/v2, https://www.ncdc.noaa.gov/cdo-web/token) along with other weather related information
• DarkSky - DarkSky is another source similar to OpenWeatherMap that provides world wide data; however it will be eclipsed and merged into an Apple product in 2023, so it doesn’t appear to be a good place to get information at this time
• Accuweather - Accuweather is another source of weather information; this one allows a limited set of 50 hits/day for free (https://developer.accuweather.com/apis)
• WeeWx - WeeWx is a general purpose posting and data gathering platform.  It is extensible with a number of different plug-ins and thus could be merged into the WWD as a data source/sink via MQTT interactions without transferring information to either CWOP or Weather Underground.  It does provide a good general purpose interface to most existing weather stations out there as well as RTL-SDR interfaces and other sensors (http://www.weewx.com/).
• Air Quality API - TBD
• Hurricane Models API - TBD
• Weather Models API - TBD

Project #27 - Developing a multi-source Web Weather Display using Microservices and DevOps

 I have been wanting to get into DevOps for a while.  Now that I am retired I should be able to find the time to immerse myself in the overall designs and setup for such a system.  I have also been studying Microservices and how that impacts the pipeline for a DevOps CI/CD approach.  Since I have a number of computer resources available to me, both RPi and Proxmox servers, I should be able to stand up a complete system fairly easily.  I also need to make it easy to modify as needed.  I am just a one person shop using a HomeLab for development, so much of this will not make sense to most people.

So for the general idea for the Web Weather Display:

• Most programming to be done with Node-Red, but allow for JavaScript and Python
• The Weather Display will be a Kiosk style web display
• Since I don’t have a Weather Station hooked into CWOP or Weather Underground, I need different sources of information
• Should be able to be updated with a display every 5 minutes
• Should show a forecast for at least 5 days
• Should allow for a display of radar data
• Should allow for a display of satellite data
• Should allow merging of micro-weather sensors to show alteration from the normal source of updates
• I want it to be able to display different models from different sources
• I want it to be able to display other environmental information such as earthquakes, hurricanes, tornado damage from different parts of the world
• I would like it to have a map display capability 
• I would like it to display data over a range of time from a backend storage medium
• I would like it to be self contained in a single RPi, or pulled to an RPi display from a remote source

For the general idea for Microservices:

• I would like to use some Microservices patterns for the development; those are TBD at this point
• I want to make the Weather Display components split into Microservices that make sense
• The Microservices should be able to be moved to a different host without causing disruption, including to different operating systems and processors
• Individual Microservices should be able to be changed without having to shut the entire set of Microservices down

For the general idea for DevOps:

• I would like to do the development on a separate Development vlan which works in conjunction with a Production vlan for the Weather Display
• The introduction of a newly tested and developed Microservice should be scripted so that I do not have to worry about forcing a change
• The movement of portions of the Microservice architecture should be fluid within the CI/CD pipeline

Some constraints for the Weather Display development:

• I should not have to buy anything additional in order to be able to complete the first version of this project
• I should be able to add additional sensors and merge their output into the displays without issue
• The first iteration of this project should use MQTT as the adhesion point, but allow for use of web services later on - so the mechanism should be hidden

HomeLabUpdate #03 - The more things stay the same, the more they change

 So here I am, a month longer into the process of updating my HomeLab, changing what I am going to do with the Proxmox server.  I have been able to add an additional 1TB SSD to the Proxmox server chassis and am now focusing in on what kinds of programs that I will be doing.  I have discovered that ZFS does indeed take a lot of memory, something that I don’t have a lot of.  I have also discovered that decent backups alleviate the need to have redundant drives that don’t contain critical information.  I have enough backups of critical information on different media that in my estimation preclude the need for redundancy like Raid1.  My critical information now resides on laptops that I backup on a regular basis and have nothing to do with the HomeLab structure in the rack.  The new 1TB SSD will become an LVM-thin component to complement the LVM-thin on the original 1TB SSD that the system resides on.  I can watch the statistics on the drives to know if they are ready to be replaced or not.  The UPS will take the shock of brownouts, spikes, and blackouts that will arrive this summer.  I have also discovered that I will not be able to force the on-board GPU to  a specific VM on the Proxmox server, so that is now off the table.

What this means is that I will be extending the current Proxmox configuration and will probably not be updating the version to 7.2.

I have decided to center on one program that I can focus my attention on and that allows me to achieve many of the goals that I have for my HomeLab.  That program will be an implementation of a Weather data gatherer focused on Docker and Node-Red.  I have already determined that I can spin up a Node-Red instance in Docker and duplicate it multiple times on the same host.  That will be the starting point for both implementing the Weather Data Gatherer and playing around with DevOps.

I recently came across some information on Microservices which seems to fit in nicely with the idea of DevOps in that you break an application down into several different parts, all communicating with each other.  It allows for spinning up new versions of the Microservice without having to stop the rest of the microservices, as well as allowing the full CI/CD chain to be implemented in development.  This should give me a software means of learning new development techniques.