Turned off the PWS from streaming data to Weather Underground and CWOP

So, while I was out on vacation, my contractor did some work on the house in preparation for sale.  One of those things involved taking down my PWS from the chimney and painting it.  Since I am going to be moving into a retirement community at the end of the month, I decided to stop transmitting data to WU and CWOP in order for the data to remain "pure".

When I got home from vacation, I simply shut down the Raspberry Pi that had been running my weather station for almost 8 years now.  I don't think that I will remove the WU and CWOP IDs just yet.  That is for another time if I can't get the station back up in a good spot.  Right now, with 8 inches of snow on my deck, I will have to wait until some thawing before I will be able to find the Tempest station in order to turn it off.

Update: I have also signed out of Weather Underground and CWOP since I am no longer sending data.

Trying to Resurrect the WeatherPi

Now that I have most of the previous parts of the network seemingly working again (have found replacements for the two Netgear GS108Tv2s that crapped out), I can concentrate on some of the other items that seemed to have gone south.  The first up is the WeatherPi that is connected to my Personal Weather Station.  It doesn't seem to be pushing data out anymore.

I am also wondering if this would be a good time to replace the WeatherPi with a USB server and move the weewx program to the HA Server.  The HA Server is also giving me problems right now, but I can deal with that later.  First off, I am going to try to get the WeatherPi to at least push data to weather underground again.

I did manage to remove the WeatherPi from underneath the TV stand and discovered that the SD Card was pretty much done with (probably a result of the power outage).  I put a new SD Card into the RPi that used to be the WeatherPi and discovered that the RPi was okay.  The new designation was PiHoleLDAP; apparently this was the one that was going to be the one that I was going to use for the PiHole, Radius, and LDAP setup using Docker.  Which reminds me, there is no reason that I cannot run weewx under Docker, I just have to research it.

There seems to be a bit of a problem with weather underground.  Since IBM took over, the API has been removed; so things like predictions and the like are no longer available to gather for an html display.  Weewx has not been updated for Python 3, so should I go to a different outlet to publish the weather station data, or should I just abandon the weather station all together?  Part of me is saying the latter, because I still have to maintain the station, look at the stats, and generally update portions of this as my time allows.  Going into retirement, it would be a scheduled activity that I would do say once every two weeks.  Not sure if it is worth the trouble.

Update: it is looking like weewx under Docker is going to be a pain to complete.  There are a number of builds on the Docker site, but all of them are rather complicated.  I am going to have to delay getting this accomplished.

Change in the Weather Station Setup

I was thinking the other day, how much I would like to take away the dependence on one RPi for my weather transmissions to Weather Underground and CWOP.  I just recently purchased a subscription to VirtualHere.  VirtualHere gives the ability for one server (in this case an RPi) to serve a USB device (in this case the Weather Station Console output) to multiple clients.  The idea here is to distribute collection of information to other programs that can deal with it.  One program would be the WeeWX server running on an RPi that performs the interface to Weather Underground, CWOP, and others.  Another program would be to extract the readings and formulate web pages.  WeeWX does a good job of formulating web pages but I want to distribute the function to a different processor so that I can push the weather updates as quickly as I can.  In addition, I can do more things with the information independent of what I am streaming, including multiple weather displays of information.

The new setup of the RPi Cluster on the shelf will be used for this purpose.  Originally, I was intent on only doing networking experiments with the Cluster, but I have had a change of heart since I am nearing retirement.  I also want to be able to incorporate other sensors into the mix to produce a well rounded output to WU and CWOP.  The new computer room with all of these components will be able to give me what I want.  In addition, I can add some VMs to the mix for experiementation on weather prediction, etc.

Weather Station Project #27 - Took some samples through material

This morning I was out in the yard setting up an experiment.  I connected the Solar Radiation/UV Sensor setup to a POE enabled cable connected to a Ethernet over Power Line adapter.  The other Power Line adapter I plugged into my computer downstairs in the computer room.  This adapter was plugged into a Pers VLAN connector on one of my GS108Ts.  By doing this, I was able to connect to the sensor setup using my laptop on the porch through wi-fi.  I am going to leave the connection downstairs so that I will be able to use the Power Line adapter in other VLANS as necessary.  It was neat to get it working.  Here is a picture of the parts on my porch table.

The values of the data that I took was as follows:

results with no cover in bright sunlight

{ "tslLUX":"65536.00", "uvVIS":"1677.00", "uvIR":"13354.00", "uvUVRaw":"761.00", "uvUV":"7.61" }

reading from site was 12.0 for UV

Kingstowne says 7.4 UV, 835 watts/m^2

results with plastic cover in bright sunlight

{ "tslLUX":"65536.00", "uvVIS":"1594.00", "uvIR":"12876.00", "uvUVRaw":"716.00", "uvUV":"7.16" }

results with polycarbonate in bright sunlight

{ "tslLUX":"65536.00", "uvVIS":"1600.00", "uvIR":"13228.00", "uvUVRaw":"720.00", "uvUV":"7.20" }

Kingstowne says 6.9 UV, 796 watts/m^2

results with soda glass in bright sunlight

{ "tslLUX":"65536.00", "uvVIS":"1627.00", "uvIR":"12333.00", "uvUVRaw":"734.00", "uvUV":"7.34" }

Kingstowne says 7.2 UV, 833 watts/m^2

results in the shade with no cover

{ "tslLUX":"2152.00", "uvVIS":"312.00", "uvIR":"962.00", "uvUVRaw":"30.00", "uvUV":"0.30" }

{ "tslLUX":"2168.00", "uvVIS":"315.00", "uvIR":"976.00", "uvUVRaw":"31.00", "uvUV":"0.31" }

Kingstowne says 6.7 UV, 817 watts/m^2

This UV data shows a loss of about 5 and a half percent in the visible spectrum, 4 and a half percent considering the IR spectrum.  The overall percent difference between my local site and the site that I am using for calibration appears to be about 2.8 percent; my value is higher.  The distance between us is about 4.13 miles so I have been assuming that the values should be about the same.

According to this data, I should not have a problem if I used the lid to my sandwich container (plastic cover in the test).  I just have to compensate for the loss; there doesn't appear to be a loss associated with the UV sensitivity as far as I can tell.  So I should multiply my UV readings by 1.0628 to get the same as the Kingstowne values?  I should multiply my uvVIS values by 0.4979 to get the watts/m^2 equivalent to the values from the site where I am comparing my values.

Weather Station Project #26 - Some more thoughts

After looking through the raw data from my sensor, I discovered a number of points where the data was "pegged".  The only thing that I can think of is that the integration time on the sensor was set too high.  This is going to require me to redo the sketch that was loaded onto the Arduino.  Thinking more about this, I should also attempt to get the final packaging completed for the sensor setup.  This means a package that is waterproof and uses POE to power it so that I have but one cable from the POE source to the sensor package.  I already have the packaging and I have a few POE splitters on order that will work well with this setup.  Now I need to come up with the final specifics.

The list of parts (from here) with some modifications are:

• Raspberry Pi B+
• Arduino Uno
• Arduino Ethernet Shield
• Arduino Proto Board Shield
• TP-Link TL-POE10R PoE Splitter w/power connector cable
• Power splitter cable (1 female to 4 males - 5.5mm x 2.1mm)
• Power adapter plug (1 female 5.5mm x 2.1mm to ISO block)
• Power plug 5.5mm x 2.1mm to micro USB cable
• Ethernet socket adapter (for mounting on wall of sandwich box)
• Clear sandwich box (see-through for RPi camera)
• RTV to pot up the holes so that water can't get in
• Mounting platform and hardware to mount sandwich box on outside

The Raspberry Pi is used to sample from the Arduino.  I have proven that I can set up a web service on the Arduino, so accessing it from within my network is a no brainer.  I have also proven that I can write a python script which is able to sample both from the Arduino based sensor set and from a Weather Underground station with the sensors that I am trying to emulate.  Ultimately the extra sensor base will include:

• Lightning sensor 
• Moisture sensor (Transpiration) 
• UV monitoring sensor
• Solar radiation sensor

I have the parts for the Moisture sensor but I am missing the lightning sensor parts.  Not sure where to find that.

Update: I got in some POE splitters yesterday so I will probably be building up the package over this weekend.  Right now I am just going to use the plastic lid.  Will need to get a crystal glass lid so that it will pass UVA and UVB through.  I don't think that UVB will be able to pass through the lid at this point.

Weather Station Project #25 - Back to the Drawing Board

Well, it looks like there is a major problem with my assumptions.  Take a look at the following graph:

The red line is the sample from my UV sensor and the green line is the sample from the PWS that has the UV sensor.  As you can see, even though my sensor is recording a similar distribution, it is way out of whack.  This is more due to cloud cover and the fact that my sensor is not looking up.  Also, when I set the sensor up, it was pointed out a window through a screen so there are some artifacts to deal with.  The levels are way off here as well. Originally I was going to setup for a sample over an eight day period but I cut that short to be able to see some data.

What I am thinking here is that I need to put the sensors into a different enclosure, one with a glass window.  Oh yeah, the solar radiation results were even worse:

Anyway, I need to go back and rethink this thing. More later.

Weather Station Project #23 - Initial Code for the Solar/UV Sensors

I thought I would take the time to put together some code that will be used in the Solar Radiation and UV Sensors.  The objective here is to at least get the sensors tracking and providing data output. I was able to cobble together the sources from the TSL2561 and the SI1145 written by Adafruit along with the source code from the Web Server written by Arduino to get something running.  I decided at the last moment to not use a Raspberry Pi but to stick with the Arduino.  I have an Arduino Uno, an Ethernet Shield, some extensions and the Arduino breadboard with the two sensors on it (see the following picture).

The code is pretty simple and surprisingly the Serial interface doesn't get in the way (it lets me do the web service without needing to have a console display):

#include <Wire.h>
#include <SPI.h>
#include <Ethernet.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_TSL2561_U.h>
#include "Adafruit_SI1145.h"

/*  This sketch is a combination of two sketches from Adafruit
 *  sensorapi for the TSL2561 lux sensor and si1145test for 
 *  the SI1145 UV sensor. The sketch captures the state of the two
 *  sensors every minute, waiting for a json request from the ethernet
 *  port.  It then provides a json string back containing the raw and
 *  computed values of the two sensors.
 *  
 *  Please note that the sketch makes use of the Adafruit Sensor API and
 *  libraries (to make the development simpler).
 *  
 *  Connections (TSL2561)
 *  ===========
 *  Connect SCL to analog 5
 *  Connect SDA to analog 4
 *  Connect VDD to 3.3V DC
 *  Connect GROUND to common ground
 *
 *  I2C Address (TSL2561)
 *  ===========
 *  The address will be different depending on whether you leave
 *  the ADDR pin floating (addr 0x39), or tie it to ground or vcc. 
 *  The default addess is 0x39, which assumes the ADDR pin is floating
 *  (not connected to anything).  If you set the ADDR pin high
 *  or low, use TSL2561_ADDR_HIGH (0x49) or TSL2561_ADDR_LOW
 *  (0x29) respectively.
 *
 */

Adafruit_TSL2561_Unified tsl = Adafruit_TSL2561_Unified(TSL2561_ADDR_FLOAT, 12345);
Adafruit_SI1145 uv = Adafruit_SI1145();

// Some more globals to store intermediate info into until asked for by the web intfc
float tslLUX;
float uvVIS;
float uvIR;
float uvUVRaw;
float uvUV;

// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = {
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED
};
IPAddress ip(192, 168, 0, 42);

// Initialize the Ethernet server library
// with the IP address and port you want to use
// (port 80 is default for HTTP):
EthernetServer server(80);

/**************************************************************************
 *    Displays some basic information on this sensor from the unified
 *    sensor API sensor_t type (see Adafruit_Sensor for more information)
 **************************************************************************/
void displayTSL2561Details(void)
{
  sensor_t sensor;
  tsl.getSensor(&sensor);
  Serial.println("------------------------------------");
  Serial.print  ("Sensor:       "); Serial.println(sensor.name);
  Serial.print  ("Driver Ver:   "); Serial.println(sensor.version);
  Serial.print  ("Unique ID:    "); Serial.println(sensor.sensor_id);
  Serial.print  ("Max Value:    "); Serial.print(sensor.max_value); Serial.println(" lux");
  Serial.print  ("Min Value:    "); Serial.print(sensor.min_value); Serial.println(" lux");
  Serial.print  ("Resolution:   "); Serial.print(sensor.resolution); Serial.println(" lux");  
  Serial.println("------------------------------------");
  Serial.println("");
  delay(500);
}

/**************************************************************************
 *    Configures the gain and integration time for the TSL2561
 **************************************************************************/
void configureTSL2561(void)
{
  /* You can also manually set the gain or enable auto-gain support */
  // tsl.setGain(TSL2561_GAIN_1X);      /* No gain ... use in bright light to avoid sensor saturation */
  // tsl.setGain(TSL2561_GAIN_16X);     /* 16x gain ... use in low light to boost sensitivity */
  tsl.enableAutoRange(true);            /* Auto-gain ... switches automatically between 1x and 16x */
  
  /* Changing the integration time gives you better sensor resolution (402ms = 16-bit data) */
  // tsl.setIntegrationTime(TSL2561_INTEGRATIONTIME_13MS);      /* fast but low resolution */
  // tsl.setIntegrationTime(TSL2561_INTEGRATIONTIME_101MS);  /* medium resolution and speed   */
  tsl.setIntegrationTime(TSL2561_INTEGRATIONTIME_402MS);  /* 16-bit data but slowest conversions */

  /* Update these values depending on what you've set above! */  
  Serial.println("------------------------------------");
  Serial.print  ("Gain:         "); Serial.println("Auto");
  Serial.print  ("Timing:       "); Serial.println("402 ms");
  Serial.println("------------------------------------");
}

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  Serial.println("Light Sensor Test"); Serial.println("");
  
  /* Initialise the sensor */
  if(!tsl.begin())
  {
    /* There was a problem detecting the ADXL345 ... check your connections */
    Serial.print("Ooops, no TSL2561 detected ... Check your wiring or I2C ADDR!");
    while(1);
  }
  
  if (!uv.begin()) {
    Serial.println("Didn't find Si1145");
    while (1);
  }

  /* Display some basic information on this sensor */
  displayTSL2561Details();
  
  /* Setup the sensor gain and integration time */
  configureTSL2561();
  
   // start the Ethernet connection and the server:
  Ethernet.begin(mac, ip);
  server.begin();
  Serial.print("server is at ");
  Serial.println(Ethernet.localIP());

 /* We're ready to go! */
  Serial.println("");
}

void loop() {
  // put your main code here, to run repeatedly:
  /* Get a new sensor event for the lux sensor */ 
  sensors_event_t event;
  tsl.getEvent(&event);

  /* Display the results (light is measured in lux) */
  if (event.light)
  {
    tslLUX = event.light;
    // Serial.print(event.light); Serial.println(" lux");
  }
  else
  {
    /* If event.light = 0 lux the sensor is probably saturated
       and no reliable data could be generated! */
    tslLUX = 0;
    // Serial.println("Sensor overload");
  }

  /* Read the Visible, IR, UV sensor */ 
  // Serial.println("===================");
  uvVIS = uv.readVisible();
  // Serial.print("Vis: "); Serial.println(uvVIS);
  uvIR = uv.readIR();
  // Serial.print("IR: "); Serial.println(uvIR);
  
  // Uncomment if you have an IR LED attached to LED pin!
  //Serial.print("Prox: "); Serial.println(uv.readProx());

  float UVindex;
  uvUVRaw = uv.readUV();
  UVindex = uvUVRaw;
  // the index is multiplied by 100 so to get the
  // integer index, divide by 100!
  UVindex /= 100.0;
  uvUV = UVindex;  
  // Serial.print("UV: ");  Serial.println(UVindex);

  // At this point formulate the latest json message to send back on web query
  Serial.print("{ "); Serial.print("tslLUX"); Serial.print(" : "); Serial.print(tslLUX);
  Serial.print(", "); Serial.print("uvVIS"); Serial.print(" : "); Serial.print(uvVIS);
  Serial.print(", "); Serial.print("uvIR"); Serial.print(" : "); Serial.print(uvIR);
  Serial.print(", "); Serial.print("uvUVRaw"); Serial.print(" : "); Serial.print(uvUVRaw);
  Serial.print(", "); Serial.print("uvUV"); Serial.print(" : "); Serial.print(uvUV);
  Serial.println(" }");

  // listen for incoming clients
  EthernetClient client = server.available();
  if (client) {
    Serial.println("new client");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println("Connection: close");  // the connection will be closed after completion of the response
          client.println("Refresh: 5");  // refresh the page automatically every 5 sec
          client.println();
          client.println("<!DOCTYPE HTML>");
          client.println("<html>");

          client.print("{ "); client.print("tslLUX"); client.print(" : "); client.print(tslLUX);
          client.print(", "); client.print("uvVIS"); client.print(" : "); client.print(uvVIS);
          client.print(", "); client.print("uvIR"); client.print(" : "); client.print(uvIR);
          client.print(", "); client.print("uvUVRaw"); client.print(" : "); client.print(uvUVRaw);
          client.print(", "); client.print("uvUV"); client.print(" : "); client.print(uvUV);
          client.print(" }");
          client.println("<br />");
          
          client.println("</html>");
          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
    // close the connection:
    client.stop();
    Serial.println("client disconnected");
  }
  
  delay(1000);
}

I did check this out by executing a web call to 192.168.0.42 and was greeted with the json message.  I will have to modify this json message later because it doesn't have the quotes around the string values but at least it is working in the short term. I am going to simply put this up in my window at home and sample through a vertical pane of glass.  If the concept is correct, I should be able to get a fairly good set of samples that at least will prove the concept.  I might have some difficulty around sunup and sundown, but it should be close.  I will start with comparing against one site.

So, I need to position the sensors to point out the window so I can start getting some samples to check. That will be followed by some Python code to extract the json message, parse into an Excel comma delimited file, extract the json message from some local websites, parse them into the same Excel file and do this over several days. I am thinking that a five minute sample should be fine. Later I can pull the file in and do a statistical analysis on the data.

Weather Station Project #22 - Added a PoE Switch to the network to support some RPi projects

I went out over the weekend and purchased a 8 port Gigabit switch with 4 PoE ethernet ports on it.  My goal is to use this switch possibly in the attic in the future where I can run ethernet wires through out and use the power from the PoE switch to both power and connect the outputs to my home network.  I am already using it to power the Gertduino/Raspberry Pi Solar Radiation/UV sensor so that I will not have to use the injector; clutter catcher that it is.  When I do get to the point of connecting the Solar Radiation/UV sensor up, I am planning on setting up a pan/tilt camera mounted next to it for weather pictures.

Weather Station Project #21 - Updated the RPi Hardware

Interesting; as it turns out it's very easy to go from the old B model to the RPi 2 without difficulty. I wanted to update my software and the version of the raspberry pi. I did an update, followed by an upgrade, followed by a dist–upgrade and was then able to take that SD card and grab an image of it. I then placed the image onto a micro-SD card which would fit on the RPi 2. It booted right up and has been outputting weather data ever since.

Weather Station Project #20 - Found a Neat Resource for the Weather Display

About 2 months ago, a post was made to GitHub at https://github.com/kmkingsbury/raspberrypi-weather-station from Kevin Kingsbury.  This is a RPi based weather station, but has a twist in that there is an avid use of the Weather Underground API for his display.  I plan on tapping into that code for use with my RPi based Weather Display.  As you know, I plan on extracting all of my information for the Display from outside sources, particularly Weather Underground.  By using some of this code my life will be made simple.

BTW, Kevin has done a fine job with the backgrounds and displays for Austin, TX.  The last time I was in Austin was for a conference in 1996 where I gave a paper on Requirements creep and Software Design Patterns.  Memories.

Weather Station Project #19 - Thinking of Switching to a USB over IP Solution for the Console

I have been having some issues with weewx over the last couple of days.  It has been almost a year (July 2014) since I installed the Vantage Vue Weather Station and weewx on a Raspberry Pi.  Everything has been going real well except for the last couple of days.  It is looking like maybe the SD card is starting to give way because the RPi has been crashing? or weewx has been crashing?  I don't know at this stage.  I need to meter the RPi to get a crash dump if weewx stops working.  I have the RPi setup to start weewx when it reboots, therefore it is more likely that weewx has stopped and the RPi remains running.  I would only know that if the crash dump says so.

In the process of thinking about setting up something here, it occurred to me to split out the USB interface on the console to a different RPi.  If I run a virtualhere USB server on an RPi, for $39, I can access the console USB from multiple clients on my network.  In particular, I can access the USB information from my MacMini, and from additional RPis in the house.  So slipping an additional RPi into the mix gives:

This does not limit my ability to use RPi2 for other USB tasks either.  Since it would be physically located in the living room (and I might be able to run both the client and the server on the same RPi) it would be convenient to have other USB devices connected as well.

Update: This might not be as smooth as I would like.  I currently have the RPi5 separated in its own VLAN in order to throttle the output (so Verizon will be happy).  Gotta think that one over.

Weather Station Project #18 - Obtained a Weather Underground API Key

I have been in the process of coming up with a way of (1) calibrating the upcoming Solar Radiation and UV sensors for my Weather Station; and (2) determining how I can display the Weather Underground information for my Weather Station on a convenient interactive home display.  So the first thing that needed to be done was to obtain an API key to Weather Underground.  They have made this extremely easy to do, especially for a developer.

I will first use the key to build a data extraction tool to gather solar radiation and UV readings from stations near me so that I can come up with an equation for my sensors that will put me in the ballpark of the more costly ones.  This will allow me to modify the weewx routines on my RPi to accept the new sensor information and then send it on to Weather Underground.

I will then use the key to put together a touchscreen style display controlled via a RPi that will display the current and forecast weather for my Weather Station.  I intend to do this as discussed in weather-station-project-9-building a weather and general info display.  The key is necessary to pull the Weather Underground information.  Of course, it could be used to display the information from any Weather Underground station, but let me build it first.

The general sequencing will be as follows for the RPi Weather Display:

Weather Station Project #17 - Waiting for some parts to Start on the Solar Radiation/UV Sensor

I just purchased a TP-Link TL-POE150S PoE Injector and a TL-POE10R PoE Splitter over the weekend.  What intrigued me was that the TL-POE150S came with a 48 volt power supply to inject power through the Ethernet cable. The TL-POE10R had a selection switch to convert down to 12, 9, or 5 volts prior to splitting the output via a 5.5mm x 2.1mm connector (cable included).  I also picked up a clear sandwich box to put the equipment into.

I ordered a power splitter, power adapter plugs, and power to micro USB adapters from Amazon.

The parts list looks like this so far:

1. Raspberry Pi B+
2. Gertduino
3. Arduino Proto Board
4. Raspberry Pi No-IR camera
5. TP-Link TL-POE150S PoE Injector w/power supply
6. TP-Link TL-POE10R PoE Splitter w/power connector cable
7. Power splitter cable (1 female to 4 males - 5.5mm x 2.1mm)
8. Power adapter plug (1 female 5.5mm x 2.1mm to ISO block)
9. Power plug 5.5mm x 2.1mm to micro USB cable
10. Ethernet socket adapter (for mounting on wall of sandwich box)
11. Clear sandwich box (see-through for RPi camera)
12. RTV to pot up the holes so that water can't get in
13. Mounting platform and hardware to mount sandwich box on outside

For the moment, I am going to forgo the Pollen Sensor and concentrate on the other two sensors.  I will update this list and add a picture as the stuff comes together this weekend.

Update: I have the power splitter cables in, still waiting on the power plug to micro USB cable to come in.  They were supposed to be delivered on Monday and it is Thursday already.  I can't complain because the adapters were only about a buck apiece.  Somehow they are tied up in Indianapolis, IN under a "Processing Exception", whatever that means.  I may end up ordering the cables from a different source.  Ok, put in another order in case this one doesn't come through.  These cables are apparently tough to find.

I added the picture below to show the new setup with the two sensors on top, on an Arduino breadboard, Gertduino, on top of a Raspberry Pi B+.  There is also a No-IR RPi camera peeking out from between the Gertduino and the RPi.


I am also wondering if I should make the unit multi-tasking.  If so, I could setup my own web service API so that the unit would respond with values from the sensors, pictures from the camera, and potentially other information gleaned from Weather Underground for my weather station.  That way I could minimize the kiosk nature of my RPi Weather Display and use unit as the main source for the display.  That also simplifies the interface with my weewx software connected to the Davis Console.

Weather Station Project #16 - Equations for the Solar Radiation Sensor

I happened upon some Solar radiation unit conversion equations from the USDA at http://www.nrcs.usda.gov/wps/portal/nrcs/detailfull/national/home/?cid=stelprdb1043619. I am very grateful that someone decided that it was in the best interests of the country to provide this information.  Anyway, now I have some basis to get values from the TSL2561 luminosity sensor.  Other conversion factors are at http://www.egc.com/useful_info_lighting.php.  According to the lbl.gov site at http://bccp.lbl.gov/Academy/workshop08/08%20PDFs/Inv_Square_Law.pdf, when converting Lux to Watts/m^2:

Converting Lux to W/m2
• There is no simple conversion….it depends on
the wavelength or color of the light.
• However, for the SUN, there is an approximate
conversion of 0.0079 W/m2 per Lux.
• Example: We read 75,000 Lux on our light
sensor. We can convert that reading to W/m2.
75,000 x 0.0079 = 590 W/m2

But now I have a check on the USDA information.  The primary value which is output from the TSL2561 is in Lux.  The Merriam-Webster definition of Lux is "a unit of illumination equal to the direct illumination on a surface that is everywhere one meter from a uniform point source of one candle intensity or equal to one lumen per square meter."

So from the above equations I take the output of the TSL2561 sensor in Lux and convert to W/m2 by multiplying by 0.0079.

Weather Station Project #15 - Calibration Procedures for the DIY Solar Radiation Sensor

First of all, I am an engineer, not a statistician. I am trying to come up with a way of calibrating a DIY solar radiation sensor on the cheap. The original question that I asked can be referenced at the My Weather Station subreddit.

First some background: I am building a solar radiation sensor (measures watts/meter²⁾ from a TSL2561 luminosity sensor. The sensor will be placed under a pane of glass, coated with white translucent paint for light dispersion. The output of the sensor will be a number, representative of lux received at my house. I have 5 personal weather stations within a 15 mile radius who have expensive solar radiation sensors from which I can obtain data readings via Weather Underground. Given that cloud cover will alter the readings in each case, I am assuming that a statistical analysis is warranted. The data is also at the same time somewhat spatially distributed. I need to get the value from my DIY sensor into the ballpark of the other sensors.

My approach so far:

1. Gather the data from the 5 sites each day for an extended period of time. 
2. Use a Chi-square test to check if the data samples are likely from within the same distribution set (note that solar radiation results tend to follow an almost bell-shaped curve over the course of a day). 
3. Use a sample T-test to establish a mean and standard deviation representative of the set which varies throughout the day. 
4. Use the mean applied against the value determined by my DIY sensor to formulate an equation (assume a linear fit for right now). 
5. Check the altered output of my DIY sensor against values from the other 5 sites in a Chi-square test to determine if I am likely from the same distribution. 
6. Run some sort of correlation against the data set to further solidify the equation that I have found.

Question: is this approach viable? Are there any glaring issues with what I am trying to do? Is there anything else that might solidify the accuracy of the calibration?

To Be Continued ...

Weather Station Project #14 - Calibration Ideas for Solar Radiation Sensor

I have been mulling over the Solar Radiation Sensor and the UV Sensor and how those would be calibrated.  I have succeeded in getting the two main sensors connected to the Arduino.  The connections for the TSL2561 (for Radiation Sensor) is done according to instructions found at Adafruit - TSL2561 Luminosity Sensor. The connections for the SI1145 (for UV Sensor) is done according to the instructions found at Adafruit-si1145-breakout-board-uv-ir-visible-sensor.  I might be able to make do with the one SI1145 sensor, but the visible sensor is not calibrated.

Unfortunately, I will need to have each of these sensors under some sort of a window material.  The UV sensor will need to go under some type of window.  From the Cancer Research UK website, "UVA mostly causes skin ageing and research has now shown that it is also likely to cause skin cancer. UVB causes redness and sunburn and is a major risk factor for all types of skin cancer.  Most glass used for windows blocks UVB but not UVA. This means that although glass might reduce the risk of sunburn, it does not prevent long term damage from UVA."  In addition, I know that polycarbonate plastic will block most UV.  I am not sure of the window material that I should use for this.  It seems appropriate to use ordinary glass since that passes half of the UV radiation.  The luminosity sensor should be okay if placed under ordinary glass. I will of course need to hard mount the sensors on top of the Arduino - possibly substituting a Gertduino on top of a Raspberry Pi to transmit the information.

When I say calibrated, I mean outputting values that are statistically in-the-ballpark of sensors costing many times more.  What I will need to do is to correlate the output of these sensors over a period of time against local area sensors costing many times more.  The idea is that whatever values come out of the sensors can be modified to be statistically within the ballpark of the highly calibrated devices.  I would assume that I would need to form some equations so that the output of my sensors match that of others within the area.  The thing about my sensors is that they will follow the same curves as other sensors, taking out for cloud cover, etc.  and should be able to be mathematically stretched to match.  This will form the basis for my calibration and over a period of time should prove out to be accurate.

Weather Station Project #13 - Thinking of a Different Way to Send Snow Related Weather Data

I am sitting in the Detroit airport waiting for my flight and it occurred to me that I might be able to complement my weather station with some additional data. What I would do is get the additional data, such as snow accumulations and do a surface extrapolation to find approximately what it should be near me. Then, I could use some laser scanning techniques to extract a free fall snow drift. I would need to shelter the area where I would determine the size from wind (so that it doesn't drift from other places, giving a false reading). By using a laser scanning technique, I could measure the height of the snow pile and thus the amount of snowfall. Before I send the information out in my normal weather stream, I could use the extrapolated point to determine if my findings are within the "ballpark" of a correct amount. This would give me some confidence in my findings. Again, this could be transmitted via a wifi link to my network to be combined with other weather information.

I should think about how I am going to interject the other three sensors with this type of thing.

Maybe I should think about opening up a blog on the Weather Underground to talk about DIY and Hobbyist Built Weather Stations.

Weather Station Project #12 - Weather Station Still Holding Up

Maybe I'm paranoid but I do check on the weather station every once in a while. This involves looking at apps on my iPad to make sure that everything is working and the station is updating. It just seems that I have missed something because the data stream is running so well.

I still have to get around to making the sensors work and integrating it into the Weather Underground data stream. Now that it is winter and around Christmas time, the development will have to wait until I am able to have time. Right now I have a ton of things going on at work and I have family responsibilities.

I do wish everyone who comes across this little blog a fond Merry Christmas!

Weather Station Project #11 - Working on Code for Other Sensors

I had some time this weekend to work on the other sensors that I will be using in the weather station.  I decided to go ahead and test out the TSL2561, the SI1145, and the Sharp  Dust Sensor.  The Arduino breadboard that I am using for the other sensors in the test suite is the Parallax Board of Education.  This breadboard plugs right into the Arduino Uno (actually on top of an Ethernet shield and then into the Uno) and provides breakout of the analog and digital lines for prototyping as well as allowing other shields to be added to the mix.  I wanted to get the software working first followed by putting the sensors on a Arduino prototyping shield.  My thought was to use the Arduino to do the sensor interface and push a serial output down the USB line to a Raspberry Pi.  Alternatively, I have a couple of Wireless Transciever modules to do the same thing.  Haven't thought that far ahead yet.  The whole breadboarded setup is shown in the image below:




The visual sensors, the TSL2561 and the SI1145, are both cantilevered over the edge of the breadboard so that I can get to the pins below for the jumper wires.




The Dust Sensor in this photo is simply sitting on top of the Board of Education since there are no electrical conductive parts on the bottom.  I simply plugged the wires into their respective locations according to prototype setups that I found online.



The wiring setup for the Sharp Dust Sensor is according to what is recorded at standalone-sharp-dust-sensor.  The only difference is that I used A3 for the analog instead of A6 which is not available on the UNO.  The connections for the SI1145 is according to the instructions found at Adafruit-si1145-breakout-board-uv-ir-visible-sensor.  The connections for the TSL2561 is according to instructions found at Adafruit - TSL2561 Luminosity Sensor. Since the I2C addresses are different between the two visual sensors, I connected the SDA and SCL connections in parallel.  The two visual sensors make use of the Adafruit sensor library.

I tested each of the sensors independently from one another using test programs that were provided.  Now I am in the process of combining the test program pieces from each of the sensors into the same sketch.  After that, I will run a test to gather data over the period of a couple of days to try and compare to known values from other weather stations.  Once that is accomplished, I will endeavor to solder the sensors into the Arduino shield and place the whole lot into the specified fin enclosure.

-- LW

Weather Station Project #10 - Starting Extra Sensors

Ok, now that the Vantage Vue is putting out good data, the next portion of the project is to set up the next three sensors.  The three sensors will be Solar Radiation, UV Index, and Pollen Count.  I'm thinking the following:

(1) Processor Setup

• Use Arduino Mega to interface to the sensors (more IO lines, a little more capable system than the Uno)
• Use Prototyping board on top of Mega for interface to the sensors

(2) Solar Radiation

• Solar Radiation sensor based on use of TSL2561 Lux Sensor from Adafruit (TSL2561)
• Solar Radiation sensor will need to be put behind a white painted glass surface
• Solar Radiation sensor uses I2C bus, read by Mega

(3) UV Index

• UV Index sensor based on use of SI1145 UV Sensor from Adafruit (SI1145)
• UV Index sensor will need to be put behind a clear glass pane, plastic pane will inhibit the UV from passing to the sensor
• UV Index sensor uses I2C bus, read by Mega

(4) Pollen Count

• Pollen Count sensor based on use of SCM GP2Y1010AU0 bought from Happy Store (B00GET4KR0)
• Pollen Count sensor actually counts density of particles greater than 2.5 um
• Pollen Count sensor puts out a voltage which will need to be read by Mega

(5) Connection to Weather Station RPi

• Connection to Weather Station RPi  done through an Addicore nRF24L01 2.4GHz wireless transceiver (B00E594ZX0), one on the Mega and one on the Weather Station RPi
• Will require python script on Weather Station RPi to read values from Addicore transceiver
• Will require code on Mega to transmit all three values at a regular interval

(6) Housing for Sensors

• Housing for Sensors should be based off of Solar Radiation shield (Radiation-Shielding-7714)
• Power comes up through the bottom of the shield
• Need two holes cut into the top; one for white painted glass for solar radiation and one for glass for UV index