Weather Station Project #15 - Calibration Procedures for the DIY Solar Radiation Sensor

First of all, I am an engineer, not a statistician. I am trying to come up with a way of calibrating a DIY solar radiation sensor on the cheap. The original question that I asked can be referenced at the My Weather Station subreddit.

First some background: I am building a solar radiation sensor (measures watts/meter²⁾ from a TSL2561 luminosity sensor. The sensor will be placed under a pane of glass, coated with white translucent paint for light dispersion. The output of the sensor will be a number, representative of lux received at my house. I have 5 personal weather stations within a 15 mile radius who have expensive solar radiation sensors from which I can obtain data readings via Weather Underground. Given that cloud cover will alter the readings in each case, I am assuming that a statistical analysis is warranted. The data is also at the same time somewhat spatially distributed. I need to get the value from my DIY sensor into the ballpark of the other sensors.

My approach so far:

1. Gather the data from the 5 sites each day for an extended period of time. 
2. Use a Chi-square test to check if the data samples are likely from within the same distribution set (note that solar radiation results tend to follow an almost bell-shaped curve over the course of a day). 
3. Use a sample T-test to establish a mean and standard deviation representative of the set which varies throughout the day. 
4. Use the mean applied against the value determined by my DIY sensor to formulate an equation (assume a linear fit for right now). 
5. Check the altered output of my DIY sensor against values from the other 5 sites in a Chi-square test to determine if I am likely from the same distribution. 
6. Run some sort of correlation against the data set to further solidify the equation that I have found.

Question: is this approach viable? Are there any glaring issues with what I am trying to do? Is there anything else that might solidify the accuracy of the calibration?

To Be Continued ...

Weather Station Project #14 - Calibration Ideas for Solar Radiation Sensor

I have been mulling over the Solar Radiation Sensor and the UV Sensor and how those would be calibrated.  I have succeeded in getting the two main sensors connected to the Arduino.  The connections for the TSL2561 (for Radiation Sensor) is done according to instructions found at Adafruit - TSL2561 Luminosity Sensor. The connections for the SI1145 (for UV Sensor) is done according to the instructions found at Adafruit-si1145-breakout-board-uv-ir-visible-sensor.  I might be able to make do with the one SI1145 sensor, but the visible sensor is not calibrated.

Unfortunately, I will need to have each of these sensors under some sort of a window material.  The UV sensor will need to go under some type of window.  From the Cancer Research UK website, "UVA mostly causes skin ageing and research has now shown that it is also likely to cause skin cancer. UVB causes redness and sunburn and is a major risk factor for all types of skin cancer.  Most glass used for windows blocks UVB but not UVA. This means that although glass might reduce the risk of sunburn, it does not prevent long term damage from UVA."  In addition, I know that polycarbonate plastic will block most UV.  I am not sure of the window material that I should use for this.  It seems appropriate to use ordinary glass since that passes half of the UV radiation.  The luminosity sensor should be okay if placed under ordinary glass. I will of course need to hard mount the sensors on top of the Arduino - possibly substituting a Gertduino on top of a Raspberry Pi to transmit the information.

When I say calibrated, I mean outputting values that are statistically in-the-ballpark of sensors costing many times more.  What I will need to do is to correlate the output of these sensors over a period of time against local area sensors costing many times more.  The idea is that whatever values come out of the sensors can be modified to be statistically within the ballpark of the highly calibrated devices.  I would assume that I would need to form some equations so that the output of my sensors match that of others within the area.  The thing about my sensors is that they will follow the same curves as other sensors, taking out for cloud cover, etc.  and should be able to be mathematically stretched to match.  This will form the basis for my calibration and over a period of time should prove out to be accurate.

Still Having Issues with Netgear G54/N150 Micro Adapters

I hate to say it but I might have to move on with this development.  I have not been able to get a functioning 8188cus driver to run in the RPi despite trying just about every known driver setup program that is out there.  I don't know what is happening here.  It seems like the newest versions of the Raspbian OS are not able to take the 8188cus driver and allow it to run.  I keep having the rtl8192cu come up in its place and that one does not allow the iw utilities to run with it.

Out of all of the 802.11n based adapters, the Netgear G54/N150 Micro was the one that worked out of the box without any changes.  But it seems that the rtl8192cu driver does not allow the adapter to perform all of it's intended functions.

Loaded OpenWRT onto an SD card for test in Reverse Router Mode

I just followed the instructions here to get an SD card image for OpenWRT.  I am going to configure it much the same as what I have done with the dual wifi adapter setups.  Then I plan to finagle some way of pushing a DMZ to the ethernet port.  More later.

Configuration Files for Specialized Router for Work

I thought that I would record the configuration files that I used for the Specialized Router here, so that I would have something to reference in the future.  The setup here assumes that I have a common SSID that I connect to both at home and at work.  At home, I have a separate SSID that I have set up on my primary router and on the WAP that I have downstairs.  The channel number is not the same in both places, but the RPi software looks for the SSID and does the connection.  At work, I use a DIR-505L router which is set to wifi hotspot mode with the same SSID but is connected to the guest wifi SSID at work.  Assuming that hostapd and isc-dhcp-server are installed, the configuration files are as follows:

(1) /etc/wpa_supplicant/wpa_supplicant.conf

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="CairParavelClone"
        psk="p19dsndgyxhrh$AM"
        proto=WPA
    key_mgmt=WPA-PSK
        pairwise=TKIP
    auth_alg=OPEN
}

(2) /etc/default/isc-dhcp-server

# Path to dhcpd's config file (default: /etc/dhcp/dhcpd.conf).
DHCPD_CONF=/etc/dhcp/dhcpd_test.conf

# Path to dhcpd's PID file (default: /var/run/dhcpd.pid).
DHCPD_PID=/var/run/dhcpd_test.pid

# Additional options to start dhcpd with.
#    Don't use options -cf or -pf here; use DHCPD_CONF/ DHCPD_PID instead
#OPTIONS=""

# On what interfaces should the DHCP server (dhcpd) serve DHCP requests?
#    Separate multiple interfaces with spaces, e.g. "eth0 eth1".
INTERFACES="wlan1"

(3) /etc/network/interfaces

auto lo

iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
iface wlan0 inet dhcp
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
#iface default inet dhcp

allow-hotplug wlan1
#auto wlan1
iface wlan1 inet static
   address 192.168.42.1
   netmask 255.255.255.0
   network 192.168.42.0

up iptables-restore < /etc/iptables.ipv4.nat

(4) /etc/hostapd/hostapd.conf

interface=wlan1
ssid=Pi_AP
hw_mode=g
channel=9
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=Raspberry
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

(5) /etc/dhcp/dhcpd_test.conf

ddns-update-style none;

default-lease-time 600;
max-lease-time 7200;

authoritative;

subnet 192.168.42.0 netmask 255.255.255.0 {
range 192.168.42.100 192.168.42.120;
option broadcast-address 192.168.42.255;
option routers 192.168.42.1;
default-lease-time 600;
max-lease-time 7200;
option domain-name "local";
option domain-name-servers 8.8.8.8, 8.8.4.4;
}

(6) /etc/iptables.ipv4.nat

# Generated by iptables-save v1.4.14 on Sat Jan 24 16:37:01 2015
*nat
:PREROUTING ACCEPT [217:54744]
:INPUT ACCEPT [4:751]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o wlan0 -j MASQUERADE
COMMIT
# Completed on Sat Jan 24 16:37:01 2015
# Generated by iptables-save v1.4.14 on Sat Jan 24 16:37:01 2015
*filter
:INPUT ACCEPT [256:19656]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [136:13904]
-A FORWARD -i wlan0 -o wlan1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i wlan1 -o wlan0 -j ACCEPT
COMMIT
# Completed on Sat Jan 24 16:37:01 2015

That is about it for right now.  Updates to follow when I get the Netgear G54/N150 nano adapters to be more stable.

Setting Up a Specialized Wireless Hotspot/Bridge Combo for Work

Givens: So, where I work at has a number of separate networks.  There is a corporate network which has a number of security setups which need to be kept pristine.  There is also a guest network which uses a capture port wireless access.  I have a copy of Kali Linux on my work laptop which runs in a VM.  I want to keep this VM separate from the corporate network but still be able to use it in my job, which involves doing some checks against lab equipment.  I also like to bring my RPis into the office to work on during lunch.

Project:  What I want to do is use a Raspberry Pi to be the intermediary with two internal network connections at work with the following characteristics:

1. EXT - provide connection to the guest wireless through a wifi adapter on the RPi; have it automatically log into the guest wireless capture page and obtain an IP; this should provide a semi-permanent connection for both internal networks.
2. NET1 - provide a bridge to the Kali Linux VM through a USB to Ethernet adapter connected to the Ethernet port on the RPi and then to a USB port on the work laptop; have it provide the EXT IP directly to the Kali Linux VM.
3. NET2 - provide a NAT firewall (opt. packet inspection) between the EXT and another wifi adapter which provides a DHCP server/Wireless Access Port with an SSID which is common to the home network.  I might split this up into 3 separate SSIDs.
4. TUN1 - provide an opt. ssh tunnel through the EXT from NET2 to the home network for vpn connection giving an external WAP connection to my home network.

I already have the makings of this project by having a wifi hotspot configuration.  This configuration uses a D-Link DIR-505L router connected to the guest network.  It however, occurred to me that maybe I should do this with one single RPi and just keep the RPi at work.  I could power this up via a USB cable plugged into my laptop dock (A to micro USB cable).

I think I might dub this one Reverse Router, because I can think of a couple of other things that I might do with it under the area of penetration testing.

Update (2015-01-25): I had problems getting the DHCP server to work correctly and consistently with the TP-Link adapters that I was using, they kept dropping out.  I finally got a sufficiently consistent connection once I switched over to the Netgear adapters. I may end up having to buy additional adapters for other tests.

Update (2015-01-27):  I changed the channel addresses so that the two Netgear wifi adapters would not be tromping on each other; for stability reasons.  I found a website that tells me how to set up multiple SSIDs on the same interface using hostapd (Multiple_SSIDs_with_hostapd). Also finally got the basic wireless access point to work with a wireless connection to my DIR-505L router.  So now I have a wireless coming in to a NAT firewall to a WAP with DHCP going out.  Right now I am copying the microSD to put onto a different microSD so that I can implement multiple SSIDs.  What I want to do there is to have three SSIDs: one accessing the house (house SSID and cert), one for guest access, and one for secure test access (TBD).  Each of these SSIDs will be set up to appear if certain files have been touched.  I have not fully determined if the Netgear G54/N150 can handle multiple SSIDs or not.  I know that hostapd can.

Update (2015-01-28):  Well I discovered that there are some issues with changing the mac address of a wifi adapter on the fly.  I did the first part of the setup from the Multiple SSIDs with hostapd article and borked the connection.  What I will probably find when I look into it is that the wifi mac address was stored somewhere and it's keeping the dhcp server from running on the WAP.

Update (2015-01-29): Nope, the Netgear G54/N150 is not capable of changing the wifi MAC address; at least from my experiments that is the case.  I wish that were not true, maybe I will try the previous wifi adapters (based on the rtl8188cu chipset).

Update (2015-01-30):  After reading a couple of things online, maybe what I have here is a driver problem for the Netgear G54/N150 nano adapters.  What seems to give it away is the problems that I have when I try to use the "iw list" command, which always comes back with an error.  In addition, the setup does not appear stable between reboots; sometimes the dhcp server comes up and sometimes it doesn't.

Weather Station Project #13 - Thinking of a Different Way to Send Snow Related Weather Data

I am sitting in the Detroit airport waiting for my flight and it occurred to me that I might be able to complement my weather station with some additional data. What I would do is get the additional data, such as snow accumulations and do a surface extrapolation to find approximately what it should be near me. Then, I could use some laser scanning techniques to extract a free fall snow drift. I would need to shelter the area where I would determine the size from wind (so that it doesn't drift from other places, giving a false reading). By using a laser scanning technique, I could measure the height of the snow pile and thus the amount of snowfall. Before I send the information out in my normal weather stream, I could use the extrapolated point to determine if my findings are within the "ballpark" of a correct amount. This would give me some confidence in my findings. Again, this could be transmitted via a wifi link to my network to be combined with other weather information.

I should think about how I am going to interject the other three sensors with this type of thing.

Maybe I should think about opening up a blog on the Weather Underground to talk about DIY and Hobbyist Built Weather Stations.