Recording weather with Arduino, Elasticsearch and Kibana 4

Recording weather with Arduino, Elasticsearch and Kibana

Miss the first part of this series?  It's here!

Part 4 - Getting data into Elasticsearch

In the previous steps we have built our weather station and configured the software.  Now it is time to start doing something with the data.

At the end of part 3 We have a system which will transmit data over an Ethernet network every minute, into our data store.  This works really well if you have a data store to send it to, and that is the first thing to get started with.

There are two excellent tools for processing this kind of data, Elasticsearch and Splunk.  In this series I am going to concentrate on Elasticsearch, and later I will come back and revisit this project using Splunk.

Getting Started with Elasticsearch

Setting up Elasticsearch is actually not that hard if you just want to learn about it.  I will take you through a very basic setup which is sufficient for the kind of data in use here.  Before trying anything more ambitious however please refer to Elastic's documentation as there is going to be a lot which I don't cover.

Elasticsearch runs on pretty much any modern operating system, so you can use it on a Mac, Windows, FreeBSD or Linux machine.  When you want to get into some of the add-ons however your choices may be limited, so for now we will run ours on a Centos 7 machine.

1.  Make sure your system is up-to-date.  For me, this is a matter of "sudo yum update -y"

2.  Make sure you have the current Java 1.8 release installed.  Your installation method will vary, but for Linux you use openjdk-1.8.0-openjdk.  If you already have Java 1.7 installed (commonly the default), you should remove this first because it is just going to cause confusion.

3.  Follow the steps that Elastic have published:  Rather than blindly repeat things from their docs, just head on over to https://www.elastic.co/guide/en/elasticsearch/reference/6.1/install-elasticsearch.html and follow the steps for your operating system.

At the very least you will need to follow the following steps:
  • Install Elasticsearch
  • Install X-Pack for Elasticsearch
  • Install Kibana
  • Install X-Pack for Kibana
Since we are dealing with such small amounts of data, this can all go onto one machine.  For serious installations you would also want to have replicas and separate the Kibana and Index machines but that's for another time.

Oh one last thing, you can get a free X-Pack for Kibana license by following the instructions at elastic.co.  This will make your life easier when we get to Kibana.

Creating the index

You interact with Elasticsearch through web API calls.  The easiest way to do this is with curl.  The source code contains the necessary command to send to your indexer, but in case you missed it:


curl -XPUT 'http://host:9200/weatherdata?pretty' -H 'Content-Type: application/json' -d'
{
    "mappings" : {
        "reading" : {
            "properties" : {
                "deviceid" : { "type" : "text" },
                "temperature" : { "type" : "float" },
                "humidity" : { "type" : "float" },
                "pressure" : { "type" : "float" },
                "dewpoint" : { "type" : "float" },
                "windspeed" : { "type" : "float" },
                "winddirection" : { "type" : "float" },
                "uv" : { "type" : "float" },
                "timestamp" : { "type" : "date" } 
            }
        }
    }
}'
If you want to add, or remove fields, it's easy enough.  Obviously, replace "host" with the name of the machine you will be connecting to.  Make sure that your weather station can identify it as well.  If you don't have your own DNS available, using IP addresses works just fine as well.

With that done, recompile the weather station code with the following:

#define _DEBUG
#define _ELASTIC
/*#define _SPLUNK */

Upload, and turn on the serial monitor function in the Arduino IDE.  After about a minute you will see a message:

Connecting to: host:9200

and after that a string of text which is the sensor readings will be sent to your monitor as well as to Elasticsearch.  Should an error occur sending to Elasticsearch, the message will be sent to the serial monitor.

Now comes the hard part.  Weather changes slowly, so you need to give the system a few hours to build up a reasonable data set to work with.  Leave everything running and take the rest of the day off.  When you come back tomorrow, you will have a decent set of numbers to work with.

Next:  Drawing Graphs


Recording weather with Arduino, Elasticsearch and Kibana 3

Recording weather with Arduino, Elasticsearch and Kibana

Miss the first part of this series?  It's here!

Part 3 - Software and Testing

One of the more amusing aspects of this exercise was to code for small platforms.

The main challenge revolves around measuring speed and rainfall, because these instruments simply generate a pulse when something happens, and the code needs to be ready to act, but at the same time, there is a degree of "housekeeping" which needs to be done.

The simplest method of doing this is to generate an interrupt whenever a pulse is received from one of the instruments.  This way, the CPU can give you the appearance of doing multiple things (like housekeeping and uploading) at once.

I like to think my coding ability is adequate, but doing this on an Raspberry  Pi, CHIP, BeagleBone etc means I have to write my ISR (Interrupt Service Routines) as kernel modules, and that is just more work than I want to commit to for my own amusement.  I needed to get back pretty much to the bare metal, which made Arduino a simpler choice.

However, if you have done any coding for Arduino, you soon realise that it is a single-threaded platform.  In essence the code you upload plus the runtime libraries are the operating system.  Don't worry, this is good news because it means we can largely do what we want with the processor, and the Atmel (now Microchip) CPUs are very flexible with interrupts and the Arduino ecosystem already has good library support.

Stay with me folks, this is going to get interesting...

We have three sources of interrupts in our system:
  • Anemometer
  • Rainfall
  • Internal Timer
The key to how all this operates is the Internal Timer.  I have used the "Timer-1" library which can be installed via the Arduino Library Manager.

Timer1 establishes an interrupt which fires every 500mS.  It is not perfectly accurate, but since the CPU is not working all that hard, it will be accurate enough for this job.  The basic loop structure of the program is as follows:

loop() {
    while(timer count < 120) {
    }
    increment minutes by 2
    collect readings
    build statistics for transmit
    transmit
    if minutes >= 1440
        reset daily statistics
}

So in essence, the CPU spends most of its time waiting for two minutes to expire.

Sensor Interrupts
When the rainfall or wind speed sensors generate a pulse, an interrupt will be raised in the CPU.  The ISR (Interrupt Service Routine) delays 50mS to debounce the input.  Unless a cyclone hits Sydney, we won't miss any pulses in this debounce routine.

After the debounce period, either the wind speed is calculated or the rain counter is incremented.  No other work is done in this time to make sure that we do as little as possible within the interrupt context.

When the timer count exceeds 1 minute (2 x 0.5s = 120 ticks), the real work begins.  All the sensors are read, and then a timestamp needs to be generated.

Timing
The timestamp is necessary because Elasticsearch cannot generate one by itself.  I could mess around with log stash and have it do so, but since we have a network connection, why not use NTP to do so?

A further gotcha, is that we need to convert the timestamp to mS for consumption by Elasticsearch.  Pretty simple, multiply the value by 1000.

However if you don't want (or can't) use NTP, you could use a small RTC module and ditch all the NTP code.

Sensor Calculations
This is the one place you are on your own.  You need four pieces of information to get useful readings from your sensors:
  • For the rain fall sensor, how many mm per "tip"
  • For wind speed, how many pulses/revolution and the circumference of the sensor
  • For wind direction, the values of resistor for each position.
Rainfall is pretty simple.  Most gauges will be between 0.1 and 1.0mm per tip event. If you can't find the value, you can always compare it with a rain gauge.  Set BUCKET_SIZE to this value.

Wind speed is going to need some math.  It's easy enough to use a multimeter or test lamp to measure pulses/rotation.  For low-cost senders the answer seems to always be 1.

The math comes in because we need to know how far a single cup travels in a revolution.  Measure the distance from the centre of the cup, to the centre of the anemometer axle.  This is the radius (r) of our circle.  Double that to get the diameter.  From high-school you might remember that the circumference of a circle is calculated as:

pi x d

If you measured the diameter in millimetres (mm), then if you divide your result by 1,000 you get the circumference in metres.  This is the value to set ROT_VALUE to.

The Wind direction resistance is for reference only at this point.  When we start feeding data into Elasticsearch, you will see how we translate it into a direction.

Networking
Much of the code for this project is to support the networking tasks.  The Ether10 is compatible with the Arduino Ethernet Shields, so the code should need little, if any, modifications.  Most of this code has come from the Arduino examples, and should be easy enough to understand.  The biggest catch is that the MAC address for the controller is hard-coded.  Later versions of Ether10 than mine have a unique IP address.  If you ran up two boards with the same code, you can going to have problems on the local network, so be sure to change the values in the byte mac[] line.

The Code

Please note that this is not always going to be the most recent version of the code.  Current releases can be found on bitbucket.  The URL is:


There are usually 2 branches of the code on Bitbucket.  The first is "master", which is tested and stable.  If you use the same basic components as I did, there's no reason it should not work for you.

The other is "develop", which is where all the fun stuff is happening.  Any new features get added via "develop", so it might not always be perfect, but it will at least compile and run for me.


/*
 * Inspired by a lot of different ideas
 * An Arduino-based weather station.
 * 
 * Owing to memory limitations, this sketch does NOT use SSL.
 * Be careful what you send over it.
 */

 /*
  * PORT ALLOCATIONS
  * PORT      DESIGNATION     DIRECTION     TYPE      DESCRIPTION
  * D13       LED_PIN         OUT           DIGITAL   On-Board LED.  General Indication
  * D12       DATA_PIN        OUT           DIGITAL   "Network" LED.  Activity indication
  * D3        RAIN_PIN        IN            DIGITAL   Rain Sensor Input - 1 pulse/tip
  * D2        SPEED_PIN       IN            DIGITAL   Speed Sensor Input - 1 pulse/revolution
  * A5        SCK             OUT           DIGITAL   I2C SCK (Clock Signal)
  * A4        SDA             IN/OUT        DIGITAL   I2C SDA (Data I/O)
  * A1        VANE_PIN        IN            ANALOGUE  Wind Direction
  * A0        UV_PIN          IN            ANALOGUE  UV Sensor
  * 
  */

/*
 *
 * Note:  You need to create the ElasticSearch Index and Mappings beforehand
 * curl -XPUT 'http://host:9200/test?pretty' -H 'Content-Type: application/json' -d'
{
    "mappings" : {
        "reading" : {
            "properties" : {
                "deviceid" : { "type" : "text" },
                "temperature" : { "type" : "float" },
                "humidity" : { "type" : "float" },
                "pressure" : { "type" : "float" },
                "dewpoint" : { "type" : "float" },
                "windspeed" : { "type" : "float" },
                "winddirection" : { "type" : "float" },
                "uv" : { "type" : "float" },
                "timestamp" : { "type" : "date" } 
            }
        }
    }
}
'

  */
#include <TimerOne.h>
#include <Ethernet.h>
#include <EthernetUdp.h>
#include <Dhcp.h>
#include <EthernetClient.h>
#include <Dns.h>
#include <BME280.h>
#include <BME280I2C.h>

//#define _DEBUG
#define _SPLUNK
//#define _ELASTIC

#ifdef _ELASTIC
#define HOST        "your-elastic-host"
#define PORT        9200
#endif
#ifdef _SPLUNK
#define HOST        "your-splunk-host"
#define PORT        8088
#endif
#define BUCKET_SIZE 0.2794   // Rain in mm per tipping event
#define ROT_VALUE   0.666667 // Distance of one rotation
#define RAIN_PIN    3
#define SPEED_PIN   2
#define UV_PIN      (A0)
#define VANE_PIN    (A1)
#define LED_PIN     13
#define DATA_PIN    12

BME280I2C bme;
EthernetClient client;
EthernetUDP udp;
unsigned long unixTime;
  

float temperature;
float humidity;
float pressure;
float windSpeed;
float dewPoint;
unsigned long tipCount;
unsigned long contactTime;
unsigned long contactBounceTime;
unsigned long totalRainfall;
unsigned long rotationCount;
unsigned int uv;
unsigned int timerCount;
unsigned int readingTime;
unsigned int totalMinutes;
int vaneReading;
int windDirection;
boolean masterStatus;

void readTemperature() {
  temperature = bme.temp(true);
}

void readHumidity() {
  humidity = bme.hum();
}

void readPressure() {
  pressure = bme.pres(1);
}

void readDewPoint() {
  dewPoint = bme.dew(bme.temp(), bme.hum(), true);
}

/*
 * Vout (mV)    UV Index
 * < 50         0
 * 227          1
 * 318          2
 * 408          3
 * 503          4
 * 606          5
 * 696          6
 * 795          7
 * 881          8
 * 976          9
 * 1079         10
 * > 1170       11+
 */
void readUV() {
//  uv = analogRead(UV_PIN);
  uv = map(analogRead(UV_PIN), 0, 1023, 0, 5000);
}

void readWindDirection() {
  vaneReading = analogRead(VANE_PIN);
  windDirection = map(vaneReading, 0, 1023, 0, 5000); // Convert to mV
}

void isr_rain() {
  if((millis() - contactTime) > 15) {
    tipCount++;
    totalRainfall = tipCount * BUCKET_SIZE;
    contactTime = millis();
  }
}

void isr_wind() {
  // Temporary.  Show that we are in the ISR
  digitalWrite(DATA_PIN, HIGH);
  if((millis() - contactBounceTime) > 15) {
    rotationCount ++;
    contactBounceTime = millis();
  }
  digitalWrite(DATA_PIN, LOW);
}

void isr_timer() {
  digitalWrite(LED_PIN, HIGH);
  
  timerCount++;
  readingTime++;

  if(timerCount == 5) {
    // Need to divide count by 60 to get m/s
    // Then multiply by 3.6 for Km/h
    windSpeed = rotationCount / 60 * ROT_VALUE * 3.6;
    rotationCount = 0;
    timerCount = 0;
  }
  digitalWrite(LED_PIN, LOW);
}

void blinkLED(unsigned int times) {
  for(int i = 0; i < times; i++) {
    digitalWrite(LED_PIN, HIGH);
    delay(75);
    digitalWrite(LED_PIN, LOW);
    delay(75);
  }
}

unsigned long inline ntpUnixTime (UDP &udp)
{
  static int udpInited = udp.begin(123); // open socket on arbitrary port

  const char timeServer[] = "pool.ntp.org";  // NTP server

  // Only the first four bytes of an outgoing NTP packet need to be set
  // appropriately, the rest can be whatever.
  const long ntpFirstFourBytes = 0xEC0600E3; // NTP request header

  // Fail if WiFiUdp.begin() could not init a socket
  if (! udpInited)
    return 0;

  // Clear received data from possible stray received packets
  udp.flush();

  // Send an NTP request
  digitalWrite(DATA_PIN, HIGH);
  if (! (udp.beginPacket(timeServer, 123) // 123 is the NTP port
   && udp.write((byte *)&ntpFirstFourBytes, 48) == 48
   && udp.endPacket()))
    return 0;       // sending request failed

#ifdef _DEBUG
  Serial.println("Waiting for ntp");
#endif
  // Wait for response; check every pollIntv ms up to maxPoll times
  const int pollIntv = 150;   // poll every this many ms
  const byte maxPoll = 15;    // poll up to this many times
  int pktLen;       // received packet length
  for (byte i=0; i<maxPoll; i++) {
    if ((pktLen = udp.parsePacket()) == 48)
      break;
    delay(pollIntv);
  }
  if (pktLen != 48)
    return 0;       // no correct packet received

  // Read and discard the first useless bytes
  // Set useless to 32 for speed; set to 40 for accuracy.
  const byte useless = 40;
  for (byte i = 0; i < useless; ++i)
    udp.read();

  // Read the integer part of sending time
  unsigned long time = udp.read();  // NTP time
  for (byte i = 1; i < 4; i++)
    time = time << 8 | udp.read();

  // Round to the nearest second if we want accuracy
  // The fractionary part is the next byte divided by 256: if it is
  // greater than 500ms we round to the next second; we also account
  // for an assumed network delay of 50ms, and (0.5-0.05)*256=115;
  // additionally, we account for how much we delayed reading the packet
  // since its arrival, which we assume on average to be pollIntv/2.
  time += (udp.read() > 115 - pollIntv/8);

  // Discard the rest of the packet
  udp.flush();
  digitalWrite(DATA_PIN, LOW);
  return time - 2208988800ul;   // convert NTP time to Unix time
}



void setup() {
  pinMode(LED_PIN, OUTPUT);
  pinMode(DATA_PIN, OUTPUT);
  pinMode(RAIN_PIN, INPUT);
  pinMode(SPEED_PIN, INPUT);
  digitalWrite(DATA_PIN, HIGH);
  
  // If using the Freetronics EtherTen board,
  // the W5100 takes a bit longer to come out of reset
  delay(100);   // This should be plenty

#ifdef _DEBUG
  Serial.begin(9600);
  while(!Serial) {
    blinkLED(1);
    delay(10);    // Wait for serial port
  }
#endif
  
  while(bme.begin() == false) {
#ifdef _DEBUG    
    Serial.println("Waiting for BME280");
#endif
    blinkLED(2);
    delay(1000);
  }

#ifdef _DEBUG
  Serial.println("BME280 is initialised!");
#endif

  byte mac[] = {
    0x02, 0x60, 0x8C, 0x01, 0x02, 0x03
  };
  while(Ethernet.begin(mac) == 0) {
#ifdef _DEBUG    
    Serial.println("Waiting for DHCP");
#endif
    blinkLED(3);
    delay(1000);
  }
#ifdef _DEBUG
  Serial.println("Network is active");
#endif

  // Set all the globals to a valid starting point.
  temperature = 0.0;
  humidity = 0.0;
  pressure = 0.0;
  uv = 0;
  totalRainfall = 0;
  tipCount = 0;
  contactTime = 0;
  contactBounceTime = 0;
  rotationCount = 0;
  timerCount = 0;
  readingTime = 0;
  totalMinutes = 0;   // Use this to determine 24 hours

  // Initialise the interrupt for the rain gauge
  attachInterrupt(digitalPinToInterrupt(RAIN_PIN), isr_rain, FALLING);
#ifdef _DEBUG
  Serial.println("Attached Rain Sensor");
#endif

  // Initialise the interrupt for wind speed
  attachInterrupt(digitalPinToInterrupt(SPEED_PIN), isr_wind, RISING);
#ifdef _DEBUG
  Serial.println("Attached Speed Sensor");
#endif

  // Set up a timer for 1/2 second intervals
  Timer1.initialize(500000);
  Timer1.attachInterrupt(isr_timer); 
#ifdef _DEBUG
  Serial.println("Attached Timer");
#endif

  sei();    // Enable interrupts
  
  digitalWrite(DATA_PIN, LOW);
}

void loop() {
  if(readingTime > 119) {    // 1 minute has expired

    blinkLED(4);
    readTemperature();
    readHumidity();
    readPressure();
    readDewPoint();
    readUV();

    digitalWrite(DATA_PIN, HIGH);
    // Let's build a stamp for the elastic store
    unixTime = ntpUnixTime(udp);

#ifdef _ELASTIC
    String data = "{ \"deviceid\": \"02608C010203\",";
    data += "\"temperature\": \"" + String(temperature) + "\", ";
    data += "\"humidity\": \"" + String(humidity) + "\", ";
    data += "\"pressure\": \"" + String(pressure) + "\", ";
    data += "\"dewpoint\": \"" + String(dewPoint) + "\", ";
    data += "\"rainfall\": \"" + String(totalRainfall) + "\", ";
    data += "\"windspeed\": \"" + String(windSpeed) + "\", ";
    data += "\"winddirection\" :\"" + String(windDirection) + "\", ";
    data += "\"uv\": \"" + String(uv) + "\", ";
    data += "\"timestamp\" :\"" + String(unixTime) + "\"";
    data += " }";
#endif

#ifdef _SPLUNK
    String data = "{ \"host\": \"02608C010203\",";
    data += "\"sourcetype\": \"arduino\", ";
    data += "\"index\": \"myiot\", ";
    data += "\"event\": { ";
    data += "\"temp\": \"" + String(temperature) + "\", ";
    data += "\"hum\": \"" + String(humidity) + "\", ";
    data += "\"press\": \"" + String(pressure) + "\", ";
    data += "\"dewpt\": \"" + String(dewPoint) + "\", ";
    data += "\"rain\": \"" + String(totalRainfall) + "\", ";
    data += "\"speed\": \"" + String(windSpeed) + "\", ";
    data += "\"dir\": \"" + String(windDirection) + "\", ";
    data += "\"uv\": \"" + String(uv) + "\" ";
    data += " }}";
#endif

#ifdef _DEBUG
    Serial.println("Connecting to: " + String(HOST) + ":" + String(PORT));
#endif    
    if(client.connect(HOST, PORT)) {
#ifdef _DEBUG
      Serial.println("Connected");
      Serial.println(data);
#endif
#ifdef _ELASTIC
      String url = "/test/reading";
#endif
#ifdef _SPLUNK
      String url = "/services/collector";
#endif
      client.println("POST " + url + " HTTP/1.1");
#ifdef _SPLUNK
      client.println("Authorization: Splunk AUTH_KEY");
#endif
      client.print("Content-Length: ");
      client.println(data.length());
      client.println();
      client.println(data);
      client.println();
      
      delay(50);
      
      if(client.connected()) {
        client.flush();
        client.stop();
      }
      digitalWrite(DATA_PIN, LOW);
    }
    else {
#ifdef _DEBUG
    Serial.println("Connect failed");
#endif      
    }
    readingTime = 0;
    // General housekeeping stuff
    totalMinutes++;
    if(totalMinutes > 1440) { // 24 hours
      totalRainfall = 0;
      totalMinutes = 0;
    }
  }
}

Next:  Getting data into Elasticsearch

Recording weather with Arduino, Elasticsearch and Kibana 2

Recording weather with Arduino, Elasticsearch and Kibana

Miss the first part of this series?  It's here!

Part 2 - Construction

Like most things you do with Arduino, construction is pretty straightforward and you can choose your own way of going about it.  I am going to document my methods and the reasons why.  Feel free to offer suggestions for improvements.

The Base Station

Budget for the project was rather limited (read non-existent), so I built it mostly from what I had on hand, or that I could obtain cheaply.

As the controller will be mounted outside of the house in order to keep the sensor runs as short as possible, it needs to be protected from the elements.  I found a medium-sized weatherproof enclosure made by Legrand at an electrical wholesaler.  It is IP55 rated so it it's mounted under cover it is quite suitable for the task.




There are a few useful tips to keep in mind when laying out your controller:

1. Give yourself some power rails

At the top of the enclosure (right side in the photo) there is a large terminal strip which supplies +5V and 0V to the components.  Since everything is low-powered, I chose to run the whole unit from the Ether10's on-board regulator.

This method gives a much tidier result, as well as frees up the Arduino from having to host all the power wires or making daisy chained connections.


2. Terminate your external connections off-board

Again, this is about neatness and durability.  You could terminate everything in Dupont pins and plug them straight into the Arduino, but it is harder to make changes.

I put a terminal block at the bottom of the enclosure, and wired from there to the Arduino.


To further ensure reliability I used a Freetronics terminal shield on top of the Ether10


3.  Use weatherproof glands for cable entry points.

I put two 12mm glands in the bottom of the enclosure, which is plenty.  It also helps to keep out bugs and spiders.  A year after installation, the inside of the enclosure is still spotless.

4. Add a little sealant to the mounting holes for the Arduino.

OK, so it's unlikely that water will get in through that gap, but it's a simple enough thing to do

If you use an Ether10, and you're going to run it with PoE make sure you set the PoE jumpers according to the manual before you put the terminal shield in place.  It's annoying to do it later.



External Sensors

The UV and Light intensity sensors need to be mounted externally, where they will get good exposure to the sun, and this posed a challenge.  How to expose them to the elements without them getting damaged?

I fitted each sensor to its own round junction box.  You could use something smaller, but these were readily available.  Drill a hole in the flat side of the box and fit the sensor.  You can see the UV sensor in the first photo.


Protecting the sensors should mean just putting a transparent cover on the top.  For the light intensity, that is true, and I fitted a clear photographic filter to the enclosure.  The UV sensor is a problem, because most materials will block UV, which is what we do not want.


Digital Camera Warehouse were able to source a Hoya "UV-Pass" filter, which is almost black in appearance.  My experiments showed that while it did lower the reading from the sensor by about 1mV it appears to be consistent.


My BME280 sensor was originally mounted inside the main enclosure with a small vent to the outside.  However, it seemed to be affected by the enclosure, so I later cut open the temperature sender from the Holman unit and fitted it there.  The advantage of this is that it comes with its own Stevenson Screen.


The signal wires from the wind speed, direction and rainfall sensors were extended and run down to the base station.


The following schematic is taken from my station.  Please note that Fritzing did not have a symbol for the SS12SD, so I used a generic ISL29102 instead.




Next:  The software


Recording weather with Arduino, Elasticsearch and Kibana

What do half a Bunnings weather station, an Arduino, a NodeMCU (or 3), Elasticsearch, a fishtank and swimming pool have to do with each other?

Nothing unless you like to monitor and measure pretty much everything around the house.  I do, and ever since I learned to fly, I have had a fascination with the weather (I am told boaties suffer from a related affliction).


I could just buy a weather station, but where's the fun in that when for the same amount of money (OK, maybe a little more) I can make my own and bore people senseless when I go on and on about it?


Actually, there is a practical purpose to this as well.  I wanted to teach myself about Elastic and Kibana.  Now you can wade through dry tutorials, but nothing in my opinion beats working with real live data.  Especially if the data is of interest to you.



The design

What do you want to measure?

This is actually pretty important (not to mention obvious), but before embarking too far; take a good hard look at what you want to achieve, because you are going to hit limitations along the way.  My list came down to:


  • Air temperature (measured and "feels like")
  • Relative humidity
  • Dew point
  • Barometric pressure
  • UV level
  • Light intensity
  • Wind speed
  • Wind direction
  • Rainfall
I also wanted to be able to measure a subset of these values (chiefly temperature and humidity) from remote locations and collate this information somehow.

How am I going to measure all of this?

There are so many low cost modules out there now which make the job much easier than a few years ago.  There's no doubt I would end up with a bunch of sensors, but at only a couple of dollars each (mostly) I get a lot of data for my money.  Let's look at the solution:

Temperature, Humidity, Pressure
There's a wealth of choices here.  For my aquarium monitor, I use the Dallas Semi DS18B20 attached to a Raspberry Pi.  They work well, and when supplied encased in stainless steel are perfect for immersion.

The DHT/MHT series (DHT11, DHT22, etc) give me relative humidity as well.  They are cheap, but rather slow and fiddly to drive.

I settled in the Bosch BME280.  It is supplied on a 10mm x 10mm PCB, gives me temperature, barometric pressure, humidity and even altitude all over an I2C connection.

Dew Point
The Dew point is used as an indication of general comfort, as well as other useful things in the petrochemical industry.  But mostly it's about the relative comfort or otherwise in the present environment.  We don't need a sensor for this if we have temperature and relative humidity.  All we need is a bit of math to work it out. Wikipedia explains it better than me.

Frost Point
In doing my research I found a little-quoted figure called the Frost point. If we know the temperature and dew point we can work out the temperature at which frost would form in the current climate.  As we get only a handful of frosts per year where we live it's not much use, but as my wife's plants don't like frost, and I can get it for the price of a little math, well why not?



UV Level
If you live in Australia, sooner or later you're going to worry about UV exposure.  I'm sure other countries are not immune either, but it's becoming common conversation these days/

Electro-Optical Components make a cheap analogue sensor called the SS12SD.  These can be obtained for around $A2.00 each on a breakout board.

Light intensity
This is related to, but not the same as the UV sensor.  Primarily it is an input to the "feels like" calculation.

In this case I am using a BH1750 sensor, which is commonly used in phones, laptops and tablets to adjust screen lighting to suit ambient conditions.

It connects over the I2C buss and handily gives a reading in Lumens so you can avoid any math.

Wind Speed
Now, you can't have a weather station without one of those turns ping-pong ball things (it's an anemometer you clown), because it gives you scientific credibility as well as the current wind speed.he bulk of them are and electro-optical or magnetic arrangement which gives you 1 or 2 pulses per revolution.  If you buy a dedicated unit (such as from Davis Instruments), there will be calibration data included.  However if you're recycling one like yours truly, you will need to work out how many metres per revolution.

Wind Direction
If you're using an off-the-shelf unit, they generally fall into one of two categories.  Either a potentiometer (variable resistor) or hall effect switches.

The potentiometer devices may have a "dead spot" in their output, and are becoming less common.  

The hall effect models use a set of hall-effect switches to switch fixed resistors into circuit, and give you an effective variable resistance without the dead spot.  However, the resolution is generally limited to 45 degrees or so.  This is quite sufficient for what I need.

Rainfall
Simplest choices are "tipping bucket" sensors or an optical device such as the Hydreon RG-11.  Both have strengths and weaknesses, and most commercial stations use the tipping bucket types, but you have to mount them securely to prevent false readings.

The final design

The controller
While all this is going on, I have to decide what controller I am going to use.  The choice is driven from a couple of perspectives, chiefly functionality and price.

I had used a Raspberry Pi in the past, but it doesn't have analogue I/O.  I could add an I2C ADC module, but the Pi is not that cheap to begin with.

The NodeMCU is another interesting option.  It has inbuilt WiFi and is very cheap, but it only gives me a single analogue input without add-on converters.

I happened to have a Freetronics "Ether10", which is and Arduino Uno compatible board with a Wiznet Ethernet interface included and it supports PoE (Power Over Ethernet), which meant one less cable I needed to run.

Sensor Channels
My final set of inputs is as follows:
  • Analogue x 2 (Wind direction and UV level)
  • I2C x 2 (Temp/RH/Pressure and Light intensity)
  • Pulse input x 2 (Rainfall and Wind speed)
  • Timer interrupt (more on this later)
Some of the sensor solution presented itself on a weekend trip to Bunnings when I found a damaged Holman iWeather unit in the bargain bin.  Display was knackered, but the sensors seem to be in good order, and for the princely sum of $25 instead of $98, it was mine.  Actually, $98 didn't seem to bad in the first place, even if I did have to ditch the display unit.

This appears to be the same design sensor unit as used by Oregon Scientific and a few other manufacturers as well.  As best as I can tell it is originally manufactured by Mi*Sol out of China.

Parts List
The following is the list of components in my original station:

Freetronics Ether10 Board x 1 (You could substitute an Arduino Uno + Ethernet Shield)
Freetronics "Screw Shield" x 1 (Optional, but recommended)
Holman iWeather outdoor assembly, or similar
Bosch BME280 on a breakout board
BH1750 Light intensity sensor on a breakout board
SS12SD UV sensor on a breakout board
Power supply to suit (or use PoE injector)
Legrand weather-proof enclosure x 1
12mm Weather-proof cable glands x 2
100V/0.25W Zener diodes x 6
Blue LED x 1
Orange LED x 1
1X/0.25W Resistors x 6
220-ohm Resistors x 2
10-way terminal strip x 2
Hookup wire
Screws, spaces, nuts all M3




Wasting your and my time

I had a really interesting experience recently which I hope might enlighten others as much as it did me: I was approached (via LinkedIn) by ...