Recording weather with Arduino, Elasticsearch and Kibana 4

Recording weather with Arduino, Elasticsearch and Kibana

Miss the first part of this series?  It's here!

Part 4 - Getting data into Elasticsearch

In the previous steps we have built our weather station and configured the software.  Now it is time to start doing something with the data.

At the end of part 3 We have a system which will transmit data over an Ethernet network every minute, into our data store.  This works really well if you have a data store to send it to, and that is the first thing to get started with.

There are two excellent tools for processing this kind of data, Elasticsearch and Splunk.  In this series I am going to concentrate on Elasticsearch, and later I will come back and revisit this project using Splunk.

Getting Started with Elasticsearch

Setting up Elasticsearch is actually not that hard if you just want to learn about it.  I will take you through a very basic setup which is sufficient for the kind of data in use here.  Before trying anything more ambitious however please refer to Elastic's documentation as there is going to be a lot which I don't cover.

Elasticsearch runs on pretty much any modern operating system, so you can use it on a Mac, Windows, FreeBSD or Linux machine.  When you want to get into some of the add-ons however your choices may be limited, so for now we will run ours on a Centos 7 machine.

1.  Make sure your system is up-to-date.  For me, this is a matter of "sudo yum update -y"

2.  Make sure you have the current Java 1.8 release installed.  Your installation method will vary, but for Linux you use openjdk-1.8.0-openjdk.  If you already have Java 1.7 installed (commonly the default), you should remove this first because it is just going to cause confusion.

3.  Follow the steps that Elastic have published:  Rather than blindly repeat things from their docs, just head on over to https://www.elastic.co/guide/en/elasticsearch/reference/6.1/install-elasticsearch.html and follow the steps for your operating system.

At the very least you will need to follow the following steps:
  • Install Elasticsearch
  • Install X-Pack for Elasticsearch
  • Install Kibana
  • Install X-Pack for Kibana
Since we are dealing with such small amounts of data, this can all go onto one machine.  For serious installations you would also want to have replicas and separate the Kibana and Index machines but that's for another time.

Oh one last thing, you can get a free X-Pack for Kibana license by following the instructions at elastic.co.  This will make your life easier when we get to Kibana.

Creating the index

You interact with Elasticsearch through web API calls.  The easiest way to do this is with curl.  The source code contains the necessary command to send to your indexer, but in case you missed it:


curl -XPUT 'http://host:9200/weatherdata?pretty' -H 'Content-Type: application/json' -d'
{
    "mappings" : {
        "reading" : {
            "properties" : {
                "deviceid" : { "type" : "text" },
                "temperature" : { "type" : "float" },
                "humidity" : { "type" : "float" },
                "pressure" : { "type" : "float" },
                "dewpoint" : { "type" : "float" },
                "windspeed" : { "type" : "float" },
                "winddirection" : { "type" : "float" },
                "uv" : { "type" : "float" },
                "timestamp" : { "type" : "date" } 
            }
        }
    }
}'
If you want to add, or remove fields, it's easy enough.  Obviously, replace "host" with the name of the machine you will be connecting to.  Make sure that your weather station can identify it as well.  If you don't have your own DNS available, using IP addresses works just fine as well.

With that done, recompile the weather station code with the following:

#define _DEBUG
#define _ELASTIC
/*#define _SPLUNK */

Upload, and turn on the serial monitor function in the Arduino IDE.  After about a minute you will see a message:

Connecting to: host:9200

and after that a string of text which is the sensor readings will be sent to your monitor as well as to Elasticsearch.  Should an error occur sending to Elasticsearch, the message will be sent to the serial monitor.

Now comes the hard part.  Weather changes slowly, so you need to give the system a few hours to build up a reasonable data set to work with.  Leave everything running and take the rest of the day off.  When you come back tomorrow, you will have a decent set of numbers to work with.

Next:  Drawing Graphs


Comments

Popular posts from this blog

Life with Deep Racer - Part 2

DIY Self-Driving - Part 9 - New Imaging

The QYT KT-7900D