Recording weather with Arduino, Elasticsearch and Kibana 4

-

Recording weather with Arduino, Elasticsearch and Kibana

Miss the first part of this series?  It's here!

Part 4 - Getting data into Elasticsearch

In the previous steps we have built our weather station and configured the software.  Now it is time to start doing something with the data.

At the end of part 3 We have a system which will transmit data over an Ethernet network every minute, into our data store.  This works really well if you have a data store to send it to, and that is the first thing to get started with.

There are two excellent tools for processing this kind of data, Elasticsearch and Splunk.  In this series I am going to concentrate on Elasticsearch, and later I will come back and revisit this project using Splunk.

Getting Started with Elasticsearch

Setting up Elasticsearch is actually not that hard if you just want to learn about it.  I will take you through a very basic setup which is sufficient for the kind of data in use here.  Before trying anything more ambitious however please refer to Elastic's documentation as there is going to be a lot which I don't cover.

Elasticsearch runs on pretty much any modern operating system, so you can use it on a Mac, Windows, FreeBSD or Linux machine.  When you want to get into some of the add-ons however your choices may be limited, so for now we will run ours on a Centos 7 machine.

1.  Make sure your system is up-to-date.  For me, this is a matter of "sudo yum update -y"

2.  Make sure you have the current Java 1.8 release installed.  Your installation method will vary, but for Linux you use openjdk-1.8.0-openjdk.  If you already have Java 1.7 installed (commonly the default), you should remove this first because it is just going to cause confusion.

3.  Follow the steps that Elastic have published:  Rather than blindly repeat things from their docs, just head on over to https://www.elastic.co/guide/en/elasticsearch/reference/6.1/install-elasticsearch.html and follow the steps for your operating system.

At the very least you will need to follow the following steps:
  • Install Elasticsearch
  • Install X-Pack for Elasticsearch
  • Install Kibana
  • Install X-Pack for Kibana
Since we are dealing with such small amounts of data, this can all go onto one machine.  For serious installations you would also want to have replicas and separate the Kibana and Index machines but that's for another time.

Oh one last thing, you can get a free X-Pack for Kibana license by following the instructions at elastic.co.  This will make your life easier when we get to Kibana.

Creating the index

You interact with Elasticsearch through web API calls.  The easiest way to do this is with curl.  The source code contains the necessary command to send to your indexer, but in case you missed it:


curl -XPUT 'http://host:9200/weatherdata?pretty' -H 'Content-Type: application/json' -d'
{
    "mappings" : {
        "reading" : {
            "properties" : {
                "deviceid" : { "type" : "text" },
                "temperature" : { "type" : "float" },
                "humidity" : { "type" : "float" },
                "pressure" : { "type" : "float" },
                "dewpoint" : { "type" : "float" },
                "windspeed" : { "type" : "float" },
                "winddirection" : { "type" : "float" },
                "uv" : { "type" : "float" },
                "timestamp" : { "type" : "date" } 
            }
        }
    }
}'
If you want to add, or remove fields, it's easy enough.  Obviously, replace "host" with the name of the machine you will be connecting to.  Make sure that your weather station can identify it as well.  If you don't have your own DNS available, using IP addresses works just fine as well.

With that done, recompile the weather station code with the following:

#define _DEBUG
#define _ELASTIC
/*#define _SPLUNK */

Upload, and turn on the serial monitor function in the Arduino IDE.  After about a minute you will see a message:

Connecting to: host:9200

and after that a string of text which is the sensor readings will be sent to your monitor as well as to Elasticsearch.  Should an error occur sending to Elasticsearch, the message will be sent to the serial monitor.

Now comes the hard part.  Weather changes slowly, so you need to give the system a few hours to build up a reasonable data set to work with.  Leave everything running and take the rest of the day off.  When you come back tomorrow, you will have a decent set of numbers to work with.

Next:  Drawing Graphs


Location: Sydney NSW, Australia

Comments

Post a Comment

Popular posts from this blog

The QYT KT-7900D

-
Image
But is it any good? As you might have noticed, I am now a freshly licensed radio amateur .  As such I do need a transceiver or two in order to get on the air.  This is the story of one of my new radios. I have a limited discretionary budget to use for my new hobby, and I have to say from the outset that my lovely wife has been very supportive.  As such, I want to try and get the best value for our money.  One evening I stumbled upon this little item on Banggood for a little over $A150 including shipping.  Covering both 2m and 70cm bands with a claimed 20W output it seemed almost too good to be true.  I bought one. About ten days later a tiny package arrived for me.  At just 98mm x 43mm x 126mm the photos don't convey quite how small it actually is.  The entire case serves as a heatsink so it is heavier than it looks, but certainly not bulky. The standard power cable comes with a cigarette lighter plug.  As I intended to run it from home usin...
Read more

Recording weather with Arduino, Elasticsearch and Kibana

-
What do half a Bunnings weather station, an Arduino, a NodeMCU (or 3), Elasticsearch, a fishtank and swimming pool have to do with each other? Nothing unless you like to monitor and measure pretty much everything around the house.  I do, and ever since I learned to fly, I have had a fascination with the weather (I am told boaties suffer from a related affliction). I could just buy a weather station, but where's the fun in that when for the same amount of money (OK, maybe a little more) I can make my own and bore people senseless when I go on and on about it? Actually, there is a practical purpose to this as well.  I wanted to teach myself about Elastic and Kibana.  Now you can wade through dry tutorials, but nothing in my opinion beats working with real live data.  Especially if the data is of interest to you. The design What do you want to measure? This is actually pretty important (not to mention obvious), but before embarking too far; take a...
Read more

Exploring Solar Power - Let there be smoke!

-
Image
Last time I introduced my new solar power project.  By the end of that post we were monitoring the power flow from the solar panel to the battery in a simple test environment.  This time we're going to dive in and start trying to control the charge power. What's MPPT? Maximum Power Point Tracking (or Transfer , depending on whom you ask) is a condition where we can transfer the highest power from the solar panel into the battery.  The key here is power, rather than voltage or current on their own.  Remember that power is the product of voltage and current, so it is possible that you will find that perhaps higher current and lower voltage (or vice versa) is the most efficient. What makes this a little problematic is that the peak power point varies constantly depending on the battery state of charge and temperature, so it is not a set and forget operation. This is why the first thing we did was to build a power monitor.  Now we can determine at any point of time...
Read more