Forem: Tej Pochiraju

Hands On With AWS IoT, Timestream and Grafana

Tej Pochiraju — Mon, 15 Mar 2021 06:44:57 +0000

Grafana has long been a favourite of ours for creating stunning visualisations. It's lightweight, easy to get started with and, over time, has added a lot of new features without losing simplicity.

Yet, we have shied from recommending Grafana to our IoT customers over, say, the TICK stack from Influx or ODFE/Kibana.

A typical IoT workflow has three main blocks on the cloud and a lot of product builders are thrown off by the complexity of integrating these while maintaining data security and access control.

Of these, Grafana only does visualisation and is spectacularly good at it. You need to find your own solutions for data ingress and storage. Sure, Grafana has plugins for most data sources you have heard of and many more you haven't. Yet, most of these require you to set up your own servers.

We previously explored integrating AWS Timestream with Grafana. Back then, we used the managed service from Grafana.com. With AWS now offering managed services for IoT ingress, time series data storage and Grafana, the integration and security challenges should largely be solved, right? Let's find out.

Setting up Grafana Instance

Like the best managed offerings from AWS, setting up and configuring Grafana is a breeze. There's a wizard led flow once you click on "Create Workspace".

AWS uses SSO for access control - this is powered by AWS Organizations behind the scenes. More about this in a bit.

IAM policies can be enabled with a click or you can configure these yourself.

Once we are done with the wizard, we enable the newly created user for access. In a couple of minutes our brand new Grafana instance is ready for use. We are brought to this friendly and useful summary.

Accessing & Configuring Grafana

While we were busy clicking Next a few times, AWS created a new Organisation, sent a confirmation email to the logged in IAM user and another invitation email to the SSO user we created.

Accepting the invitation allows us to set up a password for this user which we will need when accessing the Grafana instance.

Once we login, the interface is an AWS white-labeled version of the standard Grafana UI with a section for AWS specific data sources.

As one would expect, finding and setting up our Timestream DB as the default data source is also a matter of a few clicks.

Setting up Dashboards & Alerts

We have previously described how to send data to AWS IoT and from there into Timestream. Once that is configured, we send data using our trusty Python script.

Setting up a dashboard and panel is also exactly the same as in the previous article and we end up with, surprise, a similar panel.

The alerts workflow is the same as usual - with the exception that AWS SNS is the first option in notification channels. Once the alerts are configured, we see them on the Alerts dashboard within Grafana.

SNS is really easy to configure from within the AWS Grafana Console (where we saw the summary earlier). We will explore this integration in a future post.

Reflections

The advent of managed services from AWS (and others) has enabled cloud infrastructure to be configured rather than developed. For most IoT applications, a product builder can probably just pick one of the following configurations.

There are differences to consider, of course.

Features

ODFE has anomaly detection built-in
ODFE's authentication system supports multi-tenancy and granular data source level access controls out of the box.
- Grafana has viewer and editor roles and it's not obvious how to implement data source level access control.
Grafana allows you to ingest and visualise data from multiple data sources.
Grafana has a lot of visualisation plugins with a well documented flow for building new plugins.

Pricing

ODFE does not have a pay-as-you-go model - you pay for the hosted ElasticSearch instance(s). Expect to pay $40-$80 per month for <100,000 devices. There are no limits to the number of users you can add to the system.
AWS Managed Grafana costs $9/editor and $5/viewer per month. The prices are similar to that for QuickSight.

Ecosystem

Timestream is relatively new and the data model takes a little getting used to. Moreover, you have to use official AWS SDKs to access data for custom applications.
ElasticSearch/ODFE, being a lot more mature, have numerous SDKs to access data and build custom interfaces.

For these reasons, and to avoid vendor lock-in with Timestream, we will continue to recommend ODFE over Grafana.

Questions, comments? Connect with us at hello@iotready.co.

Anomaly Detection For IoT Using Open Distro For ElasticSearch

Tej Pochiraju — Wed, 17 Feb 2021 15:23:12 +0000

This post is the third in series on using the AWS ecosystem for IoT applications. Previously, we integrated AWS IoT with Timestream /Quicksight and ElasticSearch/Kibana.

Why

Anomaly detection is the foundation for applications such as Predictive Maintenance, which in turn is the driving force behind most industrial IoT deployments. Now that the essentials of sensors, communication, storage and visualisation have largely been solved, attention has turned to machine learning based analytics. Cue the new features from AWS IoT and Open Distro For ElasticSearch - the latter is the focus of this article.

What are we going to build?

We will:

Simulate a smart grid sensor capable of measuring current, voltage, tempterature and humidity
Train an anomaly detector in ODFE on each of these metrics or features as ODFE calls them.
Simulate various grades of anomalies and verify that detector is working fine
Integrate the anomaly detector with Kibana's alerts (previously discussed here)

Simulated Smart Grid Sensor

Our simulated sensor helps monitor and predict failures in the medium voltage (MV) transmission grid. The sensor has the following nominal specifications:

Voltage between 23kV and 25kV
Current between 0A and 600A
Temperature between 30C and 100C
Humidity between 20% and 80%

Values in these ranges are considered good. Anything outside is an anomaly.

Anomaly Detection in ODFE

ODFE uses the Random Cut Forest (RCF) algorithm for anomaly detection. RCF is an unsupervised algorithm which analyses the data and identifies patterns. Data points that do not fit into these patterns are classified as anomalies and can include, amongst others,:

Spikes
Changes in periodicity
Unclassifiable data points

Each anomaly is given a score - low scores correspond to normal and high scores to anomalous data points. Read more about RCF in these references:

ElasticSearch Instance

Assuming you followed the previous post, you will already have an ElasticSearch instance running. However, we need at least 2 CPU cores to use anomaly detection. I am using a t2.medium instance for this post.

Building Our First Anomaly Detector

Like before, we will start our simulator to inject sensor data into ElasticSearch. I started a script to simulate 21 sensors sending data every 10s.

current = random.uniform(0.0,600.0)
voltage = random.uniform(23000.0,25000.0)
temperature = random.uniform(30,100)
humidity = random.uniform(20.0,80.0)

The injected data looks a bit like this:

Initialisation / Training

The ODFE documentation has an excellent guide for setting up a detector. Following that we end up with a configuration that looks like this:

Note that:

I picked Detector Interval = 5 minutes and Window Delay = 2 minutes
- The documentation suggests smaller intervals make the system more real-time but consume more CPU, which sounds about right.
You are allowed to add up to 5 features per detector - this seems to be an ODFE limitation rather than that of the RCF algorithm itself.
I have chosen to track the max() value for each metric. You can use any of the standard ElasticSearch aggregations.
Once configured, the detector took between 30-60 minutes to initialise and go live.
I made the mistake of trying to enable the detector on a t2.small instance and kept running into an unknown error. This disappeared once I changed the instance size to t2.medium.

Anomaly Generation

Once the detector was Live, I started generating anomalies in about 60% of the data points using the following snippet. Note that I am randomly introducing anomalies into one or all of the four metrics.

r = random.randint(1,10)
if r == 1:
    current = 0
    voltage = 0
    temperature = 0
    humidity = 0
elif r == 2:
    current = 1000
    voltage = 100000
    temperature = 300
    humidity = 150
elif r == 3:
    current = 1000
    voltage = random.uniform(23000.0,25000.0)
    temperature = random.uniform(30,100)
    humidity = random.uniform(20.0,80.0)
elif r == 4:
    current = random.uniform(0.0,600.0)
    voltage = 100000
    temperature = random.uniform(30,100)
    humidity = random.uniform(20.0,80.0)
elif r == 5:
    current = random.uniform(0.0,600.0)
    voltage = random.uniform(23000.0,25000.0)
    temperature = 300
    humidity = random.uniform(20.0,80.0)
elif r == 6:
    current = random.uniform(0.0,600.0)
    voltage = random.uniform(23000.0,25000.0)
    temperature = random.uniform(30,100)
    humidity = 150
else:
    current = random.uniform(0.0,600.0)
    voltage = random.uniform(23000.0,25000.0)
    temperature = random.uniform(30,100)
    humidity = random.uniform(20.0,80.0)

The resulting timeseries charts, in their glorious randomness, look like this:

Anomaly Detection

And just like that, the detector is triggered within the first time interval. This is great - with little knowledge of machine learning and zero code, we set up a self-taught anomaly detector!

Hang On...

If you examine the anomaly grades, you will notice that the grade reduces in each time interval until the detector no longer considers the signals to be anomalous. This reminds me of an old joke,

Said the guru to his disciple, "Next year is going to be really difficult for you. You will not meet your family or friends for a long time and you will witness a lot of suffering. In fact you won't be able to step outside your own house!". "And the year after?", asked the disciple. The guru replied, "You will get used to it".

Jokes aside, why is this happening?

The anomaly detector, as mentioned above, is self-taught. And it keeps learning - even as anomalous data streams in. As the kind folk at ODFE explained to me, if 5% of your data is anomalous, is it really anomalous or, in fact, the new normal? The anomaly detector, naturally, adapts to this new normal and gives these signals a decreasing grade until they are fully normalised.

This makes sense, it's just not what I expected.

How do you solve this?

If your data has infrequent anomalies, there's nothing to fix. The existing plugin already works well!

Freezing the anomaly detection data model by stopping the learning phase should solve this problem. I have opened a feature request for this very use case.

A good suggestion from the ODFE team was to use a combination of rule based detection algorithms and ML based anomaly detection. This makes sense, especially since there are a few other issues with this domain-agnostic approach:

All signals, across all devices, in a time interval are given a single anomaly grade. We may need to classify these anomalies for priorities and, more importantly, identify the specific devices which are reporting anomalous data.
We may need different anomaly detection for each SKU. E.g. 300A current is anomalous for a sensor rated at 200A but normal for a 500A sensor. With ODFE, we would need to send data from each SKU to a different index and set up separate detectors for each.

Integrating Alerts

Once an anomaly detector has been set up, it can be used as a source in the existing Alerts plugin for ODFE. We have previously discussed this - all that changes is that we define our monitor using our new anomaly detector. Yes, that's all!

Conclusions and Next Steps

Anomaly detection is a relatively new feature in ODFE and is already really good at actually detecting anomalies and does not get tripped unless the anomalies are frequent or persistent. If the feature request is accepted and built, we are in job done territory for simple use cases.

For sensitive applications like smart grids and perhaps industrial monitoring, we are exploring solutions that combine intelligence on the cloud and at the edge. Over the coming weeks and months, we will write about our work with:

Rule based calibration and detection at the edge
Fuzzy logic based fault diagnosis at the edge
ML at the edge using projects such as TinyML

Ideas, questions or corrections?

Write to us at hello@iotready.co

Server Side Time Series Plots With Elixir Phoenix Using Contex

Tej Pochiraju — Thu, 28 Jan 2021 03:44:56 +0000

Background

At IoTReady, we are building a virtual IoT platform to help manufacturers track all of their products - whether these are born smart or not. For instance, a typical workflow we track in the Smart Grid industry looks something like this -

A manufacturer produces a batch of, say, an insulation product
The manufacturer ships certain units of this batch to a distributor
An operator buys some units of this batch
The operator installs the insulation product and captures notes and media (photos & videos)

At each stage of the flow, our mobile app scans QR codes and captures additional metadata like location and timestamps. Post installation we capture regular weather data for the installation location for analysis and preventive maintenance.

Tech Stack

Operational dashboards are a lot more fun (and useful) realtime, so we are building ours with Elixir and the Phoenix Framework. These choices deserve their own, longer, blog post. For now, we will focus on our charting library of choice.

We are big fans of Plotly and have used it extensively in the past. However, in this case we wanted to minimise JS code and do things server side as much as we could.

Step up Contex!

We discovered Contex via a blog post on the excellent Elixir School site. However, that post covers bar charts and the Contex documentation is a work-in-progress. Figuring out legends and version-to-documentation mismatches was particularly painful. Hence this post.

Our dashboard is complementary to, rather than a replacement for, BI tools like Redash or Kibana. We needed something easy to use and customise that includes some visualisation.

Quick References

The Contex documentation can be a little difficult to wrap your head around
The unit tests and samples, on the other hand, are excellent resources

Data Source

We use the OpenWeatherMap API to grab basic weather data. Our Ecto schema looks a bit like this:

schema "weather" do
  field :latitude, :float
  field :longitude, :float
  field :temp, :float
  field :humidity, :float
  field :pressure, :float

  timestamps()
end

And the query for getting recent data looks like this:

@doc """
Gets weather data points for a given latitude and longitude tuple.

## Examples
    iex> get_weather({latitude, longitude}, limit)
    {:ok, [%Weather{}]}
"""
def get_weather({latitude, longitude}, limit \\ 100) do
  q = from w in Weather,
      where: [latitude: ^latitude, longitude: ^longitude],
      order_by: [desc: :inserted_at],
      limit: ^limit,
      select: %{temp: w.temp, humidity: w.humidity, pressure: w.pressure, inserted_at: w.inserted_at}
  Repo.all(q)
end

This query returns a list of maps that look like this:

[
  %{
    humidity: 73.0,
    inserted_at: ~N[2021-01-27 17:00:01],
    pressure: 1016.0,
    temp: 297.27
  },
  %{
    humidity: 73.0,
    inserted_at: ~N[2021-01-27 17:00:01],
    pressure: 1016.0,
    temp: 297.27
  }
]

Setting up Contex

Now that we have our data, time to set up Contex. Since it's still early days for Contex, it's best to work with the master branch off the Github repo rather than the 0.3.0 release on Hex. For instance, the 0.3.0 release does not include LinePlot, which we need.

# mix.exs
defp deps do
  [
    ...,
    {:contex, git: "https://github.com/mindok/contex"},
  ]
end

Plotting the data

It's easiest to illustrate the plotting flow with code:

# Get the last 100 data points for {latitude, longitude}
weather_data = get_weather({latitude, longitude}) 


plot_options = %{
  top_margin: 5,
  right_margin: 5,
  bottom_margin: 5,
  left_margin: 5,
  show_x_axis: true,
  show_y_axis: true,
}

# Generate the SVG chart
weather_chart =
  weather_data
  # Flatten the map into a list of lists
  |> Enum.map(fn %{inserted_at: timestamp, temp: temp, humidity: humidity, pressure: pressure} ->
    [timestamp, temp, humidity, pressure]
  end)
  # Assign legend titles using list indices
  |> Dataset.new(["Time", "Temperature", "Humidity", "Pressure"])
  # Specify plot type (LinePlot), SVG dimensions, column mapping, title, label and legend
  |> Plot.new(
    LinePlot,
    600,
    300,
    mapping: %{x_col: "Time", y_cols: ["Temperature", "Humidity", "Pressure"]},
    plot_options: plot_options,
    title: "Weather",
    x_label: "Time",
    legend_setting: :legend_right
  )
  # Generate SVG
  |> Plot.to_svg()

# We are using Phoenix LiveView, so assign the chart to the socket
socket
  |> assign(:weather_chart, weather_chart)

After this, all that's left to do is to embed the SVG in the HTML view and this is all it takes:

<%= @weather_chart %>

We faced some clipping of the legend text but that was easy to fix with this CSS:

.exc-legend {
    font-size: small;
}

Here's the SVG in all its glory :-).

<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="chart" viewBox="0 0 600 300" role="img"><style type="text/css">
  text {fill: black}
  line {stroke: black}
</style>
<text x="280.0" y="20" text-anchor="middle" class="exc-title">Weather</text><text x="280.0" y="280" text-anchor="middle" class="exc-subtitle">Time</text><g transform="translate(70,40)"><g transform="translate(0, 170)" fill="none" font-size="10" text-anchor="middle"><path class="exc-domain" stroke="#000" d="M0.5, 6V0.5H420.5V6"></path><g class="exc-tick" opacity="1" transform="translate(0.5,0)"><line y2="6"></line><text y="9" dy="0.71em" dx="0" text-anchor="middle">27 Jan 09:00</text></g><g class="exc-tick" opacity="1" transform="translate(60.5,0)"><line y2="6"></line><text y="9" dy="0.71em" dx="0" text-anchor="middle">27 Jan 12:00</text></g><g class="exc-tick" opacity="1" transform="translate(120.5,0)"><line y2="6"></line><text y="9" dy="0.71em" dx="0" text-anchor="middle">27 Jan 15:00</text></g><g class="exc-tick" opacity="1" transform="translate(180.5,0)"><line y2="6"></line><text y="9" dy="0.71em" dx="0" text-anchor="middle">27 Jan 18:00</text></g><g class="exc-tick" opacity="1" transform="translate(240.5,0)"><line y2="6"></line><text y="9" dy="0.71em" dx="0" text-anchor="middle">27 Jan 21:00</text></g><g class="exc-tick" opacity="1" transform="translate(300.5,0)"><line y2="6"></line><text y="9" dy="0.71em" dx="0" text-anchor="middle">28 Jan 00:00</text></g><g class="exc-tick" opacity="1" transform="translate(360.5,0)"><line y2="6"></line><text y="9" dy="0.71em" dx="0" text-anchor="middle">28 Jan 03:00</text></g><g class="exc-tick" opacity="1" transform="translate(420.5,0)"><line y2="6"></line><text y="9" dy="0.71em" dx="0" text-anchor="middle">28 Jan 06:00</text></g></g><g fill="none" font-size="10" text-anchor="end"><path class="exc-domain" stroke="#000" d="M-6,170.5H0.5V0.5H-6"></path><g class="exc-tick" opacity="1" transform="translate(0, 170.5)"><line x2="-6"></line><text x="-9" dy="0.32em">0</text></g><g class="exc-tick" opacity="1" transform="translate(0, 155.04545454545453)"><line x2="-6"></line><text x="-9" dy="0.32em">100</text></g><g class="exc-tick" opacity="1" transform="translate(0, 139.5909090909091)"><line x2="-6"></line><text x="-9" dy="0.32em">200</text></g><g class="exc-tick" opacity="1" transform="translate(0, 124.13636363636364)"><line x2="-6"></line><text x="-9" dy="0.32em">300</text></g><g class="exc-tick" opacity="1" transform="translate(0, 108.68181818181819)"><line x2="-6"></line><text x="-9" dy="0.32em">400</text></g><g class="exc-tick" opacity="1" transform="translate(0, 93.22727272727273)"><line x2="-6"></line><text x="-9" dy="0.32em">500</text></g><g class="exc-tick" opacity="1" transform="translate(0, 77.77272727272728)"><line x2="-6"></line><text x="-9" dy="0.32em">600</text></g><g class="exc-tick" opacity="1" transform="translate(0, 62.31818181818181)"><line x2="-6"></line><text x="-9" dy="0.32em">700</text></g><g class="exc-tick" opacity="1" transform="translate(0, 46.86363636363636)"><line x2="-6"></line><text x="-9" dy="0.32em">800</text></g><g class="exc-tick" opacity="1" transform="translate(0, 31.409090909090907)"><line x2="-6"></line><text x="-9" dy="0.32em">900</text></g><g class="exc-tick" opacity="1" transform="translate(0, 15.954545454545467)"><line x2="-6"></line><text x="-9" dy="0.32em">1000</text></g><g class="exc-tick" opacity="1" transform="translate(0, 0.5)"><line x2="-6"></line><text x="-9" dy="0.32em">1100</text></g></g><g><path d="M  0.0 126.64227272727273 C  0.0 126.64227272727273 -2.333333333333333 126.65020606060605 0.0 126.64227272727273 C  2.333333333333333 126.6343393939394 15.333333333333334 126.58959848484848 20.0 126.57427272727273 C  24.666666666666664 126.55894696969698 35.333333333333336 126.53038181818182 40.0 126.5109090909091 C  44.666666666666664 126.49143636363637 55.333333333333336 126.41944393939394 60.0 126.40736363636364 C  64.66666666666667 126.39528333333334 75.33333333333333 126.41926363636364 80.0 126.40736363636364 C  84.66666666666667 126.39546363636364 95.33333333333333 126.31726363636363 100.0 126.30536363636364 C  104.66666666666667 126.29346363636364 115.33268518518518 126.30536363636364 120.0 126.30536363636364 C  124.66731481481482 126.30536363636364 137.67157407407407 126.30536363636364 140.00555555555556 126.30536363636364 C  142.33953703703705 126.30536363636364 137.67222222222222 126.30211818181819 140.00555555555556 126.30536363636364 C  142.3388888888889 126.30860909090909 157.67222222222222 126.32993636363638 160.00555555555556 126.33318181818183 C  162.3388888888889 126.33642727272728 136.67222222222222 126.32741212121213 160.00555555555556 126.33318181818183 C  183.3388888888889 126.33895151515152 336.6722222222222 126.37686666666666 360.00555555555553 126.38263636363635 C  383.33888888888885 126.38840606060604 360.00555555555553 126.38263636363635 360.00555555555553 126.38263636363635" stroke-linejoin="round" stroke-width="2" stroke="#1f77b4" fill="transparent"></path><path d="M  0.0 154.54545454545453 C  0.0 154.54545454545453 -2.333333333333333 154.54545454545453 0.0 154.54545454545453 C  2.333333333333333 154.54545454545453 15.333333333333334 154.54545454545453 20.0 154.54545454545453 C  24.666666666666664 154.54545454545453 35.333333333333336 154.4192424242424 40.0 154.54545454545453 C  44.666666666666664 154.67166666666665 55.333333333333336 155.5010606060606 60.0 155.62727272727273 C  64.66666666666667 155.75348484848485 75.33333333333333 155.51909090909092 80.0 155.62727272727273 C  84.66666666666667 155.73545454545453 95.33333333333333 156.44636363636366 100.0 156.55454545454546 C  104.66666666666667 156.66272727272727 115.33268518518518 156.66272727272727 120.0 156.55454545454546 C  124.66731481481482 156.44636363636366 137.67157407407407 155.73545454545453 140.00555555555556 155.62727272727273 C  142.33953703703705 155.51909090909092 137.67222222222222 155.62727272727273 140.00555555555556 155.62727272727273 C  142.3388888888889 155.62727272727273 157.67222222222222 155.62727272727273 160.00555555555556 155.62727272727273 C  162.3388888888889 155.62727272727273 136.67222222222222 155.75348484848485 160.00555555555556 155.62727272727273 C  183.3388888888889 155.5010606060606 336.6722222222222 154.67166666666665 360.00555555555553 154.54545454545453 C  383.33888888888885 154.4192424242424 360.00555555555553 154.54545454545453 360.00555555555553 154.54545454545453" stroke-linejoin="round" stroke-width="2" stroke="#ff7f0e" fill="transparent"></path><path d="M  0.0 13.75454545454545 C  0.0 13.75454545454545 -2.333333333333333 13.75454545454545 0.0 13.75454545454545 C  2.333333333333333 13.75454545454545 15.333333333333334 13.772575757575753 20.0 13.75454545454545 C  24.666666666666664 13.736515151515148 35.333333333333336 13.618030303030297 40.0 13.599999999999994 C  44.666666666666664 13.581969696969692 55.333333333333336 13.581969696969692 60.0 13.599999999999994 C  64.66666666666667 13.618030303030297 75.33333333333333 13.736515151515148 80.0 13.75454545454545 C  84.66666666666667 13.772575757575753 95.33333333333333 13.75454545454545 100.0 13.75454545454545 C  104.66666666666667 13.75454545454545 115.33268518518518 13.75454545454545 120.0 13.75454545454545 C  124.66731481481482 13.75454545454545 137.67157407407407 13.75454545454545 140.00555555555556 13.75454545454545 C  142.33953703703705 13.75454545454545 137.67222222222222 13.736515151515148 140.00555555555556 13.75454545454545 C  142.3388888888889 13.772575757575753 157.67222222222222 13.891060606060604 160.00555555555556 13.909090909090907 C  162.3388888888889 13.92712121212121 136.67222222222222 13.76484848484848 160.00555555555556 13.909090909090907 C  183.3388888888889 14.053333333333333 336.6722222222222 15.001212121212129 360.00555555555553 15.145454545454555 C  383.33888888888885 15.289696969696982 360.00555555555553 15.145454545454555 360.00555555555553 15.145454545454555" stroke-linejoin="round" stroke-width="2" stroke="#2ca02c" fill="transparent"></path></g></g><g transform="translate(500, 50)"><g class="exc-legend"><rect x="0" y="0" width="18" height="18" style="fill:#1f77b4;"></rect><text x="23" y="9" dominant-baseline="central" text-anchor="start">Temperature</text><rect x="0" y="21" width="18" height="18" style="fill:#ff7f0e;"></rect><text x="23" y="30" dominant-baseline="central" text-anchor="start">Humidity</text><rect x="0" y="42" width="18" height="18" style="fill:#2ca02c;"></rect><text x="23" y="51" dominant-baseline="central" text-anchor="start">Pressure</text></g></g></svg>

Since Dev.to does not render SVGs, you can look at the render here.

Where do we go from here

We think Contex is a pretty good fit for our needs. It's definitely rough around the edges and there are plenty of use cases we are yet to explore like:

interactivity (not a huge deal for us) and
realtime updates (big deal).

Realtime updates are easy to implement but we want verify impact on server performance but then again, this is not an immediate concern.

Ideas, questions or corrections?

Write to us at hello@iotready.co

From Metal To Alerts With AWS IoT, ElasticSearch 7.9 and Kibana

Tej Pochiraju — Tue, 05 Jan 2021 08:32:26 +0000

This post continues from my earlier post on AWS IoT, Timestream and Quicksight. We replace Timestream and Quicksight with ElasticSearch and Kibana.

Why

ElasticSearch and Kibana are, probably, the most commonly used stack for data analysis and visualisation. ElasticSearch is a notoriously difficult software to host and manage yourself. The managed service from AWS has long been one of the more reliable options along side the managed cloud from Elastic.co, the creators of ElasticSearch.

A lot of necessary features like authentication and alerts were only available in the enterprise edition. Until, that is, the release of Open Distro For ElasticSearch by AWS. While causing a great deal of controversy and ethical dilemma, ODFE adds authentication, access control, alerts, anomaly detection and, now, notebooks to the open source version release.

This makes ElasticSearch and Kibana:

more complete than Timestream & QuickSight with alerts and notebooks in the same UI
more capable than Timestream & Grafana with anomaly detection and notebooks
more extensible than using Timestream alone with the ability to ingest all kinds of data - not just timeseries.

What are we going to build?

We will:

Use everything we built up to step 3 of our previous post.
Set up an AWS IoT rule to route shadow updates to ElasticSearch
Use Kibana to create visualisations and alerts

Create ElasticSearch Instance

We will use the AWS ElasticSearch service that sets up ElasticSearch and Kibana with ODFE. Since, this is just a demo we will set up a single t2.small node with 10GB disk size under Deployment type = Development and testing. We will use version 7.9 of ElasticSearch which is the latest on the service as of this writing.

AWS seems to be tracking the releases much better these days. The latest release on Elastic.co is 7.10.1.

CRITICAL SECURITY NOTE

Make sure you read through the notes on network configuration and fine-grained access control for ODFE (not available for t2.small). For a production system, you should use an instance larger than t2.small to support encryption at rest and, by extension, access control.

I am using Public access here for the purposes of the demo as this instance will be live only for a couple of hours. Google ElasticSearch breaches for the ever increasing counts of exposed servers.

Create Index and Mapping

An index is the highest logical level within ElasticSearch. Think of these as similar to tables in a database.

You need an index before ingesting data.
You do not need data columns and types to be defined before ingestion.
You need to define mappings for specific columns that you want to transform during ingestion.
You need types and columns defined before analysis or visualisation.

Creating an index needs an HTTP PUT as shown below:

Note that we are specifying in the mapping that our timestamp field should be interpreted by ElasticSearch as a date (really, datetime) field.

That's it, our ElasticSearch instance is ready to receive data.

Persisting Shadow Updates

Go to AWS IoT Core -> Act -> Rules to get started. We will create a new rule SendShadowsToElasticSearch. Like before, we will add a filter and an action.

Filter

We will need add back the timestamp field and we will also need the device_id. So, our SQL query will now look like this:

SELECT 
  state.reported.cpu_usage as cpu_usage,
  state.reported.cpu_freq as cpu_freq,
  state.reported.cpu_temp as cpu_temp,
  state.reported.ram_usage as ram_usage,
  state.reported.ram_total as ram_total,
  state.reported.timestamp as timestamp,
  clientid() as device_id
FROM '$aws/things/+/shadow/update'

Action

ElasticSearch is a first class citizen in the AWS IoT actions list. So, let's select that action and configure it:

Note that we:

Selected the ElasticSearch instance we created in the previous step
Granted access to AWS IoT via an IAM rule
Added an id using the built-in newuuid() function
Specified our index
Specified a type _doc for our data which is the default built-in type for new documents

As before, we will enable the CloudWatch action in case of errors.

Our rule will finally look like this:

Querying and Visualisation Using Kibana

Let's start our simulators again and hop over to the Kibana to start analysing our data. The Kibana URL will be visible in your AWS ElasticSearch Console and should just be https://ES_DOMAIN/_plugin/kibana/.

Before we can query the data, we need to create an index pattern. Kibana should prompt you to create an index pattern as soon as you enter. If you missed that, go to Stack Management --> Index Patterns from the left sidebar.

Click on Create index pattern and walk through the wizard. Since your index has been receiving some data, the columns will already be detected. Ensure that you select timestamp as your date field.

Discover

With the index pattern in place, go to Discover in the sidebar to see your data streaming in.

If you do see similar output, we can continue to visualisations. If you don't,

Check the Cloudwatch Logs for errors
Verify that your SQL syntax is correct - especially the topic
Ensure your Rule action has the right index and an appropriate IAM Role
Verify that your Device Shadow is getting updated by going over to AWS IoT -> Things -> my_iot_device_1 -> Shadow
Look for errors if any on the terminal where you are running the script.

Visualisations & Dashboard

Once we start receiving data, visualisations within Kibana are easy to set up. Go to Visualisations in the sidebar and create a line chart for each of our metrics. Specifically,

Select the index aws_iot_demo as data source
For the Y-axis, select average as the aggregation and the metric, e.g. cpu_usage under field. Give a custom label for the axis if you want.
To enable timeseries visualisation, add an X-axis bucket with aggregation Date Histogram, field timestamp and Auto interval.
To enable distinction by device_id, add another bucket, this time with sub-aggregation Terms, field device_id.keyword and the default Order By options.
- device_id.keyword indicates that ElasticSearch is indexing the strings stored in the device_id column.

Click on Update and your chart should look similar to this:

Create visualisations for the other metrics and add them to a Dashboard from the sidebar. Your dashboard should look something like this:

Unlike QuickSight, Kibana dashboards can be embedded on websites as snapshots.

Alerts

Alerts in Kibana are independent of visualisations, unlike Grafana. They are created in two stages: monitors and triggers.

Monitors

Monitors are queries that run on a schedule. Something like the one shown below. Queries can be defined using the UI/Visual Graph, via an extraction query or even, excitingly, anomaly detection (more on this in the next section).

Triggers

Triggers compare the value returned by the monitor with specific thresholds. If the threshold is crossed, triggers can notify certain endpoints. At the moment, the built-in destinations are limited to Amazon SNS, Amazon Chime, Slack and Custom webhook. We will create a trigger without a destination so we can see the alerts in the UI.

Once an alert has been triggered, it will be activated and will show up on the Alerts Dashboard. Active alerts can be Acknowledged to let the system know that an operator is looking into the issue. Once the metric/query is no longer in breach of the threshold, the alert is Completed. For a detailed description, check the docs.

Anomaly Detection

Anomaly detection is a relatively new feature in ODFE and is getting frequent updates. As of release 7.9 on AWS, you can set up

detectors that query all or some portion of your data at regular intervals, and
models that monitor specific fields or custom queries for anomalies.

We will explore this feature in detail in a future post. For now, see this post and the docs for more on anomaly detection.

Notebooks

Kibana notebooks are an experimental feature that let you build Jupyter-like notebooks with visualisations and descriptive text. It must be stressed that these are nowhere close to the functionality of Jupyter, which of course include the power of a complete programming langauge like Python. However, they are a start and we are excited to see how they progress.

Conclusions

ElasticSearch and Kibana are large, commendable feats of engineering that power everything from our search engines to our on-demand taxi services. The stack's versatility is clear from its suitability for time-series data too.

When comparing QuickSight, Grafana and ElasticSearch+Kibana, we would recommend:

Timestream + QuickSight if you

prefer costs that more linearly scale with use
prefer using AWS in-house offerings
have data in multiple data sources that you need to analyse without duplication
don't need to visualise geospatial data

Timestream + Grafana if you

need timeseries charts and alerts
need gorgeous dashboards with easy (time) filtering
don't need to create complex BI style analyses across multiple data sources
don't need anomaly detection

ElasticSearch + Kibana if you

need flexibility in the type of charts and analyses
need anomaly detection
need to index and search text data too
don't mind duplicating your data into ElasticSearch
don't mind paying fixed monthly costs for your instance
have the ability to secure and monitor your ES instance

Ideas, questions or corrections?

Write to us at hello@iotready.co

From Metal To Alerts With AWS IoT, Timestream and QuickSight

Tej Pochiraju — Wed, 23 Dec 2020 15:19:48 +0000

AWS IoT Demo

A metal-to-alerts example of how to build an IoT enabled monitoring solution using only AWS PaaS offerings.

Why

We have been doing this workflow for 5+ years now, much longer if you count fully custom tools. So have countless others. Yet, if you asked a new engineer to build this, you can wave goodbye to a few weeks of their effort as they navigate obtuse documentation and outdated StackOverflow answers. This is our attempt to document what's possible with PaaS tools in 2020.

What are we going to build?

To achieve this, we will:

Write a Python script that monitors system metrics (CPU, Memory, Temperature, Fan)
1. You could replace this with actual hardware, perhaps run this script on an Raspberry Pi or even send FreeRTOS metrics from ESP32 but we will save that for another day.
Create multiple things on AWS IoT Core.
Send these metrics as shadow updates to AWS IoT every 10s (configurable)
Configure AWS IoT to route shadow updates to a database
Set up a visualisation tool and create dashboards using these updates
Add a few simple alerts on our visualiation tool to send notifications if system metrics cross a threshold

Git Repo Structure

master = final code + AWS IoT configuration + Grafana dashboard JSON
1_python_script = Python script without AWS IoT integration (print to console)
2_aws_iot = Python script with AWS IoT integration (shadow updates)

1 - Python Script For System Metrics

We are going to adapt this excellent blog post to create our system monitor script.

python3 -m venv venv
source venv/bin/activate
echo psutil==5.8.0 > requirements.txt
pip install -r requirements.txt
touch sysmon.py

Edit sysmon.py in your preferred text editor and add in the following functions from the blog post:
- get_cpu_usage_pct
- get_cpu_frequency
- get_cpu_temp
- get_ram_usage
- get_ram_total
Next, create a main function that calls each of these functions and populates a dictionary: payload and prints it.
1. We will also add a timestamp to the payload for use in visualisations later.
Add a while(1) that calls this main function every 10 seconds.
Add an argument parser so we can pass the interval and a device_id as command line arguments.
Run the script with python sysmon.py 10 my_iot_device_1

You should see output similar to:

2 - AWS IoT Integration

Now, we will add in the ability to send our metrics to AWS IoT. But first, we need to register our devices or things as AWS calls them.

Registering the devices

We will register the devices individually via the AWS Console. However, if you have a large number of devices to register, you may want to script it or use Bulk Registration via aws-cli or the AWS IoT Core Console.

We are using us-east-1 aka N. Virginia for integration later with Amazon Timestream which is not yet available in all regions.

Click on Create a single thing
1. Give your thing a name, e.g. my_iot_device_1
2. You can skip Thing Type and Group for this demo.
3. Create the thing
Use the One-click certificate creation (recommended) option to generate the certificates.
1. Download the generated certificates and the root CA certificate.
2. Activate the certificates.
Attach a policy and register the Thing.
1. Because we are cavalier and this is a demo, we are using the following PubSubToAny policy.
2. DO NOT use this in production!

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:*",
      "Resource": "*"
    }
  ]
}

Now repeat this a couple more times so we have a few things. I am setting up 3 devices with the imaginative names: my_iot_device_1, my_iot_device_2, my_iot_device_3.

Finally, we will rename our certificates to match our thing names so that it's easier to script together. For instance, I am using the rename utility to bulk rename my certificates:

Adding the AWS IoT SDK

Because AWS IoT supports MQTT, we could use any MQTT client that supports X.509 certificates. However, to keep things simple, we will use the official Python SDK from AWS IoT. Specifically, we will adapt the basicShadowUpdater.py sample.

Please inspect aws_shadow_upater.py for the changes we are making. Primarily, we are wrapping the functionality into 2 functions:
- init_device_shadow_handler that takes AWS IoT specific config parameters and returns a deviceShadowHandler specific to our configuration and thing.
- update_device_shadow that takes our system metrics payload and wraps it into a json structure that AWS IoT expects for device shadows.
We will also take this opportunity to modularise our code a bit by moving the main function from sysmon.py into its own separate file.
Within main.py we are reading our AWS configuration from a combination of environment variables and the local certificates.
- We only need the following: export AWS_IOT_HOST=YOUR_AWS_IOT_ENDPOINT.amazonaws.com and export CERTS_DIR=certs assuming you are keeping your certificates in certs/.
- You will probably want to create a script or .env file to set these environment variables
- For good measure, we are also verifying that the certificates actually exist.
With this done, we stitch our two modules sysmon.py and aws_shadow_updater.py together and start publishing updates. If all goes well, you should see the following in your terminal and your AWS Console (go to Thing -> Shadows -> Classic Shadow)

3 - Simulating Multiple Devices

We are done with almost all of the coding needed to get this working.

This is an easy one, open up multiple terminals/tabs and start a separate process for updating the shadow for each device. Something like this:

4 - Persisting Shadow Updates

In order to visualise, and perhaps analyse, these metrics, we need to persist them in some form of database. Thankfully, AWS IoT has a Rules Engine designed for just this purpose. The Rules Engine is essentially a message router with the ability to filter messages using an SQL syntax and send them to various destinations.

Go to AWS IoT Core -> Act -> Rules to get started.

There are 2 steps to enabling rules:

Filter: Select the messages we want to act on.
Act: Select the action(s) we want to run for each filtered message.

Filter

AWS IoT uses a reduced SQL syntax for filtering messages. Points to note:

The shadow topic we are interested in is $aws/things/thingName/shadow/update where we need to replace thingName with the wildcard +. Follow this reference on topics and wildcards.
The content of each message contains the entire state with desired and reported properties as well as other metadata. We will need to unpack the reported property to get the data we need. Follow this reference for more details.

Our SQL filter will look essentially like this:

SELECT 
  state.reported.cpu_usage as cpu_usage,
  state.reported.cpu_freq as cpu_freq,
  state.reported.cpu_temp as cpu_temp,
  state.reported.ram_usage as ram_usage,
  state.reported.ram_total as ram_total
FROM '$aws/things/+/shadow/update'

Before we can save this rule, we will also need to add an action. Actions define what to do with the filtered messages. This depends on our choice of database.

Act

AWS IoT supports a large range of actions out of the box including CloudWatch, DynamoDB, ElasticSearch, Timestream DB and custom HTTP endpoints. See the full list here.

To confirm that our messages are coming through and we are able to store them, we will use the shiny, new time series database from AWS - Timestream. We will also enable the CloudWatch action in case of errors.

Timestream DB

As of this writing Timestream is only available in 4 regions.

It's essential to create the DB in the same region as your AWS IoT endpoint as the Rules Engine does not, yet, support multiple regions for the built-in actions. _You could use a Lambda function to do this for you but that's more management and cost.

We will create a Standard (empty) DB with the name aws_iot_demo:

We will also need a table to store our data, so let's do that too:

Once this is done, we can return to the rule we started creating earlier and add a new Action.

Notes:

The AWS IoT Rule Action for Timestream needs at least one dimension to be specified. Dimensions can be used for grouping and filtering incoming data.
I used the following key:value pair using a substitution template - device_id: ${clientId()}
We are sending the device timestamp as part of the shadow update. If we include it as part of the SELECT query in the rule, Timestream will assume that timestamp is a measurement metric too.
- Instead, we will ignore the device timestamp and use ${timestamp()} as the time parameter within the Rule Action. This generates a server timestamp.
You will also need to create or select an appropriate IAM role that lets AWS IoT to write to Timestream.
Timestream creates separate rows for each metric so each shadow update creates 5 rows.

CloudWatch

This action is triggered if/when there is an error while processing our rule. Again, follow the guided wizard to create a new Log Group and assign permissions.

At the end, your rule should look something like this:

Query Timestream

Assuming we have started our simulators again, we should start to see data being stored in Timestream. Go over to AWS Console -> Timestream -> Tables -> aws_iot_demo -> Query Table. Type in the following query:

-- Get the 20 most recently added data points in the past 15 minutes. You can change the time period if you're not continuously ingesting data
SELECT * FROM "aws_iot_demo"."aws_iot_demo" WHERE time between ago(15m) and now() ORDER BY time DESC LIMIT 20

You should see output similar to the one below:

You will notice the separate rows for each metric. We will need a different query in order to combine the metrics into a single view, for instance for use with visualisation or analytics tools.

SELECT device_id, BIN(time, 1m) AS time_bin,
    AVG(CASE WHEN measure_name = 'cpu_usage' THEN measure_value::double ELSE NULL END) AS avg_cpu_usage,
    AVG(CASE WHEN measure_name = 'cpu_freq' THEN measure_value::bigint ELSE NULL END) AS avg_cpu_freq,
    AVG(CASE WHEN measure_name = 'cpu_temp' THEN measure_value::double ELSE NULL END) AS avg_cpu_temp,
    AVG(CASE WHEN measure_name = 'ram_usage' THEN measure_value::bigint ELSE NULL END) AS avg_ram_usage,
    AVG(CASE WHEN measure_name = 'ram_total' THEN measure_value::bigint ELSE NULL END) AS avg_ram_total
FROM "aws_iot_demo"."aws_iot_demo"
WHERE time between ago(15m) and now()
GROUP BY BIN(time, 1m), device_id
ORDER BY time_bin desc

Your output should look something like this -

If you do see similar output, you are in business and we can continue to visualisation. If you don't,

Check the Cloudwatch Logs for errors
Verify that your SQL syntax is correct - especially the topic
Ensure your Rule action has the right table and an appropriate IAM Role
Verify that your Device Shadow is getting updated by going over to AWS IoT -> Things -> my_iot_device_1 -> Shadow
Looking for errors if any on the terminal where you are running the script.

Pause For Breath

We have covered a lot of ground. So, let's pause and reflect. Here's what we have done so far:

Created a Python script to monitor common system metrics.
Hooked up this script to AWS IoT using the SDK and Thing certificates.
Simulated running multiples of these devices with sending a Shadow update.
Created a rule to persist these device shadows to Timestream and errors to CloudWatch.
Verified that we are actually getting our data.

Now, we only have the small matter of visualising our data and setting up alerts in case any of our metrics cross critical thresholds.

With a fresh cup of coffee, onwards...

5 - Visualisation

Storage and visualisation are, in fact, two separate operations that need two different software tools. However, these are often so tightly coupled that choice of one often dictates choice of the other. Here's a handy table that illustrates this.

Storage	Visualisation	Comments
Timestream	AWS QuickSight	See demo below
Timestream	Grafana	See demo below
DynamoDB	AWS QuickSight	Needs CSV export to S3 first
DynamoDB	Redash	Works but with limitations, demo in future post
ElasticSearch	Kibana	Works well, demo in future post
ElasticSearch	Grafana	Simpler to just use Kibana
InfluxDB	InfluxDB UI	Works well, demo in future post
InfluxDB	Grafana	Simpler to just use the built-in UI

There are, of course, numerous other ways to do this. We will focus on the first two in that table.

Timestream + AWS QuickSight

QuickSight is a managed BI tool from AWS. The official documentation to integrate Timestream with QuickSight is a little dense. However, it's pretty straightforward if you are using us-east-1 as your region.

Within QuickSight, click on the user icon at the top right and then on Manage QuickSight.
Here, go to Security & Permissions -> QuickSight access to AWS services and enable Timestream (see image below).
Next, within QuickSight, click on New Dataset and select Timestream. Click on Validate Connection to ensure you have given the permissions and confirm.
Upon confirmation, select aws_iot_demo from the discovered databases and select aws_iot_demo from the tables.
Click on Visualise

So far so good. I had to struggle for a while to understand how to get QuickSight to unpack the metrics from Timestream. Turns out, this is surprisingly easy if you follow this tutorial video from AWS. Essentially,

Create multiple visualisations
For each visualisation, add a filter on measure_name.
Click on time to add it to the X-axis. Change period to Aggregrate: Minute.
Click on measure_value::bigint or measure_value::double depending on the metric to add it to the Y-axis. Change to Aggregate: Average.
- In our case, only cpu_usage is a double.
Click on device_id to add separate lines for each device. This is added to Color in QuickSight.

That's it! My dashboard looks like this -

QuickSight is a full-fledged business intelligence (BI) tool with the ability to integrate with multiple data sources. QuickSight also has built-in anomaly detection. This makes it an incredibly powerful tool to use for IoT visualisations and analysis. We could even bring in non-IoT data such as that from an ERP. More on this in a later post!

However, QuickSight:

Does not support alerts
Does not support calculations
Has limited visualisations
Does not support dashboard embeds, e.g. in a webpage
Charges per user

Mind you, the AWS IoT rules engine can be used quite easily for alerts so you don't really need alerts in a separate tool. Having said that...

Timestream + Grafana

Grafana has long been a favourite of anyone looking to create beautiful, lightweight dashboards. Grafana integrates with a zillion data sources via input plugins and has numerous visualisations via output plugins. Grafana only does visualisations and alerts but does it really well. There are open source and enterprise editions available.

AWS has an upcoming managed Grafana service. Until then, we will use the managed service from Grafana Cloud. You could also spin up Grafana locally or on a VM somewhere with the docker image.

There's a video tutorial available but you will need to adapt a fair bit to our example. Assuming you have either signed up for Grafana Cloud or installed it locally, you should now:

Install the Amazon Timestream plugin
Back in Grafana, add a new Data Source and search for Timestream.
For authentication, we will use Access Key and Secret for a new IAM User.
- Back in AWS, create a new user with the ~AmazonTimestreamReadOnlyAccess policy attached~ admin rights. For some reason, Grafana would not connect to Timestream even with the AmazonTimestreamFullAccess policy attached.
Once the keys are in place, click on Save & Test
Select aws_iot_demo in the $_database field to set up the default DB. Try as I might, I could not get the dropdown for $_table to populate.

Now, click on + -> Dashboard and + Add new panel to get started.

Unlike QuickSight, Grafana allows you to build queries using SQL. So, for our first panel, let's create a CPU Usage chart with the following query:

SELECT device_id,
    CREATE_TIME_SERIES(time, measure_value::double) as avg_cpu_usage
FROM $__database.$__table
WHERE $__timeFilter
    AND measure_name = '$__measure'
GROUP BY device_id

We are essentially doing the same as QuickSight by defining a where clause to filter by metric and creating a time series that is grouped by device_id. The one, big, difference is the Grafana allows you to add multiple such queries to a single visualisation chart (panel in Grafana speak). Duplicating this panel and making the mods necessary, we end up with a dashboard very similar to that in QuickSight.

Notice that we get dashboard wide time controls for free!

Alerts With Grafana

Creating alerts with Grafana is surprisingly easy. Alerts use the same query as the panel and are created in the same UI.

By default, the alert is triggered on the average of the metric but you can change it a different calculation.

If you have multiple queries on a panel, you can even use a combination of queries!

Alerts can trigger notifications to various channels with built-in options for all the major chat apps, email and webhooks. For instance, if you wanted to trigger a notification within a mobile app, you would set up an API somewhere that will be triggered by a webhook configuration within Grafana. Your API is then responsible for notifying the mobile app.

For more on Grafana alerts, check out the docs.

Conclusions

Once you get around the verbose documentation, and refine your search skills, it's quite straightforward to create an end-to-end flow for most IoT use cases using purely platform-as-a-service offerings.

We have been running a deployment on AWS for a customer with ~46000 devices for 2+ years now, handling 15-20M messages monthly. All this for a fraction of the cost and attention this would need if we ran the infrastructure ourselves.

That said, I do have a few reservations and will explore those in future posts.

Ideas, questions or corrections?

Write to us at hello@iotready.co