Forem: ZP X

(MQTT Series) Part 3 - Publishing Subscribing and Topics

ZP X — Fri, 10 May 2024 11:28:12 +0000

Following up on the last introduction, let's discuss some basic concepts of MQTT.

Basic Concepts

In the very simple MQTT Hello World last time, we actually touched on a very important concept: publishing and subscribing.

It's easy to recall from design patterns, indeed, MQTT fundamentally implements an architectural publish-subscribe pattern.

Let's recall, where's the benefit of the publish-subscribe pattern? Decoupling. If the observer pattern is a low coupling between sender and receiver, then the publish-subscribe pattern completely decouples them.

Difference from Message Queues

Then what comes to mind are the various message queues in distributed applications (such as ActiveMQ, RabbitMQ, RocketMQ, Kafka, etc.), and it's easy to mistakenly think that they are similar, but their application scenarios and ranges are completely different.

First, it's important to understand that MQTT is just an application layer protocol, comparable to the AMQP protocol in message queues, with MQTT Broker corresponding to various message queues.

Cloud message queue middleware communication protocols are more complex and do not need to consider complex network conditions, but MQTT is much simpler and requires less memory and network resources;
Cloud message queue middleware communication protocols need to store messages, which will be stored indefinitely without client subscriptions, serving purposes like message buffering and smoothing peaks and valleys, whereas MQTT does not store messages, directly discarding them if there are no subscribers;
MQTT clients will receive messages as long as they subscribe to a topic with data, but this is not necessarily the case with message queues, not only do the queues need to be created first, but in the case of multiple clients subscribing to the same queue, each message will be received by only one client;

At this point, they can actually be used in combination, such as devices transmitting data to servers via the MQTT protocol, then placing it into message queues for caching to prevent data loss if the server cannot process timely.

Speaking of which, actually, ActiveMQ supports MQTT, and RabbitMQ also supports MQTT, see more details in MQTT Adapter.

Topic

Topics in MQTT are easy to understand, you can think of them like paths in HTTP protocol or Linux, but you need to remove the first "root directory" because it represents an empty root directory in MQTT.

You can send any data to any topic if you have the permission, and you can also subscribe, but note three symbols:

'+' represents a single-level directory match, it can only be placed between directories, not combined with other characters;
1. Valid examples:
  1. a/b/c/+
  2. a/+/c
  3. a/+/c/+/e
2. Invalid examples:
  1. a/b/c+
  2. a+
  3. a/+b
'#' represents a multi-level directory match, it can only be the last part of a subscription topic, if there is content before it, it must have a '/', you can also think of it as subscribing to all topics with its preceding content as a prefix;
1. Valid examples:
  1. #
  2. a/#
  3. a/b/c/#
2. Invalid examples:
  1. a#
  2. #a
  3. #/a/b
'$' is a reserved prefix for internal topics, even if you subscribe with a single '#', the Broker will not send them to you unless you explicitly subscribe, like the common $SYS topics;

Additionally, aside from testing, try not to subscribe to the '#' topic, as it's likely to cause problems when the client sends too much data.

Example

Before continuing, it's best to set up your own local test Broker to avoid interference from other people's messages on public servers.

Below, we'll use Go as an example to demonstrate message publishing and receiving.

The most commonly used library currently is paho.mqtt.golang, which can be obtained directly by using:

go get github.com/eclipse/paho.mqtt.golang

As an MQTT client, the first thing to do is establish a connection.

opts := mqtt.NewClientOptions().
  AddBroker("tcp://localhost:1883").
  SetClientID("test-client-id")

c := mqtt.NewClient(opts)
if token := c.Connect(); token.Wait() && token.Error() != nil {
  panic(token.Error())
}

defer c.Disconnect(250)

time.Sleep(time.Second)

In the example above, we established a connection with the simplest options and disconnected after a second. If you're interested in the options here, you can see MQTT Client options, where the default options are clear at a glance.

Next is publishing and subscribing, below is a very simple example:

{
    token := c.Subscribe("testtopic/#", 0, func(c mqtt.Client, m mqtt.Message) {
        fmt.Println(string(m.Payload()))
    })
    token.Wait()
    if token.Error() != nil {
       fmt.Println(token.Error())
       os.Exit(1)
    }
}

{
    token := c.Publish("testtopic/123", 0, false, "Hello world")
    token.Wait()
}

time.Sleep(10 * time.Second)

Alternatively, you can also try linking publishing and subscribing as mentioned in the first article, such as sending data on the program and receiving on the desktop client, and vice versa.

Finally

In this introductory article, we omitted connection parameters, as well as QoS and Retained two parameters during publishing and subscribing, which are very important details. They will appear in future articles (rest assured, we will let your descendants notify you of updates 🙈).

(MQTT Series) Part 2 - Setting Up a Broker

ZP X — Fri, 10 May 2024 11:24:20 +0000

In my last introduction, I briefly mentioned how to use a public Broker for testing. Obviously, you can't use a test server as a production environment server; you need one of your own.

Mosquitto

Mosquitto is arguably the most famous open-source MQTT Broker, with just enough functionality. Some advanced features like permission management require the installation of plugins, or even custom plugin development to extend its capabilities.

It also offers a public Broker for testing: https://test.mosquitto.org/

Installing it is very straightforward, just install the appropriate package, for example, on Mac brew install mosquitto, and on Linux sudo api install mosquitto. If you prefer Docker, the official image is eclipse-mosquitto. I'll skip the running details and focus mainly on its configuration¹:

Listening on the default unencrypted port 1883:

    listener 1883 0.0.0.0

If you don't want to configure user password login, here you can configure to allow anonymous connections, meaning no user password:

    allow_anonymous true

But if you configured to disallow anonymous access, then you need to set up username and password. The user password in this file can be configured using the tool provided by mosquitto: mosquitto_passwd mosquitto/config/pwfile username, and then follow the prompt to enter the password.

Additionally, you need to add this line in the configuration file:

    password_file /mosquitto/config/pwfile

Furthermore, if you need to restrict permissions for each user, you need to configure an ACL:

    acl_file /mosquitto/config/aclfile

This configuration is simple, it supports three syntaxes:

topic [read|write|readwrite|deny] <topic>, this can set permissions for anonymous client topics;
user <username>, this is used in conjunction with topic permissions;
pattern [read|write|readwrite] <topic>, this can be used for individual user permissions, where <topic> can contain %c representing the logged-in Client ID and %u representing the username;

Here's an example:²

Allow anonymous users to read all user-level topics:

    topic read #
    topic read $SYS/broker/messages/#

Allow user 'web' to read all topics:

    user web
    topic read #
    topic read $SYS/#

Clearly, this level of permissions only satisfies the most basic requirements. If you need to integrate with your platform to implement dynamic login authentication, you would need to use an auth_plugin. One officially recommended plugin is mosquitto-go-auth.

Clustering

Mosquitto itself does not support cluster deployment, but it can be implemented through the backend, see MQTT server support for details.

TLS Certificates

With increasing national requirements for privacy protection, encrypted transmission is becoming an increasingly important component, meaning all personal information transmission must be encrypted.

For MQTT, HTTPS certificates can be used because fundamentally, they are both TLS certificates and thus can be applied to MQTT as well.

If you use a certificate issued and signed by an authoritative CA, simple configuration would be:

    listener 8883 0.0.0.0
    certfile /path/to/certs/example.com.cer
    keyfile /path/to/certs/example.com.key

But if using a self-signed certificate, the client connection process is a bit more complex, requiring proper CA configuration.

Like HTTPS mutual authentication, MQTT can also use mutual authentication. In this case, when a client connects, the server will require the client to provide a certificate and use your configured CA certificate to verify the client certificate's signature.

    cafile /path/to/certs/ca.pem
    require_certificate true

Testing

Once setup is complete, you can perform simple tests using a client. However, after a basic test, most people might think it's ready for full use, but you can do more to ensure reliability.

For instance, you might estimate the number of client connections you need, the number of messages, concurrency, and message sizes to get a general range, and then perform benchmark testing.

I used the MQTT benchmarking tool, which easily tests the stress your newly setup Broker can handle.

It's fairly user-friendly; if you've used HTTP benchmarking tools like Apache Bench, you'll quickly get the hang of it. For example, from its homepage, a typical scenario is 10 clients, each sending 100 consecutive messages:

mqtt-benchmark --broker tcp://broker.local:1883 --count 100 --clients 10 --qos 1 --topic house/bedroom/temperature --payload {\"temperature\":20,\"timestamp\":1597314150}

In the output, you'll see the results of the test and can identify potential issues that were not apparent during setup. Although spending an extra hour or two might seem wasteful, discovering these issues after some usage would cost much more than these additional hours. Plus, I believe the difference between engineers isn't just in speed but in such professional diligence.

P.S.

You could also consider using a paid service to avoid maintenance labor and server costs. For instance, you could choose commercial Brokers like China's EMQX and international HiveMQ. They support both commercial and open-source versions, and you can either set up on your own servers or use their provided servers. Being commercially supported, they offer more robust features and generally a better experience.

Ref

Original published at https://blog.xizhibei.me/en/2021/10/31/mqtt-2-mosquitto-broker-setup/

(MQTT Series) Part 1 - Introduction: Hello World

ZP X — Fri, 10 May 2024 11:19:01 +0000

Introduction to MQTT

MQTT is a very simple protocol, originally designed in 1999 by two IBM engineers, Andy Stanford-Clark and Arlen Nipper, for monitoring oil pipelines. It was designed for scenarios with limited bandwidth, lightweight, and very low power consumption. At that time, satellite bandwidth was just so small and painfully expensive.¹

In modern society, although the cost of bandwidth has greatly decreased, there are still many scenarios where this protocol is needed, such as in smart homes (still part of IoT). Many small IoT devices rely on a button cell battery to function for years, making MQTT very suitable as an application layer transmission protocol.

In summary, MQTT is a client-server architecture publish-subscribe messaging transmission protocol. It is very lightweight, open, and simple, making it very easy to implement. These characteristics make it highly suitable for fields like machine-to-machine (M2M) and Internet of Things (IoT), which are limited by small memory and narrow bandwidth.²

IBM submitted version 3.1 to OASIS in 2013, and in 2014, OASIS made minor changes and released version 3.1.1.

In 2019, OASIS added many features to MQTT, such as better error handling, shared subscriptions, message content types, etc., and upgraded to version 5. These features will be discussed in dedicated chapters later on.

Hello World

First, you need an MQTT Broker. Install mosquitto … oh? You don't know what that is? Okay, let's try a simpler approach.

First, we can use some public ones, like China's EMQ (Hangzhou Yingyun Technology Co., Ltd.) provides broker.emqx.io (I must advertise for domestic software here, their MQTTX client is the most user-friendly I've used so far, and the Broker's features are also very powerful, I plan to specifically introduce server setup in a separate article).

Then, their MQTTX client, implemented in Electron, supports all platforms, just download and use.

Open the MQTTX client, let's start a simple test.

Click the + on the left sidebar to create a connection (this +, I think does not conform to interaction logic, as a creation button it should be a different level from other buttons, better placed together with new group creation);
A creation page will then pop up, fill in a name randomly, and click connect. If there are no network issues, you should be able to connect successfully (see, they know you're lazy, all the details like Broker address and port are filled in for you, which is also very valuable for us making tech products, on how to let users start using the product with the lowest cost);
Now, let's create a subscription, click add subscription on the page, in the popup dialog, fill in a somewhat random topic like test/907839342134 to avoid conflicts with others, as this is a public Broker, then click confirm.
Finally, let's publish a message. In the bottom left corner, there's an input box prompting you to enter a Topic, we enter test/907839342134, and in the content box below it, enter {"hello": "world"}, click the paper airplane below, and after sending, you will see that you have received the message you sent to yourself.

That's it for now, a very simple introduction. In the next installments, I will introduce MQTT concepts, principles, and practical applications in more detail.

Ref

Original published at https://blog.xizhibei.me/en/2021/08/29/mqtt-1-intro-hello-world/