MQTT - Large scale control!

Isn’t this some “home labber fad?”

Short answer no. But it is used extensively in homelabs. BUT it’s not some flaky bit of homebrew code (no home lab community has the money it cost!)

A little history

MQTT was built by scientists working for IBM in the 1990s. It was originally built for checking information from oil pipelines (those folks have some money!) The issues it addressed – oil pipelines are long; they are in geographically remote areas. Therefore, satellite telemetry was needed. Satellite time is expensive (oil companies are rich) but rich people don’t stay rich by throwing money to the wind. MQTT is beautifully lightweight. Now, fast forward to 2010. IBM released the specification royalty-free, the code went to the Eclipse Foundation, and the standard to OASIS – to keep it open and free to the world. This doesn’t mean people aren’t allowed to make money using it! Then (a little later on) … it became the quiet backbone for close to everything!

MQTT isn’t centralised (well no more than the Public Internet.) Anyone can download an MQTT broker server, spin it up on a VM and jump in!

MQTT uses a “publish/subscribe” technique. My client computer creates or joins a topic. If it is allowed into the topic (the brokers role) it then receives everything in that topic that is published to the broker. Maybe this might ring a few bells? If not… many MQTT implementations (including on graphical brokers!) show three bouncing dots when someone is writing to a topic in their UI… warmer? YES, it is used for all kinds of things for messaging, like Facebook Messenger, which famously runs on MQTT…

OK Instant messaging is cool, so what?

Well, because it was built for oil pipelines, the very smart people who built it originally didn’t go to tick boxes on a rigid specification, they were wise enough to keep its data structures fluid – the beginnings of the now “normal” flexible approach. If, for some reason, it doesn’t appear to support a particular datatype, it will let you put a JSON payload into a message... what is JSON I hear you say? Well essentially it’s every computer persons dream – I don’t know what will happen tomorrow, so lets just wrap it all up in a standard package, and allow that to be the flexible layer. We can communicate arrays wrapped up in nice curly brackets {} the beauty is MQTT doesn’t care about the meaning.

This means that I could send something like OSC for show control over MQTT and the data structure simply “pops out” of the client, and does its thing. Whilst NOT a discovery mechanism in and of itself (cough) what if I built a JSON structure, that contained all of the information I need to transmit about a device to the topic at the broker? A client listening to that Topic would receive that message, and from the jsonb know all about the device advertising itself… feels like discovery to me!

So why isn’t everyone doing It if it’s so great?

They are, or are planning to! In audio visual networks, we didn’t need it (if I’m frank). For 20 years we had mdns or “bonjour”. This works on a small scale – up to 200 devices. The client/server relationship was completely decentralised! This is actually the perfect answer for small standalone networks, and those that pioneered the use, were spot on for the time. A broker is a bit more of an undertaking. However not that much. A raspberry pi can form a very decent broker, and run of batteries if in a pinch! If deployed on datacentre servers, a single big broker can handle tens of millions of clients! Your Motorola R7 digital radio uses it already! Your Smart home- from lightbulbs… (sorry lamps) to your aircon and heat pump all use it.

Ah smarthome – too slow for theatre!

OK, I am fully aware of what it looks like. Consider that… your voice has to be changed to a command (this happens in a datacentre) this command then executes through the broker (hopefully on the same datacentre) however – Tuya are the big player in smart lamps, and their broker is (for here) in AWS in Luxembourg. So my google home server has to send a message from the google home broker (I suspect the nearest one to me is in Slough) which in turn has to talk to Luxembourg, and then back to my lamp in my home… no wonder it takes a moment.

In a live environment – we’d probably use the whisper API on premises to do the speech to text, run it into a local broker (in the same cluster) and then… it feels like (and I always laugh at the paradox) an “analogue” light switch (perhaps the most digital thing possible!)

But is it secure?

If you take your own broker, and run it properly – you can encrypt using the normal Internet methods. Remember, 2 minutes ago, you were happy with mdns! MQTT can be secured by certificates which you can generate yourself! I don’t know why you would want a root-CA signed public Certificate, but you can do that too if you so desire! And, if you design the topic structures and access profiles properly, then you can be assured that nobody else can see the information flow!

So, this “new thing” is it proven “reliable”?

  1. It’s not new
  2. It’s used by a lot of things
  3. It’s open source, this means there is a huge community able to fix bugs as and when they occur.

And the beauty – no silly IP addressing – your client device (if its searching via fqdn (human names instead of IP address numbers) can route its traffic, change its IP address, and carry on working! That is why google and Tuya can communicate between different countries!

← All articles