MQTT

MQTT (Message Queuing Telemetry Transport) is a lightweight and widely adopted messaging protocol that is designed for constrained devices. AWS IoT Core support for MQTT is based on the MQTT v3.1.1 specification and the MQTT v5.0 specification, with some differences, as documented in AWS IoT differences from MQTT specifications. As the latest version of the standard, MQTT 5 introduces several key features that make an MQTT-based system more robust, including new scalability enhancements, improved error reporting with reason code responses, message and session expiry timers, and custom user message headers. For more information about MQTT 5 features that AWS IoT Core supports, see MQTT 5 supported features. AWS IoT Core also supports cross MQTT version (MQTT 3 and MQTT 5) communication. An MQTT 3 publisher can send an MQTT 3 message to an MQTT 5 subscriber that will be receiving an MQTT 5 publish message, and vice versa.

AWS IoT Core supports device connections that use the MQTT protocol and MQTT over WSS protocol and that are identified by a client ID. The AWS IoT Device SDKs support both protocols and are the recommended ways to connect devices to AWS IoT Core. The AWS IoT Device SDKs support the functions necessary for devices and clients to connect to and access AWS IoT services. The Device SDKs support the authentication protocols that the AWS IoT services require and the connection ID requirements that the MQTT protocol and MQTT over WSS protocols require. For information about how to connect to AWS IoT using the AWS Device SDKs and links to examples of AWS IoT in the supported languages, see Connecting with MQTT using the AWS IoT Device SDKs. For more information about authentication methods and the port mappings for MQTT messages, see Protocols, port mappings, and authentication.

While we recommend using the AWS IoT Device SDKs to connect to AWS IoT, they are not required. If you do not use the AWS IoT Device SDKs, however, you must provide the necessary connection and communication security. Clients must send the Server Name Indication (SNI) TLS extension in the connection request. Connection attempts that don't include the SNI are refused. For more information, see Transport Security in AWS IoT. Clients that use IAM users and AWS credentials to authenticate clients must provide the correct Signature Version 4 authentication.

Connecting with MQTT using the AWS IoT Device SDKs

This section contains links to the AWS IoT Device SDKs and to the source code of sample programs that illustrate how to connect a device to AWS IoT. The sample apps linked here show how to connect to AWS IoT using the MQTT protocol and MQTT over WSS.

MQTT Quality of Service (QoS) options

AWS IoT and the AWS IoT Device SDKs support the MQTT Quality of Service (QoS) levels 0 and 1. The MQTT protocol defines a third level of QoS, level 2, but AWS IoT does not support it. Only the MQTT protocol supports the QoS feature. HTTPS supports QoS by passing a query string parameter ?qos=qos where the value can be 0 or 1.

This table describes how each QoS level affects messages published to and by the message broker.

MQTT persistent sessions

Persistent sessions store a client’s subscriptions and messages, with a Quality of Service (QoS) of 1, that haven't been acknowledged by the client. When the device reconnects to a persistent session, the session resumes, subscriptions are reinstated, and unacknowledged subscribed messages received and stored prior to the reconnection are sent to the client.

The processing of the stored messages is recorded in CloudWatch and CloudWatch Logs. For information about the entries written to CloudWatch and CloudWatch Logs, see Message broker metrics and Queued log entry.

Creating a persistent session

In MQTT 3, you create an MQTT persistent session by sending a CONNECT message and setting the cleanSession flag to 0. If no session exists for the client sending the CONNECT message, a new persistent session is created. If a session already exists for the client, the client resumes the existing session. To create a clean session, you send a CONNECT message and set the cleanSession flag to 1, and the broker will not store any session state when the client disconnects.

In MQTT 5, you handle persistent sessions by setting the Clean Start flag and Session Expiry Interval. Clean Start controls the beginning of the connecting session and the end of the previous session. When you set Clean Start = 1, a new session is created and a previous session is terminated if it exists. When you set Clean Start= 0, the connecting session resumes a previous session if it exists. Session Expiry Interval controls the end of the connecting session. Session Expiry Interval specifies the time, in seconds (4-byte integer), that a session will persist after disconnect. Setting Session Expiry interval=0 causes the session to terminate immediately upon disconnect. If the Session Expiry Interval is not specified in the CONNECT message, the default is 0.

In MQTT 5, if you set Clean Start = 1 and Session Expiry Interval = 0, this is the equivalent of an MQTT 3 clean session. If you set Clean Start = 0 and Session Expiry Interval> 0, this is the equivalent of an MQTT 3 persistent session.

Operations during a persistent session

Clients use the sessionPresent attribute in the connection acknowledged (CONNACK) message to determine if a persistent session is present. If sessionPresent is 1, a persistent session is present and any stored messages for the client are delivered to the client after the client receives the CONNACK, as described in Message traffic after reconnection to a persistent session. If sessionPresent is 1, the client does not need to resubscribe. However, if sessionPresent is 0, no persistent session is present and the client must resubscribe to its topic filters.

After the client joins a persistent session, it can publish messages and subscribe to topic filters without any additional flags on each operation.

Message traffic after reconnection to a persistent session

A persistent session represents an ongoing connection between a client and an MQTT message broker. When a client connects to the message broker using a persistent session, the message broker saves all subscriptions that the client makes during the connection. When the client disconnects, the message broker stores unacknowledged QoS 1 messages and new QoS 1 messages published to topics to which the client is subscribed. Messages are stored according to account limit. Messages that exceed the limit will be dropped. For more information about persistent message limits, see AWS IoT Core endpoints and quotas. When the client reconnects to its persistent session, all subscriptions are reinstated and all stored messages are sent to the client at a maximum rate of 10 messages per second. In MQTT 5, if an outbound QoS1 with the Message Expiry Interval expires when a client is offline, after the connection resumes, the client won't receive the expired message.

After reconnection, the stored messages are sent to the client, at a rate that is limited to 10 stored messages per second, along with any current message traffic until the Publish requests per second per connection limit is reached. Because the delivery rate of the stored messages is limited, it will take several seconds to deliver all stored messages if a session has more than 10 stored messages to deliver after reconnection.

Ending a persistent session

Persistent sessions can end in the following ways:

In MQTT 3, the default value of persistent sessions expiration time is an hour, and this applies to all the sessions in the account.

In MQTT 5, you can set the Session Expiry Interval for each session on CONNECT and DISCONNECT packets.

For Session Expiry Interval on DISCONNECT packet:

Reconnection after a persistent session has expired

If a client doesn't reconnect to its persistent session before it expires, the session ends and its stored messages are discarded. When a client reconnects after the session has expired with a cleanSession flag to 0, the service creates a new persistent session. Any subscriptions or messages from the previous session are not available to this session because they were discarded when the previous session expired.

Persistent session message charges

Messages are charged to your AWS account when the message broker sends a message to a client or an offline persistent session. When an offline device with a persistent session reconnects and resumes its session, the stored messages are delivered to the device and charged to your account again. For more information about message pricing, see AWS IoT Core pricing - Messaging.

The default persistent session expiration time of one hour can be increased by using the standard limit increase process. Note that increasing the session expiration time might increase your message charges because the additional time could allow for more messages to be stored for the offline device and those additional messages would be charged to your account at the standard messaging rate. The session expiration time is approximate and a session could persist for up to 30 minutes longer than the account limit; however, a session will not be shorter than the account limit. For more information about session limits, see AWS Service Quotas.

MQTT retained messages

AWS IoT Core supports the RETAIN flag described in the MQTT protocol. When a client sets the RETAIN flag on an MQTT message that it publishes, AWS IoT Core saves the message. It can then be sent to new subscribers, retrieved by calling the GetRetainedMessage operation, and viewed in the AWS IoT console.

Common tasks with MQTT retained messages in AWS IoT Core

AWS IoT Core saves MQTT messages with the RETAIN flag set. These retained messages are sent to all clients that have subscribed to the topic, as a normal MQTT message, and they are also stored to be sent to new subscribers to the topic.

MQTT retained messages require specific policy actions to authorize clients to access them. For examples of using retained message policies, see Retained message policy examples.

This section describes common operations that involve retained messages.

Billing and retained messages

Publishing messages with the RETAIN flag set from a client, by using AWS IoT console, or by calling Publish incurs additional messaging charges described in AWS IoT Core pricing - Messaging.

Retrieving retained messages by a client, by using AWS IoT console, or by calling GetRetainedMessage incurs messaging charges in addition to the normal API usage charges. The additional charges are described in AWS IoT Core pricing - Messaging.

MQTT Will messages that are published when a device disconnects unexpectedly incur messaging charges described in AWS IoT Core pricing - Messaging.

For more information about messaging costs, see AWS IoT Core pricing - Messaging.

Comparing MQTT retained messages and MQTT persistent sessions

Retained messages and persistent sessions are standard features of MQTT that make it possible for devices to receive messages that were published while they were offline. Retained messages can be published from persistent sessions. This section describes key aspects of these features and how they work together.

For information about persistent sessions, see MQTT persistent sessions.

With Retained Messages, the publishing client determines whether a message should be retained and delivered to a device after it connects, whether it had a previous session or not. The choice to store a message is made by the publisher and the stored message is delivered to all current and future clients that subscribe with a QoS 0 or QoS 1 subscriptions. Retained messages keep only one message on a given topic at a time.

When an account has stored the maximum number of retained messages, AWS IoT Core returns a throttled response to messages published with RETAIN set and payloads greater than 0 bytes until some retained messages are deleted and the retained message count falls below the limit.

MQTT retained messages and AWS IoT Device Shadows

Retained messages and Device Shadows both retain data from a device, but they behave differently and serve different purposes. This section describes their similarities and differences.

MQTT Last Will and Testament (LWT) messages

Last Will and Testament (LWT) is a feature in MQTT. With LWT, clients can specify a message which the broker will publish to a client-defined topic and send to all clients that subscribed to the topic when an uninitiated disconnection occurs. The message that clients specify is called an LWT message or a Will Message, and the topic that clients define is referred to as a Will Topic. You can specify an LWT message when a device connects to the broker. These messages can be retained by setting the Will Retain flag in the Connect Flag bits field during the connection. For example, if the Will Retain flag is set to 1, a Will Message will be stored in the broker in the associated Will Topic. For more information, see Will Messages.

The broker will store the Will Messages until an uninitiated disconnection occurs. When that happens, the broker will publish the messages to all clients that subscribed to the Will Topic to notify the disconnection. If the client disconnects from the broker with a client-initiated disconnection using the MQTT DISCONNECT message, the broker won't publish the stored LWT messages. In all other cases, the LWT messages will be dispatched. For a complete list of the disconnect scenarios when the broker will send the LWT messages, see Connect/Disconnect events.

Using connectAttributes

ConnectAttributes allow you to specify what attributes you want to use in your connect message in your IAM policies such as PersistentConnect and LastWill. With ConnectAttributes, you can build policies that don't give devices access to new features by default, which can be helpful if a device is compromised.

connectAttributes supports the following features:

By default, your policy has a non-persistent connection and there are no attributes passed for this connection. You must specify a persistent connection in your IAM policy if you want to have one.

For ConnectAttributes examples, see Connect Policy Examples.

MQTT 5 supported features

AWS IoT Core support for MQTT 5 is based on the MQTT v5.0 specification with some differences as documented in AWS IoT differences from MQTT specifications.

Shared Subscriptions

AWS IoT Core supports Shared Subscriptions for both MQTT 3 and MQTT 5. Shared Subscriptions allow multiple clients to share a subscription to a topic and only one client will receive messages published to that topic using a random distribution. Shared Subscriptions can effectively load balance MQTT messages across a number of subscribers. For example, say you have 1,000 devices publishing to the same topic, and 10 backend applications processing those messages. In that case, the backend applications can subscribe to the same topic and each would randomly receive messages published by the devices to the shared topic. This is effectively "sharing" the load of those messages. Shared Subscriptions also allow for better resiliency. When any backend application disconnects, the broker distributes the load to remaining subscribers in the group.

To use Shared Subscriptions, clients subscribe to a Shared Subscription's topic filter as follows:

$share/{ShareName}/{TopicFilter}

Subscriptions that have the same {ShareName}/{TopicFilter} belong to the same Shared Subscription group. You can create multiple Shared Subscription groups and don't exceed the Shared Subscriptions per group limit. For more information, see AWS IoT Core endpoints and quotas from the AWS General Reference.

The following tables compare Non-shared Subscriptions and Shared Subscriptions:

sports/tennis
sports/#

$share/consumer/sports/tennis
$share/consumer/sports/#

Non-shared Subscriptions flow	Shared Subscriptions flow

Important notes for using Shared Subscriptions

For more information about Shared Subscriptions limits, see AWS IoT Core endpoints and quotas from the AWS General Reference. To test Shared Subscriptions using the AWS IoT MQTT client in the AWS IoT console, see Testing Shared Subscriptions in the MQTT client. For more information about Shared Subscriptions, see Shared Subscriptions from the MQTTv5.0 specification.

Clean Start and Session Expiry

You can use Clean Start and Session Expiry to handle your persistent sessions with more flexibility. A Clean Start flag indicates whether the session should start without using an existing session. A Session Expiry interval indicates how long to retain the session after a disconnect. The session expiry interval can be modified at disconnect. For more information, see MQTT persistent sessions.

Reason Code on all ACKs

You can debug or process error messages more easily using the reason codes. Reason codes are returned by the message broker based on the type of interaction with the broker (Subscribe, Publish, Acknowledge). For more information, see MQTT reason codes. For a complete list of MQTT reason codes, see MQTT v5 specification.

Topic Aliases

You can substitute a topic name with a topic alias, which is a two-byte integer. Using topic aliases can optimize the transmission of topic names to potentially reduce data costs on metered data services. AWS IoT Core has a default limit of 8 topic aliases. For more information, see AWS IoT Core endpoints and quotas from the AWS General Reference.

Message Expiry

You can add message expiry values to published messages. These values represent the message expiry interval in seconds. If the messages haven't been sent to the subscribers within that interval, the message will expire and be removed. If you don't set the message expiry value, the message will not expire.

On the outbound, the subscriber will receive a message with the remaining time left in the expiry interval. For example, if an inbound publish message has a message expire of 30 seconds, and it's routed to the subscriber after 20 seconds, the message expiry field will be updated to 10. It is possible for the message received by the subscriber to have an updated MEI of 0. This is because as soon as the time remaining is 999 ms or less, it will be updated to 0.

In AWS IoT Core, the minimum message expiry interval is 1. If the interval is set to 0 from the client side, it will be adjusted to 1. The maximum message expiry interval is 604800 (7 days). Any values higher than this will be adjusted to the maximum value.

In cross version communication, the behavior of message expiry is decided by MQTT version of the inbound publish message. For example, a message with message expiry sent by a session connected via MQTT5 can expire for devices subscribed with MQTT3 sessions. The table below lists how message expiry supports the following types of publish messages:

Other MQTT 5 features

Server disconnect

When a disconnection happens, the server can proactively send the client a DISCONNECT to notify connection closure with a reason code for disconnection.

Request/Response

Publishers can request a response be sent by the receiver to a publisher-specified topic upon reception.

Maximum Packet Size

Client and Server can independently specify the maximum packet size that they support.

Payload format and content type

You can specify the payload format (binary, text) and content type when a message is published. These are forwarded to the receiver of the message.

MQTT 5 properties

MQTT 5 properties are important additions to the MQTT standard to support new MQTT 5 features such as Session Expiry and the Request/Response pattern. In AWS IoT Core, you can create rules that can forward the properties in outbound messages, or use HTTP Publish to publish MQTT messages with some of the new properties.

The following table lists all the MQTT 5 properties that AWS IoT Core supports.

MQTT reason codes

MQTT 5 introduces improved error reporting with reason code responses. AWS IoT Core may return reason codes including but not limited to the following grouped by packets. For a complete list of reason codes supported by MQTT 5, see MQTT 5 specifications.

AWS IoT differences from MQTT specifications

The message broker implementation is based on the MQTT v3.1.1 specification and the MQTT v5.0 specification, but it differs from the specifications in these ways: