IoT Architecture with AWS: Processing MQTT Messages with Persistence and API

This article presents a practical, scalable, and cost-effective solution for handling large volumes of IoT messages using AWS serverless services. It originated from a real business demand: an environmental services company needed to receive information in near real-time (though not necessarily process it immediately) from geographically distributed sensors, using MQTT as the communication protocol. Below, we detail the main challenges, the proposed architecture, service choices, and the benefits delivered.

Business Challenge

The company monitors environmental indicators for large clients, such as hydroelectric power plants. The main requirement was to capture and store field sensor data — for example, river levels — which send MQTT messages periodically. The company’s current system is a monolithic application primarily designed for human interaction (visualization, analysis, and report generation), not for large-scale ingestion.

Thus, the challenge was twofold: on one hand, to ensure these messages were reliably received and securely persisted, even with high volume; on the other, to avoid impacting the core application, which could become a bottleneck. Additionally, it was necessary to account for future growth: more sensors, more clients, and higher message frequency.

The Serverless Solution

The proposed approach was to create an intermediate solution that works as an intelligent "buffer" between IoT devices and the core application. Since the data doesn't need to be processed in real time, the architecture focuses on reliable ingestion, minimal transformation, and efficient persistence, with later scheduled consumption by the main system.

AWS IoT Core was chosen as the entry point for being a fully managed service with native support for the MQTT protocol, massive scalability, and strong integration with other AWS services. It receives messages directly from the sensors.

AWS IoT Rules acts as a message orchestrator. Using a SQL-like syntax, it transforms incoming messages, adds useful metadata like timestamp, topic, and an automatically generated uuid, and redirects them to the persistence layer:

SELECT *, timestamp() as timestamp, topic() as topic, uuid() as id, 
       timestamp() + 2592000 as expirationTime, 'pending' as status 
FROM 'devices/data'

This eliminates the need for custom code at this stage, reducing potential points of failure and accelerating development.

Amazon DynamoDB was selected for data persistence due to its ability to auto-scale with millisecond latency and native TTL (Time To Live) support. We implemented two TTL rules:

• 30 days for pending messages (not yet processed)
• 24 hours for processed messages

A Global Secondary Index (GSI) with status as the partition key and timestamp as the sort key allows the application to easily retrieve pending messages in chronological order:

messagesTable.addGlobalSecondaryIndex({
  indexName: 'status-timestamp-index',
  partitionKey: { name: 'status', type: dynamodb.AttributeType.STRING },
  sortKey: { name: 'timestamp', type: dynamodb.AttributeType.NUMBER },
});

AWS Lambda was used to implement two main endpoints: one to query pending messages, and another to confirm (ACK) message processing, updating the status to processed and reducing its TTL:

// Updates status to 'processed' and sets new expiration
UpdateExpression: 'SET #status = :status, expirationTime = :expTime',
ExpressionAttributeValues: {
  ':status': 'processed',
  ':expTime': now + 86400 // 24h in seconds
}

These functions are triggered on demand, scale automatically, and require zero server management.

Amazon API Gateway completes the cycle by exposing these functions as a secure RESTful API. The main application can, for instance, poll every 15 minutes for pending messages and acknowledge them after processing the data.

The entire infrastructure was defined using AWS CDK, ensuring version control, reproducibility, and ease of maintenance. Setting up a new environment takes just a few minutes.

A working proof-of-concept project is available here: https://github.com/epiresdasilva/iot-core-example

Benefits

From a technical perspective, this architecture eliminates the need to manage infrastructure or worry about scaling. All components are natively integrated and scale on demand. Business logic is cleanly separated and can evolve independently.

From a business perspective, the gains are significant:

• Lower infrastructure costs: no idle resources and pay-as-you-go pricing.
• Virtually infinite scalability to support growing numbers of clients and sensors.
• Greatly reduced time-to-market for new projects.
• Increased reliability and traceability with TTL and automatic logging.

Conclusion

This architecture shows how to build an efficient and resilient buffer for IoT data using a few AWS services — all serverless. With minimal maintenance effort and excellent cost-benefit, the company now has a robust ingestion pipeline, ready to scale safely.

About Me

This is Evandro Pires. I'm a husband, father of two, but also an AWS Serverless Hero, Serverless Guru Ambassador, Entrepreneur, CTO, Podcaster, and Speaker.

Cut costs and boost innovation by building a serverless-first mindset with sls.guru

Join our team and help transform the digital landscape for companies worldwide!