AWS in Practice · 05 — SNS & SQS

Resilient asynchronous communication between the project's microservices.

About this module

Here the system stops being three isolated services and becomes a truly distributed system. You'll implement the Orders → Inventory flow via events, with queues, dead letter queues, and idempotency — the fundamental building blocks.

Level: Intermediate · Estimated duration: 3-4 hours

Table of Contents

1. Why messaging?
2. SNS: pub/sub on AWS
3. SQS: managed queues
4. The SNS + SQS pattern (fan-out)
5. Standard vs FIFO
6. Dead Letter Queues
7. Idempotency in practice
8. Implementing the Orders → Inventory flow
9. CloudFormation templates
10. Monitoring and debugging
11. Practice exercises
12. Next steps

01. Why messaging?

So far, our services don't talk to each other. Let's fix that. But before implementing, it's important to understand why we use messaging instead of simple HTTP calls between services.

The problem of synchronous coupling

Imagine Orders calls Inventory directly over HTTP every time an order is created. What happens if Inventory is down? The order fails — even though the failure is in a relatively secondary service (decrementing stock could be done later).

This is temporal coupling: both services need to be up at the same moment. A problem in one brings down the other.

The messaging model

Instead of calling directly, Orders publishes an event somewhere (an SNS topic). The messaging system stores the event and delivers it to whoever is listening (Inventory's SQS queue).

Is Inventory down? No problem — the message stays in the queue. When Inventory comes back, it processes the pending ones. Orders never even knows there was an outage.

The concrete gains

• Resilience — failures in one service don't bring down others.
• Spikes absorbed — the queue acts as a buffer between producer and consumer.
• Independent scaling — producer and consumer scale separately.
• Minimal coupling — the producer doesn't need to know the consumers.
• Adding consumers is trivial — just subscribe a new queue to the topic.

The costs

• Eventual consistency — there's a delay between an event being published and processed.
• More complex debugging — you need to trace messages across systems.
• Order not guaranteed (in Standard) — events may arrive out of order.
• Duplication possible — the same message may be delivered more than once.

02. SNS: pub/sub on AWS

SNS (Simple Notification Service) is AWS's pub/sub service. Producers publish messages to a topic; subscribers receive copies.

Core concepts

• Topic — the communication point. It has an ARN, a name, and zero or more subscribers.
• Publisher — any entity (service, Lambda, CLI) that publishes to a topic.
• Subscriber — an endpoint that receives the published messages. Can be SQS, Lambda, email, HTTP, SMS.
• Subscription — the link between a topic and a subscriber. Can have filters.

Subscriber types

Important characteristics

• Fan-out: a message published to the topic goes to all subscribers.
• Push-based: SNS pushes to subscribers, there's no polling.
• Best effort in Standard: order not guaranteed, duplication possible.
• FIFO available: order and exactly-once (with more limited filters).

Subscription filters

You can configure a filter policy on a subscription so it only receives messages that meet certain criteria. For example, a queue might want only events for orders with a value greater than $1000:

{
  "totalAmount": [{"numeric": [">", 1000]}],
  "country": ["BR", "US"]
}

The publisher passes attributes when publishing; SNS applies the filters before delivering. This reduces traffic, simplifies consumers, and avoids queues filling with irrelevant messages.

03. SQS: managed queues

SQS (Simple Queue Service) is AWS's managed queue. Messages are held until they are processed — it's the buffering and decoupling mechanism.

How it works, at a high level

1. The producer sends a message to the queue (SendMessage).
2. SQS stores the message redundantly across multiple AZs.
3. The consumer polls: ReceiveMessage returns up to 10 messages.
4. The message becomes invisible for a period (visibility timeout) to other consumers.
5. The consumer processes it and calls DeleteMessage to confirm success.
6. If it isn't deleted within the timeout, the message returns to the queue and may be reprocessed.

Important concepts

• Visibility timeout — how long a message stays invisible after being received. Default: 30s. Configurable up to 12h.
• Message retention — how long a message stays in the queue if not processed. Default: 4 days. Max: 14 days.
• Long polling — the consumer waits up to 20s for new messages. Reduces cost and latency.
• Batch operations — send/receive/delete up to 10 messages in a single call. More efficient.

Important limits

For messages larger than 256 KB, use the claim check pattern: save the payload in S3 and send the ARN/URL in the queue. The consumer fetches it from S3.

04. The SNS + SQS pattern (fan-out)

Combining SNS with SQS is the most common pattern in event-driven architecture on AWS. It's worth understanding why this combination is better than either one alone.

Why not SNS directly to Lambda/service?

You can. But there are two problems:

• No buffer — if the consumer is down, the message is lost after a few attempts.
• No custom retry policy — SNS does retries, but with fixed rules.
• No replay — you can't reprocess old messages if something goes wrong.

Why not SQS directly?

SQS alone works if you have a single consumer. But if multiple services need to react to the same event, you'd have to write to multiple queues — the producer ends up knowing all the consumers. Coupling again.

The solution: SNS + SQS combined

The producer publishes to an SNS topic. Each consumer creates its own SQS queue and subscribes it to the topic. SNS does the fan-out, SQS stores with guarantees.

Figure 1 — Fan-out: the producer publishes once, multiple consumers receive.

Advantages of the pattern

• Total decoupling — the producer doesn't know who consumes.
• Each consumer has its own queue — one can fail without affecting others.
• Adding a new consumer is trivial — just create a new queue and subscribe.
• Buffer per consumer — each processes at its own pace.
• Replay possible — old messages can be reprocessed (up to the retention period).

05. Standard vs FIFO

SNS and SQS offer two types: Standard and FIFO. Knowing which to use affects the system's performance, cost, and correctness.

When to use Standard

• Independent events — order doesn't matter.
• High volume — thousands of messages per second.
• Idempotency implemented on the consumer side — duplication isn't a problem.
• The vast majority of cases — you'll only need FIFO in specific situations.

When to use FIFO

• Order is critical — event B must be processed after A.
• No idempotency and duplication is unacceptable.
• Low to moderate volume.
• Common cases: sequential workflows, financial processing, command sequences.

MessageGroupId in FIFO

In FIFO, you must specify a MessageGroupId on each message. Order is guaranteed within the same group, but different groups may be processed in parallel. Use customerId or orderId as the group ID — events from the same customer/order in order, but parallelism across customers.

06. Dead Letter Queues

Messages that fail repeatedly need to go somewhere — they can't keep clogging the main queue nor disappear. A DLQ is the solution.

The poison message problem

Imagine a malformed message that crashes your consumer. Without a DLQ:

• The consumer picks up the message, processes it, fails.
• The visibility timeout expires; the message returns to the queue.
• Another consumer (or the same one) picks it up again, fails again.
• Infinite loop — the queue is stuck processing garbage.

How the DLQ solves it

You configure an attempt limit (maxReceiveCount). When a message is received more times than that without being deleted, it's automatically moved to a secondary queue — the Dead Letter Queue.

The main queue returns to normal. You inspect the DLQ later, identify the problem, fix the consumer (or the data), and can reprocess.

Configuration

# Main queue attribute:
RedrivePolicy:
  deadLetterTargetArn: "arn:aws:sqs:...:inventory-dlq"
  maxReceiveCount: 5

maxReceiveCount = 5 means: after 5 failed attempts, the message goes to the DLQ. This number is your choice — 3 is more aggressive, 10 is more tolerant.

DLQ best practices

• Always create a DLQ for production queues. Without one, you have an operational blind spot.
• Configure a CloudWatch alarm on the DLQ's ApproximateNumberOfMessagesVisible metric.
• Longer retention on the DLQ — 14 days (the maximum), so you have time to investigate.
• Re-drive process — define when and how DLQ messages are reprocessed.

07. Idempotency in practice

Since SQS Standard delivers at least once, your message may be processed more than once. If that causes data duplication or wrong side effects, you have a serious problem. Idempotency solves it.

A rigorous definition

An operation is idempotent if running it N times has the same effect as running it once. In messaging, this means: receiving the same message 1x or 10x must produce the same final state.

Implementation strategies

1. Naturally idempotent operations

Some operations are already idempotent: SetUserEmail("foo@bar.com") can be called N times — the final state is the same. Whenever possible, model for this.

2. Event ID + processed table

Each event has a unique eventId (UUID). Before processing, the consumer checks a table to see whether that eventId has already been processed. If so, it discards it.

// Pseudocode (conceptual example only):
async function processOrderCreated(event) {
  const exists = await db.checkProcessed(event.eventId);
  if (exists) return;  // already processed, discard
 
  await db.transactWrite([
    decrementInventory(event.productId, event.quantity),
    markProcessed(event.eventId)
  ]);
}

3. Conditional writes

DynamoDB supports ConditionExpression: the operation only runs if the condition is true. Use it to avoid double writes.

08. Implementing the Orders → Inventory flow

Let's put it all into practice. When an order is created, Orders publishes order.created to an SNS topic. Inventory listens via an SQS queue and decrements the stock.

Resources to create

• SNS Topic: order-events
• SQS Queue: inventory-orders-queue
• SQS Queue (DLQ): inventory-orders-dlq
• Subscription: order-events → inventory-orders-queue
• IAM Policies: Orders can publish; Inventory can consume.

Event schema

{
  "eventId": "550e8400-e29b-41d4-a716-446655440000",
  "eventType": "order.created",
  "timestamp": "2026-01-15T10:30:00Z",
  "version": "1.0",
  "data": {
    "orderId": "ord-123",
    "customerId": "cust-456",
    "items": [
      {"productId": "prod-789", "quantity": 2, "price": 29.90}
    ],
    "totalAmount": 59.80
  }
}

Producer side (Orders)

In the Orders Service, after saving the order in DynamoDB, publish to SNS:

// Conceptual structure only, not production-ready code
import { SNSClient, PublishCommand } from '@aws-sdk/client-sns';
 
const sns = new SNSClient({ region: process.env.AWS_REGION });
 
await sns.send(new PublishCommand({
  TopicArn: process.env.SNS_ORDER_EVENTS_TOPIC,
  Message: JSON.stringify(eventPayload),
  MessageAttributes: {
    eventType: {
      DataType: 'String',
      StringValue: 'order.created'
    }
  }
}));

Consumer side (Inventory)

In the Inventory Service, poll the queue and process:

// Conceptual structure only
import { SQSClient, ReceiveMessageCommand, 
         DeleteMessageCommand } from '@aws-sdk/client-sqs';
 
const sqs = new SQSClient({ region: process.env.AWS_REGION });
 
// Processing loop (simplified)
while (true) {
  const result = await sqs.send(new ReceiveMessageCommand({
    QueueUrl: process.env.SQS_INVENTORY_QUEUE_URL,
    MaxNumberOfMessages: 10,
    WaitTimeSeconds: 20  // long polling
  }));
 
  for (const message of result.Messages ?? []) {
    await processMessageIdempotent(message);
    await sqs.send(new DeleteMessageCommand({
      QueueUrl: process.env.SQS_INVENTORY_QUEUE_URL,
      ReceiptHandle: message.ReceiptHandle
    }));
  }
}

Important: the message arriving via SNS+SQS comes with a wrapper. The actual event JSON is in JSON.parse(message.Body).Message.

09. CloudFormation templates

All this messaging infrastructure needs to be in CloudFormation, consistent with what we learned in Module 04.

Template: messaging.yaml (part 1 — SNS and DLQ)

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Messaging between Orders and Inventory'
 
Resources:
  OrderEventsTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: order-events
      DisplayName: 'Orders domain events'
 
  InventoryOrdersDLQ:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: inventory-orders-dlq
      MessageRetentionPeriod: 1209600  # 14 days

Template: messaging.yaml (part 2 — main queue)

  InventoryOrdersQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: inventory-orders-queue
      VisibilityTimeout: 60
      MessageRetentionPeriod: 345600  # 4 days
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt InventoryOrdersDLQ.Arn
        maxReceiveCount: 5
 
  InventoryOrdersSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref OrderEventsTopic
      Protocol: sqs
      Endpoint: !GetAtt InventoryOrdersQueue.Arn
      RawMessageDelivery: true
      FilterPolicy:
        eventType: ['order.created', 'order.cancelled']

Permission: SNS can write to SQS

  QueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref InventoryOrdersQueue
      PolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: sns.amazonaws.com
            Action: sqs:SendMessage
            Resource: !GetAtt InventoryOrdersQueue.Arn
            Condition:
              ArnEquals:
                aws:SourceArn: !Ref OrderEventsTopic

Outputs

Outputs:
  TopicArn:
    Value: !Ref OrderEventsTopic
    Export:
      Name: !Sub '${AWS::StackName}-TopicArn'
 
  QueueUrl:
    Value: !Ref InventoryOrdersQueue
    Export:
      Name: !Sub '${AWS::StackName}-QueueUrl'

10. Monitoring and debugging

In messaging, monitoring is more important than in synchronous code — problems manifest silently, like queues growing.

Critical metrics

Essential alarms

• DLQ not empty — if there are messages in the DLQ, there's a bug somewhere.
• Age of the oldest message > threshold — slow or stopped consumer.
• Queue growing — production faster than consumption.
• Receive/delete errors — permission or config problems.

Inspecting messages

# Get 1 message without deleting (peek)
aws sqs receive-message \
  --profile aws-curso \
  --queue-url <queue-url> \
  --max-number-of-messages 1 \
  --visibility-timeout 0
 
# Count visible messages
aws sqs get-queue-attributes \
  --profile aws-curso \
  --queue-url <queue-url> \
  --attribute-names ApproximateNumberOfMessagesVisible

Reprocessing from the DLQ

There's a feature called Dead-letter queue redrive: you select the DLQ in the console and click Start DLQ redrive — the messages return to the original queue automatically. Use it after fixing the consumer!

11. Practice exercises

12. Next steps

Your system is now truly distributed. It has producers, consumers, queues, events. The next level: hand the consumer task over to Lambda — without having to keep an EC2 polling forever.

What you learned

• Why messaging is the foundation of distributed systems.
• SNS for pub/sub and SQS for queues with guarantees.
• The fan-out pattern (SNS + SQS).
• Standard vs FIFO and when to choose each.
• Dead Letter Queues and handling poison messages.
• Idempotency: strategies and implementation.
• A complete implementation of the Orders → Inventory flow.
• CloudFormation templates for messaging.
• Monitoring and debugging in queues.

What's coming in Module 06

Lambda and event-driven processing. We'll replace the Inventory consumer (which today runs as a Beanstalk service doing polling) with a Lambda function that is invoked automatically when there are messages. We'll also add Lambda functions for sending emails, generating reports, and processing images.

Messages don't get lost unless you let them. Enjoy the journey — Module 06 is next.