Messaging: SES, SNS and SQS

Queues protect the worker

A queue separates the producer from the worker. The producer can put a job on SQS and move on. A worker receives the job, does the work and deletes the message after success.

SQS standard queues are the default choice. They scale broadly and provide at-least-once delivery. That means a message can appear again, so the handler must be safe to run twice. FIFO queues add ordering and exactly-once processing support for workloads that need it, with different throughput and naming rules.

$ aws sqs create-queue \
    --queue-name academy-messaging-demo \
    --attributes '{"VisibilityTimeout":"30"}'
$ aws sqs get-queue-url --queue-name academy-messaging-demo

The queue URL is not the ARN. Ask for the ARN when a policy or another service needs it:

$ aws sqs get-queue-attributes \
    --queue-url "$QUEUE_URL" \
    --attribute-names QueueArn VisibilityTimeout \
    --query 'Attributes'

A receive does not acknowledge. The message is hidden for the visibility timeout. Delete is the acknowledgement:

$ aws sqs send-message --queue-url "$QUEUE_URL" --message-body '{"job":"rebuild-index","attempt":1}'
$ MESSAGE=$(aws sqs receive-message --queue-url "$QUEUE_URL" --wait-time-seconds 10 --query 'Messages[0].{Body:Body,Receipt:ReceiptHandle}' --output json)
$ aws sqs delete-message --queue-url "$QUEUE_URL" --receipt-handle "$RECEIPT_HANDLE"

Do not copy the last command until the worker has completed the job. If the worker crashes before deleting, SQS makes the message visible again. That is a useful failure mode, not a duplicate bug to hide.

Poison messages go to a DLQ

Some messages fail every time: invalid JSON, a missing object or a code path nobody tested. Retrying them forever keeps a worker busy. A dead-letter queue stores messages after the source queue reaches maxReceiveCount.

$ aws sqs create-queue --queue-name academy-messaging-demo-dlq
$ DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
$ aws sqs set-queue-attributes \
    --queue-url "$QUEUE_URL" \
    --attributes "{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$DLQ_ARN\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\"}"

Inspect the policy before relying on it:

$ aws sqs get-queue-attributes --queue-url "$QUEUE_URL" --attribute-names RedrivePolicy

A DLQ is not a trash can. Give it an owner, longer retention than the source queue, an alert and a repair or replay process. Redrive only after you understand why the message failed.

Topics fan out notifications

SNS is a publish-subscribe service. A publisher sends one notification to a topic. SNS delivers it to each confirmed subscription. Email is good for a human notification. An SQS subscription is good for durable, retriable work.

$ TOPIC_ARN=$(aws sns create-topic --name academy-messaging-demo --query TopicArn --output text)
$ aws sns subscribe \
    --topic-arn "$TOPIC_ARN" \
    --protocol email \
    --notification-endpoint elonerajeev@gmail.com

The email subscription starts as PendingConfirmation. The recipient must use the link in AWS’s confirmation message before SNS will deliver notifications. Do not publish a test message to a real address unless you intend to send it.

For SQS fan-out, the queue policy must allow this topic to call sqs:SendMessage, with a condition limiting the source ARN to this topic. That policy is permission, not another queue or topic.

Email has a sending contract

SES sends email, but it is not a general-purpose notification queue. In the sandbox, the sender and recipient usually need verification and the account has low sending limits. DKIM proves domain alignment; it does not make an unverified recipient safe to mail.

$ aws sesv2 get-account --query '{Production:ProductionAccessEnabled,SendQuota:SendQuota}'
$ aws sesv2 list-email-identities --query 'EmailIdentities[].IdentityName'

Treat bounces and complaints as feedback that changes whether you may keep sending. Store no raw email content in logs unless there is a clear reason and retention policy.

Events need a router

EventBridge is for events that describe something that happened: an object was uploaded, a deployment completed or an instance changed state. An event bus receives events; rules match their shape and route them to targets.

That is different from SQS, which holds work until a consumer pulls it, and SNS, which pushes a notification to subscriptions. A system can use all three: EventBridge detects the event, SNS notifies, and SQS protects a worker.

$ REGION=ap-south-1
$ PROFILE=anythinkx
$ DLQ_URL=$(aws sqs create-queue --profile $PROFILE --region $REGION --queue-name academy-messaging-demo-dlq --query QueueUrl --output text)
$ DLQ_ARN=$(aws sqs get-queue-attributes --profile $PROFILE --region $REGION --queue-url "$DLQ_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
$ QUEUE_URL=$(aws sqs create-queue --profile $PROFILE --region $REGION --queue-name academy-messaging-demo --attributes '{"VisibilityTimeout":"30","RedrivePolicy":"{\"deadLetterTargetArn\":\"'$DLQ_ARN'\",\"maxReceiveCount\":\"2\"}"}' --query QueueUrl --output text)
$ TOPIC_ARN=$(aws sns create-topic --profile $PROFILE --region $REGION --name academy-messaging-demo --query TopicArn --output text)
$ SUB_ARN=$(aws sns subscribe --profile $PROFILE --region $REGION --topic-arn "$TOPIC_ARN" --protocol email --notification-endpoint elonerajeev@gmail.com --query SubscriptionArn --output text)
$ aws sqs get-queue-attributes --profile $PROFILE --region $REGION --queue-url "$QUEUE_URL" --attribute-names QueueArn VisibilityTimeout RedrivePolicy --query 'Attributes'
$ aws sns list-subscriptions-by-topic --profile $PROFILE --region $REGION --topic-arn "$TOPIC_ARN" --query 'Subscriptions[].{Protocol:Protocol,Endpoint:Endpoint,Status:SubscriptionArn}'
$ aws sns unsubscribe --profile $PROFILE --region $REGION --subscription-arn "$SUB_ARN" 2>/dev/null || true
$ aws sns delete-topic --profile $PROFILE --region $REGION --topic-arn "$TOPIC_ARN"
$ aws sqs delete-queue --profile $PROFILE --region $REGION --queue-url "$QUEUE_URL"
$ aws sqs delete-queue --profile $PROFILE --region $REGION --queue-url "$DLQ_URL"
$ aws sns list-topics --profile $PROFILE --region $REGION --query "Topics[?contains(TopicArn, 'academy-messaging-demo')]"
$ aws sqs get-queue-url --profile $PROFILE --region $REGION --queue-name academy-messaging-demo