Case Study of Amazon Simple Queue Service

Queues are a powerful way of combining software architectures. They allow for asynchronous communication between different systems, and are especially useful when the throughput of the systems is unequal. Amazon offers their version of queues with Amazon SQS (Simple Queue Service).

🔰 Amazon Simple Queue Service 🔰

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work.

Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

How does SQS work?

It is used to decouple various components of the application. To understand in simple terms, it is like a buffered storage in between various software components.

To understand the concept how SQS works, let’s discuss with a use case.

Consider a sender component ‘A’ and receiver component ‘B’. For example, if you have something like:

• System A — produces messages periodically in huge bursts
• System B — consumes messages constantly, at a slower pace

With this architecture, a queue would allow System A to produce messages as fast as it can, and System B to slowly digest the messages at it’s own pace. This creates a serious problem of losing messages in transit.

Messages getting lost in transit

To prevent messages getting lost, a SQS queue can be introduced to act as a buffer to prevent loss of messages. The messages are stored in the queue in transit and then delivered to the receiver.

Messages are stored in SQS Queue

This is one of the use-cases for SQS, other use cases can be decoupling the application, asynchronous processing etc.

Workflow of a message in a SQS queue

• The message is created by a producer service and sent to the SQS queue.
• The message appears in the queue for all of the possible receivers as visible. This step can be non-immediate. For example, if you configure a ‘delay’ in the message, it will stay in the queue in a delayed state and will not be available for the receivers until the delay expires.
• One of the possible receivers makes a polling of the messages of the SQS queue. This operation retrieves the visible messages from the queue and switches them to an invisible state, but does not delete them. This keeps other receivers from getting those messages if they execute a new polling.
• When the receiver ends to process the message, explicitly removes it from the queue.

SQS Use Cases

• Handle payments asynch.
• Buffer write to database (voting app)
• E-mail sender
• Application integration
• Decoupling microservices
• Batch messages for future processing
• Allocate tasks to multiple worker nodes
• Decouple current user requests from intensive background work

680 companies reportedly use Amazon SQS in their tech stacks, including Pinterest, Amazon, and Lyft.

Features of Amazon SQS

• No Data Loss: SQS ensures that the messages sent are not lost and secure. Hence, it stores them on multiple standard queues.
• Confidential data: SSE (server-side encryption) allows transmission of highly confidential and sensitive data by offering protection to the messages present in the queues with the help of Amazon key management service.
• Each Request is handled independently: In SQS, every buffered request is processed independently ie. each request is scaled transparently to handle the increase in load without any explicit instructions.
• Message Locking: SQS locks the user messages when it is being processed so that various producers can send messages to various receivers and receivers could receive messages from multiple recipients at the same time.

What is SQS used for?

The most common ways to use SQS, and of course other messaging systems, in cloud applications are:

• Decoupling microservices. In a microservice architecture, messages represent one of the easiest ways to set up communication between different parts of the system. If your microservices run in AWS, and especially if those are Serverless services, SQS is a great choice for that aspect of the communication.
• Sending tasks between different parts of your system. You don’t have to be running a microservices-oriented application to take advantage of SQS. You can also use it in any kind of application that needs to communicate tasks to other systems.
• Distributing workloads from a central node to worker nodes. You can frequently find messaging systems in the flows of distributed large workloads like map-reduce operations. For these kinds of operations, it’s essential to be able to maintain a queue of all the tasks that need to be processed, efficiently distribute the tasks between the machines or functions doing the work, and guarantee that every part of the work is only done once.
• Scheduling batch jobs. SQS is a great option for scheduling batch jobs for two reasons. First, it maintains a durable queue of all the scheduled jobs, which means you don’t need to keep track of the job status — you can rely on SQS to pass the jobs through and to handle any retries, should an execution fail and your batch system returns the message to the queue. Second, it integrates with AWS Lambda; if you’re using AWS Lambda to process the batch jobs, SQS automatically launches your Lambda functions once the data is available for them to process.

🔰 OYSTER CASE STUDY 🔰

About Oyster

New York-based Oyster.com shares unvarnished reviews of hotels in nearly 200 destinations worldwide. The company’s own investigators visit each location to assess cleanliness, amenities, service and overall quality.

What sets Oyster apart from similar sites is its extensive collection of photographs.

The Challenge

Since its 2009 launch, Oyster has published more than one million high-quality digital images. When this massive volume of images became too cumbersome to handle in-house, the company decided to offload the content to a central repository on Amazon Simple Storage Service (Amazon S3).

Oyster reprocesses its entire collection of photographic images a few times each year to update the copyright year and, if necessary, to change the watermarks. Using their previous solution, reprocessing the entire collection of photographs required about 800 hours to complete. In addition, Oyster often recreated existing images in new formats and sizes for mobile and tablet devices. Resizing existing images and adding new ones was slowing down the rate at which the company was able to process the collection.

Moreover, there were numerous software bugs in the multiprocessing solution that the company used, but since the solution didn’t scale, Oyster didn’t bother to fix them.

AWS Services Used By Oyster:

• Amazon EC2
• Amazon SQS
• Amazon Linux AMI

Why Amazon Web Services

“We were already using Amazon S3 to store the images, so using Amazon EC2 to process the images was a natural choice,” Seidman says. Chris McBride, a software engineer at Oyster, adds, “We wanted a cloud environment that could be ramped up for the large processing jobs and downsized for the smaller daily jobs.”

While the company is still running one local server, the bulk of the processing work now takes place on the AWS Cloud. Oyster is using a customized Amazon Linux AMI within Amazon EC2. Within this new environment, the company connects to Amazon S3 and Amazon Simple Queue Service (Amazon SQS) using boto, a Python interface to AWS. The images themselves are processed with the ImageMagick software available in the AMI package.

Oyster uses Amazon EC2 instances and Amazon SQS in an integrated workflow to generate the sizes they need for each photo. The team processes a few thousand photos each night, using Amazon EC2 Spot Instances. When Oyster processes the entire collection, it can use up to 100 Amazon EC2 instances. The team uses Amazon SQS to communicate the photos that need to be processed and the status of the jobs.

The Benefits

• Oyster’s old system needed approximately 400 hours to process one million photos. By using AWS, the company can process the same number of photos in about 20 hours — a 95 percent improvement.
• Oyster has also been able to reduce in-house hardware expenses by repurposing two of its old servers, which were sitting idle more than 80 percent of the time.

Conclusion

Hence, we studied that the AWS SQS helps in sending and processing the unlimited number of messages and in any region. Message payloads consist of 256KB of text which can be in any format. Send and read messages can be done simultaneously. Amazon SQS queries can share anonymously or with some specific AWS accounts. The contents of the message of AWS SQS can secure using keys managed by the AWS Key Management Service. This helps a lot for the user to use SQS in an efficient manner.