Use ElastiCache to improve Application Performance

Many applications use S3 as the backend for storing data like images, videos, files etc., As your user base grows and you start reaching millions of requests to these objects, Latency and Costs play an important role in maintaining application performance. Especially if your application requires <5 ms responses. For such applications, customers can complement S3 with an in-memory cache like ElastiCache for Redis. It reduces S3 retrieval costs and improves application performance.

Here, let me show, how you can set up an in-memory cache using ElastiCache for Redis, along with best practices to be used with S3. You can also test the performance benefits of incorporating a cache for S3. All the necessary code is written in Cloudformation using CDK.

Follow this article in Youtube

1. 🧰 Prerequisites

   This demo, instructions, scripts and cloudformation template is designed to be run in us-east-1. With few modifications you can try it out in other regions as well(Not covered here).

   • AWS CLI pre-configured - Get help here
   • AWS CDK Installed & Configured - Get help here
   • Python Packages, Change the below commands to suit your OS
     • Python3 - yum install -y python3
     • Python Pip - yum install -y python-pip
     • Virtualenv - pip3 install virtualenv

2. ⚙️ Setting up the environment

   • Get the application code

     git clone https://github.com/miztiik/elasticache-for-app-performance.git
     cd elasticache-for-app-performance

3. 🚀 Resource Deployment using AWS CDK

   The cdk stack provided in the repo will create the following resources,

   • VPC with public & private subnets, route table for private subnet
   • NAT Gateway x 1 for private subnet to communicate with the internet
   • AWS ElastiCache inside private subnet
   • S3 Bucket
   • Lambda - To ingest dummy test data into S3 & AWS ElastiCache
   • EC2 Instance in public subnet: We will use this instances to simulate a application client querying S3 or AWS ElastiCache

   Note: Most of the resources should be covered under the aws free tier, except the NAT Gateway. You can swap it out for a NAT Instance

   # If you DONT have cdk installed
   npm install -g aws-cdk
   
   # Make sure you in root directory
   cd elasticache-for-app-performance
   python3 -m venv .env
   source .env/bin/activate
   pip install -r requirements.txt

   The very first time you deploy an AWS CDK app into an environment (account/region), you’ll need to install a bootstrap stack, Otherwise just go aheadand deploy using cdk deploy

   cdk bootstrap
   cdk deploy

4. 🔬 Testing the solution

   Your ElastiCache Redis cluster is pre-loaded with a total of 200 files as a set. The same data is stored in your S3 bucket. The size of these files are slighly randomized with random string content. To know more check out the redis_data_ingester_lambda_function.

   • In the Outputs section of the Clouformation template/service, Make a note of the following paramets, we will use them later

     • Redis Host domain name - This cache cluster is pre-loaded with data for querying
     • Redis Port - `6379
     • S3 Bucket Name - This bucket is pre-loaded with data for querying
     • EC2 Instance - This server will be used to connect/query S3 and Redis

   • Connect to the EC2 instance using Session Manager - Get help here

   • Clone this repo, We need the helper_scripts in this repo

     git clone https://github.com/miztiik/elasticache-for-app-performance.git
     cd elasticache-for-app-performance
     cd helper_scripts

   • Edit the file constants.py with the values from your clouformation outputs.

   • Fetch data from Redis:

     python3 query_redis.py

   • Fetch data from S3:

     python3 query_s3.py

   • Sample results from my test,

     # Redis Query Results
     Average Latency in Microseconds: 683.2525252525253
     MAX Latency in Microseconds: 1890
     MIN Latency in Microseconds: 606
     
     # S3 Query Results
     Average Latency in Microseconds: 22621.565656565657
     MAX Latency in Microseconds: 224836
     MIN Latency in Microseconds: 10334

   • Compare the data retrieval times for both these queries. Here is a graph that i made with the data froom the results.

   

   You will notice that the queries against ElastiCache Redis were significantly quicker than the queries against S3.

5. 🧹 CleanUp

   If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well

   • Resources created during deployment
   • Delete CloudWatch Lambda LogGroups
   • Any other custom resources, you have created for this demo

   # Delete from cdk
   cdk destroy
   
   # Delete the CF Stack, If you used cloudformation to deploy the stack.
   aws cloudformation delete-stack \
       --stack-name "MiztiikAutomationStack" \
       --region "${AWS_REGION}"

   This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.

👋 Buy me a coffee

Buy me a coffee ☕, or You can reach out to get more details through here.

📚 References

1. Install NPM

2. Python Pip Redis

3. AWS Blog

4. AWS Blog: Latency Reduction

🏷️ Metadata

Level: 400