Mystique Unicorn App is a building new application based on microservice architectural pattern. One of their microservices need to have access to a persistent storage layer.
They have created this use-case as an analogy to explain their requirement. Let us say, we have a "Message Board", where any user can post messages and read messages of other users. This means, we need to store the messages from all the functions and generate be able to list/view of the messages. Ofcourse, we can use a database, but that is not the point there, "How can we use a persistent storage to share data among functions?"
Can you help them do that in Amazon API Gateway & AWS Lambda?
We can use AWS Elastic File System with AWS Lambda to achieve this.
In this article, we will build an architecture, similar to the one shown above - A simple message wall
API using API Gateway and Lambda. Any POST
message send to the api will be stored in the EFS share and any subsequent GET
requests will return all the messages in the wall.
-
This demo, instructions, scripts and cloudformation template is designed to be run in
us-east-1
. With few modifications you can try it out in other regions as well(Not covered here).- 🛠 AWS CLI Installed & Configured - Get help here
- 🛠 AWS CDK Installed & Configured - Get help here
- 🛠 Python Packages, Change the below commands to suit your OS, the following is written for amzn linux 2
- Python3 -
yum install -y python3
- Python Pip -
yum install -y python-pip
- Virtualenv -
pip3 install virtualenv
- Python3 -
-
-
Get the application code
git clone https://github.com/miztiik/lambda-with-efs cd lambda-with-efs
-
-
We will cdk to be installed to make our deployments easier. Lets go ahead and install the necessary components.
# If you DONT have cdk installed npm install -g aws-cdk # Make sure you in root directory python3 -m venv .env source .env/bin/activate pip3 install -r requirements.txt
The very first time you deploy an AWS CDK app into an environment (account/region), you’ll need to install a
bootstrap stack
, Otherwise just go ahead and deploy usingcdk deploy
.cdk bootstrap cdk ls # Follow on screen prompts
You should see an output of the available stacks,
vpc-stack efs-stack lambda-with-efs
-
Let us walk through each of the stacks,
-
Stack: efs-stack This stack will create the Amazon EFS. There are few resources that are prerequisites to create the EFS share. This stack will create the following resources,
- A VPC to host our EFS share - Deployed by the dependant stack
vpc-stack
- Security group for our EFS share allowing inbound
TCP
on ort2049
from our VPC IP range - Posix user & acl
1000
- In case you want to use OS level access restrictions, these will come in handy - EFS Access Point to make it easier to mount to Lambda and apply resource level access restrictions
- The default path for the access point is set to
/efs
- The default path for the access point is set to
Initiate the deployment with the following command,
cdk deploy efs-stack
- A VPC to host our EFS share - Deployed by the dependant stack
-
Stack: lambda-with-efs
This stack: lambda-with-efs creates an REST API with a lambda backend. This lambda function will be deployed in the same VPC as our EFS share and use the same security group(TODO: Host lambda in a independant security group). The stack mounts the EFS Access point to our lambda function, there-by enabling us to read and write to our EFS share.
Initiate the deployment with the following command,
cdk deploy lambda-with-efs
Check the
Outputs
section of the stack to access theGreetingsWallApiUrl
-
-
We can use a tool like
curl
orPostman
to query the url. The Outputs section of the respective stacks has the required information on the urls.$ GREETINGS_WALL_URL="https://2s9p0x3p53.execute-api.us-east-2.amazonaws.com/prod/lambda-with-efs/greeter" $ curl ${GREETINGS_WALL_URL} No message yet. $ curl -X POST -H "Content-Type: text/plain" -d 'Hello from EFS!' ${GREETINGS_WALL_URL} Hello from EFS! $ curl -X POST -H "Content-Type: text/plain" -d 'Hello again :)' ${GREETINGS_WALL_URL} Hello from EFS! Hello again :) $ curl ${GREETINGS_WALL_URL} Hello from EFS! Hello again :) $ curl -X DELETE ${GREETINGS_WALL_URL} Messages deleted. $ curl ${GREETINGS_WALL_URL} No message yet.
You should be able observe that we were able to read, write & delete data from the EFS share.
-
Here we have demonstrated how to use EFS along with AWS Lambda to create a persistent storage for your functions. This can be really helpful in a variety of situations. For example,
- If you are running machine language inference, lambda internal storage and layers might not be enough to host all the dependant libraries. In those cases an external storage becomes a necessity.
- Another usecase is when you want to process really huge files - unpack/zip them. Then the extra scratch space offered EFS comes handy.
If you know of other usecases for using EFS with lambda, do let me know.
-
If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well
- Resources created during Deploying The Application
- Delete CloudWatch Lambda LogGroups
- Any other custom resources, you have created for this demo
# Delete from cdk cdk destroy # Follow any on-screen prompts # Delete the CF Stack, If you used cloudformation to deploy the stack. aws cloudformation delete-stack \ --stack-name "MiztiikAutomationStack" \ --region "${AWS_REGION}"
This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.
This repository aims to teach best practices & advanced file system techniques to new developers, Solution Architects & Ops Engineers in AWS. Based on that knowledge these Udemy course #1, course #2 helps you build complete architecture in AWS.
Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here
Buy me a coffee ☕.
Level: 300