AWS Cloud has just come up with new feature in Amazon Simple Storage Service (S3) that allows you to add your own code to process data retrieved from S3 before returning it to an application
Introduction to AWS S3
Amazon Simple Storage Service (Amazon S3) is storage for the Internet. It is designed to make web-scale computing easier for developers.
Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.
AWS S3 Usecase
Amazon Simple Storage Service (S3) allows to easily share files with users, different AWS Services and with multiple applications.
The S3 Object data consumed might not be directly helpful in some cases, as different applications may need some transformation on it before using it.
Some Examples via AWS S3 Documentation: 1) a dataset created by an e-commerce application may include personally identifiable information (PII) that is not needed when the same data is processed for analytics and should be redacted.
2) if the same dataset is used for a marketing campaign, you may need to enrich the data with additional details, such as information from the customer loyalty database.
3) Redacting personally identifiable information for analytics or non-production environments.
4) Converting across data formats, such as converting XML to JSON.
5) Augmenting data with information from other services or databases.
6) Compressing or decompressing files as they are being downloaded.
7) Resizing and watermarking images on the fly using caller-specific details, such as the user who requested the object.
8) Implementing custom authorization rules to access data.
Different ways to get custom S3 Object
1) You either create, store, and maintain additional derivative copies of the data, so that each application has its own custom dataset,
2) You build and manage infrastructure as a proxy layer in front of S3 to intercept and process data as it is requested.
Both above options add complexity and costs, so the S3 team decided to build a better solution.
S3 Object Lambda works with your existing applications and uses AWS Lambda functions to automatically process and transform your data as it is being retrieved from S3. The Lambda function is invoked inline with a standard S3 GET request, so you don’t need to change your application code.
In this way, you can easily present multiple views from the same dataset, and you can update the Lambda functions to modify these views at any time.
How to Start ?
- Create a Lambda Function to transform data for your use case.
- Create an S3 Object Lambda Access Point from the S3 Management Console.
- Select the Lambda function that you created above.
- Provide a supporting S3 Access Point to give S3 Object Lambda access to the original object.
- Update your application configuration to use the new S3 Object Lambda Access Point to retrieve data from S3.
I will be sharing further Node.js Lambda Demo with S3 Object. Keep watching this space.
More detailed documentation available at: aws.amazon.com/blogs/aws/introducing-amazon..