AWS S3 File Upload + Lambda Trigger (Tutorial In Python) | Step by Step Guide
HTML-код
- Опубликовано: 11 авг 2019
- S3 is an easy to use all purpose data store. Frequently we use it to dump large amounts of data for later analysis. Using S3 Put events with Lambda, we can shift this process to an event driven architecture. In this video, I show you how to set up a S3 Put Event Trigger that invokes a lambda function every time a file is added to your s3 bucket. This allows for event driven processing and other neat applications!
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
00:12 Example Overview
01:07 Creating a Lambda Function
02:40 Lambda Code Walkthrough
03:49 Creating the S3 Trigger
06:00 Validating the S3 Trigger
06:32 Testing with a File Upload
06:56 Validating the Lambda Invocation
🎉SUPPORT BE A BETTER DEV🎉
Become a Patron: / beabetterdev
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driver Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
☁Topics covered include:
- S3 Bucket Creation
- S3 File Upload
- S3 Put Event
- S3 Event Subscription to Lambda
- Cloudwatch Logs
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#S3
#Lambda
#AWS
#Serverless
Looking to become an expert on AWS Lambda? Check out my new course: AWS Lambda - A Practical Guide
www.udemy.com/course/aws-lambda-a-practical-guide/?referralCode=F6D1A50467E579C65372
Long awaited real world use case and simplification of AWS solutions / products.
Kudos and keep it up.
Thanks for the support. If you have any topics you'd like me to cover please let me know!
Contains ALL of the details needed, yet concise and clear. Great tutorial for any one new to AWS, thank you ! :-)
You're very welcome!
This video really helps. Explanation and demo are so precise. Please keep doing this great work.
Your videos on Step Functions and Lambdas have been incredibly helpful in designing a new serverless billing ingestion process for work. Thank you so much!
You're very welcome Clayton! Super glad this was helpful :)
Short and sweet. This is the video I've been looking for, thanks heaps.
Thanks ben!
Such a simple explanation!! Thank you for your effort! At least 18k viewers were benefitted by your video!
Glad it was helpful!
I still reference old content and WOW, the explanations and video quality have come a long way in just two years.
Sometimes I look back on my old videos and think "what was I thinking!?". Glad you agree I've come a long way - It's been quite a journey!
I used to be Azure guy but just recently switched to AWS and I found your videos are very helpful! Moreover you do even better job to keep the video with specific information and don't drag it. Good work!! and keep going.
Thank you so much for your support! Its comments like yours that keep me motivated to keep going :)
clear, to the point, and useful. Thank you !!
Hurray! I vaguely understand what we did and it worked!
Your videos are really helpful to understand the concepts practically
Thanks Sweta! Glad you enjoyed :)
Thanks for this little tutorial! Didn't know you can filter for specific filetypes on the invocation.
You're very welcome Florian! Glad you learned something new!
Really awesome explanation with simple example. Keep coming.....
Thank you!
Very lean and clean :) Thanks
You're very welcome!
Great tutorial
Great presentation!
Thank you kindly!
Thanks for precise video.
You're very welcome!
Thanks for the video, I follow your videos as they are very nicely explained. Request you to please upload a video on how AWS sagemaker works?
Very well explained.
Glad it was helpful!
awesome video . very informative
Glad it was helpful!
Brilliant. Do you have a video that shows SNS Topic event upload to S3?
Thanks a lot very precise. Any info in what that code for Node would be?
Saved my job 💕
:D
Cheers from Brazil!
Thanks for watching Payador :) Stay safe!
Hi,
can you please let me know if there is any way to invoke or call AWS Appflow from lambda?
Thank you, this is very helpful. Quick question: if I am about to use Step Functions to run a few lambda functions in a batch style starting by a CloudWatch schedule event, but I also need these lambda functions to wait until particular S3 objects are uploaded to run. How does the S3 trigger work, will the lambda functions wait until they get S3 events?
Great job
Thank you Vishal!
Hi sir, thanks for the tutorial. i finished this tutorial.
but i have one scenario of json validation, while uploading a json file in the bucket using lambda it must be validate. i dont know how to do that
Thanks boss!
You bet!
How to handle if there are multiple files dropped in S3 at the same time where we need trigger one glue job? How should we handle Lambda here? Any help is appreciated.
Hey! I am facing an issue. My lambda function and S3 bucket both are in us-west-2 region but while creating event notification, I am not able to see my lambda function in the drop down list. What could be the potential issue?
thanks, good content
You're very welcome!
Duno why people would dislike such a nice video
Thanks AmjD!
Can you share a way to create a generic python lambda function which will copy data from multiple s3 buckets to corresponding tables in redshift in such a way that my credentials are not visible to anyone
Thanks for the step by step guide. I have a question. If we follow this approach, the lambda will trigger every time a new file is created in s3. So, if we get 10 files simultaneously, it's going to invoke lambda 10 times. Is it possible to invoke lambda only once in such cases?
This video was helpful in understanding how Events work. I had a doubt- Is it possible to retrieve the file path of the object? How do we write a lambda function for storing the file path to a database that is triggered every time a put event happens in a bucket?
I want to trigger a lambda for a file that’s two layers down in the s3 bucket (I.e s3/folder1/folder2/my-file.txt). How do I configure the lambda to ingest only files put in folder2?
So I was able to trigger lambda based on S3:ObjectCreated by uploading a file. But what happens when the whole S3 bucket region is down? In this case, the file won't be uploaded to the bucket and won't trigger the lambda. In this case, what do we do to still process the file when the bucket region is down?
if i wanted to call that data variable outside the function to look at the data locally, how would that look like?
Thanks for the video! How would you add an email notification to this process when the file is processed to notify that it's processed or if there was some kind of error while processing?
maybe invoke SNS and then configure the email there?
i am trying to write an xml file into s3 bucket using lambda but it is timing
out. Do you know what could be the reason?
Nice video, is it possible to do all that S3 config from a file?
@Be A Better Dev
Hi, Thanks for the video. I have a question, let's say we have a application that let user upload a file which goes to S3 and the file is processed by lambda. So in the event when user A & user B uploads a file to S3 with a negligible time difference, which file is processed first file from user A or user B? so basically, does S3 maintains the order of file to pass it to lambda to work on? Please guide. Thanks.
Hi Lalit,
Order would not be preserved - hope this clarifies!
How to configure email alert if daily routine based object is not uploaded in s3 object, can you help.
can s3 event send notification to lambda for get request ? if yes what is difference between using s3 event vs s3 object lambda ? please help me out
How to read metadata sent with the file that is uploaded
How can i get the codes and materials you use such as: the transaction.json etc
Can i take "screenshot" when the snapshots are created ?
My lambda is triggering but i am getting error in getting the bucket name. Key error. What I have to do?
Can I get the complete code?
I have tried the same python code in the lambda function, that just is giving errors at cloudwatch.
Thanks for this! I quickly fought through a couple of issues with imports and such, but woot this hit the spot! As soon as I got your demo running it was only another two hours for me to get the same running in sam. Now I just gotta figure out how to cloud formation that s3 event trigger into a sam CF template, and viola!
Glad I was able to help Tony. Sounds like you got quite a project on your hands!
It might be useful to have a video on how to use the Boto3 library. I find the documentation pretty unclear as to which option to use. For example, if I want to write to an S3 bucket, do I use resource, client, or bucket etc?
Great suggestion Sanjay, thank you!
Unable to find the phyton code in description, if you don't mind kindly share the code....
I am new to all of this, but is there a way to do all of this explained in the video via CLI?
Hi Leipa,
Absolutely you can do all of this via CLI commands. It may be a bit tricky though since you will need to find the correct API calls to make and arguments to provide.
Thanks for watching and stay safe!
Both S3-Lambda videos are good. What architecture you suggest when needing to "listen" to the s3 bucket (on local computer) and download content when a file is uploaded to s3 from another system?
Hi Igor,
Couple ways of doing this. The most robust would be S3 Put Notification -> SNS Topic -> Invoke HTTP Endpoint on your local machine. This way, your local application can get invoked by the S3 Put event!
@@BeABetterDev will give it a try very soon. Thank you!
I get YOU ARE NOT ALLOWED TO EDIT THIS SETTINGS PLZ CONTACT UR ADMIN - while I try to save the notification event setup for the bucket... Plz help
Everything is fine but the final log event getting the error. "urllib" not defined
Had the same error and just added 'import urllib' to the top!
How we generate this log in s3 bucket not in cloudwatch
thank you! i've a question: what happens if i upload the same file? lambda is executed?
Yes
I started exactly the same but I didn't get CloudWatch and S3 as layers. How to add them?
Hi elmira,
Are you certain you select a role with these permissions during lambda creation? If you select the 'create a new role' during setup it should set cloudwatch correctly.
@@BeABetterDev yes I'm sure although the whole view/console has changed now the resources/destination s etc have moved, so it's difficult to orient
Can you please share a way to create a python lambda function which will copy csv data from S3 bucket to corresponding tables in Postgres
By any chance did you figure out or have the python lambda function
can you please share me the json file link
I want to add cloud watch content to my rds table after Lambda is triggered. Is it possible and if yes How
Hi Abhishek,
You would need to use one of the cloudwatch APIs to query the data from your lambda function, and write it into your RDS table.
Hi Bro,
Can you cover terraform with s3 bucket.
Thanks in advance.
Good video, can you talk about multiples websites using aws?
I mean, if I want to serve a coffee bar and a blog, I need to use only on CloudFront distribution, or generate multiples?
I would like to see a video talking about serving multiples websites using aws, serverless prefer
Hi Lucas, thanks for the question. There are many ways to achieve this. The first thing that comes to mind is for you to use an EC2 Instance that hosts your applications. You can use the machine to host two separate stacks (LAMP, MEAN, or any other really). Here is an example (its a bit overkill in the redundancy category if you ask me), but it should help you get started: aws.amazon.com/getting-started/projects/launch-lamp-web-app/
@@BeABetterDev thanks hey!
I enjoyed your video. Can anyone help? I understood everything but I didn’t get where he edited the codes. Is that a lambda app? I mean the black screen on which he edited the codes. Is that a third party app ? Can anyone help?
Hi John the code is a lambda based application. Hope this helps clarify.
can you please do a video on how to trigger a glue etl job when a file is uploaded to s3?
Great idea for a video Pavan, I've added it to my backlog. Cheers.
@@BeABetterDev thank you so much
I am using the same code and tried with some other code I am getting an key error
{
"errorMessage": "'Records'",
"errorType": "KeyError",
"stackTrace": [
[
"/var/task/lambda_function.py",
11,
"lambda_handler",
"bucket = event['Records'][0]['s3']['bucket']['name']"
]
]
}
Can you help me with any suggestion
Hi Manas, make sure you are changing the code to account for whatever your bucket name is.
Daniel
@@BeABetterDev I am changing the code but it's not happening
Hello,
Could you please make a video on aws lmabda function with firehose
Hi Nisarg, coming soon! :)
I really like your videos, but do u think u can do tutorials with AWS CDK ?
Coming soon!!!
you ever done serverless? can you do a tutorial on setting up a local environment with serverless?
Coming soon!
where is the code ? where can we download it
Nice, however it's missing a fundamental part, get response from lambda into your app. I can't find the way...
Hello, could send the code please
where is the script? also you edited it, how do you expect a newbie to know what to do? thank U
download the object file to local system using lambda function.
Hi Prishu, thank you for the suggestion. I used it as a basis for my recent video about how to download a s3 file from lambda. See here: ruclips.net/video/6LvtSmJhVRE/видео.html
Thanks again
Hello sir we have need S3 download request count my website
Do you prefer windows or mac?
Hi John,
These days, I actually prefer Mac. I feel like I run in to less headaches when developing. Maybe I'm biased though since I use a Mac at work for most of my development.
name 's3' is not defined at line 10 in lambda function
Anyone else got the same error?
got the answer from other videos in the series
have to add
import boto3
s3 = boto3.client('s3') in the function at top
try to provide data set so that we also will implement all
Doesn't work for me. There's no indication that I put a file in s3 at all. Nothing happens, no logs, nothing
Hi Billy.
Are you sure you have precisely followed the instructions in the video including setting the appropriate IAM roles on your lambda?
Thanks,
@@BeABetterDev Yes. For some reason when I changed 'PUT' to 'All object create event' on S3 trigger it started working. Still no clue why.
@billykovalsky You might need to attach a policy to your Lambda permissions.
Steps after you created the permission to read items stored on the S3 bucket:
1. Go to IAM and click Roles
2. Click on the role name you created (I called my s3ReadOnly)
3. Click attach policy and search for CloudWatchFullAccess and attach it to your role
This should provide you with logs if an event is triggered.
If your log group isn't created( Which it should automatically generate) you can create one yourself with the naming convention: /aws/lambda/
can you please include ur source code in future videos so I dont have to type too much. pls and thx
7:26 "Uh yah let's go over to the CloudWatch log screen". Totally omitting the click-bait "View logs in CloudWatch logs" button on the Lambda screen. 🤣🙃😂🤣 The OP just thought it was a torrent phishing button placed there by the graces of a rouge site claiming they are his online AWS Lambda console.
[ERROR] KeyError: 'Records'
Beware : reading and writing to the same S3 bucket can get your bill skyrocket . So just follow each step correctly or try not to upload the file again to s3.
Very good tip!
Thanks for the video explanation. Getting below issue when running "test" for lambda code.
{
"errorMessage": "'Records'",
"errorType": "KeyError",
"stackTrace": [
[
"/var/task/lambda_function.py",
14,
"lambda_handler",
"bucket = event['Records'][0]['s3']['bucket']['name']"
]
]
}
Hi Anil,
You could possibly print out the 'event' object at the start of your lambda function to see if it contains any records. This may help you debug the problem.
@@BeABetterDev When i upload the file, it is working fine. The actual issue is - I am trying to test the function using "test". Thanks for your quick reply.
Hi Anil, in order for that to work you need to provide an input to the test event that is similar to the one that triggers the lambda. This may require some research on your end.
Alternatively you can hook up your bucket to the lambda and upload/delete/re-upload the file to trigger the lambda.
@@BeABetterDev Thank You. I just got to know that. FYIP, This is my first Lambda function.
I am not sure why you made a video if we are supposed to guess what write or click on!! What is that? Lowest quality video setting on your computer?!😅
Where is the Code, you could have put in the github.
import urllib.parse
import json
import boto3
# Initialize S3 client
s3 = boto3.client('s3')
def lambda_handler(event, context):
# 1. Get the bucket name
bucket = event['Records'][0]['s3']['bucket']['name']
# 2. Get the file/key name
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
try:
# 3. Fetch the file from S3
response = s3.get_object(Bucket=bucket, Key=key)
# 4. Deserialize the file's content
text = response["Body"].read().decode()
data = json.loads(text)
# 5. Print the content
print(data)
# 6. Parse and print the transactions
transaction = data["transactions"]
for record in transaction:
print(f"Transaction Type: {record['transType']}")
return "Success!"
except Exception as e:
print(e)
raise e
Lacks intuition so hard, kills brain cells trying to figure out.
extremely complicated compared to GCS
LIES
The python code with some which worked for me.
import json
import urllib
import boto3
def lambda_handler(event, context):
# TODO implement
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
try:
s3 = boto3.client('s3')
response = s3.get_object(Bucket=bucket,Key=key)
text = response["Body"].read().decode()
data = json.loads(text)
print(data)
transactions = data['transactions']
for record in transactions:
print(record['transType'])
return 'Success!'
except Exception as e:
print(e)
raise e
yes, for now, we should explicitly import the needed libraries.