Thank you to Darshil Parmar!. Please note that you deployed the lambda function at 39:00 minutes of the video. It is not mentioned specifically in your explanation. If not deployed it will only run the default code which will anyway run successfully with hello-world print.
thanks for the solution! i was stuck here for a long time 😅 Darshil Parmar thanks so much for the video! hopefully this solution can be pinned for others to refer to :)
What according to you will be the best resource to understand lambda in depth? I need help on that. I am working on bigdata project, but this was not my domain, learning new things and I need to learn faster. Any leads will be helpful for me. Thanks in advance. Also, please keep producing such awesome contents. Thanks a lot!!
Hi Darsheel .. great video as always and yes i did click on like 😀 ... Can you please make a video on how to create a project in Dev environment and then switch to production environment in AWS ..... Basically how to manage the code Lifecycle in AWS from Dev to Production.... or may be you can point to a resource ... Thanks
For people watching this tutorial now, AWS DataWrangler has been changed to AWS SDK for Pandas. Name has been changed but core functionality remains same
@@vishnuvardhan9082 hii, When I changed the layer to AWS SDKPandas and modified the code I found the same error Error : { "errorMessage": "Unable to import module 'lambda_function': No module named 'AWSSDKPandas'", "errorType": "Runtime.ImportModuleError", "stackTrace": [] }
Thank you to Darshil. This is for those who facing issues - 1) Replace awswrangler with awssdkpandas in the code. The rest code remains the same. 2) Add Layer : AWSDataWrangler-Python3.8 replaced it with AWSSDKPandas-Python3.8 version 10 3) Create db_youtube_cleaned db using Glue or Athena before running the code. 4) For Task timed out issue - increasing the memory along with time, for eg. time = 5 min, memory = 512 MB Hope this helps :) Tip: Guys, please go through the comments, if you are stuck. You will be able to find a solution for sure.
@snehakadam16 @Darshilparmar facing issues like "errorMessage": "Unable to import module 'lambda_function': No module named 'awssdkpandas'", "errorType": "Runtime.ImportModuleError", "stackTrace": [] help me out
it worked i think the database name should be = ""db_youtube_cleansed""" even awswrangler with AWSSDKPandas-Python3.8 version 10 and memory 256 mb is working fine for me. Thank you. But as per the video the database should get created automatically
@@DarshilParmar Function Logs START RequestId: 2b020a60-532e-4b33-9933-7cc87b5406cc Version: $LATEST An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found. Error getting object youtube/raw_statistics_reference_data/CA_category_id.json from bucket de-on-youtube-raw-useast1dev. Make sure they exist and your bucket is in the same region as this function. LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html [ERROR] EntityNotFoundException: An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.
Thank you Darshil for this wonderful video. One thing I would like to point out that might help others following this tutorial is whenever you update your lambda function, click on deploy first to actually test your changes. In my case I wasn't getting any errors and later realized that the default hello world code was still running.
Yes, I might have made mistake while editing the video, I did click on deploy and lot of people missed it. I will keep this in my mind Thank you for the feedback
@@DarshilParmar while we're at it, could you please give a solution for the EntityNotFoundException that somebody else also pointed out. I'm also getting the same error and haven't been able to resolve it. Tried creating the cleansed database in glue manually but still it is not working. Hope to get a reply. Thanks in advance :)
Just wanted to say thank you to Darshil Parmar for these projects. It's hard to find anything online that helps to this extent from end-to-end. This is great stuff! Cheers! :)
I am unable to proceed further after clicking on test getting err0r:"errorMessage": "'s3_cleansed_layer'", "errorType": "KeyError", can anyone pls tell what's the problem?
(edited) Important note on missing libraires : - AWSDataWrangler-Python3.8 is not still available - I replaced it with AWSSDKPandas-Python3.8 version 1
Amazing job - I'm just starting to use AWS because I would like to become a Cloud Engineer and this just incredible. Thank you a lot for your effort !!
Great Project Documentation to try for yourself. One little thing to add would be a rough aws cost estimate. Definitely a thing I would be looking for if I was starting.
Amazing @darshil ....It is clearly visible how much effort u have put in for ppt , video reording , storyboarding and including small small nuances and error that could be potentially faced. It can't express in words how valuable it is and how much information you are providing for the community. Really inspiring and motivating. Someone in other comment rightly mentioned It is a pure gem on RUclips
I am unable to proceed further after clicking on test getting err0r:"errorMessage": "'s3_cleansed_layer'", "errorType": "KeyError", can anyone pls tell what's the problem?
Great Stuff, You Rock boey! The entire video was very much intuitive and I must say that without a shadow of doubt that all the nitty gritties of Data is discussed in this, heading for the second part now. Worth a ⌚
prajwal, I am getting this error: { "errorMessage": "Glue table does not exist in the catalog. Please pass the `path` argument to create it.", "errorType": "InvalidArgumentValue", "stackTrace": [ " File \"/var/task/lambda_function.py\", line 40, in lambda_handler raise e ", " File \"/var/task/lambda_function.py\", line 27, in lambda_handler wr_response = wr.s3.to_parquet( ", " File \"/opt/python/awswrangler/_config.py\", line 735, in wrapper return function(**args) ", " File \"/opt/python/awswrangler/_utils.py\", line 178, in inner return func(*args, **kwargs) ", " File \"/opt/python/awswrangler/s3/_write_parquet.py\", line 719, in to_parquet return strategy.write( ", " File \"/opt/python/awswrangler/s3/_write.py\", line 313, in write raise exceptions.InvalidArgumentValue( " ] } Please help.
Thank you so much Darshil for this video. I have a question , does the glue table created automaticaly ? because i get a timout , and i think the problem is whit that . can u please provide with more information like should i create new crawel or what to run the create the cleand_table automaticly
faced the same issue , so i found out that you need to create the cleaned catalog_db in Glue then the cleaned_table will be created automaticly ; for timeout try to increase the memory to along with the time . hope it helps
Hi Darshil, great video, could you please let me know how to save to table db_youtube_cleaned as I am getting the error : "An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found." Thanks in advance :)
Thank you so much Darshil for the video! I am having an issue when trying to create a crawler, getting error : "The following crawler failed to create: "name of the crawler" Here is the most recent error message: Account 'Number of account' is denied access." Tried to check the IAM roles created, deleted recreated again, however still receiveing the same message. Would you have an idea what could be the issue?
hey darshil, everything is fine accept getting an error says "errorMessage": "An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.", "errorType": "EntityNotFoundException", how to resolve
Yeah, I got the same issue. It has to do w/ the Lambda "Configuration" tab > "Environment variables" we input @39:35. But I'm not entirely sure where the "Value" we input for each "Key" came from or is associated to?
@@naveenkonda395 just create a table in athena by tying the SQL query : create database db_youtube_cleaned first one has to create a table then only lambda will update it with the data
This is one of the biggest mistakes I found, It should be corrected because we were not told to create a new database and then the path is coming to be different. It took me 3 hours to debug this.
Hi Darshil! First of all, I would like to say "Thank you" for this tutorial. I need to mention something, I was following each steps but AWS is now different and some options are no longer available or they are so different. I can't believe that AWS platform changed so much in just one year. My question is: will you update this tutorial in the future?
If you are getting a runtime error when running the lambda function even after 3 minutes. Make sure to add import pandas as pd This will solve the issue as the AWS wrangler changed to AWS SDK Pandas
Thank you so much for making this video!!! This would be 6-7th video of yours which I've added to my playlist. I request you to post more such project videos in different domains.
Thank you Darshil for this amazing video, it was very helpful. Just completed this whole project plus did some extra work of moving data to redshift using glue job as well while creation connection and enabling vpc endpoint.:)
Bro can u plzz help me for this actually I stuck in part 1 during catalog data from1week so I can move forward if u tell my mistakes over their🙏it will big help for me
hello guys,you might be getting error at the point of testing that is because of db name has been not changed in environment variable, please take care he has forget to change db name , if you notice in athena database name is db_youtube_cleaned but it should be de_youtube_cleaned, which is giving error in lamda final testing as "Entity not found"
@@geekyprogrammer4831 facing same issue. I see a parquet file being generated in the gcs bucket, but the lambda function is timing out. Were you able to rectify it ?
Hi @DarshilParmar my lambda function is timing out, i increased the time to 15 minutes which is the max time but still it is not completing and my lambda function throwing an error. I followed the exact same steps shown in the video. Can you or someone suggest to me where I am going wrong?
Dear Darshil, Could you please let us know which architecture have you used in the demo -- Lambda architecture or Kappa Architecture. Wanted to understand more on architecture prospective. Please share your thoughts.
Hi Darshil while running the Athena job getting HIVE_CURSOR_ERROR: Row is not a valid JSON Object - JSONException: A JSONObject text must end with '}' at 2 [character 3 line 1] This query ran against the "de_database_raw" database, unless qualified by the query. error
@anusha kamath Solved it by : - Deleting data in the s3 bucket : youtube-cleaned-useast1-dev - Deleting "db_youtube_cleaned" database in AWS Glue - Recreating database in AWS Glue and name it : db_youtube_clean - Updating environment "glue_catalog_db_name" variable, rename it : db_youtube_clean - Updating environment "s3_cleansed_layer" variable in the lambda function by adding a / at the end of the path THEN -Refresh all and re execute the lambda function. - Then run the SQL query in Athena It worked by magic I don't know what was wrong, force it some time, delete, re upload, re run :)
Thanks for this great content. I'm getting errors with the lambda function: 1. Video is missing to indicate to hit the "Deploy" button. 2. After adding the layers, increasing timeout and granting permission to lambda function, I still get this: Test Event Name lambdaTestEvent Response { "errorMessage": "An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.", "errorType": "EntityNotFoundException", "stackTrace": [ " File \"/var/task/lambda_function.py\", line 39, in lambda_handler raise e ", " File \"/var/task/lambda_function.py\", line 26, in lambda_handler wr_response = wr.s3.to_parquet( ", " File \"/opt/python/awswrangler/_config.py\", line 450, in wrapper return function(**args) ", " File \"/opt/python/awswrangler/s3/_write_parquet.py\", line 666, in to_parquet catalog._create_parquet_table( # pylint: disable=protected-access ", " File \"/opt/python/awswrangler/catalog/_create.py\", line 301, in _create_parquet_table _create_table( ", " File \"/opt/python/awswrangler/catalog/_create.py\", line 152, in _create_table client_glue.create_table(**args) ", " File \"/var/runtime/botocore/client.py\", line 391, in _api_call return self._make_api_call(operation_name, kwargs) ", " File \"/var/runtime/botocore/client.py\", line 719, in _make_api_call raise error_class(parsed_response, operation_name) " ] } Function Logs START RequestId: e124c5fb-a734-417c-a227-f1ac36b93a11 Version: $LATEST An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found. Error getting object youtube/raw_statistics_reference_data/US_category_id.json from bucket de-on-youtube-raw-useast1-7011-dev. Make sure they exist and your bucket is in the same region as this function. [ERROR] EntityNotFoundException: An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found. Traceback (most recent call last): File "/var/task/lambda_function.py", line 39, in lambda_handler raise e File "/var/task/lambda_function.py", line 26, in lambda_handler wr_response = wr.s3.to_parquet( File "/opt/python/awswrangler/_config.py", line 450, in wrapper return function(**args) File "/opt/python/awswrangler/s3/_write_parquet.py", line 666, in to_parquet catalog._create_parquet_table( # pylint: disable=protected-access File "/opt/python/awswrangler/catalog/_create.py", line 301, in _create_parquet_table _create_table( File "/opt/python/awswrangler/catalog/_create.py", line 152, in _create_table client_glue.create_table(**args) File "/var/runtime/botocore/client.py", line 391, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 719, in _make_api_call raise error_class(parsed_response, operation_name)END RequestId: e124c5fb-a734-417c-a227-f1ac36b93a11 REPORT RequestId: e124c5fb-a734-417c-a227-f1ac36b93a11 Duration: 8167.89 ms Billed Duration: 8168 ms Memory Size: 128 MB Max Memory Used: 128 MB Init Duration: 3626.32 ms Can you please take a look at it?
Hey, Thanks for sharing this I made mistake in editing and missed that part For the error I’d say just create table on Athena directly Also you can join discord channel for futures queries
@@DarshilParmar Thank you. It worked. As a reference for who are getting the same error: the video is missing the creation of a database. To do so, go to Athena, create a new query with this SQL "create database de_youtube_cleaned" and run it. Lambda function should work fine after that
@@guillermojastrzebski954 massive thanks for this!! I've been searching the internet for couple of hours to rectify this. Presumed someone must have got into same issue and checked the comments section. Saved me a lot of trouble!
Hi Darshil , thank you for this beautiful and easy to understand concept , but while adding lambda layer , i can't find AWSDataWrangler for Virginia resgoin . I tried to deploy existing AWSDataWrangler layer and then add it through custom layer. I did succeed in that , but my data is not getting cleaned and getting stored on cleaned S3 bucket.
for those trying it now- 1. awswrangler name has been changed to awssdkpandas. Rest code wise - it remains the same 2. you need to have glue database created before hand, otherwise it throws error .
Hi Percy, while trying to add Aws layers, I only get 3 options - AppConfig Extension, Lambda Insights Extension, Parameters and Secret Lambda extension. Not sure what I am missing. Please help
This was a great help..one question though..when you executed this project using different AWS services S3, Athena, Glue etc.. what was the approx. cost you got after full project execution...Thanks
Most likely there won't be any charge if you are under free trial but even if they charge you it will be max 3-5$ You can raise support ticket stating you were just trying to learn about service and they won't charge you
Thank you so much Darshil for the video! I am having an issue when trying to create a crawler, getting error : "The following crawler failed to create: "name of the crawler" Here is the most recent error message: Account 'Number of account' is denied access." Tried to check the IAM roles created, deleted recreated again, however still receiveing the same message. Would you have an idea what could be the issue?
Absolute gem! Thank you for making this video. Learned a lot today. And if possible, Although I know you have your job, please try to make more of such content in future. Lots of love💛💛
I am unable to proceed further after clicking on test getting err0r:"errorMessage": "'s3_cleansed_layer'", "errorType": "KeyError", can anyone pls tell what's the problem?
Hi Darshil, Amazing content. I have a question. I am not able to find AWSwragler layer in options. Could you provide a link for downloading it so that I can custom it.
I can't find the Lambda function's AWS Datawrangler layer option. I can't even find the right arn for us-east-1. You did it at this timestamp: 45:08 Edit: AWS Datawrangler is now called AWS SDK for pandas
Hi Drashil. Thank u for the amazing work. Right now AWS doesnot have datawrangler lambda layer. so i am not able to execute the function. is there any other way to execute the function
same i am also unble to execute the lamda function. not able to test it . after the test ish shows { "statusCode": 200, "body": "\"Hello from Lambda!\"" } in the response Any suggestions ?
@DarshilParmar I tried using flatten transform in ETL job but it didn't work is it because json contains array? can you suggest me how to proceed with ETl in few words so that I can work on that
Hi Darshill, good video and thanks very much. I learned a lot. Please in your subsequent videos, do try to zoom in more often so we can get to see what you’re doing on the screen. Thanks.
Thank you Darshil for this awesome video! I have issues viewing the cleaned date in athena. i got "HIVE_UNKNOWN_ERROR: Path missing in file system location: [my path] This query ran against the "[cleaned db]" database, unless qualified by the query. " but i checked the path name are correct and i can access the parquet file locally. Can anyone help with this issue?
Sir, I just want to know how you know which code to write where you write in the lambda service, how do you know that now I have to use this package like os and so on. Is this is written i documentation or some were else
Hi Darshil, I always think of starting your project videos but I always got stuck whether aws cloud services willl be charged or it's free or is there any other alternatives
Hi @darshil #Darshil the cleansed glue table is not getting created for me but the parquet file under s3 cleansed bucket is getting created i have procceded with the correct stones the lamda function code is not changed the variables are correct the time out time 3 min 3 sec the roles are assigned correctly with s3 and glue full access Used the layer mentioned by some as aws sdk pandas ther error still is timeout or no error at all what place should i look for the reason for cleansed table not gettng create d
If you are getting a runtime error when running the lambda function even after 3 minutes. Make sure to add import pandas as pd This will solve the issue as the AWS wrangler changed to AWS SDK Pandas
Hey great video. I wanted to ask whether I will be charged for using AWS Athena coz it mentioned additional charges for using athena query when I opened it. Thanks for the video.
Thank you to Darshil Parmar!.
Please note that you deployed the lambda function at 39:00 minutes of the video. It is not mentioned specifically in your explanation.
If not deployed it will only run the default code which will anyway run successfully with hello-world print.
Yes, I have made mistake while editing the video, lot of people faced this error
thanks for the solution! i was stuck here for a long time 😅
Darshil Parmar thanks so much for the video! hopefully this solution can be pinned for others to refer to :)
@@DarshilParmar can you plz provide me a solution of error at 39:00 . i am getting error as
"{
"statusCode": 200,
"body": "\"Hello from Lambda!\""
}"
@@fahadbakshi5449 It's not an error, You need to click on Deploy button
This post needs to be pinned. I wasted todays trying to figure out how to make the lambda function work until I decided to go through the comments.
It takes a lot of effort and energy to execute the entire project and record it! I hope you find this useful and make sure you Like this video :)
What according to you will be the best resource to understand lambda in depth? I need help on that. I am working on bigdata project, but this was not my domain, learning new things and I need to learn faster. Any leads will be helpful for me. Thanks in advance. Also, please keep producing such awesome contents. Thanks a lot!!
@@aritra1414 check out AWS reinvent videos on RUclips on lambda and read white paper on lambda to understand more
Hi Darsheel .. great video as always and yes i did click on like 😀 ... Can you please make a video on how to create a project in Dev environment and then switch to production environment in AWS ..... Basically how to manage the code Lifecycle in AWS from Dev to Production.... or may be you can point to a resource ... Thanks
By following these steps is im able to build this project in azure
I respect your efforts but to be honest I didn't understand anything. please me it in simpler way or breakdown it into smaller chunk if you can
For people watching this tutorial now,
AWS DataWrangler has been changed to AWS SDK for Pandas. Name has been changed but core functionality remains same
thank you so much man, was going nuts on this! how did you know about this?
thanks man much appreciated
@@vishnuvardhan9082 hii,
When I changed the layer to AWS SDKPandas and modified the code I found the same error
Error :
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'AWSSDKPandas'",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
}
Thank you very much
thanks bhai
Thank you to Darshil.
This is for those who facing issues -
1) Replace awswrangler with awssdkpandas in the code. The rest code remains the same.
2) Add Layer : AWSDataWrangler-Python3.8 replaced it with AWSSDKPandas-Python3.8 version 10
3) Create db_youtube_cleaned db using Glue or Athena before running the code.
4) For Task timed out issue - increasing the memory along with time, for eg. time = 5 min, memory = 512 MB
Hope this helps :)
Tip: Guys, please go through the comments, if you are stuck. You will be able to find a solution for sure.
Thanks for putting this in one comment
Thank you for the amazing tutorial and putting a lot of effort @@DarshilParmar. Looking forward to more projects :)
@snehakadam16 @Darshilparmar facing issues like
"errorMessage": "Unable to import module 'lambda_function': No module named 'awssdkpandas'",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
help me out
it worked i think the database name should be = ""db_youtube_cleansed"""
even awswrangler with AWSSDKPandas-Python3.8 version 10 and memory 256 mb is working fine for me. Thank you.
But as per the video the database should get created automatically
How did you solved this ?@@vamsivenna
Darshil, you have made one of the most valuable DE learning channels on youtube. Keep up the amazing work! Thank you.
Thanks, will do!
@@DarshilParmar Function Logs
START RequestId: 2b020a60-532e-4b33-9933-7cc87b5406cc Version: $LATEST
An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.
Error getting object youtube/raw_statistics_reference_data/CA_category_id.json from bucket de-on-youtube-raw-useast1dev. Make sure they exist and your bucket is in the same region as this function.
LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
[ERROR] EntityNotFoundException: An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.
After consistently working for 2 days, finally done with the project.
How did you solve the runtime problem?
@@sembrueldorinvil4167same question bro
How did you solve runtime lamda error ?
Found the answer. Assign memory to your function. I think 500 MB should work.
@@sembrueldorinvil4167 then also same error so much frustrating it is
Thank you Darshil for this wonderful video. One thing I would like to point out that might help others following this tutorial is whenever you update your lambda function, click on deploy first to actually test your changes. In my case I wasn't getting any errors and later realized that the default hello world code was still running.
Yes, I might have made mistake while editing the video, I did click on deploy and lot of people missed it.
I will keep this in my mind
Thank you for the feedback
@@DarshilParmar while we're at it, could you please give a solution for the EntityNotFoundException that somebody else also pointed out. I'm also getting the same error and haven't been able to resolve it. Tried creating the cleansed database in glue manually but still it is not working. Hope to get a reply.
Thanks in advance :)
Thanks Ashutosh ,same thing faced by me .thanks for posting this comment ,its helpful.
@@arpansatpathi9645 was it resolved ?if yes could you post the solution?
@@arpansatpathi9645 rename 'db_youtube_cleaned' to 'de_youtube_cleaned' in env varialble
This project is insane and I'm only halfway. As someone that looked at AWS I was like HUH. You did a great job explaining it so far. :)
You can do it!
Just wanted to say thank you to Darshil Parmar for these projects. It's hard to find anything online that helps to this extent from end-to-end. This is great stuff! Cheers! :)
I am unable to proceed further after clicking on test getting err0r:"errorMessage": "'s3_cleansed_layer'",
"errorType": "KeyError", can anyone pls tell what's the problem?
Are You Solve the error@@mcaddit6802
(edited) Important note on missing libraires :
- AWSDataWrangler-Python3.8 is not still available
- I replaced it with AWSSDKPandas-Python3.8 version 1
I am having error for database db_youtube_cleaned not found can you please check once?
@@anupammathur918 same here
@@soumyaranjandash3597 becz that is not created go to athena nd create one db with that name
@@anupammathur918 Thanks this helped. I created the database in Glue and it worked
@@anupammathur918 can you please elaborate in anthena data sources are there
Amazing job - I'm just starting to use AWS because I would like to become a Cloud Engineer and this just incredible. Thank you a lot for your effort !!
Great Project Documentation to try for yourself. One little thing to add would be a rough aws cost estimate. Definitely a thing I would be looking for if I was starting.
Your content on data is pure! No match.
Thank you
Did you get charged while using AWS services under free tier?🤔
Hi, Darshil thanks for this project. Faced a lot of errors but took two days to complete the project. In the end, it is satisfying.😀
can you give the estimated cost of aws services used in this project it will be of so much help and thank you
@@vanadin8009 we can do basically for free with free tier AWS account
@@vanadin8009 Can you answer now ?
Hi.. need a help, I have free tier account and if I use Glue will it cost me?, if, yes. then how much for this work.
thankyou :)
@@ADESHKUMAR-yz2el for this project overall in one year less than $5
Amazing @darshil ....It is clearly visible how much effort u have put in for ppt , video reording , storyboarding and including small small nuances and error that could be potentially faced.
It can't express in words how valuable it is and how much information you are providing for the community. Really inspiring and motivating.
Someone in other comment rightly mentioned It is a pure gem on RUclips
These tutorials are so helpful for me! And also, they show how much effort on production you put into them. Thank you so much, Darshil!
Took me 5 days to complete this video with hands on.
But these were all worth it.
- Complete noob me
Bro help me to solve run time error @rohanchoudhary672
Thank you so much Darshil for this video. We need more such project based learning in data engineering instead of just cliche theory. 😆
Yes, more videos like this is coming
I am unable to proceed further after clicking on test getting err0r:"errorMessage": "'s3_cleansed_layer'",
"errorType": "KeyError", can anyone pls tell what's the problem?
Darshil, this is an amazingly executed project and it was easy to follow. Thanks for taking the time to put this together. Great channel.
THANKS A LOT DARSHIL FOR HELPING IN MY FIRST DATA ENGINEERING PROJECT.
Best video I have seen so far for the end to end project in big data. Thanks!
This project and you explaining it in a simple and elaborate way was awesome. Keep them coming!
hey Darshil, I rarely comment but just wanted to say a big thank u. This helped me out a lot! Looking forward to more of such content in the future :D
Excellent explanation. Each and every step is easy to follow and understandable.
Thank you Darshil, I completed this part successfully. Your content is invaluable may God bless you
bro i got stuck at 30:00 minute can u plz help me
How did you solve runtime error
Great Stuff, You Rock boey! The entire video was very much intuitive and I must say that without a shadow of doubt that all the nitty gritties of Data is discussed in this, heading for the second part now. Worth a ⌚
Hey Darshil, if you were a beginner how would you write such long code in lemda? I am a beginner and wondering how will I be able to write such codes
FYI - Now AWS Wrangler has been renamed to AWS SDK Pandas
Lifesaver ! Thanks
prajwal, I am getting this error:
{
"errorMessage": "Glue table does not exist in the catalog. Please pass the `path` argument to create it.",
"errorType": "InvalidArgumentValue",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 40, in lambda_handler
raise e
",
" File \"/var/task/lambda_function.py\", line 27, in lambda_handler
wr_response = wr.s3.to_parquet(
",
" File \"/opt/python/awswrangler/_config.py\", line 735, in wrapper
return function(**args)
",
" File \"/opt/python/awswrangler/_utils.py\", line 178, in inner
return func(*args, **kwargs)
",
" File \"/opt/python/awswrangler/s3/_write_parquet.py\", line 719, in to_parquet
return strategy.write(
",
" File \"/opt/python/awswrangler/s3/_write.py\", line 313, in write
raise exceptions.InvalidArgumentValue(
"
]
}
Please help.
Great video. Loved it and helped me build my resume. Would love to do what you do today and freelance
I have a question and i hope you will answer,
is these services free in aws free tier
Thank you so much Darshil for this video.
I have a question , does the glue table created automaticaly ? because i get a timout , and i think the problem is whit that . can u please provide with more information like should i create new crawel or what to run the create the cleand_table automaticly
faced the same issue , so i found out that you need to create the cleaned catalog_db in Glue then the cleaned_table will be created automaticly ; for timeout try to increase the memory to along with the time . hope it helps
Hi Darshil, great video, could you please let me know how to save to table db_youtube_cleaned as I am getting the error : "An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found." Thanks in advance :)
Can you timestamp the video? I can explain it easily then
@@DarshilParmar yes the error occurs at 45.37 but I think it relates to the 39.30 where the names are set
I'm facing the same issue
I'm facing the same error as well.
@@dorothysilverman7660 @Arpan Satpathi first create database name ' db_youtube_cleaned ' in glue, then run test.
i tried deploy button at 39:00 and changed time to 15 min (which is max), but it still shows timeout, anybody knows how to fix it?
among this and covid project ..which one should i try building first as a beginner ?
Thank you so much Darshil for the video! I am having an issue when trying to create a crawler, getting error : "The following crawler failed to create: "name of the crawler"
Here is the most recent error message: Account 'Number of account' is denied access." Tried to check the IAM roles created, deleted recreated again, however still receiveing the same message. Would you have an idea what could be the issue?
me too bro,
do you solve already?
I also got this error anyone solved this
WOWW. This is amazingggg. Thanks Darshil. I have just started watching the video and looking at the content got me excited.
This is the type of comment I wait for, thanks for supporting my work!
@@DarshilParmar you’re welcome
hey darshil, everything is fine accept getting an error says
"errorMessage": "An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.",
"errorType": "EntityNotFoundException",
how to resolve
Yeah, I got the same issue. It has to do w/ the Lambda "Configuration" tab > "Environment variables" we input @39:35. But I'm not entirely sure where the "Value" we input for each "Key" came from or is associated to?
@@ajtam05 have you solved this issue?
@@naveenkonda395 just create a table in athena by tying the SQL query :
create database db_youtube_cleaned
first one has to create a table then only lambda will update it with the data
This is one of the biggest mistakes I found, It should be corrected because we were not told to create a new database and then the path is coming to be different. It took me 3 hours to debug this.
at 40:48 where all all these names for environment variables coming from, and what are they even
These used in code as db name, path etc…
You can directly hard code it but best practice is to store in env
Hi Darshil! First of all, I would like to say "Thank you" for this tutorial.
I need to mention something, I was following each steps but AWS is now different and some options are no longer available or they are so different.
I can't believe that AWS platform changed so much in just one year.
My question is: will you update this tutorial in the future?
Everything is same, you just have to find right options with new UI
@@DarshilParmar Thanks for your quick response!!
If you are getting a runtime error when running the lambda function even after 3 minutes. Make sure to add
import pandas as pd
This will solve the issue as the AWS wrangler changed to AWS SDK Pandas
Great Work, Darshil!!! Next level Data Engineering knowledge provided by you in this content. ✌️ Industry level project.
Thank you for supporting
Thank you so much for making this video!!! This would be 6-7th video of yours which I've added to my playlist. I request you to post more such project videos in different domains.
Thank you, yes I will try to post such videos
Thank you Darshil for this amazing video, it was very helpful. Just completed this whole project plus did some extra work of moving data to redshift using glue job as well while creation connection and enabling vpc endpoint.:)
Part1 done. It was really informative. Waiting for part 2!
Check link in the description for that
Bro can u plzz help me for this actually I stuck in part 1 during catalog data from1week so I can move forward if u tell my mistakes over their🙏it will big help for me
@@harshalshende69do you find solution
Thank you so much for doing this.... The only one on youtube with curated data related content ❤💫
Thank you for your support and kind words
Hi Darshil, great work, I faced the issue of "Access Denied" in creating Crawler please help me out of this issue
did you solve it?
have you solved??
this is a great project with your excellent guidance Darshil. Thank you!😀
hello guys,you might be getting error at the point of testing that is because of db name has been not changed in environment variable, please take care he has forget to change db name , if you notice in athena database name is db_youtube_cleaned but it should be de_youtube_cleaned, which is giving error in lamda final testing as "Entity not found"
At 46:51, it either gives RunTime error or Timeout Error to me. Kindly help!
@@geekyprogrammer4831 I have changed db name in environment variable but it still gave RunTime error. What should I do to fix this?
@@geekyprogrammer4831 facing same issue. I see a parquet file being generated in the gcs bucket, but the lambda function is timing out. Were you able to rectify it ?
@@geekyprogrammer4831 you got the solution? Please let me know?
@@21-lengocmai97 u got the solution? Please let me know
how we have declared the environment variables at 40.49 minutes of the video?
I hope you have more such projects for us in your pipeline 😍 Please do it
Yes, I will work on it
Can u do the AWS part with Google cloud as well
Sure, I will see what I can do
What is the similar service in GCP for AWS Glue Crawler ?
Hi @DarshilParmar my lambda function is timing out, i increased the time to 15 minutes which is the max time but still it is not completing and my lambda function throwing an error. I followed the exact same steps shown in the video. Can you or someone suggest to me where I am going wrong?
I'm stuck in the same part. My lambda fuction does not execute, it always throw a timed out error. @DarshilParmar
Try increasing memory too
@@DarshilParmar Hey, Increased it to 2048MB and it took hardly 2 sec to execute. Thank you for your help. Appreciate it.
I just increased it to 500MB and it also ran very fast! Thank you both very much!
Hi Darshil,
Can explain some more details on environment variable inputs.
Google :)
Great work Darshil! 🔥🔥
Dear Darshil, Could you please let us know which architecture have you used in the demo -- Lambda architecture or Kappa Architecture. Wanted to understand more on architecture prospective. Please share your thoughts.
Thanks a loads for this video Darshil! Very much appreciated! 👏👏👏👏
Hi Darshil while running the Athena job getting HIVE_CURSOR_ERROR: Row is not a valid JSON Object - JSONException: A JSONObject text must end with '}' at 2 [character 3 line 1]
This query ran against the "de_database_raw" database, unless qualified by the query. error
Thank you so much for this darshil!...keep up the good work🙌
Thank you, I will
Excellent work. Thank you for this amazing workshop.
Thanks alot brother..needed such kind of lecture ..really helped in enhancing my skills..please make more such videos
At last when I'm running SELECT query on cleaned_statistics_reference_data, it's giving HIVE_UNKNOWN_ERROR: Path is not absolute s3//
Plz help
Hi, did you find a solution to this issue?
@anusha kamath Solved it by :
- Deleting data in the s3 bucket : youtube-cleaned-useast1-dev
- Deleting "db_youtube_cleaned" database in AWS Glue
- Recreating database in AWS Glue and name it : db_youtube_clean
- Updating environment "glue_catalog_db_name" variable, rename it : db_youtube_clean
- Updating environment "s3_cleansed_layer" variable in the lambda function by adding a / at the end of the path
THEN
-Refresh all and re execute the lambda function.
- Then run the SQL query in Athena
It worked by magic I don't know what was wrong, force it some time, delete, re upload, re run :)
Hello what should i learn before doing this project? What are the prerequisites? Thanks
Thank you so much for teaching us such valuable content! Be blessed
hello thank you for this video. I am having a problem while configuring, every time i configure I get exited out.
do you have a end to end but not with AWS? (like hadoop or spark?)
Thanks for this great content.
I'm getting errors with the lambda function:
1. Video is missing to indicate to hit the "Deploy" button.
2. After adding the layers, increasing timeout and granting permission to lambda function, I still get this:
Test Event Name
lambdaTestEvent
Response
{
"errorMessage": "An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.",
"errorType": "EntityNotFoundException",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 39, in lambda_handler
raise e
",
" File \"/var/task/lambda_function.py\", line 26, in lambda_handler
wr_response = wr.s3.to_parquet(
",
" File \"/opt/python/awswrangler/_config.py\", line 450, in wrapper
return function(**args)
",
" File \"/opt/python/awswrangler/s3/_write_parquet.py\", line 666, in to_parquet
catalog._create_parquet_table( # pylint: disable=protected-access
",
" File \"/opt/python/awswrangler/catalog/_create.py\", line 301, in _create_parquet_table
_create_table(
",
" File \"/opt/python/awswrangler/catalog/_create.py\", line 152, in _create_table
client_glue.create_table(**args)
",
" File \"/var/runtime/botocore/client.py\", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
",
" File \"/var/runtime/botocore/client.py\", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)
"
]
}
Function Logs
START RequestId: e124c5fb-a734-417c-a227-f1ac36b93a11 Version: $LATEST
An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.
Error getting object youtube/raw_statistics_reference_data/US_category_id.json from bucket de-on-youtube-raw-useast1-7011-dev. Make sure they exist and your bucket is in the same region as this function.
[ERROR] EntityNotFoundException: An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 39, in lambda_handler
raise e
File "/var/task/lambda_function.py", line 26, in lambda_handler
wr_response = wr.s3.to_parquet(
File "/opt/python/awswrangler/_config.py", line 450, in wrapper
return function(**args)
File "/opt/python/awswrangler/s3/_write_parquet.py", line 666, in to_parquet
catalog._create_parquet_table( # pylint: disable=protected-access
File "/opt/python/awswrangler/catalog/_create.py", line 301, in _create_parquet_table
_create_table(
File "/opt/python/awswrangler/catalog/_create.py", line 152, in _create_table
client_glue.create_table(**args)
File "/var/runtime/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)END RequestId: e124c5fb-a734-417c-a227-f1ac36b93a11
REPORT RequestId: e124c5fb-a734-417c-a227-f1ac36b93a11 Duration: 8167.89 ms Billed Duration: 8168 ms Memory Size: 128 MB Max Memory Used: 128 MB Init Duration: 3626.32 ms
Can you please take a look at it?
Hey,
Thanks for sharing this
I made mistake in editing and missed that part
For the error I’d say just create table on Athena directly
Also you can join discord channel for futures queries
@@DarshilParmar Thank you. It worked.
As a reference for who are getting the same error: the video is missing the creation of a database. To do so, go to Athena, create a new query with this SQL "create database de_youtube_cleaned" and run it. Lambda function should work fine after that
@@guillermojastrzebski954 This really helped me! Thanks :)
@@guillermojastrzebski954 massive thanks for this!!
I've been searching the internet for couple of hours to rectify this. Presumed someone must have got into same issue and checked the comments section. Saved me a lot of trouble!
I had to make the table in glue, and then lambda worked as well
Hi Darshil , thank you for this beautiful and easy to understand concept , but while adding lambda layer , i can't find AWSDataWrangler for Virginia resgoin . I tried to deploy existing AWSDataWrangler layer and then add it through custom layer. I did succeed in that , but my data is not getting cleaned and getting stored on cleaned S3 bucket.
Most likely some policy issue
for those trying it now-
1. awswrangler name has been changed to awssdkpandas. Rest code wise - it remains the same
2. you need to have glue database created before hand, otherwise it throws error .
Hi Percy, while trying to add Aws layers, I only get 3 options - AppConfig Extension, Lambda Insights Extension, Parameters and Secret Lambda extension. Not sure what I am missing. Please help
@Darshil
It Was An Amzaing Project. I Learned Alot From This Video. Thank You So Much, Appreciated The Work.
You are most welcome
Great Video, Thank you so much!!
Hi Darshil, please make video on how do you deliver the project to your clients after completing it.
i'm facing problem at the "aws s3 ls". I clicked it on my cmd, but still there was no activity. I'm a windows 10 user. Kindly help.
This was a great help..one question though..when you executed this project using different AWS services S3, Athena, Glue etc.. what was the approx. cost you got after full project execution...Thanks
Most likely there won't be any charge if you are under free trial but even if they charge you it will be max 3-5$
You can raise support ticket stating you were just trying to learn about service and they won't charge you
Does it cost money to use aws services? Specifically for Athena
You can set billing alarm first and also check aws free tier if it is included
The AWS tools used in this project are available in the free tier?
Great video man.
AWS has updated the Emphemeral storage of Lambda to 10 GB
Thank you so much Darshil for the video! I am having an issue when trying to create a crawler, getting error : "The following crawler failed to create: "name of the crawler"
Here is the most recent error message: Account 'Number of account' is denied access." Tried to check the IAM roles created, deleted recreated again, however still receiveing the same message. Would you have an idea what could be the issue?
Absolute gem! Thank you for making this video. Learned a lot today.
And if possible, Although I know you have your job, please try to make more of such content in future.
Lots of love💛💛
I will try my best to provide as much as I can
I am unable to proceed further after clicking on test getting err0r:"errorMessage": "'s3_cleansed_layer'",
"errorType": "KeyError", can anyone pls tell what's the problem?
How do you solve runtime error
@@mcaddit6802 yeah I am also getting same error, how did you solved it. Can you help me out
Hi Darshil,
Amazing content. I have a question.
I am not able to find AWSwragler layer in options. Could you provide a link for downloading it so that I can custom it.
Go through comments, you will find solution
Awesome tutorial project. I could complete this session.
I can't find the Lambda function's AWS Datawrangler layer option. I can't even find the right arn for us-east-1. You did it at this timestamp: 45:08
Edit: AWS Datawrangler is now called AWS SDK for pandas
Really appreciate your hardwork you bring to tthe table!!!
Thank you making my hardwork pay off by watching video
Hi Drashil. Thank u for the amazing work. Right now AWS doesnot have datawrangler lambda layer. so i am not able to execute the function. is there any other way to execute the function
same i am also unble to execute the lamda function. not able to test it .
after the test ish shows
{
"statusCode": 200,
"body": "\"Hello from Lambda!\""
}
in the response
Any suggestions ?
did you get the answer? I am facing the same issue!@@sambitkumar7621
hello Darshil instead of lamda can we do the same sort of transformation using ETL Glue job ?
Yes
@DarshilParmar I tried using flatten transform in ETL job but it didn't work
is it because json contains array?
can you suggest me how to proceed with ETl in few words so that I can work on that
Will the AWS account be free foverver, to use the above mentioned features, or will it be free only for few months
Only 12 months
Hi Darshill, good video and thanks very much. I learned a lot. Please in your subsequent videos, do try to zoom in more often so we can get to see what you’re doing on the screen. Thanks.
Thank you Darshil for this awesome video! I have issues viewing the cleaned date in athena. i got "HIVE_UNKNOWN_ERROR: Path missing in file system location: [my path]
This query ran against the "[cleaned db]" database, unless qualified by the query. " but i checked the path name are correct and i can access the parquet file locally. Can anyone help with this issue?
I'm also facing the same issue
Thank you, man.. Keep up the good work!
Hi Darshil,
When did you create the database "db_youtube_cleaned" in the video?
Do you get CreateTable operation: Database db_youtube_cleaned not found. error ? , have you resolved it?
@@anishkini6901 first create database name ' db_youtube_cleaned ' in glue, then run test.
not able to remove the timeout error
Sir, I just want to know how you know which code to write where you write in the lambda service, how do you know that now I have to use this package like os and so on. Is this is written i documentation or some were else
Practice practice and practice
When you start you will research and the more your research more things you will find
Hi Darshil, I always think of starting your project videos but I always got stuck whether aws cloud services willl be charged or it's free or is there any other alternatives
its a great one, very useful n resourceful for aspirants like me👍🏼
thank you darshil very informative content........please upload 2nd part of video
It is uploaded, check link in the description
I gave all the permissions to the role I created, but while creating the crawler, it says access denied, please help!
did you figure this out? Im having the same issue and not able to move forward.
@@SajjanDivya anyone did figure it out>??
same issue
Hi @darshil #Darshil the cleansed glue table is not getting created for me but the parquet file under s3 cleansed bucket is getting created
i have procceded with the correct stones
the lamda function code is not changed
the variables are correct
the time out time 3 min 3 sec
the roles are assigned correctly with s3 and glue full access
Used the layer mentioned by some as aws sdk pandas
ther error still is timeout or no error at all
what place should i look for the reason for cleansed table not gettng create d
If you are getting a runtime error when running the lambda function even after 3 minutes. Make sure to add
import pandas as pd
This will solve the issue as the AWS wrangler changed to AWS SDK Pandas
Hey great video. I wanted to ask whether I will be charged for using AWS Athena coz it mentioned additional charges for using athena query when I opened it. Thanks for the video.
Did anyone faced the issue where I deployed the code too but when i click test nothing happens. No error and nothing. Am i missing anything?
Hey Darshil ,i rarely comment but just wanted to tell you tht you are awesome nd you content is just amazing ❤️
Thank you Siddhi