@@AjayWadhara No worries, thanks for letting me know. Can you just confirm that the demo that you showed in this tutorial comes in Hive-style partitioning type?
First of thanks for sharing userful knowledge..Need to Know One thing I need to Move Data from RDS->DMS->S3->Query with Athena. Data is mainly date range, can DMS automatically create S3 partitions based on Year and Date... I know this thread is for athena.. but if u have experience please share that as well
Hi Jatinder, i don’t have hands on experience with DMS but yeah DMS supports date based folder partitioning when using S3 as target. Also, try to use parquet if you want to query data from Athena. I hope this helps
I am glad you liked it. Sorry, I don't have that tutorial. You can implement that very easily with Lambda. Lambda should be triggered on S3 object upload.
Thank you for the video. Simple and easy to understand. It would be great if you can make video on Lamda funtion to create partition based on S3 folder.
Can I do partition while saving a view as a table? For instance, I'm joining 4 tables and doing some logic to build a view. I want to save the view as a table, can I use partitioning in that case?
Hi Ajay thanks for the video and I have a use case of anayzing vpc flow logs using athena and glue crawler from data stored in s3 can you share your inputs to proceed on this use case..
Hi Rupesh, you can follow the same strategy that I have demonstrated, only difference would be I am using CSV, you have to use a regex while parsing the log files.
Hi bro thanks for the detailed video. Am new to AWS and learning. Just wnat to know do we need to manually create folders and place file in S3 for partitioning?
Hi, This is Trupti.. the video is very useful but i have one que that What if ur s3 data is stored as non partitioned means it has mixed data in which there are multiple years so how we will do it??
You have to write a process for the same. For new data, keep a staging bucket, trigger a lambda that partitions your data from staging to the main bucket
Best video. Useful
Thanks Ajay. well explained
Exactly what I need. Thank you for sharing!! 👏👏
Glad you liked 👍👍
Thank you! Really good content!
This made my day 🚀
Bless you mate! This really helped :)
Thanks for this video.
Do subscribe and share... it motivates me to create more content 🎯✅
One of the best - can you cover BUCKETING in Athena and maybe BUCKETING vs. PARTITIONING in Athena - Great Job!
Thank you so much for this informative video.
Can you please let me know that how can I do Hive-style partition and spark partitions here in aws
Sorry.. i am not aware much about Spark partitions and would not be able to help you on this.
@@AjayWadhara No worries, thanks for letting me know. Can you just confirm that the demo that you showed in this tutorial comes in Hive-style partitioning type?
Hi Ajay, Your explanation was Damn good! Can you please make a serious of vidoes on AWS Glue that would be helpfull
First of thanks for sharing userful knowledge..Need to Know One thing I need to Move Data from RDS->DMS->S3->Query with Athena. Data is mainly date range, can DMS automatically create S3 partitions based on Year and Date... I know this thread is for athena.. but if u have experience please share that as well
Hi Jatinder, i don’t have hands on experience with DMS but yeah DMS supports date based folder partitioning when using S3 as target.
Also, try to use parquet if you want to query data from Athena.
I hope this helps
Thank you for your sharing, do you have the tutorial for data store in S3 as /year/month/date/hour? If so, please send me the link. Thank you again
I am glad you liked it.
Sorry, I don't have that tutorial. You can implement that very easily with Lambda. Lambda should be triggered on S3 object upload.
Thank you for the video. Simple and easy to understand.
It would be great if you can make video on Lamda funtion to create partition based on S3 folder.
Thanks for the feedback
Regarding partitioning with Lambda function, I will definitely cover that in coming week
Hi, I have upload the video to create partition based on S3 folder...Do check in my uploaded videos
Can you provide the documentation for alter table command trigger in aws lambda
How did you change icon of windows 10 as xp in task bar? I also want like that.
Can I do partition while saving a view as a table? For instance, I'm joining 4 tables and doing some logic to build a view. I want to save the view as a table, can I use partitioning in that case?
Hi Ajay thanks for the video and I have a use case of anayzing vpc flow logs using athena and glue crawler from data stored in s3 can you share your inputs to proceed on this use case..
Hi Rupesh, you can follow the same strategy that I have demonstrated, only difference would be I am using CSV, you have to use a regex while parsing the log files.
Hi bro thanks for the detailed video. Am new to AWS and learning. Just wnat to know do we need to manually create folders and place file in S3 for partitioning?
Not Manually, you can write Lambda that does this for you or the process which is writing the files to S3 should do this.
@@AjayWadhara thanks for the clarification bro !
Hi, This is Trupti.. the video is very useful but i have one que that What if ur s3 data is stored as non partitioned means it has mixed data in which there are multiple years so how we will do it??
You have to write a process for the same. For new data, keep a staging bucket, trigger a lambda that partitions your data from staging to the main bucket