Superstore Data Analysis | End to End AWS Data Engineering Project for Beginners
HTML-код
- Опубликовано: 16 окт 2024
- In this video we will build a end to end data engineering project using AWS. This project will use following AWS services:
1- IAM
2- S3
3- AWS Glue
4- AWS Athena
5- AWS Quicksight
Here is the Kaggle dataset: www.kaggle.com...
Steps to create AWS account: repost.aws/kno...
Start your Data Analytics Journey : www.namastesql...
Zero to hero(Advance) SQL Aggregation:
• All About SQL Aggregat...
Most Asked Join Based Interview Question:
• Most Asked SQL JOIN ba...
Solving 4 Trick SQL problems:
• Solving 4 Tricky SQL P...
Data Analyst Spotify Case Study:
• Data Analyst Spotify C...
Top 10 SQL interview Questions:
• Top 10 SQL interview Q...
Interview Question based on FULL OUTER JOIN:
• SQL Interview Question...
Playlist to master SQL :
• Complex SQL Questions ...
Rank, Dense_Rank and Row_Number:
• RANK, DENSE_RANK, ROW_...
#aws #dataengineer #project
I will be doing more complex projects in the future. Please support this initiative by liking the video and sharing it with your friends.
Will you create only projects or as a pre-requisite you may dig deep into particular topics related to aws data engineering first and then create projects in which those topics would be used? BTW, thank you for sharing this. I liked it.
It will be helpful if you do a project on AWS using Glue S3 Lambda and Redshift
more complex projects will br very helpful sir.
Thank you, Ankit, for the informative and approachable contents to kickstart into AWS for beginners... There's much more to delve into for real-world challenges, but your dedication to taking time from your busy schedule to assist peoples is truly commendable... Your support for needy peoples is invaluable. Thanks and keep up the great work!
More to come 🫴
Great guidance I ever scene in field of AWS. I completed that project successfully. You gave best guidance while teaching. I also viewed some youtubers but their guidance worst as when you are at mid of the project some error happens and they skip that process and you stuck.
Made more videos on Data Engineering. We want your guidance sir.......Love from Maharashtra
Great video Ankit. Hope to see more real world project based videos from you. Thanks for your efforts.❤
This is so amazing for someone to excel in data engineering!!
It's a great initiative Ankit. You explained the project in a very detailed way and I am able to understand it easily. Crisp and clear. Please do more projects based on Glue jobs, SES, SNS, lambda, cloud formation, CI/CD for a project with github actions etc.
Sure
Good video for starting with AWS glue. I am looking forward for end to end ETL project using AWS glue and pyspark where we will be extracting data from a database (eg. MySQL), storing it into S3, doing some transformation on it and finally loading it into Redshift. Request you to please create one such project video.
Great suggestion!
When I worked at Anz, I saw you and became a member of your Python and SQL batch. I am your biggest admirer and follower.
For all data engineering careers, your channel is the place to go.
I'm grateful for your efforts 🎉🎉🎉🎉
Awesome, thank you! 🙂
Sir please upload complete play list on AWS and BIG Data.
🙏🙏
This helped pave a way for one of my existing projects, where I was facing a blocker.
Thank you Ankit
Great 👍
Hi Ankit, Thanks for creating AWS informative content that makes complex topics easy to understand. Much appreciated! Expecting more such videos..
Nice initiative bro
I felt in the coming videos you can add some uses of Aws service we are using in the videos.. also some more usecases..
Sure 👍
This is really a great video. Please make more by taking bigger data & show us the lifecycle like end to end project on data analysis.
Thank you , Ankit this very useful for AWS beginners. your videos are always helpful for interview
Glad to hear that😊
Sir, please continue more on this series thanks.
That's a great initiative sir, keep them coming!🎉
More to come!
Thanks for your efforts Ankit... Please continue this alonn with python problem solving video like SQL.. Waiting for python Ankit..
Thnkss alot for this Ankit bro 😎
Really like this project Ankit Bhai, Thank you very much
Thanks for liking😊
Thanks man we are waiting for this video ❤😂
Hello Ankit,
I was facing Problem in Amazon Quiksight , when i create charts in it , it is giving error "the data source or data set was deleted or become unavailable while Amazon Quicksight for ingesting the data into Spice. you can import a data to try again".
what wrong i have done?
Hi Ankit Thank you for the insight on different AWS tools. I just wanna know how Glue crawler will work in a database with multiple tables where they are related with Primary foreign keys etc, I mean does it crawl the entire schema? Also if possible make videos on how Glue is used in ETL process and can we load to AWS Redshift using Glue?
Hello Ankit, Your content is very helpful. I was seeking clarification to understand the data source needed for connecting to Quicksight for creating visualizations, as explained in this particular video. Is the process used in AWS specific to a data engineer's skill set, or is it a must-know skill for data analyst roles as well? If so, please assist us in identifying the AWS topics we need to know in order to connect a data source to Quicksight.
Thank you. Create more basic projects.
great learning source. Thanks for it
Thank you for this video..
We can upload the csv file in a s3 location and using aws glue we can create the snapshot folder.
Just a thought
Bro I am following this channel since 1 years , you are doing awesome work and want to learn aws cloud for mlops suggest me good sources thanks
Can you make a playlist on how to start with aws account for data analysis.
Like stwp 1 account creation
Redshift, lamda, s3 etc. Each service and uses foe data science. Please?
Hey Ankit, any way to automate -
1) Creating different csv files for each date.
2) Moving each file into a separate folder.
Any help will be appreciated!
Please make more videos like this, also try to include pyspark as well.
Great Initiative, Keep going @ankit..😍
I will try my best
Sir, please suggest, are you planning any courses on AWS? If yes, when will you start it?
Great initiative, thankyou ❤
On upcoming days can you make video that cover that redshift also.
And if possible can you make fetch the data from redshift to powerBI process only
Wow Amazing content as usual
Glad you think so!
Great work 👍 thank you for sharing this
Thank you! Cheers!
Good video for starters👍
Sir are you going to start any AWS course?
Awesome video and really helpful
Glad it was helpful!
Appreciate your effort.
In need of snowflake end to end project 😊
please do one video with ETL as well
Thanks, Ankit for sharing this AWS project. Please could you tell me how we can schedule the dashboards on quicksight like when the data will be updated in S3 bucket dashboard automatically should be updated? Or is it possible
You can schedule quick sight dataset refresh in quick sight only
Do you have Any plan to launch AWS data engineering course?
Hello sir, its like a incremental load instead full load .
Yes, correct
@@ankitbansal6 can you make project on data source would be MySQL, SFTP. Like realtime.
Can you also create E2E Data Engg Projects using Azure
Thank you bro appreciate your efforts 🙂
It's my pleasure
Sir do we need account of AWS , can you share any virtual playground who is giving aws platform
Sir please bring aws data engineering course also in live mode.thanks
This will include redshift?
Please create a project on Azure data engineering project.
Pyspark, Databricks and Azure data factory. By using this please creata a data engineering project
for me partitions are not getting created. What could be the reason.
Can we upload this project on github?
I am getting access denied to create the crawler, even if the regions are same and also giving the necessary polices ,if anyone faced the same error please can you tell how to resolve it
Thanks Ankit
Good One!!
Thank you very much
how to showcase this project, how to upload on github
I am unable to create an AWS GLUE .its saying access denied. I searched on google and tried every possible solution mentioned , but none of them are working
If you have logged in as an iam user then make sure you have glue access.
@@ankitbansal6 yes I tried both from iam user with glue access as well as from the root user
Is these simple data engineer project?
Yes
When is sql live starting end to end ?
March 2nd
@@ankitbansal6 thanks Ankit ! And do u also cover tableau training and python as well as I want to learn this trio ?
I'm a Talend developer.. I'm shifting my Career as AWS data engineering....this project looks simple..will the real time company data engineering pipeline will be like this onlyyy
It started with simple. Will do more complex projects 😊
thanks looking forward to it @@ankitbansal6
@ankitbansal6 This was great video covering multiple technical stack of AWS. Simple yet insightful project for beginners
Glad you liked it 😄
❤
sir how to get job after 3 years of gap
How much will this cost to run on AWS
less than 1 dollar.
@ankitbansal6 I am getting access denied error while creating crawler can you please help me in resolving this issue
Hy , is your issue resolved ?
Can u make FROM BEGINNING TO END PLAYLIST OF CLASSES OF MICROSOFT AZURE AND AMAZON AWS which we can utilise d same during work, please ankit bhansali my request for you @ankit bhansal