this is crazy ❤❤❤ wish i had seen this earlier ! is this how the whole amazon product in a actual work flow look like？ and also could you maybe make another showing azure system? pleaaase
impressive and informative video, good job, go on doing tutorials plss :) Would be very interesting to see a video about spark and snowflake on your channel!
Based on this video, Amazon EMR is a GUI layer on top of open-source tech like Apache Spark, Hadoop, and so on, enabled by AWS networking infra and storage. Automation does not come out of the box - you still need to do a lot of clicking to set things up, unless of course you know how to write Terraform or CloudFormation scripts.
Could you share more about project for data engineer beginners? I have start to learn to be a DE recently and I hope to know more about some personal project that help me to enhance my skills. Thank you so much for your sharing and waiting for your next video :> Have a good day
Why did you write your code in vscode and uploaded in AWS and again you have uploaded the same code in EMR console to run? What is difference in both steps? and how does it work?
Fantastic tutorial indeed! I did as instructed and I got two fails in deploying the 'add step' part of the EMR Cluster stage, any insights would be appreciated.
I'm learning data engineering and I find it's hard to find end-to-end data engineering projects to learn building scalable B2B data infrastructure systems that process large amounts of data. Many examples only touch the surface or handle small datasets (well, they say example). Do you have a plan to make a tutorial about actual use cases of data engineering (large and complex data, scalable, cost efficiency system, etc.)?
Yea for sure, I’ll consider making more end to end videos in the future. Hard to mimic actual use case data engineering with toy projects, but will try to bridge that gap
@@jayzern, I'm doing a project to provide a management solution for small businesses. Data such as customer appointment schedules, staff's working hours, etc., will be stored in MongoDB. I'm thinking of a data engineering component: some data will be extracted, transformed and loaded to AWS S3 for visualization on a mobile device. I think it's still simple but, perhaps, good enough
I am wondering if I only needed to do PySpark, is EMR the best tool or is it overkill and Glue serverless would be good enough with a lot less to manage and fewer configurations to worry about. Is it possible to enable better performance with all the options in EMR?
@@syedmehdi5125 bc most companies want senior data analysts or graduate students when it comes to data science. You’ll waste more money chasing a data science job than you’ll make
Hey, we typically create VPCs over EMR clusters for more networking control and better security. If I rmb correctly, here we defined a public subnet and internet gateway which connects to S3. You could also use private subnets to avoid attaching internet gateway, but it's trickier to setup and can incur NAT gateway charges. The video is just an example
Just for the information to all the learners this is not how things to be done in tech industries....you need to understand Terra form scripts along with jenkins which deploys aws services....you will not get access to go on management console and play around and do stuff.
Your content is valuable but I find that your presentation is too fast. Sometimes, you fastforward steps too quickly. It makes the video look not lengthy but it's challenging for audience to closely follow your steps.
That’s super helpful feedback, I’ll try to slow down on the steps and talk more in future videos. Hopefully my latest video is slower. How the audience feels matters a lot
eah good in EMR AWS but an absolute rookie in Videography and equipment use manual focus since you are stationary.... your autofocus keeps looking for something and change light set-up
![AWS EMR Tutorial [FULL COURSE in 60mins]](http://i.ytimg.com/vi/v9nk6mVxJDU/mqdefault.jpg)
I hope you create more videos about AWS services. Loved the way you explain things, perfect for beginners.
This is an outstanding tutorial. Thank you for making this!
Best tutorial I've seen so far. Was confused between Glue and EMR for a future projects requiring big compute power with control over each node.
So sad your channel doesn't have more tutorials like this :( thank you so much!
Same thoughts me too
This is so goood :). Please keep making these kind of videos! Hello from Seattle
Thanks Isaac from Seattle! Appreciate your support
Great work mate, very crisp!
Thanks man!! Love ur content
Absolutely enjoyed watching the entire video. I felt this video is gonna be great start to understand EMR. Thanks for making it jay
We need more videos Jaaay 🙏🏻💪🏻 You're awesome dude!
awesome explanation, simple , subtle and to the point!
Very well explained EMR video, thank you
Thanks for this video, useful info.
Very clear! Thank you for sharing this excellent tutorial!
thank you!! I watched the RUclips demo and it was really helpful. I also want to study spark on eks
great tutorial! can’t wait to see more
Hey, thank you so much!!.. you really explain very well!
gnarly stuff man! great content.
this is crazy ❤❤❤ wish i had seen this earlier ! is this how the whole amazon product in a actual work flow look like？ and also could you maybe make another showing azure system? pleaaase
Excellent tutorial
Go ahead bro....CONGRATS TUTO
I can see you are brilliant.
Very well explained, kudos
impressive and informative video, good job, go on doing tutorials plss :) Would be very interesting to see a video about spark and snowflake on your channel!
That was FAST, you are subscribed :D
Any vids related to "Amazon Managed Workflows for Apache Airflow"???
You killed it. Loved it! Extremely useful
Thank you man! Hope to create more
this is amazing, thank you!
thank you, this is great tutorial
Great Article ! Thanks for sharing..
Based on this video, Amazon EMR is a GUI layer on top of open-source tech like Apache Spark, Hadoop, and so on, enabled by AWS networking infra and storage. Automation does not come out of the box - you still need to do a lot of clicking to set things up, unless of course you know how to write Terraform or CloudFormation scripts.
Your video is very interesting!
Hope you release many new videos :)
nice job, great tutorial
nice, keep going
More tutorials 🙏
the type of video that makes me wanna quit the field because of how bad i feel about the level I am in , but its a very helpful video though
Are you working as data engineer right now?
@@shafiq_ramli I am looking for a job now as a Data Engineer
@@shafiq_ramliif u could help me find one 😢😅
Hey! Great video. Can you also recommend alternatives for EMR that are available in the market?
Thanks for sharing man 👌
Great video
Great!! Thank u so much!
Great job
Could you share more about project for data engineer beginners? I have start to learn to be a DE recently and I hope to know more about some personal project that help me to enhance my skills. Thank you so much for your sharing and waiting for your next video :> Have a good day
Thanks .. Good stuff
Great VIdeo
thanks for the video
bro thank you i survive
great video! can you make also for AWS Glue? Thank you!
Why did you write your code in vscode and uploaded in AWS and again you have uploaded the same code in EMR console to run?
What is difference in both steps? and how does it work?
💗
Great tutorials! thanks for the headup! do you have a git repo or more notion notes? would like some guidance
More videos on Streaming, Airflow and Spark
Fantastic tutorial indeed! I did as instructed and I got two fails in deploying the 'add step' part of the EMR Cluster stage, any insights would be appreciated.
Hi bishop, I got this error too just now - did you manage to fix it? afaik I did everything the same
when writing the spark script, does it ever change or the skeleton layout remains the same? i truly appreciate this and i cannot wait for more
is this free to use or do i need to have a licensed software in order to use? this is quite interesting.
I'm learning data engineering and I find it's hard to find end-to-end data engineering projects to learn building scalable B2B data infrastructure systems that process large amounts of data. Many examples only touch the surface or handle small datasets (well, they say example). Do you have a plan to make a tutorial about actual use cases of data engineering (large and complex data, scalable, cost efficiency system, etc.)?
Yea for sure, I’ll consider making more end to end videos in the future. Hard to mimic actual use case data engineering with toy projects, but will try to bridge that gap
@@jayzern, I'm doing a project to provide a management solution for small businesses. Data such as customer appointment schedules, staff's working hours, etc., will be stored in MongoDB. I'm thinking of a data engineering component: some data will be extracted, transformed and loaded to AWS S3 for visualization on a mobile device. I think it's still simple but, perhaps, good enough
I am wondering if I only needed to do PySpark, is EMR the best tool or is it overkill and Glue serverless would be good enough with a lot less to manage and fewer configurations to worry about. Is it possible to enable better performance with all the options in EMR?
And thank you for this video - I’m studying for AWS certification and it was helpful to see your demonstration
From where you learn that coding part 😢
why do you need to create a VPC?
VPC is for nodes. It allows them to communicate between each other and the master node.
Please post more videos
I hav done masters of science in biotech, 38 yers of age, want to switch to data science...how shud i do it??? Plz reply.....
Do projects and add them in your resume. Try upwork and do some projects as freelancers. Keep applying
Don’t
@@Ved3sten y , plz reply...
@@syedmehdi5125 bc most companies want senior data analysts or graduate students when it comes to data science. You’ll waste more money chasing a data science job than you’ll make
Hey why did you create vpc?
Hey, we typically create VPCs over EMR clusters for more networking control and better security. If I rmb correctly, here we defined a public subnet and internet gateway which connects to S3. You could also use private subnets to avoid attaching internet gateway, but it's trickier to setup and can incur NAT gateway charges. The video is just an example
Isnt using EMR notebook one of of the ways to trigger EMR job?
Yes it is! Wanted to keep things simple in the video so didn't include it
Can you provide your pyspark code ???
Don't stop
Just for the information to all the learners this is not how things to be done in tech industries....you need to understand Terra form scripts along with jenkins which deploys aws services....you will not get access to go on management console and play around and do stuff.
Your content is valuable but I find that your presentation is too fast. Sometimes, you fastforward steps too quickly. It makes the video look not lengthy but it's challenging for audience to closely follow your steps.
That’s super helpful feedback, I’ll try to slow down on the steps and talk more in future videos. Hopefully my latest video is slower. How the audience feels matters a lot
nice try but its not working
Let me know how I can help
I can add a step for the spark application@@jayzern
Check if
1. the Spark script is encrypted when you upload it inside S3
2. any typos (line 41 should be "add_argument")
I had tried. but it's not working for me @@jayzern
Send me a DM on instagram @jayzern or linkedin, happy to pair up
eah good in EMR AWS but an absolute rookie in Videography and equipment use manual focus since you are stationary.... your autofocus keeps looking for something and change light set-up
Fair point 👍 will work on lighting and camera setup more next time
Love the ways how you demonstrate! so clear and easy to understand! Thanks for sharing @jayzern
thanks so much