this is crazy ❤❤❤ wish i had seen this earlier ! is this how the whole amazon product in a actual work flow look like？ and also could you maybe make another showing azure system? pleaaase
impressive and informative video, good job, go on doing tutorials plss :) Would be very interesting to see a video about spark and snowflake on your channel!
Could you share more about project for data engineer beginners? I have start to learn to be a DE recently and I hope to know more about some personal project that help me to enhance my skills. Thank you so much for your sharing and waiting for your next video :> Have a good day
Fantastic tutorial indeed! I did as instructed and I got two fails in deploying the 'add step' part of the EMR Cluster stage, any insights would be appreciated.
I'm learning data engineering and I find it's hard to find end-to-end data engineering projects to learn building scalable B2B data infrastructure systems that process large amounts of data. Many examples only touch the surface or handle small datasets (well, they say example). Do you have a plan to make a tutorial about actual use cases of data engineering (large and complex data, scalable, cost efficiency system, etc.)?
Yea for sure, I’ll consider making more end to end videos in the future. Hard to mimic actual use case data engineering with toy projects, but will try to bridge that gap
@@jayzern, I'm doing a project to provide a management solution for small businesses. Data such as customer appointment schedules, staff's working hours, etc., will be stored in MongoDB. I'm thinking of a data engineering component: some data will be extracted, transformed and loaded to AWS S3 for visualization on a mobile device. I think it's still simple but, perhaps, good enough
Hey, we typically create VPCs over EMR clusters for more networking control and better security. If I rmb correctly, here we defined a public subnet and internet gateway which connects to S3. You could also use private subnets to avoid attaching internet gateway, but it's trickier to setup and can incur NAT gateway charges. The video is just an example
@@syedmehdi5125 bc most companies want senior data analysts or graduate students when it comes to data science. You’ll waste more money chasing a data science job than you’ll make
I am wondering if I only needed to do PySpark, is EMR the best tool or is it overkill and Glue serverless would be good enough with a lot less to manage and fewer configurations to worry about. Is it possible to enable better performance with all the options in EMR?
Just for the information to all the learners this is not how things to be done in tech industries....you need to understand Terra form scripts along with jenkins which deploys aws services....you will not get access to go on management console and play around and do stuff.
Your content is valuable but I find that your presentation is too fast. Sometimes, you fastforward steps too quickly. It makes the video look not lengthy but it's challenging for audience to closely follow your steps.
That’s super helpful feedback, I’ll try to slow down on the steps and talk more in future videos. Hopefully my latest video is slower. How the audience feels matters a lot
eah good in EMR AWS but an absolute rookie in Videography and equipment use manual focus since you are stationary.... your autofocus keeps looking for something and change light set-up
I hope you create more videos about AWS services. Loved the way you explain things, perfect for beginners.
This is an outstanding tutorial. Thank you for making this!
So sad your channel doesn't have more tutorials like this :( thank you so much!
Same thoughts me too
This is so goood :). Please keep making these kind of videos! Hello from Seattle
Thanks Isaac from Seattle! Appreciate your support
Best tutorial I've seen so far. Was confused between Glue and EMR for a future projects requiring big compute power with control over each node.
Great work mate, very crisp!
Thanks man!! Love ur content
Very well explained EMR video, thank you
We need more videos Jaaay 🙏🏻💪🏻 You're awesome dude!
Absolutely enjoyed watching the entire video. I felt this video is gonna be great start to understand EMR. Thanks for making it jay
awesome explanation, simple , subtle and to the point!
Excellent tutorial
thank you!! I watched the RUclips demo and it was really helpful. I also want to study spark on eks
Very clear! Thank you for sharing this excellent tutorial!
Hey, thank you so much!!.. you really explain very well!
gnarly stuff man! great content.
great tutorial! can’t wait to see more
Very well explained, kudos
Go ahead bro....CONGRATS TUTO
this is amazing, thank you!
I can see you are brilliant.
this is crazy ❤❤❤ wish i had seen this earlier ! is this how the whole amazon product in a actual work flow look like？ and also could you maybe make another showing azure system? pleaaase
You killed it. Loved it! Extremely useful
Thank you man! Hope to create more
nice job, great tutorial
Great Article ! Thanks for sharing..
thank you, this is great tutorial
Your video is very interesting!
Hope you release many new videos :)
nice, keep going
impressive and informative video, good job, go on doing tutorials plss :) Would be very interesting to see a video about spark and snowflake on your channel!
Great video
Thanks for sharing man 👌
That was FAST, you are subscribed :D
Any vids related to "Amazon Managed Workflows for Apache Airflow"???
Great job
More tutorials 🙏
Great!! Thank u so much!
Could you share more about project for data engineer beginners? I have start to learn to be a DE recently and I hope to know more about some personal project that help me to enhance my skills. Thank you so much for your sharing and waiting for your next video :> Have a good day
great video! can you make also for AWS Glue? Thank you!
Thanks .. Good stuff
thanks for the video
Fantastic tutorial indeed! I did as instructed and I got two fails in deploying the 'add step' part of the EMR Cluster stage, any insights would be appreciated.
Hi bishop, I got this error too just now - did you manage to fix it? afaik I did everything the same
the type of video that makes me wanna quit the field because of how bad i feel about the level I am in , but its a very helpful video though
Are you working as data engineer right now?
@@shafiq_ramli I am looking for a job now as a Data Engineer
@@shafiq_ramliif u could help me find one 😢😅
Great VIdeo
bro thank you i survive
Great tutorials! thanks for the headup! do you have a git repo or more notion notes? would like some guidance
More videos on Streaming, Airflow and Spark
I'm learning data engineering and I find it's hard to find end-to-end data engineering projects to learn building scalable B2B data infrastructure systems that process large amounts of data. Many examples only touch the surface or handle small datasets (well, they say example). Do you have a plan to make a tutorial about actual use cases of data engineering (large and complex data, scalable, cost efficiency system, etc.)?
Yea for sure, I’ll consider making more end to end videos in the future. Hard to mimic actual use case data engineering with toy projects, but will try to bridge that gap
@@jayzern, I'm doing a project to provide a management solution for small businesses. Data such as customer appointment schedules, staff's working hours, etc., will be stored in MongoDB. I'm thinking of a data engineering component: some data will be extracted, transformed and loaded to AWS S3 for visualization on a mobile device. I think it's still simple but, perhaps, good enough
From where you learn that coding part 😢
Hey why did you create vpc?
Hey, we typically create VPCs over EMR clusters for more networking control and better security. If I rmb correctly, here we defined a public subnet and internet gateway which connects to S3. You could also use private subnets to avoid attaching internet gateway, but it's trickier to setup and can incur NAT gateway charges. The video is just an example
is this free to use or do i need to have a licensed software in order to use? this is quite interesting.
when writing the spark script, does it ever change or the skeleton layout remains the same? i truly appreciate this and i cannot wait for more
Please post more videos
why do you need to create a VPC?
VPC is for nodes. It allows them to communicate between each other and the master node.
I hav done masters of science in biotech, 38 yers of age, want to switch to data science...how shud i do it??? Plz reply.....
Do projects and add them in your resume. Try upwork and do some projects as freelancers. Keep applying
Don’t
@@Ved3sten y , plz reply...
@@syedmehdi5125 bc most companies want senior data analysts or graduate students when it comes to data science. You’ll waste more money chasing a data science job than you’ll make
Love the ways how you demonstrate! so clear and easy to understand! Thanks for sharing @jayzern
💗
Isnt using EMR notebook one of of the ways to trigger EMR job?
Yes it is! Wanted to keep things simple in the video so didn't include it
I am wondering if I only needed to do PySpark, is EMR the best tool or is it overkill and Glue serverless would be good enough with a lot less to manage and fewer configurations to worry about. Is it possible to enable better performance with all the options in EMR?
And thank you for this video - I’m studying for AWS certification and it was helpful to see your demonstration
Don't stop
Just for the information to all the learners this is not how things to be done in tech industries....you need to understand Terra form scripts along with jenkins which deploys aws services....you will not get access to go on management console and play around and do stuff.
Your content is valuable but I find that your presentation is too fast. Sometimes, you fastforward steps too quickly. It makes the video look not lengthy but it's challenging for audience to closely follow your steps.
That’s super helpful feedback, I’ll try to slow down on the steps and talk more in future videos. Hopefully my latest video is slower. How the audience feels matters a lot
nice try but its not working
Let me know how I can help
I can add a step for the spark application@@jayzern
Check if
1. the Spark script is encrypted when you upload it inside S3
2. any typos (line 41 should be "add_argument")
I had tried. but it's not working for me @@jayzern
Send me a DM on instagram @jayzern or linkedin, happy to pair up
eah good in EMR AWS but an absolute rookie in Videography and equipment use manual focus since you are stationary.... your autofocus keeps looking for something and change light set-up
Fair point 👍 will work on lighting and camera setup more next time
thanks so much