AWS EMR Big Data Processing with Spark and Hadoop | Python, PySpark, Step by Step Instructions

In this video, I gave an overview of what EMR is and its benefits in the big data and machine learning world. I then provided a step by step instruction on how to spin up an EMR cluster and do a spark submit job on it to process data from a Stack Over Flow survey.
Support the channel plz 😊: www.buymeacoffee.com/felixyu
Instruction on how to create a key pair on aws: docs.aws.amazon.com/AWSEC2/la...

[@aimeeyu1991]

Good stuffs! Your videos are always so detailed and informative 👍🏻

[@jasper5016]

Wow, Felix, you are a fantastic teacher. Really happy to find your channel. Thanks

[@maulikpatel7116]

Wow, so informative!!! Thanks so much for teaching me how to do this!!!

[@kkb_now_i_have_a_handle]

Crisp and to the point, thanks Felix please keep it up.

[@FelixYu]

tyty

[@privatestuff4432]

Thanks! Your video is far, far better than most out there!
Note to other viewers: If you decide to try this yourself, you should download the 2020 survey results, and not something newer. The fields are different in newer versions of the survey.

[@BlazinEdit]

very good and undervalued content keep up with the good work man :) you are helping a lot! I bet your channel will start growing soon.

[@FelixYu]

Thank you!! This means a lot :)

[@brothermalcolm]

awesome, one of the few 30min tutorials for big data on aws that actually worked!

[@FelixYu]

glad that u found it helpful!! 👍

[@mrmen556]

This is what I was looking for. Thanks for the video.

[@FelixYu]

glad that u found it helpful!! 👍

[@shashikantkrishna4156]

Very informative! 🙂Thank you so much!

[@sakshi0070]

thank you sir for this session....love from india🙏

[@rajach6636]

Very well explained 👏👌

[@sudippandit9855]

Excellent!!

[@khandoor7228]

Great content!

[@orpat007]

Awesome! Thanks!

[@WolfmaninKannada]

sir your explanation is very clear ...i request you to make end to end project videos on aws etl

[@krishnabisen2666]

Thank you!

[@joaovitor12full]

Wow, That's amazing

[@FelixYu]

Thank you!!

[@electricalsir]

Thank you so much sir for detail explanation it will be very useful to us ❤❤ thanks a lot

[@FelixYu]

Glad that u found it helpful!!

[@Chennai59]

Outstanding . Thanks dude

[@FelixYu]

Glad u like it

[@simpledataengineer5231]

Awesome video!

[@FelixYu]

glad that u found it helpful!!

[@abhishekchandrashukla3814]

Good stuff sir!

[@VamsiKrishna-vf5gm]

Really great stuff... excellent presentation !!

[@FelixYu]

thank you..glad that it's helpful

[@amadeeyo8181]

Great tutorial man keep it up

[@PraveenKumar-ic5zo]

The best...Awesome.

[@FelixYu]

Thanks a lot!

[@ParijatKar]

Thanks for the AWS EMR Configuration details. How the underlying S3 or HDFS is distributing data blocks for parallel processing? How redundancy and parallelism can be configured? I have logs from airline equipment for the last 30 years, equivalent to 1 PB. I want to use all of it to identify failures with indicators.

[@letscodewithvivek5191]

Great Tutorial

[@FelixYu]

Thank you

[@AndrewPa]

well done Felix :-)

[@xiaocuizhang6879]

thank for the sharing !

[@FelixYu]

Glad that u found it helpful

[@ajinkyahatolkar294]

Very nicely explained

[@FelixYu]

thank you!! glad that u found it helpful!!

[@santoshraju3546]

Very informative. Can you do a deep dive of aws emr?

[@ririraman7]

Awesome tutorial brother

[@FelixYu]

tyty 😄

[@johnmcgettigan4882]

Yu-demy! thank you!

[@FelixYu]

Haha glad that u found it helpful!!

[@amitchaurasia2691]

Very good to start with EMR hands-on

[@FelixYu]

Glad that u found it helpful!!

[@sayedaweshrahman4608]

Clean...👍

[@FelixYu]

Thank you!!

[@SahilP-yv4fv]

good tute

[@chaithanyamannem5040]

good stuff Felix ..do u have video of migrating on premises to cloud bigdata cluster ?

[@chandnimirchandani3040]

Do you have any document to map the property graph model to Hadoop

[@hikariuchiha977]

hi what is the difference in 4 applications? when u create the cluster there is 4 options, how do you know wchich one to select?

[@dimitris_k2841]

god job mate

[@FelixYu]

Thanks mate!!

[@andresnicolasrodriguezsanc4586]

thanks!

[@FelixYu]

Glad that u found it helpful

[@shbhmara4915]

Very good

[@FelixYu]

thanks mate

[@user-nk6ov5rk5h]

Great Video !! However, right now I think you do have to set up an IAM role for accessing your S3 bucket is it not ?

[@user-tf4il6gs9w]

Thanks for the video ....can you help me with dependency files like the python uses other module ...how to go about that? when i want to submit with spark submit in EMR

[@deenadayalmuli3364]

excallent

[@FelixYu]

Thank you!!

[@poolput1]

How to set application with database ?

[@ParijatKar]

When I am doing big data then I am really using B I G data. What are the challenges at the Peta Byte level?

[@burakevrentug]

Felix, thank you for your sharing but the application interface is changed. So I can't do your application. Will you share your video updated version?

[@josemanuelgutierrez4095]

I have a question , do you have any course explaining pyspark ?? or any recomendations maybe

[@elitedecor8510]

If we terminate Amazon got going to charge money...what is the police to use it for free for practice purpose

[@surajthallapalli4227]

Hi, the EMR outputs many files as part_0000,....and so on in S3. But i want just one output file after pre-processing it in spark EMR. How to do that?

[@FelixYu]

U can do something like
df.repartition(1).write.mode(“overwrite”).parquet(“locationPath”)

[@surajthallapalli4227]

@@FelixYu Thanks a lot

[@blosun641]

When I try to use command "spark-submit" I get "-bash: spark-submit: command not found"
Is there any solution to this ? I´m using Putty on Win

[@FelixYu]

On the EMR creation (4:10 of the video), did u choose the last combination that has spark and Hadoop??

[@viniciusfigueiredo6740]

Friend, how do I save Jupyter notebooks in my EMR?

[@eshanpandey4186]

I guess there's an option to enable jupyterhub right when you are initializing the EMR cluster

[@simrangulani4299]

I don't have vs code installed. What should I do?

[@FelixYu]

U don’t have to use vs code to write the code. U can use any IDE for it. Or u can just google vscode to download it

[@krishnasanagavarapu4858]

[@raulgutierrez5862]

Excellent demo! What was the AWS cost of running this demo?

[@FelixYu]

I didn’t check but prob less than $1

[@chaitanyarishabh1857]

Good stuff, however seems like someone is sleeping and snoring in background 😴

[@SahilP-yv4fv]

who is snoring in the background