How spark works internally | what are the different components and how do they interact

This video talks about how spark components work together and what is a Job, Stage and task. We will walk through a spark program to see how stages and tasks are created.
0:00 Intro
0:36 A spark program walk through
1:25 creation of RDD
2:37 Execution
4:51 Spark Stages
6:55 Spark Tasks
9:00 Executor and task parallelism
12:10 spark component hierarchy

[@HARISHKUMAR-vo2cp]

Just like "Spark" your explanation is very powerful and professional. Your topics help very much to revise before attending interviews.

[@BigDataThoughts]

Thanks harish

[@vigneshkumar3076]

Seriously, well presented - Simple and sufficient.

[@BigDataThoughts]

Thanks Vignesh

[@ramsiddu007]

Thank you for sharing your expertise @Shreya Pal, appreciate!

[@BigDataThoughts]

Thanks jai

[@suniguha]

Very nice and compact explanation. Thank you.

[@BigDataThoughts]

Thanks sunip

[@vishwassrinivas474]

Thank you for this awesome !!! content

[@BigDataThoughts]

Thanks vishwas

[@chanakyamishra4795]

Awaited topic ✅✅ Thank you 👏👏

[@BigDataThoughts]

Thanks chanakya

[@rajdas3889]

Thanks so much for sharing, I got this from a fb AI Community. thanks

[@BigDataThoughts]

Thanks Raj

[@sanyoge389]

Really really helpful.. I am following your videos regularly. Pls keep doing the brilliant job and Thanks alot 🙏

[@BigDataThoughts]

Thanks

[@abhisekhmishra4029]

best before interviews🙂...thanks a lot

[@BigDataThoughts]

Thanks Abhishek

[@salmansayyad4522]

Hi Shreya, the way you explain things is very clear and to the point. Thank you very much for such a nice explanation. Please make some videos on real time big data project working on cloud.

[@guptaashok121]

Nice

[@subhashreebehera2994]

Best explanation till date ❤️

[@BigDataThoughts]

Thanks Subhashree

[@mohammadrizwan830]

Super Explanation 👌👌👌

[@BigDataThoughts]

Thanks Mohammed

[@nikhilgupta110]

Nice and simplistic approach to explain something which is very confusing and complex at first , you can also consider adding other things like, how spark engine perceive our code e.g Logical to Physical to cost effective plan. And How exactly no. Of shuffle operations, wide transformation has relation with No. of spark jobs and stages.

[@BigDataThoughts]

Thanks Nikhil

[@yaswanthkumar9960]

superb explanation.

[@BigDataThoughts]

Thanks yaswanth

[@machhindrasmathematics2551]

Insightful

[@sandippatel6999]

very nice explanation

[@BigDataThoughts]

Thanks sandip

[@GuruVideoAC]

Well done

[@BigDataThoughts]

Thanks

[@abhishektambewagh6156]

Great explanation ma'am!!

[@BigDataThoughts]

Thanks

[@yashawanthraj8872]

Can Node/Thread have more partition than no of executors, if yes where the no of partition information will be stored.

[@thelifehackerpro9943]

Job is a sequence of stages, jobs are divied into tasks

[@samk_jg]

Hi , I would like to know, do you offer any course on spark ? what if someone wants to learn from you online , please let me know , thanks!

[@gauravk4050]

Mam please change your mic, your content is very gud

[@BigDataThoughts]

Thanks gaurav. This is a old video. New ones are with better audio quality