Understanding Stages in Spark UI for a Spark Job | Spark Interview Questions

Hi Friends,
In this video, I have explained Spark internal machanism on how driver creates the logical plan and then how it will turn to the run-time execution plan, how the stages will be created etc details.
Please subscribe to my channel for more interesting learnings.

[@vanajar7345]

Explained clearly. It took me 10 -15 video before reaching to this video and no other video was as clear as your's

[@sravanalakshmipisupati6533]

Thanks a lot.

[@naveennoel9496]

Explained in a very clear manner. Thanks.

[@sonutyagi149]

You Channel is Gold Mine to me - Thank you So So Much .

[@sushmamc8904]

You have explained it so well. Thank you so much

[@sravanalakshmipisupati6533]

Thanks a lot :)

[@vaishnavidhoke8174]

Amazingly explained! Thankyou so much for making this, it’s extremely helpful, kudos!!

[@bhavanivani448]

Nice explained, really helpful for interview

[@sravanalakshmipisupati6533]

Thank you 😊

[@nagendran-wn6ws]

Nice explanation ..appriciated madam

[@technologyexplorer9193]

Nice explanation.
Could you please explain little bit more on how exactly on what bases tasks are created. For each stage

[@sravanalakshmipisupati6533]

Thank you very much. Sure.

[@sriadityab4794]

Can you please explain more with good example on how to identify time taking jobs and optimizing them?

[@sravankumar1767]

Nice explanation sravani

[@sravanalakshmipisupati6533]

Thanks a lot Sravan :)

[@reddammareddy2999]

Super mam

[@joyo2122]

now i get it

[@vishnureddym5389]

Hi @sravana, it was good explanation. I have a doubt in this. In the second job we can see two stages and the first stage is skipped which means it is broadcasted or cached right?. But job 1 executed the same and in the second job again that stage executed or first job itseems cached and using the result as input in the second job second stage?

[@sravanalakshmipisupati6533]

Hi Vishnu, the cached data will be sent to second stage as input.

[@vishnureddym5389]

Thank you for the reply. My question is we have two jobs . Job 1 is having one stage and job 2 is having two stages. So job 1 stage 1 is same as job 2 stage 1 or it is different?. If same then can we use job1 stage1 results as input to job2 stage 2.

[@sravanalakshmipisupati6533]

@@vishnureddym5389 , @ 3:30, you can see that there are 2 jobs present - Job 31 & 32. Job 31 cached result is sent to exchange at the end and you can see that the input is read from exchange in job32. This way, Job1 output is sent to job2 as input. Also, job2 will have the stage1 (which is greyed out) and stage2.

[@vishnureddym5389]

@@sravanalakshmipisupati6533 so job 31 stage out put is sending to job 32. But job32 stage1 is not executing again right which same as job 31 output

[@sravanalakshmipisupati6533]

@@vishnureddym5389 Yes, Job 32 stage 1 will not be executed again, that's why it is showing as greyed out. The cached output from exchange will be considered directly as input to stage2.

[@kubersharma1971]

You have nailed it . Would be looking to connect with you over Linkedin pls share your LinkedIn profile link

[@sravanalakshmipisupati6533]

@kubersharma1971 Thank you.
www.linkedin.com/in/sravana-lakshmi-pisupati-b2b6478a

[@ittrainingsjobsimmigration2301]

Very well demonstrated ...if you have any Instagram I'd / fb id please share to clear certain doubts

[@sravanalakshmipisupati6533]

Sravana Lakshmi Pisupati is my ID.