Spark Performance Tuning | EXECUTOR Tuning | Interview Question

#Spark #Persist #Broadcast #Performance #Optimization
Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more
Click here to subscribe : / @techwithviresh
About us:
We are a technology consulting and training providers, specializes in the technology areas like : Machine Learning,AI,Spark,Big Data,Nosql, graph DB,Cassandra and Hadoop ecosystem.
Mastering Spark : • Spark Scenario Based I...
Mastering Hive : • Mastering Hive Tutoria...
Spark Interview Questions : • Cache vs Persist | Spa...
Mastering Hadoop : • Hadoop Tutorial | Map ...
Visit us :
Email: [e-mail]
Facebook : / tech-greens
Twitter :
Thanks for watching
Please Subscribe!!! Like, share and comment!!!!

[@RohanKumar-mh3pt]

Very Nice and clear explanation before this video i was very confused regarding executor tuning part now after this video it is now crystal clear.

[@TheFaso1964]

Dude. I feel like I knew nothing about spark in particular before I got my hands dirty with your performance improvement solutions.
Appreciate a lot, got my subscription. Cheers from Germany !

[@TechWithViresh]

Thanks a lot :)

[@nivedita5639]

Very very helpful. Thanks

[@ranju184]

excellent explanation. Thanks

[@aneksingh4496]

As always best !!! Please include some real simulation example s

[@sankarn6016]

Nice Explanation!!
can we use this approach for tuning/triggering multiple jobs in cluster ??

[@fahad_ishaqwala]

Excellent videos brother. Much Appreciated. Can you do a video on Performance Tuning for Spark Structured Streaming jobs as well.

[@TechWithViresh]

Surely, Working on a video for the same.

[@whatever-genuine7945]

How to allocate executers, core and memory if there are multiple jobs running on the cluster?

[@DilipDiwakarAricent]

If not configure , so what will be the default number choose by spark.

[@KNOW-HOW-HUB]

To process 1TB data what could be the best approach we have to follow

[@giyama]

This calculation is for just one job, what would be the calculation for multiple jobs running simultaneously?
And how to calculate based on the volumetry?
(Great job btw, tks!)

[@SidharthanPV]

Dynamic allocation is currently supported. You can set the max limit, yarn takes care of managing it in case of multiple instances running parallel.

[@SpiritOfIndiaaa]

thanks bro , really wonderful explanation.... bro , can you make some vid on how to analyze Stages , Physical Plans etc on SparkUI ...based on that how to fix the issues regarding optimization ... its always confusing a lot to interpret these sql explain plans?

[@TechWithViresh]

Thanks very much, check out the video on stage details

[@SpiritOfIndiaaa]

@@TechWithViresh i dont find it, any url plz

[@manisekhar4446]

According to your eg. How much GB if data can be processed by spark job??

[@snehakavinkar2240]

How to decide these configurations for a certain volume of data? Thank you.

[@TechWithViresh]

idea is to make sure max 5 tasks per executor, and the partition size is within the memory allocated to exec

[@mdmoniruzzaman703]

Hi, 10 nodes means including the master node?
i have a configuration like this:
"Instances": {
"InstanceGroups": [
{
"Name": "Master nodes",
"Market": "SPOT",
"InstanceRole": "MASTER",
"InstanceType": "m5.4xlarge",
"InstanceCount": 1
},
{
"Name": "Worker nodes",
"Market": "SPOT",
"InstanceRole": "CORE",
"InstanceType": "m5.4xlarge",
"InstanceCount": 9
}
],
"KeepJobFlowAliveWhenNoSteps": false,
"TerminationProtected": false
},

[@anusha0504]

What are advanced spark technologies

[@user-vl1ld3be3n]

What if I have multiple spark jobs in parallel in on spark session

[@umeshkatighar3635]

What If each node has only 8cores?? How does spark allocate 5cores per jvm ?

[@snehakavinkar2240]

Is there any upper or lower limit to the amount of memory per executor?

[@TechWithViresh]

depends on the total memory resource available in your cluster.

[@inferno9004]

@5:10 can you explain how 20GB + 7% of 20GB is 23GB and not 21.4GB ?

[@rockngelement]

calculation mistake bhai, anyway it doesn't affect the info in this video

[@sivavulli7487]

Hi Sir , thank you for your nice explanation but if only one job is running over the cluster , that is more meaningful and understandable ..what if there are so many jobs running on the same cluster ??

[@TechWithViresh]

Based on the executor params passed for the each , that defines the container boundaries or running scope for that.If there are not enough resources available to be allocated, then that job(s) would be in queue.

[@sivavulli7487]

@@TechWithViresh so executor core can run only one job task at a time ..so if that is the case , in your examples , if there are 2 jobs on the same cluster, we need to take half of the resources mentioned in that video or better to take whatever you mentioned ..then first job runs successfully , it will take second job??( Until first job completed, second will be in queue).. could you please suggest best approach...alltogather before giving spark resource configurations for any job , just if we look at the cluster configuration is enough or need to look at how many other jobs running on the same cluster??

[@TechWithViresh]

@@sivavulli7487 Yes, we should take into account, how many concurrent jobs need to be run .How better approach followed these days to have interactive clusters for each job..

[@sivavulli7487]

@@TechWithViresh okay ..thank you sir ..if possible , pls can you make a video how to give the resources if there are multiple concurrent jobs running on the same cluster...

[@rikuntri]

One executor is having four core so it can handle one task or 4 at a time

[@the_high_flyer]

No of cores = no of parallel task

[@girijapanda1306]

7% of 21GB = 1.4 GB am I missing something here

[@KiranKumar-cg3yg]

Means what I know is nothing.

[@RAB-fu4rw]

7% of 21 gb is 3gb ????? how come it is 1.47 GB how did u arrive at 3 GB ???

[@komalkarnam1429]

Yes had the same question

[@divyar7991]

For yarn you can choose between 6 to 10 per