Bhai mera bhai 😍 Abto hazaro students aayenge bhai ke pass par Apne sabse pehle student ko mat bhulna bawa😜 Very proud of you bhai... And i can guarantee every1 here that he is the best teacher that there is❤️
Buddy! You got a new sub here. Loved your detailed explanation. I see no one explaining the query plain this detail and I believe this is the right way of learning. But I would love to see an entire Spark series.
no one teaches detailed way complex things like you no matter what please spread you're knowledge to world i am sure there must be people learn from you , remember you as master life long who settled in it job like me
Bro, I am beginner but i was able to understand everything. Really great content and ur explanations was also amazing. Please continue doing such great videos. Thanks a lot for sharing .
This is really informative, such details are not even present in the O'Reilly Learning Spark Book. Please continue to make such content. Needless to say but I have already subscribed.
By watching your first 15mins of youtube video and I am awed beyond my words. What a great explanation @afaqueahmad. Kudos to you! Please make more videos of solving real time scenarios using PySpark & Cluster configuration. Again BIG THANKS!
Just 10 minutes into this notebook and I am awed beyond my words. What a great explanation Afaque. Kudos to you! Please make more videos of solving real time scenarios using Spark UI and one on Cluster configuration too. Again BIG THANKS!
One of the cleanest explanation I ever come across on the internals of Spark. Really appreciate all the effort you are putting into making these videos. If you don't mind, May I know which text editor are you are using when pasting the Physical plan?
Thank you for taking the time to create such an in depth video for Spark Plans. This is very helpful ! Would you also be able to explain Spark Memory Tuning ? How do we decide how much resources to allocate (driver mem, executors mem , num executors , etc for a spark submit ? Also Data Structures Tuning, Garbage Collection Tuning ! Thanks again !
Thanks for the kind words @crystalllake3158 and the suggestion; currently the focus of the series is to cover all possible code level optimization. Resource level optimisations will come in much later, but no plans for the upcoming few months :)
I loved your explanation and understood it very well. Could you help me to understand at 23 mins, if we have join key as cid and group by region. how the hash partitioning works. will that consider both?
Can you please prepare a video showing storage anatomy of data during job execution cycle? I am sure there are many aspiring spark students who may be confused about the idea of RDD or dataframe and how it access data through apis (since spark is in memory computation) during job execution. It will help many upcoming spark developers.
Hey @udaymmmmmmmmmm, I added this video recently on Spark Memory Management. It talks about storage and responsibilities or each of memory components during job execution. You may want to have a look at it :) Link here: ruclips.net/video/sXL1qgrPysg/видео.html
Hi Afaque. A suggestion. You could start from the beginning to connect the DOTS! Like if in your scenario we have X Node machine with Y workers and Z exectors and if you do REPARTITION and fit the data like this then this could happen. Otherwise the Machine would sit idle and so on.
At the very end of the video 38:36, we see that the cast("int") filter is present in the parsed logical plan and Analyzed logical plan. I am a little confused as to when we refer those plans. Can you please explain?
I am doing coalesce(1) and getting error as : Unable to acquire 65536 bytes of memory, got 0. But when i am doing repartition(1), it worked. Can you please explain what happens internally in this case?
if it is doing local aggregation before shuffling the data then why it will throw out of memory error while taking count of each key when the column has huge distinct values
I have doubt when the data will be distributed to executor is it before scheduling the task or after scheduling the task and who assign the data to executor
Hey @sangu2227, this requires an understanding of transformations/actions and lazy evaluation in Spark. Spark doesn't do anything (either scheduling a task or distributing data) until an action is called. The moment an action is invoked, Spark creates a logical -> physical plan and Spark's scheduler divides the work into tasks. Spark's driver and Cluster manager then distributes the data to the executors for processing :)
Hi sir i came across a doubt Consider the executor size 1gb/executor. We have 3 executors and intially 3 gb data gets distributed across 3 executors each executor is having 1gb partition after various transformations we came across a requirment to decrease the number of partitions to 1 partition for that we will use repartition(1) or coalesce(1). In this scenario all the 3 partitions will merges to 1 partition each partition is having size of 1 gb approximately. Collectively all the partitions size is 3 gb approximately. When repartition (1) or coalesce(1) all the 3 gb data should sit in 1 executor having capicity of 1gb only. So here the data is execeeding the executor size what happens in this scenario. Could you please make video on this requesting sir.
Hi @bhargaviakkineni, In the scenario you described above where the resulting partition size (3 GB) exceeds the memory available on a single executor (1 GB), Spark will attempt to spill data to disk. The spill to disk is going to help the application from crashing due to out-of-memory errors however, there is going to be a performance impact associated, because disk IO is slower. On a side note, as a best practice, It’s best to also think/re-evaluate the need to write to a single partition. Avoid writing to a single partition, because it generally creates a bottleneck if the sizes are large. Try to balance out the partitions with the resources of the cluster (executors/cores). Hope that clarifies :)
You mentioned that for coalesce(2) shuffle will happen, but later you mentioned that shuffle will not happen in case of coalesce hence no partitioning scheme. Could you please explain it in detail?
So, coalesce will only incur a shuffle if its a very aggressive situation. If the objective can be achieved by merging (reducing) the partitions on the same executor, it will go ahead with it. In case of coalesce(2), its an aggressive reduction in the number of partitions, meaning that Spark has no other option but to move the partitions. As there were 3 executors (in the example I referenced in the video), even if it reduced the partitions on each executor to a single partition, it would end up with 3 partitions in total, therefore it incurs a shuffle to have 2 final partitions :)
Hello @afaqueahmad7117, thanks for the great video. While explaining repartition, you mentioned you’ve a video on the AQE. Please can you link that as well?
Can't wait for that@@afaqueahmad7117 These RUclips videos are so much more helpful. Hats down one of the best ones that explain the Spark performance tuning and internals in a very simplest of forms possible. Cheers!
200 is the default number of shuffle partitions. You can find the number here in this table by the property name "spark.sql.shuffle.partitions" spark.apache.org/docs/latest/sql-performance-tuning.html#other-configuration-options
Local distinct on cust id doens't make sense and couldn't understand. How globally it does distinct count if the count is already computed. The reasoning behind why cast doens't push down predicate is not clearly explained and just as it's mentioned in the doc
Hi Sir Please Make a Detailed course on apache spark which include every aspect of spark for Data Engineer role Please make sure there are a lot of begineer course here in market keep the course from intermediate level to advance level. Please tr to make video in Hindi it will be very helpful.
🔔🔔 Please remember to subscribe to the channel folks. It really motivates me to make more such videos :)
Your content is really awesome with out any further thought, and its very advance level pyspark understanding
Bro, This is really nice, I just love the way you teach and very very good content. Bahut bahut Sukriya and ton of love
Thanks a bunch. To my knowledge, no one has explained Spark explain function this detailed level. Very in-depth information.
Bhai mera bhai 😍 Abto hazaro students aayenge bhai ke pass par Apne sabse pehle student ko mat bhulna bawa😜
Very proud of you bhai... And i can guarantee every1 here that he is the best teacher that there is❤️
This is the most detailed explanation I have ever seen.
Appreciate it man @ridewithsuraj-zz9cc :)
Buddy! You got a new sub here.
Loved your detailed explanation. I see no one explaining the query plain this detail and I believe this is the right way of learning. But I would love to see an entire Spark series.
Thank you @snehitvaddi for the kind appreciation. A full-fledged, in-depth course on Spark coming soon :)
@@afaqueahmad7117 Most awaited. Keep up the 🚀
My today's well spent 40 mins. Thanks for the knowledge sharing.
Awesome video as always. Would really appreciate more videos explaining how DAG's can be read
no one teaches detailed way complex things like you no matter what please spread you're knowledge to world i am sure there must be people learn from you , remember you as master life long who settled in it job like me
Bro, I am beginner but i was able to understand everything. Really great content and ur explanations was also amazing. Please continue doing such great videos. Thanks a lot for sharing .
@thecodingmind9319 Thanks for the kind words, means a lot :)
Absolute gem ❤❤ would like to have video on handling real time scenarios (handle slow running job, oom etc)..
This takes me back to me YaarPadhade times. Great work Bhai much love!
After looking for some time for best material which truly explains this topic, and try to dig deep enough you clearly delivered, thanks Afaque.
Glad it was helpful, appreciate it :)
rare content! please don't stop making these
Amazing content.. I am a newbie into Spark but I am hooked.. Sir plz post the continued series.. awaiting for your video posts.. Amazing teacher
You are a gem bro. The content that you bring here is terrific. ❤❤❤
Thanks man, @yashwantdhole7645. This means a lot!
Proud of you brother, looking forward to more of such videos. Great job!
This is really informative, such details are not even present in the O'Reilly Learning Spark Book. Please continue to make such content. Needless to say but I have already subscribed.
Excellend content, please make more videos like this with deep understanding of "how stuff works"... Highly Appreciate it. Love from 🇵🇰
Thank you @MuhammadAhmad-do1sk for the appreciation, love from India :)
Bhai bhot bhadia content banaate ho. Love your vdos. Please keep it up. You have great teaching skills.
Bohot shukriya bhai sahab!
Thanks! From Brazil
Appreciate it, thanks man! :)
Great explanation of such a complex topic, thanks and keep up the good work.
Thanks man @Dhawal-ld2mc :)
one of the best videos i came across on spark query plan explanation. Thank you! :)
Appreciate it @myl1566, thank you!
It's great to see such useful contents in spark... an its helpful to understand clearer with your notes! you rock.... Thankless thanks !!
This is one of the best video about Spark I have seen recently!
Beautifully explained. Many concepts got cleared. thanks a lot.Keep going.
a lot of knowledge in just one video
Appreciate it @user-pq9tx6ui2t :)
Explanation is so good
Thank you @satyajitmohanty5039 :)
Afaque, THANK YOU SO MUCH FOR THESE VIDEOS!!
They are so amazing for a fast paced learning experience.
Hope you soon upload much more!!
It's a great video with a great explanation. Awesome. Thank you for such a detailed explanation. Please keep doing such content.
By watching your first 15mins of youtube video and I am awed beyond my words.
What a great explanation @afaqueahmad. Kudos to you!
Please make more videos of solving real time scenarios using PySpark & Cluster configuration. Again BIG THANKS!
Hey @VenuuMaadhav, thank you for the kind words, means a lot. More coming soon :)
By far best content i have seen on explain query thing!!! Keep it brother. Good luck!
Glad, you liked it, thank you! :)
One of the best videos I have seen on Spark, waiting for your Spark Architecture Video
Just 10 minutes into this notebook and I am awed beyond my words.
What a great explanation Afaque. Kudos to you!
Please make more videos of solving real time scenarios using Spark UI and one on Cluster configuration too. Again BIG THANKS!
Hi @mohitupadhayay1439, really appreciate the kind words, it means a lot. A lot coming soon :)
One of the cleanest explanation I ever come across on the internals of Spark. Really appreciate all the effort you are putting into making these videos.
If you don't mind, May I know which text editor are you are using when pasting the Physical plan?
Many thanks for the kind words @venkatyelava8043, means a lot. On the text editor - I'm using Notion :)
Great content with practical knowledge. Hats off to you !!!
Nice explanation
Explained the concept really well!
Thanks for such an in-depth overview!! helps a lot to grow!!
I am sure that down the line, in a few years, you will cross 100k subscribers. Great content BTW.
Hey @CoolGuy , thanks man! Means a lot to me :)
"God bless you! Great video! Learned a lot"
This is pure gold, congrats bro , keep the good work
Thank you @garydiaz8886, really appreciate it! :)
Very useful, video man, thanks for explaining things in so much details, keep doing the good work.
please do more vedios bro. love this one
Thank you @varunparuchuri9544, really appreciate it :)
Very useful and explaining complex things in easy manner . Thanks and expect more videos from you
Thank you so much for making this video. this is really very helpful.
Bro, you dropped this👑
Underrated pro max!
Great Video!
Appreciate it @jjayeshpawar, thank you!
This is really good, thanks so much for this explanation!
this is gold. Thank you very much!
@user-meowmeow1 Glad you found it helpful :)
Amazing content! Thank you for sharing!
Thank you @crazypri8, appreciate it :)
Great explanation!!Keep uploading such quality content bro
Great Content. Nice and Detailed!!
Thank you @shaheelsahoo8535, appreciate it :)
Very Good explanation...Keep Going
Thank you!
Thank you for taking the time to create such an in depth video for Spark Plans. This is very helpful !
Would you also be able to explain Spark Memory Tuning ?
How do we decide how much resources to allocate (driver mem, executors mem , num executors , etc for a spark submit ?
Also Data Structures Tuning, Garbage Collection Tuning !
Thanks again !
Thanks for the kind words @crystalllake3158 and the suggestion; currently the focus of the series is to cover all possible code level optimization. Resource level optimisations will come in much later, but no plans for the upcoming few months :)
Thanks ! Please do keep uploading, love your videos !
Great content brother. Please post more 😁
Great video thanks for sharing. I definitely subscribe
Excellent job 🙌
Thanks @prasadrajupericharla5545, appreciate it :)
Great explanation.
This was awesome!
I loved your explanation and understood it very well. Could you help me to understand at 23 mins, if we have join key as cid and group by region. how the hash partitioning works. will that consider both?
Can you please prepare a video showing storage anatomy of data during job execution cycle? I am sure there are many aspiring spark students who may be confused about the idea of RDD or dataframe and how it access data through apis (since spark is in memory computation) during job execution. It will help many upcoming spark developers.
Hey @udaymmmmmmmmmm, I added this video recently on Spark Memory Management. It talks about storage and responsibilities or each of memory components during job execution. You may want to have a look at it :)
Link here: ruclips.net/video/sXL1qgrPysg/видео.html
You are awesome man❤
Great explanation man! Thank you! What's the editor that you use in the video to read query plans?
Thanks @venkateshkannan7398, appreciate it. Using Notion :)
Bro please make more videos !!!
Great explanation! I love the simplicity of it! I wonder what is the app you use for having your Mac as a screenshot that you can edit with your iPad?
Thanks @kvin007! So, basically I join a zoom meeting with my own self and annotate, haha!
Hi Afaque.
A suggestion.
You could start from the beginning to connect the DOTS!
Like if in your scenario we have X Node machine with Y workers and Z exectors and if you do REPARTITION and fit the data like this then this could happen.
Otherwise the Machine would sit idle and so on.
quality content
Thank you!
You were too good!
Thank you!
At the very end of the video 38:36, we see that the cast("int") filter is present in the parsed logical plan and Analyzed logical plan. I am a little confused as to when we refer those plans. Can you please explain?
I am doing coalesce(1) and getting error as : Unable to acquire 65536 bytes of memory, got 0.
But when i am doing repartition(1), it worked. Can you please explain what happens internally in this case?
if it is doing local aggregation before shuffling the data then why it will throw out of memory error while taking count of each key when the column has huge distinct values
Thanks for the content and when can we expect new video?
Coming soon, in the next few days! :)
I have doubt when the data will be distributed to executor is it before scheduling the task or after scheduling the task and who assign the data to executor
Hey @sangu2227, this requires an understanding of transformations/actions and lazy evaluation in Spark. Spark doesn't do anything (either scheduling a task or distributing data) until an action is called.
The moment an action is invoked, Spark creates a logical -> physical plan and Spark's scheduler divides the work into tasks. Spark's driver and Cluster manager then distributes the data to the executors for processing :)
Hi sir i came across a doubt
Consider the executor size 1gb/executor. We have 3 executors and intially 3 gb data gets distributed across 3 executors each executor is having 1gb partition after various transformations we came across a requirment to decrease the number of partitions to 1 partition for that we will use repartition(1) or coalesce(1). In this scenario all the 3 partitions will merges to 1 partition each partition is having size of 1 gb approximately. Collectively all the partitions size is 3 gb approximately. When repartition (1) or coalesce(1) all the 3 gb data should sit in 1 executor having capicity of 1gb only. So here the data is execeeding the executor size what happens in this scenario. Could you please make video on this requesting sir.
Hi @bhargaviakkineni, In the scenario you described above where the resulting partition size (3 GB) exceeds the memory available on a single executor (1 GB), Spark will attempt to spill data to disk. The spill to disk is going to help the application from crashing due to out-of-memory errors however, there is going to be a performance impact associated, because disk IO is slower.
On a side note, as a best practice, It’s best to also think/re-evaluate the need to write to a single partition. Avoid writing to a single partition, because it generally creates a bottleneck if the sizes are large. Try to balance out the partitions with the resources of the cluster (executors/cores).
Hope that clarifies :)
You mentioned that for coalesce(2) shuffle will happen, but later you mentioned that shuffle will not happen in case of coalesce hence no partitioning scheme. Could you please explain it in detail?
So, coalesce will only incur a shuffle if its a very aggressive situation. If the objective can be achieved by merging (reducing) the partitions on the same executor, it will go ahead with it. In case of coalesce(2), its an aggressive reduction in the number of partitions, meaning that Spark has no other option but to move the partitions. As there were 3 executors (in the example I referenced in the video), even if it reduced the partitions on each executor to a single partition, it would end up with 3 partitions in total, therefore it incurs a shuffle to have 2 final partitions :)
@@afaqueahmad7117 Thanks for clarification.
Hi Afaque, how can I download the data files you are using? I want to try it hands on :)
Should be available here: github.com/afaqueahmad7117/spark-experiments :)
Hello @afaqueahmad7117, thanks for the great video. While explaining repartition, you mentioned you’ve a video on the AQE. Please can you link that as well?
Thanks @nijanthanvijayakumar, yes that video is upcoming in the next few days :)
Can't wait for that@@afaqueahmad7117
These RUclips videos are so much more helpful. Hats down one of the best ones that explain the Spark performance tuning and internals in a very simplest of forms possible. Cheers!
In exchange hashpartitioning what is the significance of number 200? what does that mean?
200 is the default number of shuffle partitions. You can find the number here in this table by the property name "spark.sql.shuffle.partitions" spark.apache.org/docs/latest/sql-performance-tuning.html#other-configuration-options
Local distinct on cust id doens't make sense and couldn't understand. How globally it does distinct count if the count is already computed. The reasoning behind why cast doens't push down predicate is not clearly explained and just as it's mentioned in the doc
Hi Sir, you mentioned that you referred AQE before. Can I get that link ? I want to know about AQE
Yes, I will be releasing the video in the next few days. :)
@@afaqueahmad7117 Thank you sir.
What drawing board are you using for those notes?
Using "Notion" for text, "Nebo" on iPad for the diagrams
@@afaqueahmad7117cool thx!
Fab Cotenet
Great Work buddy keep it up .... love your content, very simple to understand @Afaque Ahmed
Thanks a ton!
Hi Sir Please Make a Detailed course on apache spark which include every aspect of spark for Data Engineer role Please make sure there are a lot of begineer course here in market keep the course from intermediate level to advance level. Please tr to make video in Hindi it will be very helpful.