Hi Afaque, thank you for another great explanation, I have a question, since AQE splits skewed partitions into smaller ones, is salting still useful when AQE is enabled ?
Thanks for great content, You should of used Salt bae gesture when you said salting :) Is Slating still good approached if join is happening between two large datasets with hundreds of million rows? Explode will increase the number of rows for one dataset. Let's say 100,000,000 * 200 Salet_Number = 20,000,000,000 Rows
I have read and watched many things related to salting but this visual explanation just makes it really easy to comprehend it, plus really well articulated. Waiting for more videos to learn from :) Also could you recommend some books or other resources that have enabled you to attain this level of knowledge, Thanks!
Hey @user-nz7uh1qo5o, many thanks for the kind words, it means a lot to me, and, glad to know that the video was helpful. Most of the content is based on my work experiences + good ad-hoc content on Medium to which I could relate. My only humble suggestion is to be ruthless, get your hands dirty, question everything that's happening and search the internet if anything doesn't makes sense :)
Hi, can you please help me why are we considering salt number 3 or4. Is this should be equal to number of shuffle partitions that we have in our data or the distinct values that we have in our dataset.Please explain.
can you show us if salting in aggregations was really worth it ? I'm skeptical that too many shuffles in salting will deteriorate the performance with salting.
Hey @akshaybaura, there will indeed be a performance dip due to shuffles when using Salting, but, without Salting you're at the risk of either: a. Getting OOM (out of memory) errors. b. Your jobs running 5-10x slower because fewer resources (cores and memory) are being used while the others remain underutilised. However, even when using Salting, the performance largely depends on factors like the size of dataset and the correct use of Salt Number.
@afaqueahmad7117 - Great topic and amazing explanation - Looking forward to learning more from you. One suggestion is to create more videos related to designing idempotent data pipelines, backfilling missed window data, simulating different production failures and how to approach them, coz I see more people are doing interview focused videos. These topics will mentor both entry level and mid level Data engineers to gain confidence in Data Engineering field
Glad you liked the video and the explanation! Really appreciate your feedback. Yes, all of that is in the roadmap, but for the upcoming year. The initial plan is to cover all aspects related to Performance Tuning + Foundations.
Hey @gudiatoka, I wish it was so, but just in case you're referring to AQE as the solution, it isn't always very helpful, so you still need to resort to salting.
@@afaqueahmad7117 yes AQE and partition is useful and in case of larger dataframe when salting key applied to lower df it duplicated records making it more skewed then the concept of salting not valid at least for me...may be it servers different
Hi Afaque, thank you for another great explanation, I have a question, since AQE splits skewed partitions into smaller ones, is salting still useful when AQE is enabled ?
Thanks for great content, You should of used Salt bae gesture when you said salting :)
Is Slating still good approached if join is happening between two large datasets with hundreds of million rows? Explode will increase the number of rows for one dataset. Let's say 100,000,000 * 200 Salet_Number = 20,000,000,000 Rows
This is excellent Spark content videos. It is prefect explanation on Spark performance concept.
Many thanks @dhavaldalasaniya, this means a lot, appreciate it :)
Hi, I saw in Spark UI that AQE sometimes handles the data skew. Does enabling AQE can be a solution to the data skew problem.
Precious 30 minutes, quality content
Thank you @sasadsasadsad, appreciate it :)
Excellent Content..!! Loved It. Such detailed explanation on Salting Technique with Graphical Representation.
Appreciate it @arghyakundu8558 :)
I have read and watched many things related to salting but this visual explanation just makes it really easy to comprehend it, plus really well articulated. Waiting for more videos to learn from :)
Also could you recommend some books or other resources that have enabled you to attain this level of knowledge, Thanks!
Hey @user-nz7uh1qo5o, many thanks for the kind words, it means a lot to me, and, glad to know that the video was helpful. Most of the content is based on my work experiences + good ad-hoc content on Medium to which I could relate. My only humble suggestion is to be ruthless, get your hands dirty, question everything that's happening and search the internet if anything doesn't makes sense :)
I never see such a detailed explanation. Thank you very much @afaque Ahmad. Is there any way we can get the document.
Appreciate it @janb4637, let me try and put it on GitHub :)
Hi, can you please help me why are we considering salt number 3 or4. Is this should be equal to number of shuffle partitions that we have in our data or the distinct values that we have in our dataset.Please explain.
Nice explaination
Hey @Afaque, great content as usual, but I thought this video could be a little concise, great work anyways!
Thank you @rgv5966 for the appreciation. Tried my best to keep it concise, but will take your feedback :)
what if the values are unique in join 1 to 1 join? will it create skew
Not easy to understand, but it great
Thanks for this. Love from 🇵🇰
Appreciate it @MuhammadAhmad-do1sk, Love from India :)
can you show us if salting in aggregations was really worth it ? I'm skeptical that too many shuffles in salting will deteriorate the performance with salting.
Hey @akshaybaura, there will indeed be a performance dip due to shuffles when using Salting, but, without Salting you're at the risk of either:
a. Getting OOM (out of memory) errors.
b. Your jobs running 5-10x slower because fewer resources (cores and memory) are being used while the others remain underutilised.
However, even when using Salting, the performance largely depends on factors like the size of dataset and the correct use of Salt Number.
@afaqueahmad7117 - Great topic and amazing explanation - Looking forward to learning more from you. One suggestion is to create more videos related to designing idempotent data pipelines, backfilling missed window data, simulating different production failures and how to approach them, coz I see more people are doing interview focused videos. These topics will mentor both entry level and mid level Data engineers to gain confidence in Data Engineering field
Glad you liked the video and the explanation!
Really appreciate your feedback. Yes, all of that is in the roadmap, but for the upcoming year. The initial plan is to cover all aspects related to Performance Tuning + Foundations.
hash(1,0)%3 how to calculate?
0%3
@vinothvk2711 is right. As outlined in the video, we're assuming h(1, 0) = 0, so it's equal to 0 % 3 = 0
BRO PAAN CHABAKE BOL RAHA HAIN KYA EXPLANATION IS SUPER
Haan bhai @shivoham5939 paan chaba raha tha :)
@@afaqueahmad7117 awesome 👌
@@afaqueahmad7117 bro i need one on one call with you i will pay as much you want plzzz
@@afaqueahmad7117 I have interview I need guidance
After 3.0 salting is not useful
Hey @gudiatoka, I wish it was so, but just in case you're referring to AQE as the solution, it isn't always very helpful, so you still need to resort to salting.
@@afaqueahmad7117 yes AQE and partition is useful and in case of larger dataframe when salting key applied to lower df it duplicated records making it more skewed then the concept of salting not valid at least for me...may be it servers different