Spark Performance Tuning | Handling DATA Skewness | Interview Question

#Spark #Persist #Broadcast #Performance #Optimization
Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more
Click here to subscribe : / @techwithviresh
About us:
We are a technology consulting and training providers, specializes in the technology areas like : Machine Learning,AI,Spark,Big Data,Nosql, graph DB,Cassandra and Hadoop ecosystem.
Mastering Spark : • Spark Scenario Based I...
Mastering Hive : • Mastering Hive Tutoria...
Spark Interview Questions : • Cache vs Persist | Spa...
Mastering Hadoop : • Hadoop Tutorial | Map ...
Visit us :
Email: [e-mail]
Facebook : / tech-greens
Twitter :
Thanks for watching
Please Subscribe!!! Like, share and comment!!!!

[@desparadoking8209]

Very informative video 👍🙂

[@sahiba2227]

7:24, repartition does full shuffling and hence creates equal size partitions. i.e It always guarantess the equal sized partitions.

[@brothermalcolm]

Great video, like the pace, like the presentation.

[@TechWithViresh]

Glad you liked it!

[@SpiritOfIndiaaa]

thank you so much , really good , so what is the difference b/w isolation salting and salting ? and what is difference b/w , isolation map join & map join ??

[@Dsmehra379]

thank you so much for this video, but i am not able to find the 2nd part of this video.. Can you please comment the link for the part 2 video

[@AdityaKommu1]

Hello,
Your videos are very good,
Can you please do a video on incremental data load and full data load by taking an example?

[@vijeandran]

Hi Viresh, Thanks for the video.... How can we achieve salting technique in Pyspark?

[@rishigc]

@TechWithViresh I simply love your videos. I have watched your other tutorial videos too. They are awesome. I am interested in knowing how to do Iterative Broadcast Join with the SQL API. Any help is highly appreciated. Can you pls advise.

[@Gecasomx]

Thanks for the video, no part 2 tho?

[@ramkumarananthapalli7151]

Thank you for making this video. Could you suggest on which column mean, medium and the mode are calculated?

[@bentchow]

The columns are those that are being shuffled by such as the join columns or group by columns. There is data skew when the distribution is not normal.

[@udaynayak4788]

Hi Viresh, can you please share the link for part 2

[@harshvardhansolanki1466]

Thank you so much for the video. I seek some clarification though.
In your example you did mapPartition. Means for each partition of different keys, you updated the key with salt. But still the records remained in the respective partitions only. How will those records be shuffled across partitions for equal distribution?

[@TechWithViresh]

Partition will change with the change in the key, as it is essentially the hascode of key+salt now.

[@harshvardhansolanki1466]

@@TechWithViresh I tried it so I believe a new DF will have to be created and REPARTITIONED again! in order for the records to be shuffle by updated salted keys. It wont just trigger shuffle on key update in mapPartition function! That only makes sense.

[@vivekpuurkayastha1580]

If the partition key is non numeric then how to perform salting? like your tower ids were numeric, but if instead of being 1, 2, .. they are to be A, B, ...

[@jaiharsad7121]

Hi sir, pls upload the spark interview question videos which were present earlier.. I'm not able to find them in your playlist

[@TechWithViresh]

All the videos are uploaded, please check:)

[@ayanbizz]

Nice explanation.A couple of questions 1) Repartitioning does ensure the data distribution is not skewed (unlike coalesce) 2) You said repartitioning uses the hash value to distribute the data (are you talking about bucketing ?)

[@TechWithViresh]

There are two provided partitioners in Spark 1. Hash partitioner and 2. Range partitioner.Default is Hash one.

[@harshvardhansolanki1466]

If you repartition on column, there you can get skewed data. If you repartition by number of parts then distribution may be almost equal.

[@ajaywade9418]

video from 11:30, we are adding random key to exiting towerid key
for Example. tower id: 101 and salt key : 67 then 101+67= 168 hash value of the 168 would be a final value right.
what in case of partition column is string datatype. ??

[@TechWithViresh]

Incase of strings, we can add surrogate keys, based on string column values and then do the salting.

[@thanoojbharateeyudu3786]

We could loose our key join by Salting key adding random numbers
If we want to do join with the same key then problem
May be join key could be the different on other than salted column

[@chilukapavan6344]

Awesome video 🙏...can you pls share part2 video link

[@TechWithViresh]

Coming Soon!! , Thanks :)

[@pareshpal3533]

@@TechWithViresh when ?

[@aneksingh4496]

have u uploaded part 2 of this

[@TechWithViresh]

Check out other videos in the playlist for performance optimization and executor tuning.

[@Hk-eo5yr]

can u share part 2 video

[@TechWithViresh]

Coming Soon!! , Thanks :)

[@rishigc]

where is Part 2 ?

[@bhuneshwarsingh630]

please give some solid coding example with explaination