Spark Accumulators | Custom Accumulators with Demo | Session - 2 | LearntoSpark

In this video, we will learn about the Spark Accumulators and learn how to create a custom accumulators with one example.
Git Repo:
sample-Dataset: github.com/azar-s91/dataset/b...
scala code:
github.com/azar-s91/learntosp...

[@manjulavelayutham8046]

Thank you so much for your videos! They helped me a lot to clear my interviews ! Keep up the good work!! 🙂

[@shyamsundar8665]

Why we took this usecase
How it is different from
df.filter(df.age

[@prakashsaravanana3616]

Good one Azar !!!! I have one question Why associative and commutative operation is limitation in accumulator?

[@rajasekhar582]

Hi, azar hope ur doing well, can show all above videos in shell script in spark with scala

[@MrManish389]

How to calculate number of partitions required for a 10 GB of data, and for repartitioning and coalesce please help??

[@ShivamSingh-sm2oy]

devide by 128

[@ShivamSingh-sm2oy]

repartition u cn use to temper current partitions either increase or decrease , whereas coalesce can only be used for decreasing the no of partitions

[@MrManish389]

@@ShivamSingh-sm2oy or 256 , if 1 block size is of 256

[@ShivamSingh-sm2oy]

@@MrManish389 correct