Performance Tuning in Spark

If you need any guidance you can book time here, topmate.io/bhawna_bedi56743
Follow me on Linkedin
/ bhawna-bedi-540398102
Instagram
bedi_foreve...
You can support my channel at: bhawnabedi15@okicici
Here are the links you might need to re check!
JOIN STRATERGIES IN SPARK
• 35. Join Strategy in ...
CHOOSE RIGHT CLUSTER CONFIGURATION
• 22. How to select Work...
• Databricks Cluster Cre...
CORRECTLY PARTITION THE DATA
• Partitions in Data bricks
• 8. Delta Optimization...
Z-ORDER/COMPACTING
• 8. Delta Optimization...

[@tanushreenagar3116]

So nice its helps a lot

[@EDWDB]

Thanks Bhawna, can you please make a video on monitoring and troubleshooting spark jobs via UI

[@CoolGuy]

Bucketing, salting are also good optimization techniques.

[@AyushSrivastava-gh7tb]

Hi Bhawna. Your videos have helped me immensely in my databricks journey and I've nothing but appreciation for your work.
Just a humble request, could you also please make a video on Databricks Unity Catalog??

[@cloudfitness]

Yes already done with a playlist in UC 😀

[@oldoctopus393]

1) 0:54 - not correct. DataSets and DataFrame has to be serialized and de-serialized as well, but since these APIs impose structure on data collection these processes could be faster. Overall RDDs provide more control to Spark in terms of data manipulations;
2) not all DataFrames could be cached;
3) UDFs could be converted into native JVM bytecode with help of Catalyst optimizer. You may use df.explain() to see something like "Generated code: Yes" or "Generated code: No" in the output

[@krishnasai7550]

Hi bawana,
I learned somewhere we cannot uncache the data but we can unpersist so we use persist more inplace of a cache. but here you mentioned we can uncache. I'm bit confused which is correct?

[@AbhinavDairyFarm]

Please share this ppt that will help us

[@stevedz5591]

How can we optimize spark Dataframe write to CSV it takes lot of time when it's a big file. Thanks in advance

[@RohitSharma-ny1oq]

Mem ur voice like #Soote ko jga d

[@cloudfitness]

Hahhahha...yeah agree😂