23. Databricks | Spark | Cache vs Persist | Interview Question | Performance Tuning

#Cache, #Persist, #DatabricksOptimization, #SparkOptimization, #CachevsPersist, #DatabricksInterviewQuestions, #SparkInterviewQuestions, #DatabricksInterview, #DatabricksPerformance,
#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure

[@omprakashreddy4230]

Only few people have ability to teach in way that even novice can understand. Hats off to you.
Keep going !!!

[@rajasdataengineering7585]

Thank you for your encouraging words

[@Poori1810]

can not agree more

[@pavithraeshwar8881]

This is the explanation thank you for share the knowledge sir👏

[@rajasdataengineering7585]

Thanks and welcome

[@stepup2me1]

You have very good way of explaining the concepts. Thank you!

[@rajasdataengineering7585]

Thank you Chetan

[@rockykefunday2707]

you are the real raja bro , super

[@rajasdataengineering7585]

Thank you bro

[@gulsahtanay2341]

Thank you for sharing your knowledge with us!

[@rajasdataengineering7585]

My pleasure! Thank you

[@rahulpandit9082]

I found many videos on RUclips regarding Cache and Persist, but nobody explain like the way you did...

[@rajasdataengineering7585]

Thank you Rahul

[@coolraviraj24]

You explained it so simply...
i hope will be able to explain to the interviewer the same way u did😅

[@rajasdataengineering7585]

Thank you! All the best!

[@joyo2122]

your videos are the best

[@abinaya7704]

Your videos are making wonders!!

[@rajasdataengineering7585]

Thank you

[@kamalbhallachd]

Good 👍

[@rajasdataengineering7585]

Thank you! Cheers!

[@tanushreenagar3116]

Nice content sir

[@rajasdataengineering7585]

Thanks!

[@turanfair9364]

Best teacher!!! Thank you sir 🙏🏻

[@rajasdataengineering7585]

Thank you Turan

[@justvenkyy...3423]

this is too good . please keep doing. can you post on processing small file problem with spark?

[@rajasdataengineering7585]

Thanks 👍🏻
Sure will post a video for small file problem

[@vutv5742]

Great explaination 🎉

[@rajasdataengineering7585]

Glad it was helpful! Keep watching

[@shakthimaan007]

But where and how do we define these? Can you please add a short demo?

[@kamalbhallachd]

Knowledge session

[@rajasdataengineering7585]

Thanks Kamal

[@iamkiri_]

Raja, I really appreciate your explanation :)

[@rajasdataengineering7585]

Glad to hear that! Thanks for your comment

[@swathi6472]

Please make Video on Salting in Performance optimization

[@rajasdataengineering7585]

Sure will create a video on salting technique

[@pankajchikhalwale8769]

I guess you have at least an M.Tech. + M.Ed. degrees.
Expert in Spark and Amazing Teacher.
Sir, Tussi Grett Ho !

[@rajasdataengineering7585]

Thank you Pankaj! Hope you like the tutorial

[@pankajchikhalwale8769]

@@rajasdataengineering7585, So far I have watched 9 out of the 22 videos in the "Databricks Performance Optimization" playlist. It is very detailed. Like it.

[@rajasdataengineering7585]

Glad you like it!

[@ranjithajit4717]

Can you add the examples for creating persist in the description?

[@sanjayr3597]

Very good playlist which I have come across.. Could you please provide example with practical example because I was watching some videos regarding this and what I noticed was when we df.cache() then by default it is MEMORY_AND_DISK SER ..there was no just MEMORY_AND_DISK it was always SERIALIZED ..need to know the reason on this.

[@RamaiahChenna]

Hi Sir, we want vidoe for performance issues and solutions while develope the notebook
what are the issue comes

[@vlogsofsiriii]

Hi Raja. I have one doubt.
Cache - will store the data in memory means is it onheap memory ??
Persist - Will store the data in onheap and off heap both ??
Is it correct ??

[@rajasdataengineering7585]

Yes that's correct. Cache always stores in memory but persist has flexibility of memory or disk

[@vlogsofsiriii]

@@rajasdataengineering7585 memory means here onheap rgt and disk means offheap??

[@rajasdataengineering7585]

No onheap and offheap both are memory and disk is different. I have already posted a video on onheap vs offheap. Pls watch that video

[@vlogsofsiriii]

@@rajasdataengineering7585 thank you 😊

[@premsaikarampudi3944]

Hi, I was asked to prepare for Spark for my next role in the same company I am working, Is this learning series enough ?

[@rajasdataengineering7585]

Hi, yes this is more than enough if you complete all these videos

[@sravanthiyethapu9970]

Hi Raja, u said that persist will use both memory and disk. Here memory means both on and off heap memory????

[@rajasdataengineering7585]

By default, it is cached at on-heap memory. But if off-heap memory is enabled and jvm memory(on-heap) is full, off-heap memory would be used for caching remaining partitions

[@aayushdesai532]

great video sir! one question - is disc memory same as off heap memory?

[@rajasdataengineering7585]

No, off heap and in disc both are different. Off heap memory is part of RAM. on heap is controlled by jvm while off heap is controlled by os itself

[@suresh.suthar.24]

Best Explanation. but i have 1 question like cache() is a transformation or action ?

[@rajasdataengineering7585]

Cache is an action

[@tunyestark2633]

@@rajasdataengineering7585 No, cache is not an action.It is an transformation, please do try it out.

[@Uda_dunga]

Try to make videos under 10 mins sir

[@rajasdataengineering7585]

Sure, will do

[@MrPerikala]

how to avoid the duplicate rows while joining large datasets

[@rajasdataengineering7585]

Drop_duplicates or distinct can be used to remove duplicates