66. Databricks | Pyspark | Delta: Z-Order Command

[@shreeyashransubhe2537]

Sir, I have gone through lots of videos but never understood the concepts so simple yet very detailed manner. Thank you very much. I have shared your playlist with my colleagues too. They also liked it very much.

[@rajasdataengineering7585]

Thank you for your valuable comments. Really appreciated

[@pratikparbhane8677]

Great Explain , Understood OPTIMISE , VACCUM() AND Z-ORDERING in One Video

[@rajasdataengineering7585]

Glad it was helpful!

[@AnupGupta-05]

Hi Brother you are best teacher, the way you explain its best, keep up the good work

[@rajasdataengineering7585]

Thank you!

[@ravisunkara6664]

Awesome explanation on Z-ordering. Greatly appreciated your efforts making this video.

[@rajasdataengineering7585]

Thank you

[@annaduraip3182]

Great, thank you. You have explained in simpler way to understand anyone.

[@rajasdataengineering7585]

Thank you

[@rohitdanda]

Your videos are so simple that a kid can also understand. Thanks and salute sir🖖 for putting so much effort and making videos and helping us!

[@rajasdataengineering7585]

Thanks for your comment. Glad to know it helps data engineers

[@mukilanlakshmanan8968]

Sir, I love your teaching method, you have explained it in detail.

[@rajasdataengineering7585]

Thanks Mukilan! Glad to hear that

[@navdeepjha2739]

Invaluable explanation sir! I went through many blogs but couldn't get it. You made it crystal clear😊

[@rajasdataengineering7585]

Glad to hear that! Thanks for your comment

[@NileshPatil-b3u]

Sir, Thanks for explaining in a very simple manner.

[@rajasdataengineering7585]

Thanks and welcome

[@ajaykiranchundi9979]

A very well explained . The way you broke down the data to explain the same is amazing. I am sure it would have taken good time to put it together. Indebted to you brother.

[@rajasdataengineering7585]

Thanks Ajay👍🏻

[@sundarkannan1856]

Thank you and very clear and helps
to understand what is Zorder method in delta table

[@rajasdataengineering7585]

Glad it was helpful! Keep watching

[@SumitAmbatkar]

i watched your nearly all playlist i loved your teching style, how to ogrip on concept, your explaination are fabulous keeping doing sir, best of luck. we are always here for you
Thank you..:)

[@rajasdataengineering7585]

Thank you,Sumit! Keep watching

[@RanjeetkumarYadav]

Amazing and very intuitive example. Thank You!!

[@rajasdataengineering7585]

You're very welcome! Keep watching

[@YogeshBiguvu2208]

Excellent explanation with Examples.....Thank you so mcuh sir..

[@rajasdataengineering7585]

You are most welcome! Glad it helps

[@sraoarjun]

Indeed an awesome video !! Great explanation !!

[@rajasdataengineering7585]

Glad you liked it! Thank you

[@terrificmenace]

Thank you sir 🙏🏻 I went through many udemy courses but never understood these concepts. Ur explanation is very good and easy to understand many many thanks sir 🙏🏻 🙏🏻

[@rajasdataengineering7585]

Thank you 👍🏻

[@rahul_chilukamari]

one such good video with neat explanation.

[@rajasdataengineering7585]

Thank you

[@shwetac2929]

you teaching methos is very good ....this video clear my all doubt

[@rajasdataengineering7585]

Glad to hear that

[@AFSARAHMED4]

Excellent Explaination Sir

[@rajasdataengineering7585]

Thanks

[@dineshwaditake5248]

Nicely explained !!

[@rajasdataengineering7585]

Glad it was helpful!

[@arabajshaikh8411]

Excellent, Thank you so much.

[@rajasdataengineering7585]

Glad it was helpful! You are welcome

[@omprakashreddy4230]

Your videos are definitely creating great impact. Thank you for that.
Can you also please explain df.explain() command in great detail with examples.

[@rajasdataengineering7585]

Happy to hear that it's creating impact on data engineers. Thank you
Sure, will post a video on explain plan

[@rajasdataengineering7585]

Hi Omprakash, created a video on explain plan as per your request. Hope it helps you - ruclips.net/video/6NrVQTbkndU/видео.html

[@vivek05117gece]

very well explained. Kudos to you.

[@rajasdataengineering7585]

Glad it was helpful!

[@gil.0007]

Very nicely explained 🎉

[@rajasdataengineering7585]

Thanks, glad it was helpful!

[@saurav0777]

Thanks for uploading . Very nice explanation

[@rajasdataengineering7585]

Thanks

[@FreakONcW1]

Extremely helpful video.

[@rajasdataengineering7585]

Thanks Kinjal! Glad to know it was helpful!

[@tanushreenagar3116]

Very nice sir 👌 cleared my concept now

[@rajasdataengineering7585]

Thank you

[@chaitanyanagare757]

Great video content.. Thank you so much

[@rajasdataengineering7585]

Glad you liked it! Keep watching

[@bharatbhojwani4144]

Your work is quite awesome. It will be great incase you can share your notebooks too.

[@tanushreenagar3116]

PERFECT CONTENT SIR

[@rajasdataengineering7585]

Thanks Tanu!

[@manjit_singhh]

Very nice explanation 🙂

[@rajasdataengineering7585]

Thanks

[@sathyahisto]

good Explaination, liked it when you demonstrated with excel. Just one suggestion syntax for zorder seems to be changed to "Zorder by ()"

[@rajasdataengineering7585]

Yes, you are right. Thanks

[@amiyarout217]

great explanation

[@rajasdataengineering7585]

Glad it was helpful! Keep watching

[@satheeshkumarak6708]

Hello Sir, How do you determine the number of columns to be used in Z order and whether or not to use a particular column for Z order provided that you have calculated the cardinality percentage of all the columns?

[@3a8saisamireddi61]

detailed explanation👏

[@rajasdataengineering7585]

Thank you 🙂

[@mohitupadhayay1439]

Raja please try to create a full project where all these optimizations can be shown at full scale.

[@rajasdataengineering7585]

Sure Mohit, will do!

[@ravulapallivenkatagurnadha9605]

Please continue this videos

[@venkatasai4293]

Thanks for the great explanation Raja. So are the statistics collected on all the columns ? What if we want to query on other columns ? Will it work ?

[@rajasdataengineering7585]

Yes Venkata, it will work first 32 columns. If your table contains more than 32 columns and you want to collect statistics for those columns, we can configure that separately

[@venkatasai4293]

@@rajasdataengineering7585 ok . So zorder is similar to bucketing right ? Colocating the data into same set of files ? If two tables contains same key and if we zorder them on the key While joining the data it will fetch only required files into the executor ?

[@shankar1556]

Hi Azar,
Thank you for explanation.
I have a dought. in this example it shows that z-order create new partitions with sorting emp_id. Does z-order really create new partitions?

[@rajasdataengineering7585]

Hi Shankar, this is Raja.
When we perform z-order, data is being co-located within same set of files. It is not shuffling the data, nor creating new partitions

[@purnimasharma9734]

Hi Raja, how is the partition column determined e.g. how does it know that you have to use emp_id here? Is it based on the predicate column?

[@purnimasharma9734]

Never mind, when I watched your video completely, I found out.

[@rajasdataengineering7585]

Great

[@TheDataArchitect]

What about using multiple columns in z-order?

[@viniciusguimaraessantana5455]

thank you very much.

[@rajasdataengineering7585]

You are welcome!

[@anuragpaudyal3297]

awesome😃

[@rajasdataengineering7585]

Thank you! Cheers!

[@TotuBabyBird]

Great!

[@rajasdataengineering7585]

Thanks

[@kunalr_ai]

Watching on Oct 24 for job interview. Not selected yet

[@rajasdataengineering7585]

All the best for your interview tomorrow (Oct 24th)

[@aswaniyettapu9992]

Can u do one video on lead and lag in pyspark..?

[@rajasdataengineering7585]

Sure, will post a video on lead and lag very soon

[@rajasdataengineering7585]

Hi Aswani, have posted a video on lead and lag function today as per your request

[@aswaniyettapu9992]

Tq so much

[@ravulapallivenkatagurnadha9605]

Please do video on how to convert pandas data pipilines to spark data pipiy

[@rajasdataengineering7585]

Sure will do