Wooow awesome super marvelous or if there are any other words in English which can praise in depth of concept you have explained.. Thank you very very much sir
I have a question - please tell me whether my question is right or wrong. . I have a pipeline in databricks and it runs 4 times in every 24 hours in production environment. 8am, 2pm, 8pm, and 2am. At those 4 times - there may be other different applications also running in the same databricks production cluster. Say - at 8am 10 other jobs are running, at 2pm no other job is running, at 8pm 5 other jobs are running, and at 2am 4 other jobs are running. If after all the logical and physical planning is done and if multiple physical plans are created and cost model is applied - then - will there be different physical plans, which will be executed at 8am, 2pm, 8pm, and 2am respectively ? In other words - will the choice of physical plan actually executed at 8am, 2pm, 8pm, and 2am depend on actual run-time work-load/volume of work at that time (i.e. 8am, 2pm, 8pm, and 2am) in production environment ?
Why is exchange happening after sort-merge join ? I see both the tables are initially exchanged and then sorted which means same DEPT-ID records are on the same partition that enabled sort-merge join. So, I can't understand why exchange partitioning is done after sort-merge join.
I requested this a few days ago and here you are with awesome explanation. This shows how much you care about the community :)
Thank you for your kind words 🙂
Honestly, I was intimidated my the length of the video but then after watching it. It was very simple :) Thanks
Glad it helped!
very nice explanation raja..Thank you for your efforts to make this video..looking forward to more videos ...It is very useful
Sure Ram, will post more videos
Hands down best df.explain() video I've watched so far.
Glad it was helpful!
Wooow awesome super marvelous or if there are any other words in English which can praise in depth of concept you have explained.. Thank you very very much sir
Thank you Sonu!
Great explanation sir, keep going😊
Thank you! Keep watching
Very clear explanation, thanks.
Glad it was helpful!
Nice explanation Raja 👌 👍 👏
Thanks Sravan!
Awesome content sir
Thank you
I have a question - please tell me whether my question is right or wrong.
.
I have a pipeline in databricks and it runs 4 times in every 24 hours in production environment. 8am, 2pm, 8pm, and 2am.
At those 4 times - there may be other different applications also running in the same databricks production cluster.
Say - at 8am 10 other jobs are running, at 2pm no other job is running, at 8pm 5 other jobs are running, and at 2am 4 other jobs are running.
If after all the logical and physical planning is done and if multiple physical plans are created and cost model is applied - then - will there be different physical plans, which will be executed at 8am, 2pm, 8pm, and 2am respectively ?
In other words - will the choice of physical plan actually executed at 8am, 2pm, 8pm, and 2am depend on actual run-time work-load/volume of work at that time (i.e. 8am, 2pm, 8pm, and 2am) in production environment ?
Why is exchange happening after sort-merge join ? I see both the tables are initially exchanged and then sorted which means same DEPT-ID records are on the same partition that enabled sort-merge join. So, I can't understand why exchange partitioning is done after sort-merge join.
This is great content. Is it possible to version control data frame query execution plans?
Good one 👍
Thanks 👍🏻
Well Explained !!!!! 😄
Glad you liked it! Thanks
really nice, thankyou so much!!!
Thank you! Glad you find it useful
Vera level thala!!(G*d level explanation)
Thank you
@@rajasdataengineering7585 Bro..can you explain diff between lineage graph and dag?
Bcoz both are diff ryt?
Both are same
Thank you!