spark sql engine in spark | Lec-8

[@dineshpandey5008]

I have watched so many videos of spark but never find such explanation... amazing ..... You have very deep knowledge about spark.... solute....

[@lucky_raiser]

bro, why do not you ask people to like , share, and subscribe. such a great content, it needs to be reached to more audience.

[@jaypurohit5566]

What a Great explanation.. This series is so useful. Highly appreciate your efforts.

[@mohammedasif4349]

Tum bahut mast kaam karta hai Manish Bhai

[@RahulRajSingh-z4i]

This is series is liquid gold amazing content❤

[@AkankshaAher-o1i]

Your teaching is awesome, understable and very neet to point. Thanku you for your efforts and teching this for free. 😊

[@Azure-Mahesh]

wonderful explanation on DAG & LAZY EVALUATION, Thank you

[@BlissOfSoul07]

How cutely you said Byeee😉

[@goldykarn5922]

Thank You Bhaiya for this amazing series

[@manish_kumar_1]

Directly connect with me on:- topmate.io/manish_kumar25

[@DataForgeAcademy]

Great session, explained in depth. Thanks

[@darshakkumarranpariya867]

It's a great series, sir. Highly appreciated

[@engineerbaaniya4846]

Best Playlist for SPark Heads off to you man

[@yashsharma8687]

its "Hats off to you" :)

[@Matrix_Mayhem]

Thanks Manish! Really informative lecture.

[@HanuamnthReddy]

Buhuth archa bhathayavhain.. thank you nhaya ghee..😊

[@rohitgaikwad2266]

Thanks Manish . Detailed Explanation.

[@adityakvs3529]

bhai catalyst optimizer is used in job level or entire application level

[@yashwantdhole7645]

Manish bhai hamara AQE, Dynamic Partition Pruning ye saare optimizations 'Optimised Logical plan' ke phase me bante hai?

[@rohishzade4107]

We cannot directly say Spark SQL engine is a compiler, its not a traditional compiler, but it incorporates some compiler-like behavior.

[@neerajCHANDRA]

Very nice Manish

[@lokeshrathi5500]

What is the use of Physical Plan, what steps are required to create them?

[@rajnandinipadhy2533]

please make a detailed video on catalyst optimizer and tungsten engine

[@mission_possible]

Thanks for the sessions ...can you please make video on Adaptive Query Execution

[@bangalibangalore2404]

Cost based model banane ke liye usko pata kaise chalta hai ki yeh operation zyada computation cost karega? Uske liye to spark ko actually us process ko karke dekhna parega? Ki metadata se andaja laga leta hai?

[@meetpatel.47]

Where our catalog metadata is stored?
Means where it store metadata file?
Spark doesn't support data storage.

[@mayankkandpal1565]

nice explanation.

[@ankitachauhan6084]

very nice explanation

[@ChandanKumar-xj3md]

Hey Manish, waiting for the next spark lecture.

[@CctnsHelpdesk]

what is DataFrame/Dataset APIs . is it same as Dataframe while loading the data to dataframe or some kind of API used to read the data and save into dataframe

[@MayankKumar-if2bs]

Great session so far❤

[@raghunandan8390]

You could give definitions for each concept

[@DpIndia]

Nice Videos

[@anshukumari6616]

Thanks Manish !!

[@sankuM]

Your series of #spark fundamentals is really helpful, @manish_kumar_1! 🙌🙌 Keep continuing this.. thanks on behalf of all existing/buddying data engineers! :)

[@riptideking]

why cont catalog comes before unresolved logical plan? this may reduce the time and cost.

[@dexterslab7750]

what is the difference between spark engine (which you have explained in spark ecosystem lecture) and spark SQL engine.
and if they are same then why are we calling it spark SQL engine.?

[@IamDanish99]

I think he mentioned it as spark sql engine. But as far as I know catalyst optimizer is an optimisation framework within Spark sql engine.

[@bhavyabhagwani5407]

@@IamDanish99 what are you referring to while learning spark?

[@Abhishektyagi-v3c]

Thanks Manish for this , was waiting for your next video since last two days. But Please also make video on handling and working with other file formats, such as Parquet, ORC, AVRO it will also be useful.

[@manish_kumar_1]

Next video will be on this

[@ManishSharma-fi2vr]

❤Thank You!

[@adityakvs3529]

Bhai can we meet one meeting i have some doubts in the catalyst optimizer, which i tried resolve online but couldn't

[@manish_kumar_1]

Yes you can book on topmate

[@saisreekar7414]

Sir will all these steps will happen in application driver?

[@manish_kumar_1]

Yes

[@ajaypatil1881]

bhaiya please spend little bit more time on explaining potential interview questions at the beginning

[@manish_kumar_1]

Sure

[@ajaypatil1881]

thank you@@manish_kumar_1

[@mohitdaxini3067]

Request - Nested json reading without any udf

[@manish_kumar_1]

Learn explode function in pyspark. You will get it