10:32 hey @@manish_kumar_1, I feel there shouldn't be any comparison b/w DW & Spark because these 2 are very different entities. DW helps store processed data while Spark actually helps process the data to land in DW. Don't you feel so??? BTW what was the work that kept you away for a month???
You are correct. I just explained because some of the people may get confused that if parallel processing is there and also data is stored in columnar based file format, so why can't we use spark. That was the motivation behind comparing these two. I was out of station due to job requirements.
Manish ji, I really love your content i am following your Azure series. i really appriciate aapne jis tarah hum subko inte easy langauage me sub kuch explain kiya bahot se content mujhe bahot dino se clear nhi ho rahe the but aapke channel ko join karne k baad bahot hi sahi tarike se clear ho gya hai.. i really wanted to connect with you... Thanks a lot for doing this for us... And you motiviate to us... it's not that easy but if you have good mentor so journey bcame joyfull.. that making for us... :)
Which is better data analyst vs data engineer who earn more and better future and can someone from non it background can become data engineer as a fresher
Hello brother! Love your content. I was going through your spark playlist. I have a question. I have 2 csv files that I have stored by partitioning on a key. Both the files are partitioned on same key/key combination (values are same, although the actual name of the column could be different). Now I want to join the two, by reading them by spark.read , creating temp views and running basic sql against the temp views. I want to ask if there is a way to ensure that the partitions having the same key from both the files are stored on the same node when spark reads them to increase join speed?
Brother , i have a doubt . I am in a tier2 college . I want to pursue data engineering but does big tech companies hire data engineers who are just beginners or do they prefer masters and experienced people. After competitive coding should i do data engineering roadmap or any development. Please reply bhaiya ❤
Yes keep learning all the required skills for DE. You will get one. No need to go for masters, just keep in mind that openings may be less for Beginner
I think you don't want to understand the similarities and difference between these two tech stack. I don't find anything wrong here. As I said both uses distributed computing to solve business use case. There are multiple points where we can compare these two. Even Architect does the same thing before designing the solution, which will serve better.
@@manish_kumar_1 yes You are right both uses distributed computing framework. My question is how we can use spark as a Datawarehouse solution. Spark is the general purpose in memory compute engine. But we can't use Datawarehouse on top spark.
Spark series was delicate content thank-you for that really excited enough to get along with the DW series!
10:32 hey @@manish_kumar_1, I feel there shouldn't be any comparison b/w DW & Spark because these 2 are very different entities. DW helps store processed data while Spark actually helps process the data to land in DW. Don't you feel so??? BTW what was the work that kept you away for a month???
You are correct. I just explained because some of the people may get confused that if parallel processing is there and also data is stored in columnar based file format, so why can't we use spark. That was the motivation behind comparing these two.
I was out of station due to job requirements.
got it...🙌🙌@@manish_kumar_1
Very good and easy-to-understand content. Keep it up, Manish.
Areey maan gye guruji
Good one.
Very well explained ❤
Manish ji, I really love your content i am following your Azure series. i really appriciate aapne jis tarah hum subko inte easy langauage me sub kuch explain kiya bahot se content mujhe bahot dino se clear nhi ho rahe the but aapke channel ko join karne k baad bahot hi sahi tarike se clear ho gya hai.. i really wanted to connect with you... Thanks a lot for doing this for us... And you motiviate to us... it's not that easy but if you have good mentor so journey bcame joyfull.. that making for us... :)
Great to see you after long gap
very well explained . Please regular videos upload krein
I will try my best
Thankyou
Commenting first being first viewer just to say your work is awesome!!!
Have talked to you over linkedin a few times, and you are as nice a person as a teacher
Thank you so much 😀
Finally you are back after a long gap.
Yes
Do we need to learn data warehouse inspite if spark sql and scala hive ?? Please suggest
Which is better data analyst vs data engineer who earn more and better future and can someone from non it background can become data engineer as a fresher
in today's scenario Snowflake have all ai/ml capabilities, streaming and even handles all type of data, so very less difference with spark now
bring something on delta lake
❤
Hello brother! Love your content. I was going through your spark playlist. I have a question. I have 2 csv files that I have stored by partitioning on a key. Both the files are partitioned on same key/key combination (values are same, although the actual name of the column could be different). Now I want to join the two, by reading them by spark.read , creating temp views and running basic sql against the temp views. I want to ask if there is a way to ensure that the partitions having the same key from both the files are stored on the same node when spark reads them to increase join speed?
Why do we need Data Warehouse when lakehouse architecture is there?
Usko v discuss karenge aage ke videos me
very good manish bhai . bhai ab next video kab upload karo gaye
Aaj aa rha hai🙂
Data is updating in bw 12-2 pm . How to avoid duplicacy while writing in datawarehouse ??
Merge can help you to avoid duplicacy
is your's data warehouse series is enough to crack azure data engineering data warehouse interview.
Yes
Brother , i have a doubt . I am in a tier2 college . I want to pursue data engineering but does big tech companies hire data engineers who are just beginners or do they prefer masters and experienced people. After competitive coding should i do data engineering roadmap or any development. Please reply bhaiya ❤
After one year you can come into Data Engineering domain. But as fresher it is hard but not impossible.
Yes keep learning all the required skills for DE. You will get one. No need to go for masters, just keep in mind that openings may be less for Beginner
Are the videos uploaded daily or not ?
No 3 videos per week on an average
Der kardi 1 month se wait kar rahe hai
Sorry for the delay
Why you are comparing data warehouse to spark?
spark is different thing
This is awkward, here you are telling Spark
I think you don't want to understand the similarities and difference between these two tech stack. I don't find anything wrong here.
As I said both uses distributed computing to solve business use case.
There are multiple points where we can compare these two. Even Architect does the same thing before designing the solution, which will serve better.
@@manish_kumar_1 yes You are right both uses distributed computing framework.
My question is how we can use spark as a Datawarehouse solution.
Spark is the general purpose in memory compute engine.
But we can't use Datawarehouse on top spark.
Thankyou @manish_kumar_1 for such simplified explanation 🙂