Видео

window = Window.partitionBy("cust_id").orderBy("flight_id").rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing) df1 = df.withColumn("origin", first("origin").over(window))\ .withColumn("destination", last("destination").over(window)) df1.select(["cust_id","origin","destination"]).dropDuplicates().show() This is a short approch

What was the experience for this candidate? I mean people with how much experience can expect this qs

Don't say simple for everything....we see difficulties here

Thank you. Pls keep posting videos on pyspark interview questions

I have a question on threadpool in spark. When we use threadpool executor, all threads are running on same node? Like only on driver node? Or Will it utilize all the workers in the cluster? Can you please clarify ?

null_df=null_df.agg(sum(when(col('id')=='Null',1).otherwise(0)).alias('id'),sum(when(col('name')=='Null',1).otherwise(0)).alias('name'),sum(when(col('age')=='Null',1).otherwise(0)).alias('age')) null_df.display()

Pretty simple

Bro Can You provide an Roadmap for me I'M an fourth year student previously i have knowledge on mysql,python,pandas,powerbi,excel But Wanted learn data engneering for better options Can you Guide me or can you suggest me any courses

Can you please post the dataset in the comment section it breaks the flow and sorry to say but quite annoying

❤❤❤❤❤❤❤

bro also attach csv file in the description it will be better we can prac. although u r doing great job

once we migrated the data from source to destination -> how do we validate that data whether we have migrated correct data (how to compare source and destination data) could you ans this

Your voice very low , increse the Audio

Hi Pritam Saha, I'm also from TCS. Recently I'm continuously watching your pyspark Interview Series, Really helps alot to develope my problem solving skills in Pyspark, Great work brother. If possible try to create a interview preparation series for python as well, for SQL there are lot of platforms and other channels are there but for Python only few. So would appreciate if you start for python as well thank you brother.

Can you please make a similar video using pytest framework for testing databricks notebooks

Nice content brother, usefull for all the aspiring Data Engineers.

hello.... i am looking for Data engineer roles... mostly openings are in accenture,mindtree, deloiite, ....but how to check these kind of product based openings..

very disturbed explanation.. seems due to linguistic oddities

1. window_spec = Window.partitionBy("product").orderBy("sale_date") wdf = df.withColumn("2nd day pre sales" , lead("amount" , 2).over(window_spec))\ .withColumn("3rd day pre sales" , lag("amount" , 2).over(window_spec))

Thanks for sharing🎉

Can you please create video on DLT streaming tables. I'm facing issues while using SCD1. My bronze notebook is seperate and Silver notebook is seperate. I'm facing issues while calling bronze table as stream and loading into silver.

thanks for sharing. keep up the good work!!