71. Databricks | Pyspark | Window Functions: Lead and Lag

[@jagadeeswaran330]

Great explanation sir, thank you!

[@rajasdataengineering7585]

You are welcome, Jagadeeswaran! Glad it was helpful!

[@sravankumar1767]

Nice explanation Raja 👌 👍 👏

[@rajasdataengineering7585]

Thank you Sravan!

[@sharmadtadkodkar2038]

Great content. But not able to find all your videos under your page. Can you please add them in one page and share the link.

[@rajasdataengineering7585]

You can find all videos under video tab

[@rajunaik8803]

Hi Raja, you are not mentioning dataframe name while performing window definition. Will it automatically point to dataframe created in previous cell? And what if my notebook has multiple dataframes?

[@rajasdataengineering7585]

Hi Raju,
Dataframe must be mentioned to apply window function. I would have defined window logic separately and applied that later in a dataframe. It cannot be applied automatically on any dataframe

[@rajunaik8803]

@@rajasdataengineering7585 my bad, thanks Raja

[@travelfoodlife1449]

So first logic will be given and when we call data frame that will apply logic and perform the action right sir 🙌

[@rajasdataengineering7585]

Yes that's correct

[@aswaniyettapu9992]

Have you posted videos on sort merge join, autobroad cast join and shuffle join..? if not can you post that video

[@rajasdataengineering7585]

Have already posted video on Broadcast join
ruclips.net/video/HDiXK3Gl-hs/видео.html
yet to post on sort merge and shuffle join

[@sachinchandanshiv7578]

Hi Sir,
Stuck in one scenario. Please assist
input datafram is as below
id description
1 "abcd
pqe
rrr
qqq"
2 "ttt
ppp
ooo
www
iii"
3 "aaa
ppp
eee
zzz
rrrr"
4 "ssss
jjjj"
output dataframe should be as below last line of element
id description
1 qqq
2 iii
3 rrrr
4 jjjj

[@rajasdataengineering7585]

Hi Sachin, split function can be used for this requirement

[@sachinchandanshiv7578]

@@rajasdataengineering7585 Yes Sir. By using split I am able to seperate line but I am unable to extract last element from dataframe cell. bcuz last line can be at any position. last line may be at 2nd, 3rd, 5th or any position.

[@rajasdataengineering7585]

Try to use index[-1].
If not, we can calculate the number of elements and find the max of it using UDF

[@sachinchandanshiv7578]

@@rajasdataengineering7585
Done ! Thanks 🙂
def get_last_ele(str1):
str1=str1.split("
")
f1=[]
for i in str1:
f1.append(i)
return f1[-1]
stringUDF = udf(lambda m: get_last_ele(m))
df1.withColumn("last element", stringUDF("name")).show()

[@rajasdataengineering7585]

Welcome 🙂
I will try to post optimized solution if any when I get sometime.
Happy to know that you solved this requirement in quick time!

[@manikanta-zq1yg]

Thank you

[@rajasdataengineering7585]

You're welcome