33. fill() & fillna() functions in PySpark | Azure Databricks

[@starmscloud]

Good One Maheer !
May be you can replace different null values different columns using key-value pair . something like this .
inputData = [(1,'Alex',5000,None),(2,'Liam',None,'uk'),(None,'Duckett',7000,'Eng'),(4,'Hope',7800,None)]
inputSchema=["Id","Name","Sal","Loc"]
inputDF = spark.createDataFrame(inputData,inputSchema)
inputDF.na.fill({'Id':'-1000' , 'Loc':'unknownLoc'}).show()

[@manu77564]

Thanks bhaiii. Would you please explain date functions in azure.

[@polakigowtam183]

Good Vedio Maheer

[@vineethsetty3222]

Hello maheer can we handle nulls in this way .withcolumn("gender",expr("coalesce (gender,-1)")) here if gender is there value remains same if there us null then will be replaced with -1

[@starmscloud]

Yes.. You can do that . In fact , you can use it with selectExpr as well where you don't need to import expr function

[@starmscloud]

inputData = [(1,'Alex',5000,None),(2,'Liam',None,'uk'),(None,'Duckett',7000,'Eng'),(4,'Hope',7800,None)]
inputSchema=["Id","Name","Sal","Loc"]
inputDF = spark.createDataFrame(inputData,inputSchema)
inputDF.selectExpr( "*","coalesce(id,-1) as newID","coalesce(name,'unknown') as newName","coalesce(Sal,-1) as newSal","coalesce(Loc,'unknownLoc') as newLoc").show()

[@vineethsetty3222]

@@starmscloud thanks Manoj

[@shuaibsaqib5085]

@@starmscloud Nice..

[@sravanthibijivemula9798]

Could you please make more videos on azure databricks