Converting CSV to Nested JSON/ Dictionary format in PySpark | Realtime scenario

Hi Friends,
In this video, I have explained some sample python code to convert csv file and convert the records into JSON format.
github.com/sravanapisupati/Sa...
Code- github.com/sravanapisupati/Sa...
Please subscribe to my channel for more interesting learnings.

[@svsphaneendra]

Thank you so much for this video 🙏

[@sravankumar1767]

Nice explanation sravana

[@sravanalakshmipisupati6533]

Thank you Sravan, hope this is what you were looking for.

[@sravankumar1767]

@@sravanalakshmipisupati6533 can you please share Email. I will share my user story, actually I don't understand

[@sravanalakshmipisupati6533]

@@sravankumar1767 [e-mail]

[@soumyakantarath4551]

Thanks - if possible please upload the datasets you are using in the playlist - some are not there.

[@sravanalakshmipisupati6533]

Sure.

[@sravanalakshmipisupati6533]

Added files, please check.

[@soumyakantarath4551]

thank you.@@sravanalakshmipisupati6533

[@ardataengineering6521]

💣🙏👍

[@ambadasshinde6670]

such kinds of Q's will be asked in Interview ? because its purely based on usage of inbuilt functions provided by pyspark.
can you please let me know how to prepare for such kind of Q's for interview ?

[@sravanalakshmipisupati6533]

Its a real-time scenario. And, yes they might ask this in coding interviews as well. Good Hands-on is the only option for preparing for coding rounds.

[@andheprashanth4686]

How to use that myformat variable to save data to hive table in same format

[@sravanalakshmipisupati6533]

Hi Prashanth,
Please use the below -
Dataframe.saveAsTable("db_name.hive_table_name")
You can add partitionBy() also if you want.
Dataframe.write.partitionBy("col1").saveAsTable("db.table")

[@andheprashanth4686]

@@sravanalakshmipisupati6533 I need to use that myformat variable(which is list in form of json) to store data in hive using that same format.I hope u understood my question

[@sravanalakshmipisupati6533]

@@andheprashanth4686 create a dataframe out of myFormat json and then you can write it as hive table.
Dataframe= spark.CreareDataframe(myFormat)

[@rajdeepsinghborana2409]

Hello ma'am , how to read PDF files and cleaning data in spark - Scala

[@sravanalakshmipisupati6533]

Thank you for watching the video. I'll get back to you on this.

[@ayaanshdas1412]

Getting __init__ missing positional argument error while performing groups and aggregation on my df

[@sravanalakshmipisupati6533]

Please share the code which you are trying to execute.

[@ayaanshdas1412]

@@sravanalakshmipisupati6533 df=['key','id','name','email']
df1=df.groupby("key").agg(f.map_from_entries(f.collect_list(f.struct("id","name","email"))).alias("info"))
df printing showed good, but while grouping and aggregating am facing that error.

[@sravanalakshmipisupati6533]

@@ayaanshdas1412 Please create dataframe through SparkSession and try.