6. How to Write Dataframe as single file with specific name in PySpark |

In this video, I discussed about writing dataframe as single file with specific name in pyspark.
Link for Azure Synapse Analytics Playlist:
• 1. Introduction to Azu...
Link to Azure Synapse Real Time scenarios Playlist:
• Azure Synapse Analytic...
Link for Azure Data bricks Play list:
• 1. Introduction to Az...
Link for Azure Functions Play list:
• 1. Introduction to Azu...
Link for Azure Basics Play list:
• 1. What is Azure and C...
Link for Azure Data factory Play list:
• 1. Introduction to Azu...
Link for Azure Data Factory Real time Scenarios
• 1. Handle Error Rows i...
Link for Azure Logic Apps playlist
• 1. Introduction to Azu...
#PySpark #Spark #Databricks #PySparkLogic #WafaStudies #maheer #azure #AzureSynpase #AzureDatabricks #azure

[@reniguha1]

I was so frustrated and was not able to find the solution until i watched this video, you are my Guru now on 👏

[@Sinfully__beautiful]

Thank you for this information! This is a very common scenario and I’ve been looking for an answer for a long time.

[@WafaStudies]

Thank you 😁

[@anupgupta5781]

this is really important video, I was trying to find a workaround because we dn't have direct method available in pyspark for achieving this, Thanks bro

[@WafaStudies]

Welcome 😊

[@VinayKumar-st9iq]

Completed Playlist. Hope you will add more scnario based questions like this.

[@sabarishjothi9557]

Thanks a lot!! Key command which was very difficult to find in google except for your video!!

[@angelacalzadobeltran5689]

Works!! And was exactly what I needed and couldn't find anywhere. Thank you so much! 👏

[@tatha143]

hi...watched your adf ,adb playlist......thanks for all the work .. your videos helped me to crack azure interview...

[@shubhamunhale5762]

Thanks for the information brother.
Really helpful for me.

[@mnaveenvamshi3651]

Awesome tutorial bro, you are a great teacher, a big like and new subscriber.

[@sonamkori8169]

Thanks Maheer, plz add more scenario based questions

[@WafaStudies]

Sure 😊

[@narendrakishore8526]

Very useful information 👍

[@HanumanSagar]

Hi Bro..Can we do copy directly like dbutils.fs.cp( source path to dest path)?
Instead of using for loop and if condition?

[@WafaStudies]

Yes u can. But part file name we should get it first right? So we used for loop and if condition to get filename

[@pinur_paglami]

bro, i need to write a dataframe to a csv file on a network fileshare. how to dp that? please help

[@UPavan07]

what will be the case if we store two files(.csv) in same location then the if condition gives us two different names.

[@starmscloud]

Good One Maheer !

[@WafaStudies]

Thank you ☺️

[@emach4392]

a very good video but dbutils does not seem to work on notebook in lakehouse fabric

[@stevedz5591]

Hi Maheer can we have one vedio on ADF pipeline Orchestration

[@muvvalabhaskar3948]

Hi if I am doing same with parquet instead of CSV I am getting error like py4j security error any idea how to workaround this one

[@baranidharanselvaraj9381]

Superb bro thanks a lot 🎉

[@RR.G]

I see this code is copying each csv file to a different name, but not creating a single csv file for all the data

[@nagarjunak1296]

Hi bro, How long it takes to cover all Azure Synapse analytics course ?

[@kundankumar5395]

Will it not degrade the performance while writing the dataframe?

[@Vikasptl07]

Yes for large file it may cause driver failure, Coalesce will do shuffle and move it to one partition and then save it.. pandas will collect it to driver node and save it. If df size is large then it is not advisable to to save it under one file but if needed then yes for small files we can use.

[@huzischannel]

This doesnt work for me in Synapse notebook. Getting below error. I am not sure we need to define any library for this.
NameError: name 'dbutils' is not defined

[@nuthalapativenkatanaveenku4795]

It would help if you used mssparkutils instead of dbutils in a Synapse notebook. To do this, you can import mssparkutils with:
from notebookutilities import mssparkutils
Alternatively, you can directly use mssparkutils without importing it, and it will work fine.

[@mhaya1]

Bro, can you share sample datasets.

[@muvvalabhaskar3948]

Can we do same for parquet file

[@sahityamamillapalli6735]

Yes we can do, I have tried its working

[@singhanuj2803]

Brilliant

[@Vikasptl07]

Convert df to pandas df and save it as one file

[@WafaStudies]

Yes we can that too. I will cover this in next video 🙂

[@Vikasptl07]

Great work you are doing on RUclips 🙏

[@starmscloud]

Pandas DF will be a Problem when the file size is huge as it always runs on a Single Node cluster .

[@Vikasptl07]

@@starmscloud Yes for large file it may cause driver failure, Coalesce will do shuffle and move it to one partition and then save it.. pandas will collect it to driver node and save it. If df size is large then it is not advisable to to save it under one file but if needed then yes for small files we can use.

[@starmscloud]

@@Vikasptl07 That's Correct

[@Basket-hb5jc]

Isnt this very inefficient