Pyspark Tutorials 3 | pandas vs pyspark || what is rdd in spark || Features of RDD
HTML-код
- Опубликовано: 25 июл 2020
- #RanjanSharma
This is third Video with a difference between Pandas vs PySpark and Complete understanding of RDD.
Covering below Topics:
What is PySpark ?
Why Pyspark when We have Pandas a PowerFul API and difference between them
What is RDD how it processes Data ?
Important Features of RDD
Stay tuned for Part 4 Video of Installation of Apache Spark and Pyspark in local Environment.
BIG DATA IS PROBLEM and HADOOP IS A SOLUTION
Hit the Like button if you really liked the video.
PPT is uploaded in to the Google Drive Link and Github link
Python Playlist: ruclips.net/user/playlist?list...
AI PlayList: ruclips.net/user/playlist?list...
Join Whatsapp Group for AI : chat.whatsapp.com/IB6fQBEcZAd...
Telegram Group : www.t.me/@MachineLearningIndia
Subscribe my Channel / ranjansharma
Google Drive: drive.google.com/drive/u/1/fo...
Github : github.com/iamranjan/youtube-...
*** Connect with me on below Channels ***
LinkedIn: / iamranjan
Medium : / iamranjansharma
Instagram : / iamranjan.sharma
Email : iamranjan.sh@gmail.com
Keep Practicing :-)
Happy Learning !!
#MachineLearning #Python #artificialIntelligence #dataScientist #DeepLearning #intelligence #BuisnessIntelligence #Ranjan #RanjanSharma
#Pyspark #SPark #ApachePyspark #apacheSpark #hadoop #bigData #MAPREDUCE #PysparkMachineLearning
that's what I call quality content. Very logically presented and instructed.
Very neat and clear explanation. Thank you so much.!! .**SUBSCRIBED**
one more thing ..how can someone dislike anyone's efforts to produce such helpful content. please respect the hard work.
thanks So nice of you :) . Keep sharing and Exploring bro :)
Great sir ...happy for clearing the concepts
Keep watching..thanks bro . Keep sharing and Exploring bro :)
Excellent Content, Thank you Ranjan.. Subscribed :D
excellent explanation,, strong hold on concepts,,
Glad you liked it! thank you :)
Nice explanation
great explanation
Glad you think so! Buddy keep exploring and sharing with your friends :)
Great thank you for this explanation
Thanks :) Keep Exploring :)
@@RanjanSharma I just got a job offer for a data engineer working with databricks spark. Your video definitely helped me in the interview. Thank you again.
@@JeFFiNat0R Glad i could help you 😊
Content wise great videos.. way of explaining can be improved.
Glad you think so!Thanks :) Keep Exploring :)
Hi Ranjan, thank you for the great series and excellent explanations. I have two questions:
1) In the video at 5:05, you mention that PySpark requires a cluster to be created. However, we can create Spark Sessions locally as well if I am not mistaken. When we run spark locally, could you please explain how PySpark would outperform pandas? I am confused about this concept. You can process data using various cores locally, but your ram size will not change right?
2) In the previous video you mentioned that Apache Spark computing engine is much faster than Hadoop Map Reduce because Hadoop Map Reduce reads data from the hard disk memory during data processing steps, whereas Apache Spark loads the data on the node's RAM. Would there be a situation where this can be a problem? For example, if our dataset is 4TB and we have 4 nodes in our cluster and we assign 1TB to each node. How will an individual node load 1TB data into RAM? Would we have to create more nested clusters in this case?
I've same doubt. How spark would store TB's of data in ram
you are too fast in explaining things. Could you please slow down a bit ? btw, good work
Thanks for your visit .. Keep Exploring :)
in my further videos , i have decreased the pace.
@9:19 min RDD in memory computation? Panda does in memory isn't it? do RDD also do in-memory.. may be i lost somewhere with point can you explain this minute difference please?
Audio is low compared previous 2 videos.
Why are you still using RDDs and not the Spark SQL Dataframe API?
This video was just for explanation of RDD. In next video, I will be explaining SQL DataFrame.
**gj**