In this video you referred to serializers such cPickle and Marshall (10:18), as well as RDD serializer (15:23). I was reading Spark's documentation and under the tuning guide it mentions only 2 types of serializers, Java serializer and Kyro serializer, so I am confused about what serializers are present in Spark?
1:47 so you mentioned that pyspark is based on the python programming language but it was built with the scala language? I am confused. Does it use 2 languages?
Hi. PySpark is based on the Python programming language, but it is built using Scala. PySpark uses two languages but in different ways: 1. Python for writing applications and interacting with the Spark framework. 2. Scala for implementing the components of PySpark.
Apache Spark (underlying technology) is made in Scala. PySpark has been released in order to support the collaboration of Apache Spark and Python, it actually is a Python API for Spark. You can think of it as a wrapper around the original library. TLDR: Pyspark is not (made) in Scala, it's just a Python API on top of original Spark library
![Seungmin "그렇게, 천천히, 우리(As we are)" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/kAzmhLHePqU/mqdefault.jpg)
In this video you referred to serializers such cPickle and Marshall (10:18), as well as RDD serializer (15:23). I was reading Spark's documentation and under the tuning guide it mentions only 2 types of serializers, Java serializer and Kyro serializer, so I am confused about what serializers are present in Spark?
Thanks brother❤
1:47 so you mentioned that pyspark is based on the python programming language but it was built with the scala language? I am confused. Does it use 2 languages?
Hi. PySpark is based on the Python programming language, but it is built using Scala. PySpark uses two languages but in different ways:
1. Python for writing applications and interacting with the Spark framework.
2. Scala for implementing the components of PySpark.
@@MindmajixThanks for the explanation. At 7:33, in the diagram is it supposed to be sparkcontent or sparkcontext?
Apache Spark (underlying technology) is made in Scala. PySpark has been released in order to support the collaboration of Apache Spark and Python, it actually is a Python API for Spark. You can think of it as a wrapper around the original library.
TLDR: Pyspark is not (made) in Scala, it's just a Python API on top of original Spark library
Very useful ❤❤❤❤
very helpful
thanks...
Thankyou very much 😊
Thompson Harbor