01. Pyspark Setup With Anaconda Python | DataBricks like environment on your local machine | PySpark
HTML-код
- Опубликовано: 19 авг 2022
- #spark
#pysparktutorial
#pyspark
#talentorigin
In this video lecture we will learn how to setup PySpark with python and setup Jupyter Notebook on your local machine.
Spark Tutorial, Pyspark Tutorial, Python Tutorial
Books I Follow:
Apache Spark Books:
Learning Spark: amzn.to/2pCcn8W
High Performance Spark: amzn.to/2Goy9ac
Advanced Analytics with Spark: amzn.to/2pD57Ke
Apache Spark 2.0 Cookbook: amzn.to/2pEbAUp
Mastering Apache Spark 2.0: amzn.to/2udDEUg
Scala Programming:
Programming in Scala: amzn.to/2uiTGfl
Hadoop Books:
Hadoop: The Definitive Guide: amzn.to/2pDheH4
Hive:
Programming Hive: amzn.to/2Gqwz7o
HBase:
HBase The Definitive Guide: amzn.to/2Gj9rI2
Python Books:
Learning Python: amzn.to/2pDqo6m 
Наука
The only video that gave the instructions clearly and that worked. Thank you so much, you are the best
Great job with the video. Thank you.
Thanks for the video !
Thank you for this. I appreciate it
love from Nigeria
U did great job. Thanks for sharing video. Waiting for more stuff on python sprak or databricks
You sure can expect tutorials on pyspark, python and data analytics in coming months. Thanks for showing interest 👍🏻
Good video. First half a little slow for those of us that know how to create a conda environment but good after that.
wherer are the conf files located when installing with conda ? there are not in /opt/spark/conf obviously, but where then ? Thanks
Thanks Babai
Unable to found kernel when executing jupyter kernelspec list command
i am getting this error can you tell me how to fix it
pyspark-env\python.exe: No module named ipykernal
Hi sir ples help me
Everything went fine but after running sparksession i am getting errors
Like:
Py4JJavaError
Can you help me to fix it? [List Kernel Specs] WARNING | Native kernel (python3) is not available No kernels available
Thanks for the video! Why is a new environment needed for pyspark - couldn't it be installed into the base environment?
You can install in your base environment too. But while using python there might be dependency issues with other projects you might be working on.
For example if you are working on Artificial Intelligence models which might require a package of version x, but your pyspark might require same package but of version y. In this case if you have a single environment for all your projects soon maintenance would become challenge.
Hence it's always a best practice to create a new environment for each project you work.
Only collection() is working fine for me, Other function like take() throwing an error - Could not serialize object: IndexError : Tuple index out of range. Please help me
Did you get solution for it?
Please help if you have got any solution
Can we connect to a remote hadoop cluster and run pyspark program with yarn from this
Yes you can.
I followed the video and got the error when import pyspark ....error is pyspark module not exists...can anybody help me how to fix the pyspark
But this much information won't help to install pyspark in anaconda. We need to install correct version of jdk and python and Spark and setting env variable path etc. After that if we are lucky it will work otherwise watching tons to videos to finding solution of errors
thank ,an
this is the feedback i get. what do i do? Collecting package metadata (current_repodata.json): failed
CondaSSLError: OpenSSL appears to be unavailable on this machine. OpenSSL is required to
download and install packages.
Even I got same error,, is this resolved
For me it is showing only python 3 not pyspark-env please help
Excellent video. I followed all the steps what you have done in this video. But for below command
(pyspark-env) C:\Users\sri>jupyter kernelspec list
Getting below error
'jupyter' is not recognized as an internal or external command,
operable program or batch file.
Please let me know how can we solve this issue.
did u resolved the issue
just use - conda install ipykernel : fallow next from the video
@@dhanureddy thanks bro!!
(pyspark-env) C:\Users\sri>jupyter kernelspec list
Getting below error
'jupyter' is not recognized as an internal or external command,
operable program or batch file.
For above error, you need to run this command
>conda install -c anaconda ipykernel
then ipykerne packages will be installed on "pyspark-env" environment.
Hi santosh, i have faced same error as you specified and foolwed the command you have provided, it then had some installation and downloads done
but when i am importing pyspark in notebook its showing module not found can you please help me?
thank you so mucch mate much love
Thanks for the wonderful video! I am getting below error while creating spark session like you showed here.
RuntimeError: Java gateway process exited before sending its port number
I am also getting the same error
@@mrrobot111 Did you figure out what to do??
@@Antoniolavoisier1 same error. any solution?
Could anyone figure this out? Do we need to install Java?
I have a problème with Java i Think.
your 3th line is note OK for me.
"My error is : "Java Gateway process exited before sending its port Number"
My java version is "19.0.2 2023" 2023-01-17
I can't solve this problem.
How could I do?
The rest of your tutorial is great
The notebook hangs when I run the cell to create sparksession..Help pls
There are multiple reasons for this.
Before you run any cell confirm that Kernel is in ready state, if the hardware is decent this could be a common issue.
Getting below error:
Anaconda3\envs\pyspark-env\python.exe: No module named ipykernel
Please help
Please execute the below command before running the step where you are getting this error
pip install ipykernel --user
Getting this error after creating SparkSession : Py4JError: org.apache.spark.api.python.PythonUtils.isEncryptionEnabled does not exist in the JVM
sir i followed correctly but spark createing is not coming..............file is not found error comeing
I can help if you can provide me exact error or screenshot for your error.
'Builder' object has no attribute 'getorCreate'
@TalentOrigin I am getting below error
RuntimeError: Java gateway process exited before sending its port number
When i am running this command
spark = SparkSession.builder.appName("Practise").getOrCreate()
can you help me with this?
Did you figure this problem out please?
thank ,an