Local Install Spark, Python and Pyspark

How to install spark, python and pyspark locally.
blog.hungovercoders.com/datag...
Below are the links, code and paths referenced throughout.
Python
www.python.org/downloads/
Remember to tick set PATH on install.
Java
java.com/en/download/help/win...
Set JAVA_HOME system environment variable to be C:\Program Files\Java\{jre version}
Spark
spark.apache.org/downloads.html
Unzip the tar file twice and place in C:\Spark
In environment system variables set a SPARK_HOME environment variable to be C:\Spark.
In environment system variables add a new Path to be %SPARK_HOME%\bin.
Hadoop
github.com/cdarlint/winutils
Download the winutils.exe.
Add a C:\Hadoop\bin folder.
Add winiutils to this folder.
In environment system variables set a HADOOP_HOME environment variable to be C:\Hadoop.
In environment system variables add a new Path to be %HADOOP_HOME%\bin.
Confirm Spark
Open command prompt with admin privileges
You should just be able to run "spark-shell" from anywhere as you have done environment variables above and it will just work.
Local Spark UI
localhost:4040/
Pyspark
code.visualstudio.com/
py -3.9 -m venv .test_env
.test_env\scripts\activate
pip install pyspark
pyspark
.test_env\scripts\deactivate

[@jeevitrasree]

Very much useful. Thank you very much for this knowledge sharing!

[@user-gn1pt8jp6r]

Short and best video helping debug Spark installation and re-installing from scratch

[@hczr]

Half a day of trying with several videos and articles, only yours worked for me. Thank you so much!

[@jphumani]

Thanks! Agree with everyone else. ONLY ONE to tell you exactly how to do it in VScode. THANKS a lot

[@TopperDillon]

I fully agree with @Cesar Vanegas Castro. This is the only video that shows how to integrate VS Code PySpark into a local Spark installation. Thanks a lot for sharing mate!

[@manofqwerty]

This has nothing to do with VSCode really, the setup here is editor agnostic. The exact same workflow works with PyCharm and will work for any other editor.

[@cesarguitars]

This is the only video that helped me to run properly through Vscode pyspark integrated into an environment, thanks

[@hassanessam375]

thanks a lot it was very smooth and easy for me unlike what usually happens when installing pyspark

[@Tech_world-bq3mw]

Excellent video, it worked for me.

[@MrSchab]

very useful, concise and to the point. thanks a lot!

[@senthilkumarpalanisamy365]

This is helpful, thank you

[@tawfiqmoradhun8007]

very good video ! thanks

[@babatundeomotayo8460]

This was really helpful. Thanks man🙏🏽

[@juanrandsonian2949]

Great video! Thanks a lot

[@biloloonguesamuel1940]

great!!!

[@bdev1444]

For all who have the "The term 'pyspark' is not recognized as the name of a cmdlet, function, script file, or operable program" error in the the visual studio editor: Restarting visual studio might help as that updates the environment variables. Also I restarted as administrator. I don't know which one did the trick, but it is working now

[@Laupfa36]

If you've got a problem at the last part, where DataGriff created the environment in the VSCode Terminal, make sure to check that your terminal is using command prompt (cmd) instead of powershell. That did the trick for me!

[@jasminechen6671]

Nice video! However, for those who try to implement Spark recently, please lower to Spark 3.1.2 version because it didn't work firstly when I installed Spark 3.2.1

[@juanrandsonian2949]

3.3.0 is working for me (Windows 11 + python 3.10.6)

[@manofqwerty]

I have had success with PySpark 3.4 (Windows 10 + Python 3.11.3)

[@user-eg1ss7im6q]

is it possible to run a pyspark scripts with a jupeter notebook? it helps to create a follow up clip for how to run a script with it.

[@tetricko]

when i use vscode terminal i try pyspark it give me this error:
pyspark : The term 'pyspark' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ pyspark
+ ~~~~~~~
+ CategoryInfo : ObjectNotFound: (pyspark:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
but if i use it in cmd it works why is this happening?

[@juneaoliveira6623]

good evening!
in the terminal visual studio code is not very readable can you please detail. the time is 09:51
Thank you!

[@urnal2432]

Run "spark shell" in cmd ? I run but dont recognizing like a cmdlet...

[@01Didisek]

Someone can have some struggle with the script (PowerShell says "execution of scripts is disabled on this system") - as I did :] .
In that case try this:
In cmd as an Administrator run the command 'powershell Set-ExecutionPolicy RemoteSigned'.
After you are done, run 'powershell Set-ExecutionPolicy Restricted'.

[@louatisofiene9114]

The system cannot find the path specified :(

[@shaneelsharma]

Same here, when I run the python file with session builder and everything... it runs fine in standalone shell but no from VS Code Debug or Run Script configs... Any idea ?

[@shaneelsharma]

I think i found the solution ... while I wrote the .py file in vs code i went ahead and used spark submit command as C:\Spark\spark-2.4.5-bin-hadoop2.7\bin\spark-submit .\yourPyFile.py
I think we can create add configurations for such things in vs code itself by tweaaking run & debug configurations