❌ Apache Spark & Jupyter on Google Cloud Dataproc Cluster ❌ Spark + Jupyter + Dataproc

[@DecisionForest]

Hi there! If you want to stay up to date with the latest machine learning and deep learning tutorials subscribe here:
ruclips.net/user/decisionforest

[@shaz-z506]

Good tutorial, could you please create a video on how to use spark-submit to execute jobs on all available cluster in a big data scenario on GCP.

[@fernandes7949]

first congratulations on sharing knowledge.
A question in the case I already have a hadoop cluster created and would like an IDE to facilitate the creation of script in pyspark and even scala.
Wouldn't google have anything already installed free? What do you recommend me. Thank you Fernandes Brasil

[@DecisionForest]

Thank you Fernandes! Well you can use Jupyter Notebooks on Dataproc for exactly this purpose. You can also integrate with VSCode: cloud.google.com/code/docs/vscode.

[@patricciaavila]

How do connect the PySpark to BigQuery?

[@manojselvakumar4262]

How to access the Spark UI from Dataproc?

[@calendr13]

I have windows, how can I do for the scripts?

[@imayzoo]

Thanks for the video, very easy

[@DecisionForest]

You're welcome! Happy it helped.

[@miladto]

I could not find the bash script to create cluster on Dataproc

[@DecisionForest]

Hi Milad, the link to the download is in the description, it contains all the scripts.

[@puggyk4220]

I tried to use edf format file