How to Run a Spark Cluster with Multiple Workers Locally Using Docker

In this video, I'll show you how you can use Docker Desktop to run a Spark cluster that mimics a production environment on your local machine!
/ setting-up-a-spark-sta...

[@rasmusandreasson1548]

The king! Thank u for good content!

[@thedataguygeorge]

Thanks so much Rasmus!

[@not_saboor]

Thanks for this !

[@thedataguygeorge]

No problem Saboor!

[@Levy957]

i love your videos

[@thedataguygeorge]

Thanks Levy! Love your support!

[@ccc_ccc789]

Thanks

[@thedataguygeorge]

No problem!

[@early-riser18]

Thank you for the rundown - very helpful. Could you add a link to the code written please? Some code in the Dockerfile is hidden by the right-side screen fold and has to be guessed. Thanks :)

[@mayowaoludoyi5425]

Thank you for this walkthrough video. How can I establish connection to a relational database like Oracle from "dockerised" spark like this. I understand there is a different set up that requires JDBC. Where does it fits in this your setup?

[@thedataguygeorge]

Hey, you would add it similarly to how I connect to snowflake in other scripts, where you use the python ODBC drivers to establish connections to relational db's like Oracle

[@not_saboor]

Can you explain the part on Jinja Templating you mentioned in 3:40

[@thedataguygeorge]

Sure! What specifically about it are you interested in learning more about?

[@imanitrecruiterineurope4142]

Hi!
It seems that the applications aren't taking any resources and are stuck in a loop on my end. What could be the cause?

[@Jalabulajunx]

you don't have a directory like requirement, how will req/req.txt work?

[@thedataguygeorge]

With Spark, you'll typically initiate a spark session and provide a list of requirements you need for that particular session

[@csmithDevCove]

What about connecting sparp-nlp to this

[@thedataguygeorge]

You would just want to add it to be installed within the docker image!

[@rafaellourenco4599]

Bro, you skiped all the bugs stuff

[@thedataguygeorge]

Sorry was solving them off camera but will make sure to show more of the troubleshooting process next time!

[@rafaellourenco4599]

@@thedataguygeorge can you share a repo with this project?

[@Jalabulajunx]

I am always getting entry point.sh not found, has anyone figured it out?