Well, I had been searching for a course where we can do the spark development using an IDE and run complete end to end testing and deploy to Azure Databricks. For past 3 years I didn't find anything but today I watched this video and I can say that atleast a beginning point. In case you can create more videos on Azure Databricks development locally and deployment. I assure you you will be the king
Thank you for this walkthrough video. How can I establish connection to a relational database like Oracle from "dockerised" spark like this. I understand there is a different set up that requires JDBC. Where does it fits in this your setup?
Hey, you would add it similarly to how I connect to snowflake in other scripts, where you use the python ODBC drivers to establish connections to relational db's like Oracle
Thank you for the rundown - very helpful. Could you add a link to the code written please? Some code in the Dockerfile is hidden by the right-side screen fold and has to be guessed. Thanks :)
Why all images names are 'da-spark-image' ? I get a pull access denied. This toturial is amazing and exactly what I was looking for, but can't make it happen
So I got it working finally, even while getting the the pull access denied prompt. My issue ended up being the line "RUN curl downloads.apache.... " in the Dockerfile had an error I needed to fix for it to work.
Thank you so much for this video! Very helpful!
No problem, hope you got it running!
I haven't tried this yet, but if it works, you are a prince amongst men.
Hahahaha let me know how it goes my man!
@@thedataguygeorge I just tried it and it worked for me, great job.
This also seems like a good introduction to Dockers. I am definitely getting a feel for the advantages of the tool
Yes great way to get started!
Well, I had been searching for a course where we can do the spark development using an IDE and run complete end to end testing and deploy to Azure Databricks. For past 3 years I didn't find anything but today I watched this video and I can say that atleast a beginning point. In case you can create more videos on Azure Databricks development locally and deployment. I assure you you will be the king
The king! Thank u for good content!
Thanks so much Rasmus!
Thanks for this !
No problem Saboor!
i love your videos
Thanks Levy! Love your support!
Can i submit multiple custom jobs to the cluster at the same time?
Yes!
Thank you for this walkthrough video. How can I establish connection to a relational database like Oracle from "dockerised" spark like this. I understand there is a different set up that requires JDBC. Where does it fits in this your setup?
Hey, you would add it similarly to how I connect to snowflake in other scripts, where you use the python ODBC drivers to establish connections to relational db's like Oracle
do you have this on a repo so that we can take a look at the whole thing?
Also wondering how you addressed the JAVE_HOME path setup. My error message is that it's not set.
Thank you for the rundown - very helpful. Could you add a link to the code written please? Some code in the Dockerfile is hidden by the right-side screen fold and has to be guessed. Thanks :)
Can you explain the part on Jinja Templating you mentioned in 3:40
Sure! What specifically about it are you interested in learning more about?
So this tutorial would also help me set this up to where I'm running computations on a server?
Definitely!
Hi!
It seems that the applications aren't taking any resources and are stuck in a loop on my end. What could be the cause?
you don't have a directory like requirement, how will req/req.txt work?
With Spark, you'll typically initiate a spark session and provide a list of requirements you need for that particular session
So the spark workers could be more physical computers or multiple vm's?
Multiple vm's!
Missed the part where you need to add the spark conf file
Good Call out!
Why all images names are 'da-spark-image' ? I get a pull access denied. This toturial is amazing and exactly what I was looking for, but can't make it happen
I also have this issue
So I got it working finally, even while getting the the pull access denied prompt. My issue ended up being the line "RUN curl downloads.apache.... " in the Dockerfile had an error I needed to fix for it to work.
I am facing the same issue... what changes are required to make it work? Please help.
@@Sudo801 Thank you for this comment!
I am not able to create .env.spark file in macbook m1, please sgare solution
Why aren't you able to create it?
so you came up with this all by yourself?
All by reading severally articles online lol
What about connecting sparp-nlp to this
You would just want to add it to be installed within the docker image!
Is it possible to build this without makefile? It is challenging to install it on windows...
WSL?
Thanks
No problem!
Please also explain daemons vs tasks
Will do in a future video!
Bro, you skiped all the bugs stuff
Sorry was solving them off camera but will make sure to show more of the troubleshooting process next time!
@@thedataguygeorge can you share a repo with this project?
I am always getting entry point.sh not found, has anyone figured it out?