How to Submit a PySpark Script to a Spark Cluster Using Airflow!
HTML-код
- Опубликовано: 22 окт 2023
- Master the intricacies of deploying PySpark scripts on Spark clusters with our comprehensive guide, leveraging the power of Airflow. Delve into step-by-step procedures, best practices, and common pitfalls to ensure a smooth execution. Understand the synergy between PySpark's distributed computing capabilities and Airflow's robust workflow management. This tutorial is designed for both beginners and seasoned professionals looking to optimize their big data processing tasks. Subscribe for in-depth insights into Spark, Airflow, and the dynamic world of data engineering!
registry.astronomer.io/provid...
airflow.apache.org/docs/apach...
Thank you so much for the vid ! Both the format and the subject are awesome !
Thanks so much Vlad!
Love this new format
Thanks! Good to know!
Thank you very much for the useful video.
Thanks so much for the appreciation!
this worked well !
thank you so much 😃
Glad it helped!
Best Tutorial
Thanks Sandeep!
nice vid bro but if i got my airflow installed in docker how can i get the SparkSubmitOperator Package
Add it to your requirements file and restart your environment
Where are you running spark cluster is it running locally
Why did we use spark://master
in connection im confused how is docker container connecting to spark cluster running on your laptop ?
Yes i'm running Spark locally! That's why I'm able to connect to it that way!
@@thedataguygeorge How can i check port as you have used maybe it is different for me or not?
how am i supposed to send data directly into snowflakes? I mean, if i'm processing Petabytes how am i supposed to retrieve them in my local machine and then push them into snowflakes?
You could chunk it by ingesting smaller pieces of the larger data set, or spin up a remote spark cluster with a ton of compute!
Hi Data Guy, the video is so insightful. I am just struck at understanding the Host & Port details you provided while creating the connection in Airflow Webinterface. How to configure those while bringing up the airflow container to submit a pyspark script. Kindly help me with this one.
Thanks! What specifically are you having trouble with? Is the Host/Port combo for the Spark connection not working?
I followed you but have one error
JAVA_HOME is not set
So I have to install Java in my Container?
In your Spark cluster yes!
I am thinking of any consequences, if using python operator directly and call out a pyspark code within, and fill all the ins and outs within a given task. Do you think this works?
Tentatively yes, but what do you mean by fill all the ins and outs of a given task?
@@thedataguygeorge Thanks for the reply. I meant the input args and in/out data referenced by a Dag / connected tasks.
how can i connect to 2 remote servers at the same time when i have to run multiple jobs
Thank you.
plz show setup spark cluster
Would you like to OSS Spark locally or cloud based?
kubernetes based spark job submit is best for real production grade, very less videos on spark airflow kubernetes job submit in dynamic@@thedataguygeorge
@@thedataguygeorge locally
Can we connect databricks community edition to airflow ?
I believe so but not 100% sure since I know CE is pretty limited!
hi Can you add another episode to it describing how to submit pyspark on a cloud provider cluster like databricks AWS-EMR GCP-Dataproc etc,, because what i have seen is just like astronomer for airflow, people use clusters on cloud providers rather than installing/running spark on bare metal or VMs
Yeah forsure! I have an Azure Databricks instance available, would that work for ya?
wow man that astro dev init killed me
I am making the docker image myself every time lol
is there any video you have to help with the astro package and its tips and tricks?
Even better than a video, check out this guide to getting started with Astro!
@@thedataguygeorge Hi there! It seems the link for the guide wasn't attached to your comment for some reason... Can you please re-attach it? Thanks! 🙏
Where can I get code ??
Hadn't hosted it yet but will do!
Hi the data guy
make some knowledge transfer session on spark structured streaming with kafka integration pls.
Thank you 😅
Definitely, I'll put it in the schedule!
Thank you for the video, Kindly assist with this error: Cannot execute: spark-submit --master yarn --num-executors 1 --total-executor-cores 1 --executor-cores 1 --executor-memory 2g --driver-memory 2g --name arrow-spark --queue root.default sparksubmit_basic.py. Error code is: 1.; 96442
Could you please give me a little more context?
getting this error Error when trying to pre-import module 'airflow.providers.apache.spark.operators.spark_submit' found in test_dag.py: No module named 'airflow.providers.apache
Hey man, as a fellow data analyst I really enjoy your videos, but you gotta stop those premiers… it’s really annoying to see an interesting video in my subfeed and then everything it’s just a premier for like in 5 days… I mean of course you can use this feature once in a while but please not 5 times a week, it’s really frustrating.
Ok good to know, will stop! Sorry about that, I just did it so people could see the upcoming schedule but didn't realize it was so prominent in your feed!
@@thedataguygeorgeThanks man!
I totally agree!
Love the content and the 10min short and no-bs is just amazing. But the scheduled thing is very annoying
No problem, thanks for letting me know!
Hi how do we setup connection between two dockers so that airflow can trigger task in spark docker. Can you make video on it