Видео

Hello Rakesh, Please dont rush to complete the video, the importance is to learn while you watch it. I would recommend to watch 1 hour daily for next 6 days and practice along with it. And its generic PySpark, mllib is not covered it in. But once you understand this learning mllib will not take time. Don't forget to share this with your network ♻️

@easewithdata thank you sir u know around 2.5 hrs in 0.75x I just completed .30 min of video bcz I want to gain knowledge rather than completing video am I preparing for ds is it enough to skill ? And sir if u have time to create tutorial on AWS or azure cloud services keeping in context of data science and analytics ?

Thank you so much for your kind words. I know my reach and subscription rate is not good, I want people to learn all basics and understand the concepts. Please make sure you share this will all your friends and share this with your network over LinkedIn. Your help means a lot 🥰

@@easewithdata sure sir am great full and told my friends family to subscribe please upload more videos

Details already mentioned in the comments of the video.

Kafka allows you to work over real time, you can use api, python code, java sdk etc to publish data over Kafka

Declarative framework like DLT allows you to create table without schema and manage it automatically as per your data.

SCD is a type of dimension to maintain data in DWH. COPY INTO is to bring data into Data Lake from utilizing file sources.

Thank you, please make sure to share with your network 😃

Kudos 👏 I hope you loved your journey of knowledge 😊 Dont forget to share this with your friends over LinkedIn and tag us ♻️

@@easewithdata yes sure

For Associate Exam preparation - Buy udemy exam preparation by Derar alhussein. This should be enough to clear association certification. Dont forget or recommend the content to your network over LinkedIn ♻️

@ thank you for the sharing the information.🙏 Sure will recommend and share

Thank you 👍 Please make sure to share with your network over LinkedIn ♻️

Thank you 👍 Don't forget to share this with your network over LinkedIn ♻️

Databricks is built on spark and basic fundamental is to have multiple nodes(machine) to process data so we have to use cluster there's is no option and cluster are nothing but bunch of vms. In case if adf and synapse u are taking that are also internally invoked the vms to process our jobs it's just that it is hided from user and azure manage that and we just see it as runtime environment there

Everything required compute/process/cpu in the background for execution, sometimes this is completely abstracted from the end users. Even ADF and Synpase has that in background. So does all RDMS systems, this is why you install them on machines. ADB allows you to configure it as per your requirement, thus you see VM's being spinned up and used.

Yes, enough to get you started and make you comfortable with Spark.

I was wondering if you could make a video on how to perform upserts in PySpark, especially when working with subqueries in MERGE or UPDATE statements. Handling correlated subqueries and updates with complex joins is often tricky. Would love to see your insights on optimizing such operations!

Thank you 👍 You can checkout the Databricks Zero to Hero series to learn more about Merge in Delta tables. I will try to cover optimizations in future videos. ruclips.net/p/PL2IsFZBGM_IGiAvVZWAEKX8gg1ItnxEEb Don't forget to share with your network over LinkedIn ♻️

@@easewithdata can I practise these databricks playlist in Fabric notebooks?, please let me know

@@Shreekanthsharma-t6x Yes, if it allows you to run PySpark code.

@@easewithdata thanks a lot

Thank you 👍 Don't forget to share with your network over LinkedIn ♻️

Thank you ❤️ Dont forget to share this with your network over LinkedIn ♻️

Thanks subham. I did not see this update. I updated the dockerfile using ChatGpt. and it worked. # Base Python 3.10 image FROM python:3.10-bullseye # Expose ports EXPOSE 8888 4040 # Change shell to /bin/bash SHELL ["/bin/bash", "-c"] # Upgrade pip RUN pip install --upgrade pip # Install OpenJDK RUN apt-get update && \ apt-get install -y --no-install-recommends openjdk-11-jdk && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # Fix certificate issues RUN apt-get update && \ apt-get install -y --no-install-recommends ca-certificates-java && \ apt-get clean && \ update-ca-certificates -f && \ rm -rf /var/lib/apt/lists/* # Install nano and vim RUN apt-get update && \ apt-get install -y --no-install-recommends nano vim && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # Setup JAVA_HOME -- useful for Docker commandline ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/ ENV PATH $JAVA_HOME/bin:$PATH # Download and Setup Spark binaries WORKDIR /tmp RUN wget archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz && \ tar -xvf spark-3.3.0-bin-hadoop3.tgz && \ mv spark-3.3.0-bin-hadoop3 /spark && \ rm spark-3.3.0-bin-hadoop3.tgz # Set up environment variables ENV SPARK_HOME /spark ENV PYSPARK_PYTHON /usr/local/bin/python ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.5-src.zip ENV PATH $PATH:$SPARK_HOME/bin # Fix configuration files RUN mv $SPARK_HOME/conf/log4j2.properties.template $SPARK_HOME/conf/log4j2.properties && \ mv $SPARK_HOME/conf/spark-defaults.conf.template $SPARK_HOME/conf/spark-defaults.conf && \ mv $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh # Install Jupyter Lab, PySpark, Kafka, boto & Delta Lake RUN pip install jupyterlab==3.6.1 pyspark==3.3.0 kafka-python==2.0.2 delta-spark==2.2.0 boto3 # Change to working directory and clone git repo WORKDIR /home/jupyter RUN git clone github.com/subhamkharwal/ease-with-apache-spark.git # Fix Jupyter logging issue RUN ipython profile create && \ echo "c.IPKernelApp.capture_fd_output = False" >> "/root/.ipython/profile_default/ipython_kernel_config.py" # Start the container with root privileges CMD ["jupyter", "lab", "--ip=0.0.0.0", "--allow-root"]

Thank you 💓 Don't forget to share with your Network on LinkedIn ♻️

Thank you for letting me know, I fixed it. Please download the latest Dockerfile from github.com/subhamkharwal/docker-images/tree/master/pyspark-jupyter-lab and try again. Please let me know if that works

Please download the latest Dockerfile from github.com/subhamkharwal/docker-images/tree/master/pyspark-jupyter-lab and try again. Please let me know if that works

Thank you so much 💓 Don't forget to share this with your network over LinkedIn ♻️

Thanks 💓 Don't forget to share with your Network on LinkedIn ♻️

Thank you 👍 Don't forget to share with your network over LinkedIn ♻️

You can use PySpark notebook by running this command in docker docker pull jupyter/pyspark-notebook or use Databricks community edition

Absolutely, this is how spark works. But it you want to explain the complete process this is the video - ruclips.net/video/CYyUuInwgtA/видео.htmlsi=eRNtot_osWZ-DvlY

OK 🙋

I will cover this later in this series.

This series currently on hold.

Absolutely join works with Hive. If you select more than one column then the hashing will happen with combination of both columns.

Glad you found it helpful 😊 Please make sure to share with your network over LinkedIn

Glad you enjoyed the video. Make sure to share it with your network over LinkedIn! ♻️

Here you can find resources on DLT META - databrickslabs.github.io/dlt-meta/

Thanks bhai ❤

Glad you're finding the playlist helpful! 🥰 I already put your comment on my LinkedIn, please make sure to share with your network ♻️ www.linkedin.com/posts/subhamkharwal_spark-pyspark-dataengineering-activity-7276494540339888128-0xo9?

If you want to learn PySpark, just follow along the video 👉

Shubham - me too. Facing issue - unable to locate package openjdk--11jdk

I am not able to recreate the issue. Can u please help me with steps. I will try to recreate the new image.

Till then please use PySpark Jupyter notebooka

Please download the latest Dockerfile from github.com/subhamkharwal/docker-images/tree/master/pyspark-jupyter-lab and try again. Please let me know if that works.

Please make sure to share with your network over LinkedIn ❤️

Thank you, please make sure to share this with your network ♻️

Already available in Video description

@@easewithdata Thanks

Thank you 😊 Please make sure to share with your network ♻️

In this example we didn't enable any deletion and truncate on SCD2 table. It was only enabled for SCD1 with property

@easewithdata That makes perfect sense, Shukriya 🙏