Видео

Hi Bro, I have a question, My cluster config is 65 nodes each with 16 cores and 128 GB Memory. I am reading a file of 7 days of total partitions of 4500 (some are unevenly partitions as well) of 500GB data, and join and filters happens and it becoms 1.3 TB and i have given shuffle partition of 10000 (each of size 128MB). So, if i give core - 5 and totalm executors - 195 based on the calculation you provided. How will it process the data, will it be able to process 500GB data and dumps 1.3 TB. If yes, how it will be done? Should i do coalesce(). Or how i can approach this?

Currently watching your playlist one by one. Great content. Very detailed explanation. In the first scenario you had 5 executors and with 4 cores each. If you have 1500 shuffle partition how they are going to be accommodated.

I have learned somewhere that the max partition size can only be 128mb in spark. Isnt that contradict to what you mentined when explaining about 300GB example? Just asking to Correct myself if wrong.

Hey @afaqueahamd7117 , The explanation is excellent I watched all of your videos and they way you explain things in detail actually makes to watch it again and again whenever I am attending an interview. Eagerly waiting for your next videos. Kudos🙌

you are a lifesaver bro i have learnt this concept so many time but i didn't get it very well but in your video i totally understand how to calculate the resource for spark job

Sir still waiting for a dedicated video for driver and executor omm in very detailed. Question on if the file is 100 gb, and can we sort it? If yes will there be data spill, basically a interview quesion for 8+ year exp

I love you bro for such crisp explanation, the way to experiment and teach helps a lot!

Nice explanation

Great Work Bro.

amazing job bro

Hey Afaque, awesome video as always! quick question. If we have the chance to increase the memory of the spark execution container, how can we decide between assigning that extra memory to the on heap memory or assigning it to the off heap memory if at the end the total available memory is always the sum of these two? I know you mentioned that off heap memory is not affected by the garbage collection process, but is also slower that on heap memory, so wouldn't it be better if we always assign all possible memory to the on heap memory right from the beginning instead of waiting for the off heap memory to come into play?

Awesome video as always. Would really appreciate more videos explaining how DAG's can be read

Industry level content.

Is this applicable to cloud platform?

I never see such a detailed explanation. Thank you very much @afaque Ahmad. Is there any way we can get the document.

Awesome video! I've been searching for something like this and all the other videos I found don't get to the point and neither explain things as good as you do. I'm definitely subscribing and sharing this with other DE's from my team, please keep posting content like this!

Very informative video afaque. Please create more videos.

What if the nodes are of unequal size?

Excellent.. Keep going..

Am.not able to see spill details in spark 3.5.2 UI?

Excellent Content..!! Loved It. Such detailed explanation on Salting Technique with Graphical Representation.

Hi Sir Please Make a Detailed course on apache spark which include every aspect of spark for Data Engineer role Please make sure there are a lot of begineer course here in market keep the course from intermediate level to advance level. Please tr to make video in Hindi it will be very helpful.

Thank you for this video !!

Sir please do a video on executor out of memory in spark and driver out of memory in spark

This is helpful, but I still have a few doubts. 1. If Broadcast join is immune to skewness, why there is Salting technique? 2. In the Broadcast join example, the customer dataset appeared to be outside of any executor. Where is it actually stored? How can we specify its storage location?

Want to understand better, Read this: Spark CacheLLM Response: 1. **Why Use Caching?** - Caching can significantly improve performance by reusing persistent data instead of recomputing it - It helps avoid redundant computations on the same dataset across multiple actions 2. **Lazy Evaluation and Caching** - Apache Spark uses lazy evaluation, where transformations are not executed until an action is triggered - Caching can help by materializing the result of a long sequence of transformations, avoiding recomputation 3. **Spark's Lineage Graph** - Spark tracks the lineage of transformations using a Lineage Graph - Caching breaks the lineage, reducing the size of the graph and improving performance 4. **Caching vs. No Caching** - The demo shows a significant performance improvement when caching is used, as seen in the Spark UI 5. **Persist and Storage Levels** - The `persist()` method is used for caching, with different storage levels available - Storage levels like `MEMORY_ONLY`, `DISK_ONLY`, and combinations control memory/disk usage and replication - Choose the appropriate storage level based on your requirements and cluster resources 6. **When to Cache?** - Cache datasets that are reused multiple times, especially after a long sequence of transformations - Cache intermediate datasets that are expensive to recompute - Be mindful of cluster resources and cache judiciously 7. **Unpersist** - Use `unpersist()` to remove cached data and free up resources when no longer needed - Spark may automatically unpersist cached data if memory is needed If you liked it, Upvote it. NarutoLLM Response

Great tutorials 🙏, please create more videos on spark from beginners point of view

Your channel is so underrated, Please dont stop

i like very much of your videos, it's insightful. can you please make series/videos on Spark interview oriented questions. Thanks in advance

Hi Afaque. A suggestion. You could start from the beginning to connect the DOTS! Like if in your scenario we have X Node machine with Y workers and Z exectors and if you do REPARTITION and fit the data like this then this could happen. Otherwise the Machine would sit idle and so on.

Perfect explanation & perfect examples throughout the playlist, Bhai mere Change data capture aur Slowly changing dimension jo bhi apply hote hain project me uska bhi khel samza de.

a lot of knowledge in just one video

I spent INR 42000 on a Big Data course but could not understand this concept clearly because the trainer himself lacked clarity. Here I understood completely.

Great explanation of such a complex topic, thanks and keep up the good work.

Great explanation, Thank you, But how would we know how to configure exact (at least best) "spark.sql.shuffle.partitions" at run time? Because each run/day the volume of the data is going to be changed. So, how do we determine the data volume at run time to set the shuffle.partitions number?

Awesome explanations! pls share us more relevant videos

Dead gorgeous stuff.

Hey Afaque Great tutorials. You should consider doing a full end to end spark project with a Big volume of data so we can understand the challenges faced and how to tackle them. Would be really helpful!

Not easy to understand, but it great

Useful information

Thanks for the video! I also have a question: when I execute complex query, there will be multiple stage with different shuffle write sizes, which do I have to take in consideration for the computation of the optimal number of shuffle partitions?

All the concepts are clearly explained. Please do more videos.

Your teaching skill is very good, please make a full series on PySpark, that'll be helpful for so many aspiring data engineers.

Please make one on AWE aswell

Excellent 🎉 👍 appreciate your effort

Correct me if I'm wrong, but these calculations consider the execution of only one job at a time. How do the calculations change when there are multiple jobs running in a cluster, as often happens?

Buddy! You got a new sub here. Loved your detailed explanation. I see no one explaining the query plain this detail and I believe this is the right way of learning. But I would love to see an entire Spark series.

Why to give such a large fraction (0.4) to User memory as in the end when the transformations will be performed in a particular stage , whether we give it a user defined function or any other function execution memory will be only used . So Whats exactly the role of User Memory ??

could you please make video on stack overflow like what are scenario when it can occur and how to fix it

Thanks for the content, really appreciate it. My understanding is AQE take care of Shuffle Partition Optimization and we don't need to manually intervene (starting spark 3) to optimize shuffle partitions. Could you clarify this please?