Hi sir please answer my question. Interviewer asked me this question Q : if rdd fails at stage 5 what will happen and how do we handle this situation recover data and fix the problem
there are two types of problem that could happen, recoverable and unrecoverable situation, in case of unrecoverable one you have to go through the logic and update it accordingly. while in case of recoverable issue you don't have to perform anything as it is resilient, spark will internally re-executes all the stages and finally create the RDD
Hi @Data Savvy , I I do have a doubt on partition concept . By default any file stored in HDFS will be distributed. Say 1 GB file with cluster split size as 256 MB , do have 4 different pieces internally. When I create an RDD from this file , would the partition you mentioned here is the same 4 pieces or Spark do have another mechanism to define its on partitions. In simple words , is HDFS split and spark RDD partitions are same ?
Could you please arrange the spark interview question playlist ..videos are not in order very random arrangement
Waiting for next video....
Hi sir please answer my question. Interviewer asked me this question
Q : if rdd fails at stage 5 what will happen and how do we handle this situation recover data and fix the problem
there are two types of problem that could happen, recoverable and unrecoverable situation, in case of unrecoverable one you have to go through the logic and update it accordingly. while in case of recoverable issue you don't have to perform anything as it is resilient, spark will internally re-executes all the stages and finally create the RDD
@@abhishek5643 thanks
Hi @Data Savvy , I
I do have a doubt on partition concept .
By default any file stored in HDFS will be distributed. Say 1 GB file with cluster split size as 256 MB , do have 4 different pieces internally. When I create an RDD from this file , would the partition you mentioned here is the same 4 pieces or Spark do have another mechanism to define its on partitions.
In simple words , is HDFS split and spark RDD partitions are same ?
Not all heroes wear capes
Finished 27 videos,
Awesome... Great dedication... M happy this was useful to you :)
Very much useful and will now start following another series that you have started
@@DataSavvy great knowledge...I appreciate you alot
Thanks Shyam... Your words are very encouraging :)
@@DataSavvy I am looking for some support with spark. Plz let me know if you are interested thank you.