04. On-Heap vs Off-Heap| Databricks | Spark | Interview Question | Performance Tuning

Поделиться
HTML-код
  • Опубликовано: 11 янв 2025

Комментарии •

  • @datawallah1289
    @datawallah1289 2 года назад +4

    Thank You for such a wonderful explanation. Really helps a large community of people persuing in big data

    • @rajasdataengineering7585
      @rajasdataengineering7585  2 года назад

      Thank you Atul

    • @abhinavsingh1173
      @abhinavsingh1173 Год назад

      @@rajasdataengineering7585 Your course it best. But problem with you course is that you are not attching the github link for your sample data and code. Irequest you as your audience please do this. Thanks

  • @Wiztech360
    @Wiztech360 Год назад +2

    Great content, I was wondering if you can provides the notes or the ppt's of the tutorials.

  • @nsrchndshkh
    @nsrchndshkh 3 года назад +1

    Beautifully explained, keep up the good work. Waiting for your next video.

  • @RadomName3457
    @RadomName3457 Год назад +1

    very good tutorial so far, thanks sir!

  • @sandhyarani-i5k
    @sandhyarani-i5k Год назад

    Great! can you please tell how to handle the out of memory exception in spark

  • @akashhudge5735
    @akashhudge5735 2 года назад +4

    why no one is telling how to remove the data from off-heap memory? what happens when off-heap memory is full as there is no automatic garbage collection? but explanation is good.

  • @varalaxmi1742
    @varalaxmi1742 Месяц назад

    Hi, In this video you say serializing and deserializing overhead is there for off heap memory, but in previous videos you said, we can avoid serialization and deserialization in off heap memory. Could you please clarify which one is correct?

  • @LearningBigdata
    @LearningBigdata Год назад +2

    Thank you for the awesome series. The videos are missing in the series. After 4th video, the videos are missing and out of 100+ video series, only 65 seems to be available and videos are missing in-between. Could you please suggest?

    • @rajasdataengineering7585
      @rajasdataengineering7585  Год назад

      All videos are available but they are segregated based on a playlist. If you look at videos under "videos" tab instead of playlist, you can find 113 videos overall

    • @LearningBigdata
      @LearningBigdata Год назад

      Thank you so much for the reply. May I ask for your LinkedIn or any other way to get in touch with you, please?@@rajasdataengineering7585

  • @Ustaad_Phani
    @Ustaad_Phani 4 месяца назад +1

    Nice explanation sir

  • @epicure07
    @epicure07 5 месяцев назад +1

    Very good video, thank you

  • @ritchie6655
    @ritchie6655 4 месяца назад

    There is Garbage Collector to clear the memory in on heap Memory, what happens when the Off-heap memory becomes full?

  • @nic-iv4ds
    @nic-iv4ds 2 года назад +1

    Great content, can you please show some example of using huge dataset without off-heap memory and with off-heap memory

  • @YoSoyWerlix
    @YoSoyWerlix Год назад +1

    Hi Raja! Great content, shall we follow the playlist as it is or by following the numbers before the title? (Asking for the best order for a beginner) thank you!

    • @rajasdataengineering7585
      @rajasdataengineering7585  Год назад +1

      Hi, thanks for your comment. You can follow the number before the title

    • @dineshdeshpande6197
      @dineshdeshpande6197 11 месяцев назад

      @@rajasdataengineering7585 , The video sequence is missing for the numbers in title, makes its quite confusing for the order to follow. The playlist has 65 videos but numbers in title are 100+ . PL help here to follow the proper order.

  • @suresh.suthar.24
    @suresh.suthar.24 Год назад +1

    Thank for video, but i have small doubt like when we use persist() method with storage level off-heap then data will be stored in techyon or worker node's off-heap memory ?

  • @suriyams3519
    @suriyams3519 Год назад +1

    HI raja, 1. when we enable off-heap memory gc will not come in to this picture ?- 2.) gc memory will be used when off heap memory is not enabled? 3.) off-heap used to store the intermediate results instead of disc?

  • @pavithrakumaraguru8487
    @pavithrakumaraguru8487 8 месяцев назад +2

    It would be great if you could share the notebooks and PPTS used on videos.

  • @saravninja
    @saravninja 2 года назад +1

    Good explanation!!

  • @sophiayue2484
    @sophiayue2484 2 года назад

    Well Done! Is it possible you might share the slide with us? Thanks.

  • @sivasankar2968
    @sivasankar2968 3 года назад +1

    Very help full

  • @vishalaaa1
    @vishalaaa1 Год назад +1

    excellent

  • @suriyams3519
    @suriyams3519 Год назад +1

    off heap memory and memory overhead are same?

    • @rajasdataengineering7585
      @rajasdataengineering7585  Год назад

      No both are different

    • @suriyams3519
      @suriyams3519 Год назад

      @@rajasdataengineering7585 Thanks::::)

    • @shivanireddy1428
      @shivanireddy1428 10 месяцев назад

      @@rajasdataengineering7585 can you also share what is overhead memory and how is it included in executor memory

  • @gulsahtanay2341
    @gulsahtanay2341 10 месяцев назад

    Very helpful

  • @satishkoppisetty927
    @satishkoppisetty927 3 года назад

    please do video on skew data

    • @rajasdataengineering7585
      @rajasdataengineering7585  3 года назад

      Sure Satish, will post videos on skew data

    • @rajasdataengineering7585
      @rajasdataengineering7585  3 года назад

      Hi Satish, as per your request, posted a video on data skew topic. Hope it is useful to you
      ruclips.net/video/EQhldyLWPwI/видео.html

  • @NikhilGosavi-go7be
    @NikhilGosavi-go7be 3 месяца назад +2

    done

  • @dq5507
    @dq5507 2 года назад

    Hey subtitles will be more helpful. PLEASE ADD IT.