Understanding Databricks & Apache Spark Performance Tuning: Lesson 02 - Spark Hardware

[@Ali-q4d4c]

amazing explanation. are you planning to make more videos on performance tuning?

[@BryanCafferky]

Yes. It's on my list. :-) Thanks

[@MoulayC.-qv3rn]

Thanks, very useful and up to date.

[@BryanCafferky]

You're welcome!

[@rahulrokkam8174]

Thanks Bryan!

[@BryanCafferky]

You're Welcome.

[@Noobsmove]

I think the main reason not to use photon, is that it costs extra^^

[@BryanCafferky]

I have not seen that there is a higher cost to using Photon but even if that's true, the time saves running the VMs should outweigh the cost. See this blog. www.databricks.com/product/photon

[@Noobsmove]

@@BryanCafferky Yeah it's a little cheeky. If you enable it, the DBU goes up. I have photon disabled for light workloads

[@BryanCafferky]

@@Noobsmove Interesting. Guess its in the fine print. 🙂