A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets - Jules Damji

"Of all the developers' delight, none is more attractive than a set of APIs that make developers productive, that are easy to use, and that are intuitive and expressive. Apache Spark offers these APIs across components such as Spark SQL, Streaming, Machine Learning, and Graph Processing to operate on large data sets in languages such as Scala, Java, Python, and R for doing distributed big data processing at scale. In this talk, I will explore the evolution of three sets of APIs-RDDs, DataFrames, and Datasets-available in Apache Spark 2.x. In particular, I will emphasize three takeaways: 1) why and when you should use each set as best practices 2) outline its performance and optimization benefits; and 3) underscore scenarios when to use DataFrames and Datasets instead of RDDs for your big data distributed processing. Through simple notebook demonstrations with API code examples, you'll learn how to process big data using RDDs, DataFrames, and Datasets and interoperate among them. (this will be vocalization of the blog, along with the latest developments in Apache Spark 2.x Dataframe/Datasets and Spark SQL APIs: databricks.com/blog/2016/07/1... databricks.com/glossary/what-...)
Session hashtag: #EUdev12"
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: databricks.com/product/unifie...
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / databricks
Instagram: / databricksinc Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. databricks.com/databricks-nam...

[@chiranjibghorai6950]

Such a pleasure to hear him talk!

[@techoral2261]

Now i know about RDDs, DataFrames and Datasets. Thanks for explaining it more precisely. Appreciated.

[@tdkboxster]

What an amazing talk! Crisp and Clear! truly impressed.

[@rahultiwari2860]

Thanks for in-depth explaining RDD DF And DS...

[@ctriz76]

this is a brilliant and fluid explanation

[@mjrajeshmj]

Excellent talk. Thanks Jules Damji.

[@sayandbhattacharya1100]

I had a nice learning time thanks for the talk!

[@daviduzumaki]

this guy is such a good speaker

[@harshtiku3240]

An excellent talk by a clear master.

[@puja9689]

Amazing presentation. Very intuitive..Thanks Boss!

[@lbasavaraj]

What a brilliant talk!! Thanks

[@aliwaheed906]

Amazing talk, I left off Spark to move in to ML when there was only RDD, I came back and see DataFrame in Spark and I am totally confused, your video helped a lot, Thank you

[@abhiganta]

This is best and clear talk on 3 APIs

[@soufianebenkhaldoun7765]

Very well explained !! Thank's

[@AllForLove3]

Amazing talk! very well explained indeed.

[@jijotitus1755]

Amazing Talk. Thank you!

[@TheTambourinist]

Thanks for the video. Very understandable!

[@shemantkr]

it was very insightful, such talks really helps developer why/how one should use structure API

[@pauliewalnuts6734]

so good!!! thanks for this

[@Dyslexic_Neuron]

excellent explanation!! :D

[@shannithssachin]

Great Talk

[@BasemKhalaf-uj7cc]

Thank you!

[@anibaldk]

Only 300 likes for such an informative, crystal clear talk??

[@Blobonat]

Very good talk!

[@vipultyagi1369]

brilliant talk!

[@shankarsr1]

awwwwesome talk thanks!

[@tableauvizwithvineet148]

Nice and informative video

[@nareshgb1]

I am wondering how the "type safe" feature combines with the "unstructured data" that is the nature of data in the systems that spark would be used in.

[@varundosapati7148]

I was trying out the example you mentioned @10:46 and as i am getting compile time error, I had to rewrite the final statement as below.
parsedRdd.filter( content => content._2 == "en").map(filteredContent => filteredContent._3).reduce(_+_).take(100).foreach(reducedContent => printf(s"$reducedContent._1: $reducedContent._2"))
I would really appreciate if you can review above statement

[@meravchkroun4197]

Thanks!
Can you attach the links here?

[@nithints302]

wow

[@AbhijeetSachdev]

[@ernesthert1898]

No SS

[@ernesthert1898]

Hibud

[@Chris_zacas]

This was amazing! Pretty well explained!
Thanks!

[@goodyoyo0214]

Amazing Talk