Data Wrangling with PySpark for Data Scientists Who Know Pandas - Andrew Ray

"Data scientists spend more time wrangling data than making models. Traditional tools like Pandas provide a very powerful data manipulation toolset. Transitioning to big data tools like PySpark allows one to work with much larger datasets, but can come at the cost of productivity.
In this session, learn about data wrangling in PySpark from the perspective of an experienced Pandas user. Topics will include best practices, common pitfalls, performance consideration and debugging.
Session hashtag: #SFds12
Learn more:
Developing Custom Machine Learning Algorithms in PySpark
databricks.com/blog/2017/08/3...
Introducing Pandas UDF for PySpark
databricks.com/blog/2017/10/3...
Best Practices for Running PySpark
databricks.com/session/best-p...
Session Overview:
- Why?
- What Do i get with pyspark?
- Primer
- Important Concepts
- Architecture
- Setup
- Run
- Load CSV
- View Dataframe
- Rename Columns
- Drop Column
- Filtering
- Add Column
- Fill Nulls
- Aggregation
- Standard Transformations
- Keep it in the JVM
- Row Conditional Statements
- Python when Required
- merge/join dataframes
- Pivot table
- Summary Statistics
- histogram
- SQL
- Make sure to
- Things not to do
- If things go wrong
- Thank you
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: databricks.com/product/unifie...
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / databricks
Instagram: / databricksinc Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. databricks.com/databricks-nam...

[@fiddlepants5947]

Really nice how we see pandas and pyspark functions side-by-side!

[@javonnii436]

yea I thought the same!

[@AlessandroBottoni]

Fantastic introduction to PySpark for beginners. Hope to see Andrew Ray again on the stage for other presentations.

[@ratkush]

Must watch Q n A session in the end. I loved it.

[@enes-the-cat-father]

Thank you for such a great presentation for beginners!

[@kevinlin5486]

This a great video. Exactly what I'm looking for thanks very much.

[@tanishasharma3665]

he provided with a really good comparison between the two!

[@thedarkknight579]

Thank you so much for the Session ❤️

[@VishalSharma16]

Super helpful, thanks for sharing!

[@toygraphers240]

Thank you very much for your contribution.

[@ZenvilleErasmus]

Cool talk and key differences nicely illustrated.

[@harjeetkumar4632]

Here are some more videos on spark Spark Interview Questions: ruclips.net/p/PL9sbKmQTkW05mXqnq1vrrT8pCsEa53std

[@alexnim4873]

great presentation!

[@willwright5181]

Great intro!

[@pratikmehta1152]

Volume is low! :(

[@konstantinrebrov675]

use detachable speakers

[@goedzo4361]

Really helpful

[@santil.7072]

Does it mean that using pyspark sql is the best practice in data wrangling using spark?

[@abrahamf80]

My path to data was a little bit unsual to say the least, started to work in the financial industry using databricks and now on side projects started to work on pandas... funny that I actually used this video backwards hehe

[@elliottharris4526]

Would this be a good tool for combining large numbers of csvs into a single dataframe quickly and then performing manipulations on that dataframe before outputting a single csv?

[@raphaels2103]

19:12, now pandas has an SQL support

[@musasall5740]

by just downloading and writing this code it will not work. You have to create a session.

[@1over137]

PySpark is great with it's read only. It all goes badly wrong when you try and write anything with a typed schema.

[@Arjun147gtk]

I think I need a soundbox on full volume to hear this.

[@jaspreet0305]

I've the same issue, thanks to the captions, I saved a lot of money

[@francischab2262]

7:49

[@Rabixter]

Whats with the volume?

[@xiaoyunzhang6878]

Nebraska Alumni

[@krishnakishorepeddisetti4387]

Which is better in databricks environment?? Python or R or SQL..reply in comments

[@jimbocho660]

Most people seem to find SQL better.

[@trespittman1055]

Too quiet please fix

[@Tyokok]

great tech video, but volume really ...

[@kaixianghuang8589]

LOL good presentation, but unprepared for the Q &A

[@TheBjjninja]

Why did someone ask about uDF? What does UDF have to do with spark?

[@Drivebyeasy]

Hey Andrew could you send me your Github link

[@Atlas-ck9vm]

Just use koalas.