Professional Preprocessing with Pipelines in Python

In this video, we learn about preprocessing pipelines and how to professionally prepare data for machine learning.
◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾
📚 Programming Books & Merch 📚
🐍 The Python Bible Book: www.neuralnine.com/books/
💻 The Algorithm Bible Book: www.neuralnine.com/books/
👕 Programming Merch: www.neuralnine.com/shop
🌐 Social Media & Contact 🌐
📱 Website: www.neuralnine.com/
📷 Instagram: / neuralnine
🐦 Twitter: / neuralnine
🤵 LinkedIn: / neuralnine
📁 GitHub: github.com/NeuralNine
🎙 Discord: / discord
🎵 Outro Music From: www.bensound.com/

[@vzinko]

Rather than creating a class for each step, another much easier approach is to make use of sklearn's FunctionTransformer. This basically allows you to write a custom function and turn it into a transformer object, which can then be fed through a pipeline as per normal

[@isaacandrewdixon]

This was awesome and very informative. Many thanks from a machine learning novice!

[@dmitriidavs4181]

Fantastic video, always wondered the reasoning behind using classes in ml, thank you!!!

[@randomfinn404]

For those who noticed that the encoder seems to sort the values alphabetically and messes up the job column names, instead of manually typing column names you can do:
matrix = encoder.fit_transform(X[['Job']]).toarray()
column_names = sorted([i for i in df['Job'].unique()])
This will also work if there are more /new jobs and values added and makes a column for each unique value while keeping the order.
Good tutorial in any case!

[@jacksummers3918]

Use
pd.get_dummies(X.Job, prefix="Job")
Much neater

[@ShortsSmith]

G.... Thank You... I was hoping to that some one noticed it...
I'm glad that I got the Better version ❤

[@onecarry1532]

Hey man, great channel! Love the topic based tutorials ❤️
Video Suggestion: Can I suggest you attempt making a video on: Using Python and the Tree Algorithm to make an autocomplete Python CLI program.
Haven’t seen this anywhere and I guess it’s a great way to understand why the Tree algorithm might be the best solution for an autocomplete program.
Thanks! Sure we all appreciate what you do for the community ♥️ 🌻

[@nathanhaynes2856]

Nice. For this example I might use the ColumnTransformer class, its perfect for dropping columns and integrating imputers and scalers on select features.

[@Deacc]

This video is pure gold. Thank you so much!

[@niv_syt6315]

I remember when I took courses from udemy in ML and took more time from this video, keeps to continue creating more videos from the same subject.

[@manyes7577]

wow this technique is amazing. thanks for sharing us with brilliant knowledge

[@nikulnayi3271]

Thank you so much nicely explained
with what you showed i created pipeline and dumped it as pikle file but when i tryinng to load that model and using it. i have been facing an error : AttributeError: Can't get attribute 'NullEncoder' on

[@vlplbl85]

I find using FunctionTransformer much easier. It turns each of your custom functions into a transformer and you don't need to write a class, but just a function.

[@tharakawickramasinghe3762]

Thank you. This is very helpful.

[@apheironnn]

That was really helpful, thanks!

[@MrTactics26]

Sick video bro! 😎

[@Juzz_RSA]

Thank you, this was informative 😁

[@jelcroospockt]

I would really like to find a tutorial on how to pass arguments to an pipeline function you created yourself, like the namedropper. So i can use the gridsearch to try out dropping different features.

[@juandiegoorozco5531]

really useful, thank you very much

[@rohscx]

What is the opening song of this videos name?

[@gasfeesofficial3557]

bro great video!!

[@allanmachado2011]

Thank you!

[@sviteribuben7245]

Very usefull! Thx!

[@pakaponwiwat2405]

Thank you, sir!

[@__wouks__]

I think your feature encoder has some faulty logic for the "Job" column. The df2 for example shows 1 x writer, 3 x programmer and 1 x teacher, but afterwards there isn't even a "teacher" column. And if you were to recreate the single columns using 1 or 0 from the features you created you wouldn't get the same dataframe.

[@thomasgoodwin2648]

With an eye towards the love that programming has gotten from the ml community lately, it occurs to me that perhaps ml could also be used more in the data preprocessing role.
For example: Choosing encoding types, handling missing values, flattening, etc could all be automated.
Just a thought.
2nd random thought. I know random noise has been added to features in an attempt to get the models to generalize better but did not fare well.
However I have not seen that anyone has tried simply using noise generators (normal, gaussian, etc) as individual features and allowing the model itself to choose when and where noise might be effective.

[@736939]

16:42 I think it's wrong to use fit_transform in transform method, because it will cause to memory leakage, after you divide data into two parts train/test - where transform on the test dataset will recalculate imputer.

[@falkstankat6511]

Yeah, thought the Same

[@juanbetancourt5106]

Great!

[@aayushpatel2904]

Thanks Sir

[@nachoeigu]

I have a big one question: What is the difference of build a Machine Learning application with Pipeline and to build a machine learning application with a OOP technique? I see that it is the same.

[@adriandiaz5688]

Yeah, this is a great video but that's something I'm curious about as well.

[@MalcombBrown]

Could you use the get_dummies pandas method for the One Hot Encoding?

[@lexcheshir6416]

yep

[@o1techacademy]

Awesome

[@dilshodfayzullayev924]

where do you work #admin

[@slothner943]

Are you swedish? 😮

[@bellabella-tv8zg]

1st