Pydantic is the Python Package I Wish I'd Learned Earlier
HTML-код
- Опубликовано: 26 апр 2022
- Pydantic is the data validation we need in Python. It allows us to define a model and set the data types for each field, making it not only easier to work with in our IDE but also checking the types at runtime, which is something that is not in Python as standard.
I've found that it helps when dealing with any external data that we would want to process in our applications, like user submitted data or data from files we have received outside our own ecosystem. So far its been a great addition to the way I code and I think you will love it too!
Patreon: / johnwatsonrooney
Scraper API I use: www.scrapingbee.com?fpr=jhnwr
Proxies: iproyal.club/JWR50
Hosting: Digital Ocean: m.do.co/c/c7c90f161ff6
Gear I use: www.amazon.co.uk/shop/johnwat...
Twitter / jhnwr 
Наука
![Moneybagg Yo - WHISKEY WHISKEY [feat. Morgan Wallen] (Official Audio)](http://i.ytimg.com/vi/kJamyq0WplE/mqdefault.jpg)
I rarely write any comments, but I just wanted to thank you for the content. I've had very basic understanding of webscraping with requests, but recently I've had a project to scrape as many websites as possible and thanks to you I learned about scrapy, pipelining to database, user-agents and much more. Thank you so much!
Thank you! I’m glad you’ve enjoyed my videos and that they were useful to you, thanks for watching!
You do a fantastic explaining concepts and give great examples. Thank you
Quick note of thanks: I just landed my first programming job which is django development. Before I learned any web dev though, I was always using your scraper tutorials and building my own scrapers; working on my python chops. Those were some of the most formative things in terms of actually getting the confidence that I could learn and write code. I've been quiet on this channel while learning all the web dev stuff but I wanted to say thanks again for all the help you've provided along the way.
That’s great Matt, congratulations on your new job! I’m really pleased to have helped you in some way on your path to becoming a django dev - thank you for letting me know I appreciate it.
Thanks for explaining everything so clearly!
Very nice tip and intro to Pydantic, I'll give it a try, thank you
John you are amazing, thank you very much for all the knowledge you share, cheers from Chile!
Many thanks!
You might not know the true impact you are making upon us when making these videos, but it is substantial. I can’t thank you enough for what you are doing.
great work, this is definitely a handy tip
Best Chanel ever, thank you so much
THAT WAS GOOD! Thanks man
I've been using pydantic for year. The validator is a killer feature that needs to be built in python next version.
Not in a programming mode but I like watching a master home his craft.
I stoped python for a few weeks, and have been learning html,css,js for about 3 weeks. It really help a lot to understand scrapy. Now i'm back to learn more about scrapy skills!
I just scrapy all your video titles, wanted to learn one by one.You uploaded 146 videos, a heavy job omg. I couldn't directly scrapy from youtube，maybe there is something wrong with proxy.(live in cn and have to use VPN to get to youtube or google).So I just copy down the html element and use bs4 to parse it.
Perfect thank you
Thank you so much for this John! 🙏🥳 Great topic.
I also watched your SQLModel video: thanks for that too!
More videos about storing more complex data structures in an ORM setup are appreciated.
My goal is to store scraped data that is enhanced with API data to the Django database.
Not sure how to setup the route from JSON + other JSON to the Django ORM yet.
Thanks very kind!
I had actually heard of Pydantic several months back but never explored it further, don't know why, just kind of gave it a first glance and moved on.
I did not know how useful it could be hahaha I'm definitely going to use this!
Thanks!
Optional doesn’t mean it can be omitted. You need to set a default value for it to truely be optional, but if the default is None then the Optional allows the mixing of list and None
Yes you are right I made a mistake there
It's like dataclasses with validation. The only issue I have is that it's a bit rigid. Once it's a pedantic model you cannot change anything inside it. So you need to make the transformations to the incoming data (if any, and yes, there can be) before you instanciate it. But if you're doing that, then you might as well be using dataclasses to begin with and perform the transformations on the go. The problem with that is that you lose the validation, you will have to write it yourself.
FYI: There is a Pydantic PyCharm plugin that will autocomplete the class variables you type at about 3:33 where you create the item object.
Oh great thanks I didn’t know about that
awesome
amazing!
Thank you!
Brother, you should make more videos on these libraries other than just web scraping. I love your videos but I couldn't find other library videos.
Nice 🤠 but I Have a question how modify the Json (e.json) when the exception is raise
I had literally just said aloud "yeah, I don't want to type all that out" and then you said "I can hear you saying ''I don't want to type that out all the time". :))
Haha yup!
Not sure if you have tried this, but you can create a top class with a root attribute to hold the list of items. Instead of using list comp. in fact this is what the pydantic Jason to model generators do.
Like a products model to hold list of product models
You can do this same thing with dataclass mixin module
So is this similar to a json schema validator except that this is operating on a dict? For incoming requests, we're using a json schema validator to ensure the structure and type match against a json schema. I'm wondering what this will bring to the table should I replace schema validation with pydantic.
Hey John, are you going to do more tutorials on Go? You are a Gofer now after all 😁!
Haha yes I will, don’t worry
how about taking it a step further and doing using dataclasses? Instead of using basemodel? Would that work too?
from pydantic import dataclasses
You can achieve all of these with dataclass also
Hey, what tech stack do you use for your blog?
Hugo static site generator, GitHub and netlify
Love your video and learn today. I am not clear of at 9:37 product[**item] , will you please explain what "**" mean here in the list comprehension. Look forward to your reply. Thanks
In Python, ** is used to unpack dictionaries/key-value pairs. This is useful in a few areas, I mostly see it used in function/method parameters. def function(*args, **kwargs): would let you call function(blah=True) and kwargs would be a dictionary that looks like: {"blah": True}
@@VolundMush thank you so much
John, pls explain what does it mean in list comprehension this Product(**item)?
We are loading the data into the Pydantic model called Product
@@JohnWatsonRooney thx
@@JohnWatsonRooney I guess you are trying to unpack the dictionary/json data structure item?
for example u have 1 excel sheet and it consist of 10000 data in it. Later when we import that excel file in pycharm or jupiter notebook. if i run that file i will get an Index range also know as Row labels. my python code should be able to read that ten thousand row labels and should be able to separate / split into 10 different excel sheet files which will have 1000 data in each of the 10 saperated sheet.
other example is, if there is 9999 data in 1 sheet then my python code should divide 9000 data in 9 sheet and other 999 in other sheet without any mistakes.
i am asking this because in my data there is not any unique values for my code to split the files using .unique
plz help i have search the whole YT , stackoverflow, and github tooo from 3 days
I find this video shallow and pydrntatic
-Peter Griffin
I wanna thank you for teaching the world.
Please I'm python beginner I my next step is to build web scraper.can you please help me how do I go about it.
Thanks! I have loads of web scraping with Python content on my channel if you have a look I did a basics scraping guide not that long ago
why not simply use a dataclass, they have types as well.
Ha ha python is moving towards java...
Great content, but watching someone type is annoying AF! Why not type it up beforehand, then hide and reveal lines as you go?!
Different styles, but fair enough - my typing skills aren’t the best
Don't show off your typing skills, but use copy and paste. I find it particularly annoying to cancel and correct your typing in my mind, while you do it!
@validator("sku")
def sku_length(cls, value):
assert value == 7
return value
must be better
I’ve always worked on the assumption that assert is for use in tests not code logic
@@JohnWatsonRooney fair enough