The most important Python script I ever wrote
HTML-код
- Опубликовано: 31 май 2024
- The story of my first and most important automation script, plus an example of what it would look like now.
✅ WORK WITH ME ✅
johnwr.com
➡ COMMUNITY
/ discord
/ johnwatsonrooney
➡ PROXIES
www.scrapingbee.com/?fpr=jhnwr
proxyscrape.com/?ref=jhnwr
➡ HOSTING
m.do.co/c/c7c90f161ff6
If you are new, welcome. I'm John, a self taught Python developer working in the web and data space. I specialize in data extraction and automation. If you like programming and web content as much as I do, you can subscribe for weekly content.
⚠ DISCLAIMER
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.
➡ TIMESTAMPS
00:00 Story
03:21 Code Start 
Наука
![Roddy Ricch - Survivors Remorse [Official Music Video]](http://i.ytimg.com/vi/J5IeFzr2GgE/mqdefault.jpg)
At a job I had, one of the more important clerical staff lost 4 hours a week, comparing Folder A to Folder B, and Folder B to Folder C, and nothing what had changed with what folders were added, or moved. I wrote a script that would daily create a directory listing, DIFF with yesterdays listing, produce a report of ADDs, MOVEs and DELETEs and email it to them daily, then make today's listing "yesterdays listing" for the next run. This saved them 4 wasted work weeks a year. In addition they had the changes daily now instead of weekly. Once that was available it turned out there were 7 or 8 other people who benefited from this information. I just changed the program to email them all. It is amazing how one little script can save so much time.
"Automate Everything":
1. Automate task A
2. Automate task B
3. Automate execution for task automation of A and B
4. Find a way to automate the automation in step 3
5. Keep building the automation tree
Not Python- but I automated a task using VBA in the first few weeks of my current role that I have no doubt was the reason they couldn’t keep someone in my role for more than a few weeks before they ran for the hills. Still using it most days and it literally saves me 20 hours a week.
What was the task?
Taking MRP downloads from our ERP system - organizing them by part and creating nice legible worksheets for each part. It also pulls in relevant master data/specific order comments from the network into each sheet that are relevant for each part.
Yep VBA has served me like this for 25 years. Now starting to learn python but within the use cases for excel it’s hard to beat
Python is awesome.
I automated a task with Python that was repetitive, that took about an hour every time. Now instead of me creating multiple CSV files from one large file with Excel manually (Excel struggles with a lot of data), Python does it with the run of a command in just a second.
Same story 🙂 Python automation has also helmed me make new friends in every department in the company. Your channel is great. Thank you.
how does it helm you? what does helm mean?
Likely meant helped and not helmed@rollinOnCode
Same experience, I used to work as a Data Entry Officer at an e-commerce company (nothing fancy here) but I can't stress this enough, 60% to 70% even till 80% of my tasks can be automated from changing product price, disabling it, adding promoted price for certain duration, changing its status to in/out of stock, changing product's description with a pre-defined description based on a given google sheet. Imagine doing this manually for 100+ of products, no thanks! I even helped our brand managers by fetching 100+ of production information from other vendors and competitors.
input() is a nice alternative to sleep you may like - it waits until you press enter in the shell, so you can wait as little or as much as you like.
I DIDN'T KNOW YOU COULD DO THIS!!! I mean... now that I think about it, I knew there were -programs- that did this... like Q&A testing type packages. This looks relatively simple, though!!! I had no idea! This is great! I also have a weekly thing I have to do for work that is tedious, and I've been thinking it could be scripted... but just assumed that would be way over my head. This will definitely help me get started! Thank you so much for posting this! (So glad it shows up on my feed!)
Well, that was such a cool and informative video that I had to leave a comment to help your RUclips algorithm. Well done :)
Thanks for that!
Nice one and fascinating. While I was working on a tedious, repetitive task, I started looking into Python. I had a little experience with C#, but it just did not feel right, and Python was simple to adapt. I've never been regretting my Python journey since.
It really is great for automation tasks
Great example and helps inspire ideas for me.
Nice video as always. It would be cool to also see a video about solving captcha!
Thank you so much for sharing your experiences and story John.
In my case main python use in my life was also webscrapping from Amazon to find deals.
I have improved a lot in my code and generally over time with lots of learning sources
Would appreciate some content about automation of these python scripts please ?
And may be dynamic content too (where one code runs other code, or passes data to other code at a dynamic time)
Very good info, thanks!
You changed my life. Only your videos. No joke.
I mean there was a lot of grind. Except these videos made it possible.
Came for the hope of automating parts of my work. Stayed for the CS talk!
The CS references really hit home for me. 🤣🤣🤣🤣 great video!
When worlds collide lol nice to see a Prime nugget mentioned. Great video man!
Great tutorial, thanks!
Nice, video. I need to start looking into automating more things before I get automated.
Always be one step ahead hah
Same here 👏 In my case I had to copy-paste social media data into a clunky spreadsheet. I started with some simple Python scripts to automate whatever I could, ended up in love with programming and am currently building large-scale analytics apps for my company. Your channel has been huge help throughout the years! Thank you 🙏
Thank you for watching your very kind
Fantastic video
Love these quick and dirty scripts!
Very good!
I did the same thing but with Selenium for python, ultimately I wasn’t allowed to use it departmentally because selenium was not in the approved software for the company but it sure made my job a lot easier
I was fortunate enough that I was allowed to and trusted too I know not everyone will be though
Been watching your videos for a while. Very informative and easy to understand for a newbie like me. Self taught programmer?
Hey - thanks for watching, appreciate it! Yes I am self taught!
@@JohnWatsonRooney great 🙂 background in computer science or something else totally different?
Incredible! Would you consider keep making Playwright videos? I think there's a lot of demand for scraping AI web search (particularly Google Gemini responses).
Sure, I can
looks good / useful, but does this shopping cart software have a python API available, that would be easier to pull data from rather than resort to screen scraping.
This is awesome but if you have access to APIs rather than UIs it's a lot more efficient to do it without any UI involved at all. Ideally software developers would be involved in helping eliminate so much busy work in the first place. I try to be helpful to other departments in that way where some small changes to a tool someone uses every day can make all the difference.
Hi John, what do you use for such a smooth windows management?
Thanks for your video. I'm following along and getting stuck at 8:52 with `page.get_by_role("button", name="Login").click()`. Has this happened for anyone else? Any thoughts on how to troubleshoot this issue? Thanks for any assistance you can provide.
Does the system not have an export or an api? Would be much easier than hacking around the front end. If there was no export then i can see its use - or even running things in the UI - ive used test software to automate things for me before :) Its great seeing the computer do what you woudl be doing normally while you get on with something else more productive.
What resource did you come across at 1:03? Apologies, the title and author all ran together.
that is awesome. filed for ref.
Where / how does original orders - “for order in orders:” get populated before you populate via csv reader?
May I ask why you chose this method over the playwright codegen method you used in your past vid?
Sometimes codegen doesn’t pick the best selectors so I often do it myself
As a person who doesn't understand much about coding: is having your username and password openly in a script the norm?
It kinda... "feels" wrong...
I assume you could make those two things an interactive prompt to type in everytime you start the program instead of having it written in the code permanently.
But that won't help much if it is scheduled to run automatically at certain times without user interaction...
Again, I don't understand much about programming, so I'm curious if that is a security concern in the "real world"?
You’re right, it isn’t. It’s a security risk. But I wanted to keep it as simple as possible here. I would store the user and pass in environment variables or similar. But as I designed this script to only run locally with me there I didn’t worry about it
@@JohnWatsonRooney Thank you very much for your reply!
Good to know! Then this is something I have to look further into, should it come to a practical application for me.
But at least my "feeling" wasn't wrong. That's something, I guess.
Keep up the good work, I enjoyed the video. :)
Can you do a video on how you decide what selectors you use in playwright - get_by_placeholder? get_by_role? query_selector? xpath?
Yea I had a lot of problems in them too, where like one goes wrong and code then needs to be changed again.
I know in selenium we could just put literally all of them inside of a
try:
except Exception as E:
pass
Method and that way we can have it try it all in selenium, by xpath, by ID etc.
Didn't really know how to do similarly with playwright but I think there might be something similar and that way we can put them all in some sort of try, except blocks ?? Although I don't really know if it works with playwright or not (try, except)
Hey man thank you so much. So I would love to actually be able to automate stuff as you do without necessarily following tutorials ,could you share your thought process or perhaps make a video on how one can become good as you. I do understand Pythona and coding in general. I am interested in expanding my knowledge within automation scripts. Any pointers?
Read the book “Automate the Boring Stuff with Python” or look into the course “Automate everything with Python”, do not get caught up in watching short tutorials like this, although this video was very informative, you need to learn how to learn yourself. Cheers mate
Helix editor master race
I'm a newbie data entry, I want to automate data processing and I need to start from the last page of the data and then export it to excel. Can I do that using this technique? thank you!
Dude ngl i immediately click on your videos it has help me learning programs that help me with work thank you i appreciate your videos
Thanks that’s very kind
What colorscheme are you using? It's gorgeous.
It’s kaolin-dark, my new favourite
@@JohnWatsonRooney cool, thank you. I'm excited to try it out. Right now, I'm using Nord for everything, and I really like it. But it's time to switch things up
Might be a silly question, but in this particular example you selected the particular "Orders", e.g. 3000, 3001, 3001. Now imagine it starts at 3000, however it keeps on going and you don't know when it stops. What would you write in the code?
that's called a while loop with a variable incrementing until an error is reached
How do you manage captcha ?
Superb, wouldn't be even better if you start with "playwright codegen" and tune it afterwards?
Yes that does help - i did that in my other automation video but in this I wanted to build it up myself and show it that way
Love playwright codegen!! Only limitation..... I want to use a Chrome Extension and have playwright activate buttons within the extension..... Unfortunately playwright uses Incognito, for which you cannot use extensions.... UNLESS SOMEONE OUT THERE HAS THE ANSWER TO THIS?? 😱
Nice . Is there any tutorial on Installtion of neovim and Installtion of python libraries please let me know
sudo apt install neovim
Subd 🤙🏻
Would you be available to meet on a call via Zoom at your hourly rate for a period of time you'd be available. I have a question regarding scraping Python API and I've watched all your videos that gave me an excellent overview I just have a few specific questions and would greatly appreciate your time helping me solve these couple of issues. Thank you!
I do some automations of some activities that are repetitive and boring. On my case I used a bash script.
Quick question, dose opencart allow for these scraping activities in their policy or they can band the IP, i know we are using playwrite with sleep time to mimic human but good to know if they anti-scrape bot or not
I did just this and now work remotely making six figures at the same company
All my scripts are in bash employing mostly awk, grep, sed, wc, find, expect, xargs, psql or sqlplus,
hello, I have watch several of your webscraping videos. those demo was straight foward. I dable a bit in python, nothing really to talk about on that. but could you just give a demo on: lets say I wanted to scrape a header and the associated images of that header. the headers by using some keyword from the website that has this as their headline : 'samsung new galaxy (some fancy name ) is a market hit says CEO. and the image have some text that is attatched to it that states: source getty images or somthing like that. to get those header and images returned. Can I just give all the header and image text as a keyword so beautiful soup could parse and ger the url for the image and assiciated header or header with assiciated image url then return the valueof the image and header. why I ask this?. because I don't want to be looking for those element manually (google inspect tool) I just need to add keywords and hit run then the code automatically find those and return them.
why 20 minutes ?
if you do this, all the difficult part is about cloudflare
The video was pleasant, but the title didn't give any info on what the video was about.
You appear to have reading comprehension issues. Seek help
Really funny the company I work for uses the exact same angular admin theme. I think it’s called Color Admin.
The only thing to note is that they might have an API that you can access that make your automation more stable by taking the UI out of it.
Yes its something great, and feels like you are so developed 😂😂, when you see someone do something that takes like hours while you can automate it, and let computer do it, while you are saving your energy and time to important things
6th 😂
1st like
🎉
Beat me to it .
Try it again with Puppeteer
Nande?
Thanks this really gives a great outline to pull and analyze more efficiently. Exactly the type of thing I needed to see to get me back on the keys 🫡