I have been developing Python programs for a year, and have studied this subject for many many years before. I just wish every single tutorial on difficult topics were as good as this series is. Absolutely amazing. All the important details are here and important parts explained, I never had this much of an easy time following a coding series.
It has been two years already, where is the continuation? Why did you abandon your channel? You explain better than all my teachers! You're the best! It has been 2 years already, perhaps you have improved object detection, bot performance, and their movement between points, or maybe there's something else interesting to see!
Glad you look forward to these! I've got a big work project due at the end of the month, so it'll probably be another two weeks before the next one unfortunately.
Hey, I'd like to thank you very much. I've been wanting to learn to code for a while but have felt so small compared to how powerful programming can be. I'm extremely thankful for your videos. I'm now taking what I've been learning from you and other talented people to help me with my robotic/AI ideas I've been dreaming of for a long time now. Who knew it took learning code by playing games would be one of the best ways for me to learn but it helped a ton. Thank you! I still get annoyed with the code still but I take a break and a few hours or the next day I've figured it out.
@@LearnCodeByGaming Hello bro! Awesome video. I'm learning a lot from it. Whell, i'm trying to crop the screesnhot to read just re LifeValue with pyteserract. But when i try, i cannot cut the image properyly cDC.BitBlt((0, 0), (self.w, self.h), dcObj, (self.cropped_x, self.cropped_y), win32con.SRCCOPY) When i change this, nothing changes. Please help 😁🙏
I already have a code that works, from the another project which saved the screenshot in HD or SSD. but too slow. Then when i try in this Real-Time the FPS goes from 4 to 0.70. I will try to do some Multiprocessing to this slow function run alone
I created a farming bot for an mmo that clears a stage of bad guys, and at the end runs around to look for a relic and go to it to leave the mission. I used 3 needle images at different distances, but it is extremely hard for the bot to find since the relic has fire coming out of it and the level is on a fiery magma rock surface. Im really excited for the next video, because this has really been the final frontier for my bot that I can't seem to get through as well as id like. Thank you for the great content.
I stopped already coding 10 years ago and doing other things. I started playing a game some month ago which consumes a lot of time and look for opportunities for automate it... and agian think about to start coding with the new tools, like py. Its a loop which will never ends... thank you for sharing that tutorial.
Games are supposed to be fun, I hate when they add grinds. But then automating the grinds can become more fun than the game itself. Thanks for watching!
I nearly cried when you said the solution for the black screen issue is to just default to a full screen capture. The specific game I'm trying this with won't work with that method, and it also has the black screen bug, so I guess I just can't do my project. This video series is like the 20th resource I found for this and looked the most promising. If this is the only solution you have, I guess I'm on a fool's errand here
Finally dude i have been waiting for this so long I have watched the video entirely but don't stop this series please. Waiting for next one, by the way, nice video
Very nice series, I'm really enjoying it and learning at lot too. Thanks for putting all the effort into these videos. I'm strugling a bit to adapt the WindowCapture code to my Ubuntu OS, but hopefully I will get there. I can feel Deep Learning coming and I can't wait to see it :)
@@LearnCodeByGaming Setup an interesting function to help randomize click locations for targets found with the opencv techniques you've gone over. Hoping this will help with potential detection for services I may or may not be using this technology with ;)
you should definitely try machine learning with yolov5 for object detection. this is much more powerful than template matching and can address some of the issues you mentioned at the end. also would make for a great vid! solid video already, well deserved sub!
If you're having trouble viewing your vision window because it's too large, you can resize your screenshot just before calling imshow by using screenshot = cv.resize(screenshot, (new_width, new_height))
Careful around 7 minutes when he's first testing the live capture of the new image, in the previous tutorials he used PNG and made note of it as important. In this one he's switched to JPG, and CV2 will error if you use PNG as it's expecting the images in certain formats.
i do have the exact problem now, but i dont understand where i have to change something, because at that point there is nowhere any mention of filetype
What I usually do with vision algorithms is wrap the code with try catch blocks and listen for 'keyboard exceptions' or something like that for me to be able to have that "press q to close" functionality instead of sleeping for 1ms with the cv.waitKey() function because that can be hurting your fps. Might be a good thing to try out to improve the performance.
Thanks for keeping tabs on the channel! Will probably be two weeks before the next one is up. Got a big client project due at the end of this month that I need to focus on first.
Oh finally we're getting close to the end 😂. Yeah so I really want to make an gathering ai. So as your video has helped me alot since the game I was trying to make an bot is albion online and you're working on it. Sooo if it's ever possible can you make a video where the code identifies the nodes ( which it already does ) and then makes our character move towards it on its own and collect it. All on itself because that's what I wanted to do. And so far I'm also stuck on this place because I don't know coding alot so your video is helping me a lot ....that's why if you can teach how to make the character move towards the identified item that would be helpful because I want to make a lot of ai that do the gathering on their own and staring with albion ...I'll have a great start as it has a lot of problems soo it's a good place to start and you're good at explaining the" why, what and how" of it.....so I'm really looking forward to the movement part of the series
I found one (possible) bug that may throw people off when using this version of the code. Please let me know if I'm misunderstanding, or if there is a better way of input validation for matchTemplate. Here is the scenario, In vision.py, under the .find() method, line 29 If you start the script using a needle img you know won't work and matchTemplate CANNOT locate the needle image in our haystack window, it will throw an error and stop the script with "(-215:Assertion failed) (depth == CV8U || depth == CV_32F) && type == _templ.type() && img_dism()
I also found this an issue unfortunately. In my case, it also presents an error even when searching for an image that should be in the feed. I am not as familiar with OpenCV, but it might have something to do with dropping the alpha channel and making the image C_CONTIGUOUS because if you comment these 2 lines out: # img = img[..., :3], # img = np.ascontiguousarray(img) in the windowcapture.py file, the program will run. Doing this however, I am now experiencing a new issue which I haven't had time to look into yet where it fails draw rectangles around an object the system logs as detected.
@@orion7317 also having this issue and don't really understand how to get past this and I'm also sure the issue has to do with the changes done to the img in the windowcapture class. Please let me know if you guys came up with a more functional workaround please!
I deleted img = img[...,:3] and img = np.ascontiguousarray(img) from window capture class and now it somewhat works, the fps is down from 30 to 5, but no window pops up
@@Daniel-nv9fl Great its also work for me, thank you! I notice when my needle image is PNG I need comment or delete line with img = img[...,:3] and img = np.ascontiguousarray(img) in windowcapture.py. But if I have image in JPEG format I don't need to delete or comment it. @Alekaditez Ben in video also has FPS drop when start program so it's part of journey :)
you also could do what i did in a game, just modify a image file of the thing you search in the game, like a mod, and the n recognize that. Also, you could use a lot of templates for searching in a for loop, and when one templates matches, then use that template only until the algorithm doesnt recognize the template anymore because of changes in day, night, etc, this way you could keep your performace but also make the code adaptable to the situations, thanks for the tutorials, you rock man. what do you think of those sugestions ?
What would the for loop look like? I have a pretty good script down but I’m tying to understand an efficient way to cycle through multiple needle images or templates or something
@@hiscockanthony97 it is hard to explain in words XD, but you could make a for loop that runs trough a list of your needle images and make a recognition, if the recognition is done with one image, break the for loop, return the image you need in a variable manner. and enter an outer while loop where you use the image that generated a match for image recognition till you cant find the image again, then you do the for loop again, it could be done inside a function
Awesome videos! Is Video #6 coming soon? Would like to learn how to click random points within the object and learn how to create steps. such as, Step 1.Find Wood Step 2.Click Wood Step 3.Gather wood Step 4. sleep for 10 minutes Start all over again.
Hey again Ben, needless to mention this series is just epic! Thanks! I have a few questions. 1. For simplicity, let's say u want to detect one of your 2D icons from that top right menu bar (for example that mail icon). After you took a screenshot of that icon and implemented this into your code, you are now successfully detecting the icon. Now, if I have a different monitor with different resolution (thus size matters) and I'm trying to run the same code with the same icon screenshot, it wouldn't work, right? So my question is, how can take a picture of an object and make sure it will be detected among all screen resolutions? Maybe there's a way to convert the sampled image (the mail icon) and the surface image (the game screen frame) to some sort of a "binary pattern" that doesn't rely on pixel color so that they will be treated equally and the size won't matter? 2. How do you use a png image (transparent background, and non-'squared') as your target object to detect? Thanks!
Hey Shay. For #1, I can't find anything OpenCV has built-in for that kind of multi-scale template matching, so you'd need to do it yourself. Basically you'd make a loop that resizes your needle image, looks for that using match template, and then continues that process until your image is found. That might be faster if you first convert the images to black and white (so the binary pattern like you suggest) using something like canny edge detection. For #2, my understanding is that OpenCV can load transparent png files no problem with imread(), as long as you provide the IMREAD_UNCHANGED flag. But imshow() and matchTemplate() will give you some trouble. Match template has an optional mask argument, and I think you'd need to provide that along with your needle image in order to correctly ignore those pixels that aren't of interest.
@@LearnCodeByGaming I've already found a possible solution to (1) that indeed involves scaling the image up/down. Working on it a the moment. Thanks a lot man!!
I have had an issue with Chrome giving a black screen when captured. If you disable Hardware Acceleration, the capture of Chrome as a window works fine.
This series is great! I have followed along and learnt a lot, but I have ended up stuck! I am getting an error even if I copy and paste your code, for a opencv assertion around "(-215:Assertion failed) (depth == CV_8U || depth == CV_32F)...." I am struggling to fix this so bad!
Would the image detection method be effective for detecting in-game text? Specifically, the goal is to locate an item named [DARK ARMOUR], which has a 1/100 chance of appearing on the screen when clicking. To achieve this, it is necessary to click rapidly to cycle through all ARMOUR types until [DARK ARMOUR] appears on the screen. The question remains, however, as to the speed at which clicks must be made in order to not miss this specific type. I hope I was clear to what I wanted
Dear Ben, amazing video! I had one quick question about the beginning (part 1) of this series: the results don't show the whole picture, but only the top left area of the original (I saw this is the same for you in this video). My screen is 1920x1080 And I search an image around pixel(1900,1000), however, this part of the picture is not in the result (and therefore also not in the max_loc and max_value). Have you got any clue how to solve this? Huge thankyou in advance, kind regards. You're doing amazing!
One if the issues i've faced jus to share is that when importing pyautogui in main.py, it changes to 100% dpi when the windows system is at 125% dpi. Will need to change your system to 100% dpi if not matches screen just has the actual screen on top left.
I had the same problem, but one possible solution(which I used) is that if you know the area the match will show up(for example gui), try limiting the search area to be smaller and to the general vicinity of the occurance. This is due to the fact that match template is essentially opencv matching pixel for pixel to check for an exact match(to my understanding, maybe wrong). so if the pixel count of your 'haystack' is smaller, the less it impacts the framerate(the faster it searches through the image). An alternative solution other than limiting the size of the 'haystack' is to reduce the resolution of both the 'haystack' and the 'needle', putting them under the exact transformation(or the matching won't work) or just extract your needle from the already processed 'haystack', this solution is less accurate, if the resolution is too low, the exact set of the pixel will have a higher chance to occur therefore picking up false positives. Either way, the goal is to reduce the total pixel count and dimension for opencv to process through to reduce the processing time. Hope that solves your probelm~
So with the fix at the beginning of this video, I'm able to make my program work. But the issue being that if I want it to work effectively, as in the screen is still usable while its doing its thing, I need to be able to capture that window specifically. So my question is, understanding that this has aged, has there been a development in finding a solution to do just that? My bot currently searches my game to determine when a match queue has popped and then accepts it for them. So not worrying about covering the screen would be a boon.
How could I check a specific pixel color? Do I need to use coords from the game window or the window capture window. How would I do that? Also amazing video! was thinking prayer_loc = prayer_orb.find(screenshot,.8,'points') new_cord = (prayer_loc[0][0],prayer_loc[0][1]) new_x,new_y = wincap.get_screen_position(new_cord) if np.any(screenshot[new_x+12, new_y] == 0): prayer()
Hey after all the coding I get this error: cv2.error: OpenCV(4.5.3) C:\Users unneradmin\AppData\Local\Temp\pip-req-build-q3d_8t8e\opencv\modules\imgproc\src\templmatch.cpp:1164: error: (-215:Assertion failed) (depth == CV_8U || depth == CV_32F) && type == _templ.type() && _img.dims()
I've been trying to implement this with RuneLite and I've run into some.. interesting issues. Initially, it captured perfectly. Then when I shifted some files around it broke and was capturing a black screen. I wasn't concerned because I read the pinned comment on the previous video. Now however, the capture is artifacting as if there was a GPU driver failure. It's very bizarre and I'm not quite sure how to fix it.
Hey Ben, came back here to ask another question :) 1. Lets say my screen is 1920x1080. Now, if I only want to capture and process 100x100 pixel of the center of my screen - will performance (time to capture and process - essentially FPS) will be faster/higher? 2. Regarding the same subject - how exactly do I only capture a certain part of the screen (lets stick with the example of that 100x100 in the middle of my screen)? I've noticed I can play with the self.w and self.h (and the offsets) of the WindowCapture class but I always have a black screen leftovers and the FPS seems to be low/slow. love your vids! Thanks!
@@alekaditez3830 not necessarily an aimbot, but a snap-to-target in a certain radius of it (on the screen). So not the "usual" aimbot you were thinking of but tnx :)
I'm uncertain if I'm just not understanding this correctly but for the lines that specify the needle_img_path and haystack_image, etc, am I supposed to be substituting say haystack for screenshot, if so where is that specified, because as far as I can tell it isn't specified in main or in get_screenshot in windowcapture? As well, do I need to change the needle_img to whatever my template is? For instance in the self.needle_img = cv.imread(needle_img_path) should it be self.needle_img = cv.imread('template.png', cv.IMREAD_UNCHANGED). If anyone knows the answer to this it would be greatly appreciated thanks.
Nevermind, I got it solved, I came upon the issue of it struggling with the match template feature but I solved it by taking cv.IMREAD_UNCHANGED out of the cv.imread part.
@@tinatipton3291Thank you so much I had the cv2.error: OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\templmatch.cpp:1164: error: (-215:Assertion failed) (depth == CV_8U || depth == CV_32F) && type == _templ.type() && _img.dims()
Hey I have a question. My code detects the objects then I have it set up to click on the object. However it starts spam clicking because it keeps continuing through the while loop. Ideally I’d like to be able to run a sleep function after I click and detect again for another needle after a few seconds, but when I try that the screen capture window crashes.
I am trying to combine this with pyautogui to click on the object that vision detects. I am finding if I just use the cordinates that vision returns, the mouse moves to a point above what I want to click. I just did some subtraction to adjust for this but I can't figure out why this adjustment is necessary.
I've got something I would love to have you try. Its a problem I've been working towards solving with opencv image recognition. Can you think of a way to ignore transparent pictures of an image? I'm trying to find a near stationary object but the background around the object and moving and changing. Any suggestions for this issue would be fantastic. I can get it to recognize sometimes if I check multiple images but often its too slow and misses the object before it disappears.
Hey so I just noticed that matchTemplate() has an optional mask parameter. I think if you provide that mask to ignore the parts of your needle image that should be transparent, that should give you the results you're looking for. If you try this out let me know how it goes!
I would like to count how many times player click on "buy" button. But once "buy" button is hold, count is double up, triple up and so on. It is possible to manipulate somehow so that it is counted once, or i should implement any "on click" event? regards
Any chance you can help with this? I am getting this error after changing the Window name and changing the image used to check the screen shot with. error: (-215:Assertion failed) points = vision_limestone.find(screenshot, 0.5, 'rectangles') result = cv.matchTemplate(haystack_img, self.needle_img, self.method)
I have a weight trained for chickens in rs, its 300mb or so though and seems pretty slow to load, some people have made weights down to 30mb that work somehow. Opencv is probably your best bet for now, not to mention getting a tensorflow version, cuda, etc to all work together is a nightmare.
EX E would you be able to answer a couple questions I have about opencv? I’m trying to do a similar thing as a first starter project but feel like I’m missing something. If ya can add me on discord: ToeAsiago#4748
I have been developing Python programs for a year, and have studied this subject for many many years before. I just wish every single tutorial on difficult topics were as good as this series is. Absolutely amazing. All the important details are here and important parts explained, I never had this much of an easy time following a coding series.
Looking sharp with the new hair my man!
Thanks! Pretty stoked that my haircut placed opened back up last week.
Awesome Video. Please how can i increase FPS ? Im running at 1.5fps. all functions works but fps its low. help please
It has been two years already, where is the continuation? Why did you abandon your channel? You explain better than all my teachers! You're the best! It has been 2 years already, perhaps you have improved object detection, bot performance, and their movement between points, or maybe there's something else interesting to see!
i feel the same man
Yeah, finally a new video. I've been waiting for this video for 2 weeks. When are new videos coming?
Glad you look forward to these! I've got a big work project due at the end of the month, so it'll probably be another two weeks before the next one unfortunately.
Your videos are better than any paid service. This is great!
Your contents are top notch. You deserve more subs!
Thanks!
Hey, I'd like to thank you very much. I've been wanting to learn to code for a while but have felt so small compared to how powerful programming can be. I'm extremely thankful for your videos. I'm now taking what I've been learning from you and other talented people to help me with my robotic/AI ideas I've been dreaming of for a long time now. Who knew it took learning code by playing games would be one of the best ways for me to learn but it helped a ton. Thank you! I still get annoyed with the code still but I take a break and a few hours or the next day I've figured it out.
Taking breaks is an important part of coding, for sure. Glad my videos have been helpful for you!
@@LearnCodeByGaming hello bro would you give me a hand? A thousand thanks
@@LearnCodeByGaming Hello bro! Awesome video. I'm learning a lot from it. Whell, i'm trying to crop the screesnhot to read just re LifeValue with pyteserract. But when i try, i cannot cut the image properyly
cDC.BitBlt((0, 0), (self.w, self.h), dcObj, (self.cropped_x, self.cropped_y), win32con.SRCCOPY)
When i change this, nothing changes. Please help 😁🙏
I already have a code that works, from the another project which saved the screenshot in HD or SSD. but too slow. Then when i try in this Real-Time the FPS goes from 4 to 0.70. I will try to do some Multiprocessing to this slow function run alone
I love the tutorials and I am just watching and not even coding along (I only program in Java)
I created a farming bot for an mmo that clears a stage of bad guys, and at the end runs around to look for a relic and go to it to leave the mission. I used 3 needle images at different distances, but it is extremely hard for the bot to find since the relic has fire coming out of it and the level is on a fiery magma rock surface. Im really excited for the next video, because this has really been the final frontier for my bot that I can't seem to get through as well as id like. Thank you for the great content.
That sounds awesome! I think I've got some ideas in the next couple videos that will help you.
I stopped already coding 10 years ago and doing other things. I started playing a game some month ago which consumes a lot of time and look for opportunities for automate it... and agian think about to start coding with the new tools, like py. Its a loop which will never ends... thank you for sharing that tutorial.
Games are supposed to be fun, I hate when they add grinds. But then automating the grinds can become more fun than the game itself. Thanks for watching!
you're one of my favorite programmers on youtube! thanks for the video and nice hair!
Wow this video is worth watching. I can't believe that a lecture with this quality is just uploaded on RUclips for free!
I nearly cried when you said the solution for the black screen issue is to just default to a full screen capture. The specific game I'm trying this with won't work with that method, and it also has the black screen bug, so I guess I just can't do my project. This video series is like the 20th resource I found for this and looked the most promising. If this is the only solution you have, I guess I'm on a fool's errand here
Finally dude i have been waiting for this so long
I have watched the video entirely but don't stop this series please.
Waiting for next one,
by the way, nice video
You're an EXCELLENT teacher. Thank you!
Very nice series, I'm really enjoying it and learning at lot too. Thanks for putting all the effort into these videos. I'm strugling a bit to adapt the WindowCapture code to my Ubuntu OS, but hopefully I will get there. I can feel Deep Learning coming and I can't wait to see it :)
man this channel is fooking amazing
extremely underrated
tyy benn mateee!
You're welcome!
This is great stuff, commenting so youtube algorithms show it to more people :)
I learned more on Python from you than in our actual university. lmao
I'm 3 years late but thank you for making this series.
Cant wait to watch the video. Thanks Ben!
No problem, thanks for watching!
Yes it's here!!!!!!!! Thanks Ben
Yes, finally here! Thanks for watching Joe
@@LearnCodeByGaming Setup an interesting function to help randomize click locations for targets found with the opencv techniques you've gone over. Hoping this will help with potential detection for services I may or may not be using this technology with ;)
@@s1ack3r07 Awesome!
@@s1ack3r07 you could use pyautogui click relative position function, and use the random library withing a range to click it.
Please come back, we miss you
You are awesome "Ben", thanks for amazing learning session!
you should definitely try machine learning with yolov5 for object detection. this is much more powerful than template matching and can address some of the issues you mentioned at the end. also would make for a great vid! solid video already, well deserved sub!
any resources on where to start on that @EGK?
If you're having trouble viewing your vision window because it's too large, you can resize your screenshot just before calling imshow by using
screenshot = cv.resize(screenshot, (new_width, new_height))
As usual, amazing content!
Man, you are amazing, as a beginner I learned a lot.
Careful around 7 minutes when he's first testing the live capture of the new image, in the previous tutorials he used PNG and made note of it as important. In this one he's switched to JPG, and CV2 will error if you use PNG as it's expecting the images in certain formats.
i do have the exact problem now, but i dont understand where i have to change something, because at that point there is nowhere any mention of filetype
@@locutusvonborg2k3 Don't change anything, just make sure your needle image is in JPG format
took me 5 hours to figure this one out
Great video, can't wait for the next
quick tip: use Windows+Shift+S to grab cropped screen regions directly without the need to fiddle with ms paint afterwards
I have been following line by line. This is great!
Yes!
Glad you like these!
Its cool, man
Please don't stop) waiting for new videos
I'm writing on java , but your videos helping me to understand how it works
Still getting a black screen. The script certanly finds the window, but its all black. Anyone else with the same problem?
yeah i do too
I have the same problem
You can’t minimize the window you’re capturing. Could that be the issue?
@@Brakerjw That's not my issue in any case. Even just maximised and in focus, it's not working
Anyone fond the solution to this problem
Yesssss!!!
OMG, really amazing way to learn python
thanks a lot
This channel is incredibly useful, thanks a bunch bro
THANKS for this AWESOME tutorial!!!!
I really like the way you teach, nice video series!
What I usually do with vision algorithms is wrap the code with try catch blocks and listen for 'keyboard exceptions' or something like that for me to be able to have that "press q to close" functionality instead of sleeping for 1ms with the cv.waitKey() function because that can be hurting your fps. Might be a good thing to try out to improve the performance.
Interesting idea here. Wondering if you could provide a code snippet for this concept. Thanks
Great work Ben !
Thank you!
Dude you’re amazing!
Yaaay! Thanks thanks!!
Thanks for following these videos!
yessssssssss, dudeeee
NICE!!!! I was looking forward to this!!! When the next video will go up? :D
Thanks for keeping tabs on the channel! Will probably be two weeks before the next one is up. Got a big client project due at the end of this month that I need to focus on first.
@@LearnCodeByGaming No problem, I will be here when it comes
Videos are 10/10 great content.
Another great video Ben...
yolov3 object detection tutorial would be great
Yeah it would be! That's definitely on my list of video ideas. Thanks for letting me know what you want to see.
Oh finally we're getting close to the end 😂. Yeah so I really want to make an gathering ai. So as your video has helped me alot since the game I was trying to make an bot is albion online and you're working on it. Sooo if it's ever possible can you make a video where the code identifies the nodes ( which it already does ) and then makes our character move towards it on its own and collect it. All on itself because that's what I wanted to do. And so far I'm also stuck on this place because I don't know coding alot so your video is helping me a lot ....that's why if you can teach how to make the character move towards the identified item that would be helpful because I want to make a lot of ai that do the gathering on their own and staring with albion ...I'll have a great start as it has a lot of problems soo it's a good place to start and you're good at explaining the" why, what and how" of it.....so I'm really looking forward to the movement part of the series
Yeah that's where I plan on ending this series, with the automated collection working. I think it'll be three more videos total.
@@LearnCodeByGaming yeah I saw that . You told that somewhere ...I forgot where I saw that. But I'm following you for this series 😄
I found one (possible) bug that may throw people off when using this version of the code. Please let me know if I'm misunderstanding, or if there is a better way of input validation for matchTemplate.
Here is the scenario,
In vision.py, under the .find() method, line 29
If you start the script using a needle img you know won't work and matchTemplate CANNOT locate the needle image in our haystack window, it will throw an error and stop the script with "(-215:Assertion failed) (depth == CV8U || depth == CV_32F) && type == _templ.type() && img_dism()
I also found this an issue unfortunately. In my case, it also presents an error even when searching for an image that should be in the feed. I am not as familiar with OpenCV, but it might have something to do with dropping the alpha channel and making the image C_CONTIGUOUS because if you comment these 2 lines out: # img = img[..., :3], # img = np.ascontiguousarray(img) in the windowcapture.py file, the program will run. Doing this however, I am now experiencing a new issue which I haven't had time to look into yet where it fails draw rectangles around an object the system logs as detected.
@@orion7317 also having this issue and don't really understand how to get past this and I'm also sure the issue has to do with the changes done to the img in the windowcapture class. Please let me know if you guys came up with a more functional workaround please!
I deleted img = img[...,:3] and img = np.ascontiguousarray(img) from window capture class and now it somewhat works, the fps is down from 30 to 5, but no window pops up
@@jugarucristi1 so its pointless
@@Daniel-nv9fl Great its also work for me, thank you! I notice when my needle image is PNG I need comment or delete line with img = img[...,:3] and img = np.ascontiguousarray(img) in windowcapture.py. But if I have image in JPEG format I don't need to delete or comment it.
@Alekaditez Ben in video also has FPS drop when start program so it's part of journey :)
Thanks man. Really helpful
Can't get my Find click positions right. I'm getting back an error:
result = cv.matchTemplate(haystack_img, needle_img, cv.TM_CCOEFF_NORMED)
cv2.error: OpenCV(4.4.0) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-h4wtvo23\opencv\modules\imgproc\src\templmatch.cpp:1163: error: (-215:Assertion failed) (depth == CV_8U || depth == CV_32F) && type == _templ.type() && _img.dims()
You are a legend.
you also could do what i did in a game, just modify a image file of the thing you search in the game, like a mod, and the n recognize that. Also, you could use a lot of templates for searching in a for loop, and when one templates matches, then use that template only until the algorithm doesnt recognize the template anymore because of changes in day, night, etc, this way you could keep your performace but also make the code adaptable to the situations, thanks for the tutorials, you rock man.
what do you think of those sugestions ?
Yeah those are really good ideas!
@@LearnCodeByGaming thanks man, keep the good work, i am a big fan.
What would the for loop look like? I have a pretty good script down but I’m tying to understand an efficient way to cycle through multiple needle images or templates or something
@@hiscockanthony97 it is hard to explain in words XD, but you could make a for loop that runs trough a list of your needle images and make a recognition, if the recognition is done with one image, break the for loop, return the image you need in a variable manner. and enter an outer while loop where you use the image that generated a match for image recognition till you cant find the image again, then you do the for loop again, it could be done inside a function
Thank you dude
bro you do an amazing work, trank you!!
YOUR CODE IS GENIOUS
Awesome videos! Is Video #6 coming soon? Would like to learn how to click random points within the object and learn how to create steps. such as,
Step 1.Find Wood
Step 2.Click Wood
Step 3.Gather wood
Step 4. sleep for 10 minutes
Start all over again.
excellent content, congratulations
Great videos! Do you plan to use this with machine learning?
Thanks! I'm not going to get to machine learning in this series, but I definitely want to do a lot of machine learning content on my channel.
Dude you are genious!
When I run the code instead of one window a lot of windows open up. How can I fix this?
Hey again Ben, needless to mention this series is just epic! Thanks!
I have a few questions.
1. For simplicity, let's say u want to detect one of your 2D icons from that top right menu bar (for example that mail icon). After you took a screenshot of that icon and implemented this into your code, you are now successfully detecting the icon. Now, if I have a different monitor with different resolution (thus size matters) and I'm trying to run the same code with the same icon screenshot, it wouldn't work, right? So my question is, how can take a picture of an object and make sure it will be detected among all screen resolutions?
Maybe there's a way to convert the sampled image (the mail icon) and the surface image (the game screen frame) to some sort of a "binary pattern" that doesn't rely on pixel color so that they will be treated equally and the size won't matter?
2. How do you use a png image (transparent background, and non-'squared') as your target object to detect?
Thanks!
Hey Shay. For #1, I can't find anything OpenCV has built-in for that kind of multi-scale template matching, so you'd need to do it yourself. Basically you'd make a loop that resizes your needle image, looks for that using match template, and then continues that process until your image is found. That might be faster if you first convert the images to black and white (so the binary pattern like you suggest) using something like canny edge detection.
For #2, my understanding is that OpenCV can load transparent png files no problem with imread(), as long as you provide the IMREAD_UNCHANGED flag. But imshow() and matchTemplate() will give you some trouble. Match template has an optional mask argument, and I think you'd need to provide that along with your needle image in order to correctly ignore those pixels that aren't of interest.
@@LearnCodeByGaming I've already found a possible solution to (1) that indeed involves scaling the image up/down. Working on it a the moment. Thanks a lot man!!
@@ShayRubachOfficial Hi. Did you find a way to use a png image as the target?
@@s0ld1er16 oh that was so long ago I couldn't even remember :/ sorry
@@ShayRubachOfficial what about first question? :D
I got a black screen again
I have had an issue with Chrome giving a black screen when captured. If you disable Hardware Acceleration, the capture of Chrome as a window works fine.
Thank you sensei.
This series is great! I have followed along and learnt a lot, but I have ended up stuck! I am getting an error even if I copy and paste your code, for a opencv assertion around "(-215:Assertion failed) (depth == CV_8U || depth == CV_32F)...."
I am struggling to fix this so bad!
Funnily I just figured it out! it was an issue inside of the __init__ for the cv.imread. I fixed it by removing the "cv.IMREAD_UNCHANGED"
@@PanicNvm What did you change it too? I'm having the same issue
@@matthew4030 I just removed it 🙂
@@PanicNvm Perfect thanks so much! I'm working again!
@@PanicNvm do u mean inside the vision __init__ im struggling pretty hard with this too
Awesome!
Solution to this error "error: (-215:Assertion failed) (depth == CV_8U || depth == CV_32F) && type == _templ.type() && _img.dims()
Thank you!! I was about to have a mental breakdown trying to figure out what the issue was LOL
Nice
Ur my inspiration
I have a black Result Screen what I make false?
did you fix this issue, i got black screen too.
Great tutorial series! I have a question...What if I wanted to capture from a my Avermedia Capture Card instead of game window?
GREAT!
very nice explanation
Can you explain how to send commands via adb?
window capture is not working for me, i think its because the window name is too long? is there a way i can make it find the process instead?
Would the image detection method be effective for detecting in-game text? Specifically, the goal is to locate an item named [DARK ARMOUR], which has a 1/100 chance of appearing on the screen when clicking. To achieve this, it is necessary to click rapidly to cycle through all ARMOUR types until [DARK ARMOUR] appears on the screen. The question remains, however, as to the speed at which clicks must be made in order to not miss this specific type.
I hope I was clear to what I wanted
Hi! I did everything just as in the tutorial, but when capturing the Google Chrome window I still have only black screen. Any tips how to fix it?
Dear Ben, amazing video! I had one quick question about the beginning (part 1) of this series: the results don't show the whole picture, but only the top left area of the original (I saw this is the same for you in this video). My screen is 1920x1080 And I search an image around pixel(1900,1000), however, this part of the picture is not in the result (and therefore also not in the max_loc and max_value). Have you got any clue how to solve this? Huge thankyou in advance, kind regards. You're doing amazing!
Good stuff
One if the issues i've faced jus to share is that when importing pyautogui in main.py, it changes to 100% dpi when the windows system is at 125% dpi.
Will need to change your system to 100% dpi if not matches screen just has the actual screen on top left.
Yeah that's good to note. I think Windows defaults to 125% for some reason and that has caused problems for many people.
amazing thank u
Is it possible to feed the program a file of "needles" (multiple images) without an impact to frame-rate?
Thanks for the video!
I had the same problem, but one possible solution(which I used) is that if you know the area the match will show up(for example gui), try limiting the search area to be smaller and to the general vicinity of the occurance. This is due to the fact that match template is essentially opencv matching pixel for pixel to check for an exact match(to my understanding, maybe wrong). so if the pixel count of your 'haystack' is smaller, the less it impacts the framerate(the faster it searches through the image). An alternative solution other than limiting the size of the 'haystack' is to reduce the resolution of both the 'haystack' and the 'needle', putting them under the exact transformation(or the matching won't work) or just extract your needle from the already processed 'haystack', this solution is less accurate, if the resolution is too low, the exact set of the pixel will have a higher chance to occur therefore picking up false positives. Either way, the goal is to reduce the total pixel count and dimension for opencv to process through to reduce the processing time. Hope that solves your probelm~
@Learn Code By Gaming Can we draw these rectangles as an overlay on the actual game screen instead of inside the python window?
Interesting idea...so you can just watch what the bot is thinking as it works...
When Vision appears, I get a black screen. How can I fix this please!
So with the fix at the beginning of this video, I'm able to make my program work. But the issue being that if I want it to work effectively, as in the screen is still usable while its doing its thing, I need to be able to capture that window specifically. So my question is, understanding that this has aged, has there been a development in finding a solution to do just that?
My bot currently searches my game to determine when a match queue has popped and then accepts it for them. So not worrying about covering the screen would be a boon.
I changed the image as well as the window where it is supposed to look for it but it gives me errors I don't even know.
How could I check a specific pixel color? Do I need to use coords from the game window or the window capture window. How would I do that? Also amazing video! was thinking
prayer_loc = prayer_orb.find(screenshot,.8,'points')
new_cord = (prayer_loc[0][0],prayer_loc[0][1])
new_x,new_y = wincap.get_screen_position(new_cord)
if np.any(screenshot[new_x+12, new_y] == 0):
prayer()
Hey after all the coding I get this error:
cv2.error: OpenCV(4.5.3) C:\Users
unneradmin\AppData\Local\Temp\pip-req-build-q3d_8t8e\opencv\modules\imgproc\src\templmatch.cpp:1164: error: (-215:Assertion failed) (depth == CV_8U || depth == CV_32F) && type == _templ.type() && _img.dims()
hi , it worke once for me, now i have just black creen ny ideas ?, sometims it show statc img from other window
I've been trying to implement this with RuneLite and I've run into some.. interesting issues. Initially, it captured perfectly. Then when I shifted some files around it broke and was capturing a black screen. I wasn't concerned because I read the pinned comment on the previous video. Now however, the capture is artifacting as if there was a GPU driver failure. It's very bizarre and I'm not quite sure how to fix it.
Hey Ben, came back here to ask another question :)
1. Lets say my screen is 1920x1080. Now, if I only want to capture and process 100x100 pixel of the center of my screen - will performance (time to capture and process - essentially FPS) will be faster/higher?
2. Regarding the same subject - how exactly do I only capture a certain part of the screen (lets stick with the example of that 100x100 in the middle of my screen)? I've noticed I can play with the self.w and self.h (and the offsets) of the WindowCapture class but I always have a black screen leftovers and the FPS seems to be low/slow.
love your vids!
Thanks!
If you're trying to make an aimbot you want to be capturing your entire screen. Other than that I'm not sure.
@@alekaditez3830 not necessarily an aimbot, but a snap-to-target in a certain radius of it (on the screen). So not the "usual" aimbot you were thinking of but tnx :)
I'm uncertain if I'm just not understanding this correctly but for the lines that specify the needle_img_path and haystack_image, etc, am I supposed to be substituting say haystack for screenshot, if so where is that specified, because as far as I can tell it isn't specified in main or in get_screenshot in windowcapture? As well, do I need to change the needle_img to whatever my template is? For instance in the self.needle_img = cv.imread(needle_img_path) should it be self.needle_img = cv.imread('template.png', cv.IMREAD_UNCHANGED). If anyone knows the answer to this it would be greatly appreciated thanks.
Nevermind, I got it solved, I came upon the issue of it struggling with the match template feature but I solved it by taking cv.IMREAD_UNCHANGED out of the cv.imread part.
@@tinatipton3291Thank you so much I had the
cv2.error: OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\templmatch.cpp:1164: error: (-215:Assertion failed) (depth == CV_8U || depth == CV_32F) && type == _templ.type() && _img.dims()
@@h4m719 it fixed? I need help it
Hey I have a question. My code detects the objects then I have it set up to click on the object. However it starts spam clicking because it keeps continuing through the while loop.
Ideally I’d like to be able to run a sleep function after I click and detect again for another needle after a few seconds, but when I try that the screen capture window crashes.
I am trying to combine this with pyautogui to click on the object that vision detects. I am finding if I just use the cordinates that vision returns, the mouse moves to a point above what I want to click. I just did some subtraction to adjust for this but I can't figure out why this adjustment is necessary.
Someone else had this problem and it turned out they had their zoom level set to 125% somewhere in the Windows settings. You might look into that.
I've got something I would love to have you try. Its a problem I've been working towards solving with opencv image recognition. Can you think of a way to ignore transparent pictures of an image? I'm trying to find a near stationary object but the background around the object and moving and changing. Any suggestions for this issue would be fantastic. I can get it to recognize sometimes if I check multiple images but often its too slow and misses the object before it disappears.
Hey so I just noticed that matchTemplate() has an optional mask parameter. I think if you provide that mask to ignore the parts of your needle image that should be transparent, that should give you the results you're looking for. If you try this out let me know how it goes!
I would like to count how many times player click on "buy" button. But once "buy" button is hold, count is double up, triple up and so on. It is possible to manipulate somehow so that it is counted once, or i should implement any "on click" event? regards
Any chance you can help with this? I am getting this error after changing the Window name and changing the image used to check the screen shot with.
error: (-215:Assertion failed)
points = vision_limestone.find(screenshot, 0.5, 'rectangles')
result = cv.matchTemplate(haystack_img, self.needle_img, self.method)
genius
Please go train a model and search for it in Albion
Ganja Kurier Good Idea the last Tutorial is 3 Years old.
I want to cover a few other techniques first, but I may get to machine learning on this project down the line. Thanks for watching!
@@LearnCodeByGaming I hope so because your videos are the bset
I have a weight trained for chickens in rs, its 300mb or so though and seems pretty slow to load, some people have made weights down to 30mb that work somehow. Opencv is probably your best bet for now, not to mention getting a tensorflow version, cuda, etc to all work together is a nightmare.
EX E would you be able to answer a couple questions I have about opencv? I’m trying to do a similar thing as a first starter project but feel like I’m missing something. If ya can add me on discord: ToeAsiago#4748