Wordpress Robots.txt Best Practices - How to Optimize WordPress Robots.txt [Still true in 2023!]

Improve your site management with these WordPress Robots.txt best practices. The robots txt file, if one exists on your site, tells bots what they are and are not allowed to access. These bots include search engine crawlers, SEO software and even malicious software.
Discover how to create and optimize the WordPress robots.txt file in this tutorial. This tutorial is the robots.txt file explained for beginners.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: www.example.com...
Grab your free 17-Point WordPress Launch Checklist PDF: wplearninglab....
Robots.txt Google Search Console link: support.google...
Black Hole for Bad Bots tutorial: • [Real Results] Black H...
Always backup your site just in case (or use a staging site), here's how: • WordPress Backup, Rest...
Join our private Facebook group today! / wplearninglab
Here are 20+ reasons why I host all my sites with SiteGround: wplearninglab....
Post videos of your WordPress success using the hashtag #WPLLCommunity!
Here's the link for Elementor Pro (aff): wplearninglab....
I hope this information helps you! If you have any questions leave a comment below or ping me @WPLearningLab on Twitter.
WP Learning Lab Channel: www.youtube.com...

[@wplearninglab]

If you enjoyed this video don't forget to the like it and subscribe! Then check out this playlist about speeding up WordPress: ruclips.net/video/cBDY-CXWmI0/видео.html

[@franshescomviran5741]

Bloody brilliant mate! Cheers from Australia! 🎉

[@weldingempire]

Thank you so much. I was having a lot of trouble trying to have ahrefs crawl my site and adsense. wonderful and clear tutorial.

[@17aig]

As always thank you so much for this tutorial

[@wplearninglab]

You're welcome Nissim, thanks for watching! Let me know if you have any questions :)

[@personaldronerepair6141]

You Rock !!
Thank you for your efforts and time in!!

[@wplearninglab]

Thanks, you rock too!

[@WilliamFernandesvideos]

Congrats and thanx !

[@wplearninglab]

Thanks and thanks for watching!

[@TheBlades2012]

Hi, When I put my sitemap on just like you did it comes up with the red X on the left?

[@victoragburum1858]

Every time I crawl my site with semrush, it keeps detecting errors that my some pages could not be crawled, pls does anyone have an idea how to fix that issue

[@worldtraveler303]

Thanks so much!!!! 🙏🏿🙏🏿🙏🏿🙏🏿🙏🏿🙏🏿

[@rgkh1503]

Hi there, thanks for the video, but this seems not to be the best practice anymore (cf usage of meta tags)..any update of this video planned?

[@Lettucelu]

Thank you!!!!

[@MH-jp1fv]

awesome thank you!

[@Electronzap]

Good stuff :)

[@wplearninglab]

Thanks Paul and thanks for watching!

[@mahmoudgh2030]

Thanks

[@tevinkiprop1883]

Hey so I have a large website and when I click on to coverage I have the discovered- but not indexed yet message also when I try to use the URL inspection it displays noindex. kindly help since only a few of my pages are being indexed

[@shelbythomas7744]

What is your favorite contact form and spam blocker for that form?

[@wplearninglab]

Happy forms is pretty awesome and the free version is quite powerful.
For free forms I also like Contact Form 7. I know it's old and a bit clunky, but you can build out lots of simple form types and there are plenty of addons for more functionality.
If the form needs to have calculations done on data entered in the fields I like Gravity Forms.
I like Recaptcha v3 and/or honeypot fields to reduce spam.
..I hope that helps, let me know if you have any further questions. Thanks for watching!

[@17aig]

Could you please add links to the Robert text tester and the blackhole Thanks in advance

[@wplearninglab]

For sure, here you go:
Robots.txt Google Search Console link: support.google.com/webmasters/answer/6062598?hl=en
Black Hole for Bad Bots tutorial: ruclips.net/video/0KAzGDHq-1M/видео.html

[@nachogalvezantunez1915]

Great Bjorn! Really thankfull! My site got broken with the "500 Internal Server Error" after adding the "Really Simple SSL" Plugin to my Wordpress and Google Search Console is telling me that's because my server (at Google Cloud Platform) bloqued Google search engines (Googlebot) so they can not find my domain. Google Search Console though the robots.txt taster tool is prompting me to go back to the last usefull "robots.txt" downloading it to my computer and uploading it to my server. I can´t find any "robots.txt" file at my "html" directory; acording to your video I just have to upload the one downloaded from "robots.txt taster" tool, is this right? I really apreciate very much in advanced and hope you can help me. Best regards.

[@frankstone]

Thanks to your previous tutorial, I have installed IThemes Security and activated the "hide backend" feature. Will the robots.txt file provide additional site security that is not already installed?

[@wplearninglab]

Hi Frank,
If you use the "hide backend" feature of iThemes Security you wouldn't need the user agent rules in the robots.txt file. The only thing that may benefit you is having the link to the sitemap in the robots.txt file. But you can also add your sitemap directly to the webmaster tools in major search engines instead.
I hope that helps :) Let me know if you have any further questions. Thanks for watching!

[@mustafatankus9343]

hello, only the homepage of my site comes up in google searches. None of the articles on my site appear in google searches. Do you have any idea about the reason? I registered the property on google search console, submitted the sitemap to google. I checked the robots.txt file but I can't seem to solve this problem in any way.

[@budsgrower9970]

I have the same problem. I hope someone will answer

[@GBFragrances]

Hi! How do I edit the robot txt file that's blocking Google to crawl my pages?

[@Cognitoman]

Should be in the public part of the website

[@siddhant2002]

Do you have any video to how to make money as wordpress developer

[@wplearninglab]

Hi Sidhant,
I don't at the moment. From my experience, you'll need to (1) have the development skills, which it sounds like you have and then (2) you have to go out and find clients. Part 2 is the hardest for most people. You have to go where your ideal clients meet up (conferences, meet ups, etc.) or you have to go door to door with an offer they can't refuse.

[@pahtashow]

How to make a sitemap in 2022 wordpress ?

[@guadadis]

Hi! Do you know what could be happening with my file robots.txt? Every so often it automatically switches to
user-agent: *
Disallow:

[@mauwiks]

Mine too

[@Cognitoman]

Prolly something with your server from your hosting company

[@starsproduction9676]

It's the robot.txt make your blog post show in search results