What is Robots.txt & What Can You Do With It
HTML-код
- Опубликовано: 13 июл 2024
- Want to block certain file types or restrict search engine crawlers from crawling certain parts of your website? That’s what a robots.txt file is for & through this video tutorial you will learn how to use robots.txt file like an expert. rankmath.com/kb/how-to-edit-r...
Note: If you do not see the "Edit robots.txt" feature in Rank Math's General Settings, make sure to use the "Advanced Mode" of Rank Math, which can be found at the top-right corner of Rank Math's Dashboard.
Chapters (Timestamps):
00:00 - Introduction
01:25 - Several Use Cases of Robots.txt
03:03 - Language Of Search Engines
03:16 - User-Agent
03:44 - Disallow Rule
03:57 - Allow Rule
04:25 - Crawl-Delay Rule
04:34 - Sitemap Directory
04:44 - Robots.txt syntax
05:43 - The Location Of Your Robots.txt
06:20 - Managing Robots.txt with Rank Math
07:04 - Default Robots.txt Rules
07:31 - The Structure of a Robots.txt File
08:29 - Block Filters and Sorting Pages
09:44 - Block Internal Search Pages
11:20 - Blocking Certain Paths of Your Website
11:45 - Block Certain File Types On Your Website
12:41 - Block All Web Crawlers Except One
13:06 - How To Debug Robots.txt Errors
14:39 - Outro
List Of Search Engine Crawlers And Their Associated Names:
► www.robotstxt.org/db.html
Robots.txt Testing Tool By Google:
► www.google.com/webmasters/too...
We hope to have made this Robots.txt tutorial simple and easy for you to understand and follow. It can seem intimidating at first, but it becomes rather easy if you take some time to understand it. All you need is just to understand the language search engines are speaking.
Anyway, don’t forget to subscribe to our channel if you want to learn more about how to beat your competitors on search engines:
► ruclips.net/user/RankMathvi...
Rank Math, the Swiss Army Knife of WordPress SEO, can take your SEO game to the next level. Rank Math already goes above and beyond other top-rated SEO plugins. Do you want to see the magic unfold for your website?
► Visit Rank Math today: rankmath.com
Here's are 40+ unique features that set Rank Math apart:
► rankmath.com/blog/unique-rank...
Rank Math Links:
► Knowledge Base: rankmath.com/kb/wordpress/seo...
► Support: rankmath.com/support/
Connect with Rank Math:
► / rankmathseopluginwordp...
► / rankmathseo
► / rankmath 
Хобби
Love the way you explained and it's really straightforward
Hello, and thanks for your comment. Glad you liked our video. Feel free to get in touch if you have questions.
Wow, thank you so much, Jack. I was hacked recently and experienced a serious drop in traffic. Your tips are helping me slowly climb back up. The best part is that these minute tweaks actually work.
Hello and thank you for your comment.
Glad that helped. We hope that your experience with plugin excels.
Please feel free to reach out to us again in case you need any other assistance.
It is a very useful video for me. Thank you. 👍
Hello, and thanks for your comment. Glad you liked the video. Feel free to get in touch if you have questions.
In-depth video explaining everything related to robot.txt! THANKS Mate🙌
Hello, and thanks for your comment. Glad that helped. Feel free to get in touch if you have questions.
I appreciate the information
Hello, and thanks for your comment. Glad that helped. Feel free to get in touch if you have questions.
Thanks, it was a very informative video
Hello, and thanks for your comment. Glad that helped. Feel free to get in touch if you have questions.
Jack: May I suggest that you make a new video series about how to optimize rank math schema (I have already seen other existing videos which are hard to follow) in bulk for elementor pro ecommerce pages and sites with many pages? I love your videos and your clear, systematic concise explanations.
Hello, Thank you for your comment.
We have taken note of your topic suggestion.
Thank you so much for the feedback and your valuable suggestions.
We really appreciate them.
Thank you...
Hello, and thanks for your comment. Glad that helped. Feel free to get in touch if you have questions.
thanks ❤❤❤
Hello, and thanks for your comment. You are always welcome. Feel free to get in touch if you have questions.
But I could not understand if I use the wildcard (*) for User-agent, then it includes all bots, and why do I need to include another User-agent, msnbot, in the case as the video shows? I understand it's optional, but is there any use in adding another User-agent if I use a wildcard at the top?
Hello, thank you for your comment.
If you’re referring to the example shown at 3:35, the first rule (User-agent:* Disallow: /) will prevent all search engine crawlers from crawling your website.
The next rule (User-agent:Googlebot Disallow: ) is specifically addressing Googlebot and mentioning there is no directory disallowed, and hence Googlebot can crawl the entire website.
We don’t recommend blocking all search engine crawlers unless it is a test or staging site.
You can also refer to more examples here: rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/
Hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.
thanks
Hello, and thanks for your comment. You are always welcome. Feel free to get in touch if you have questions.
Thanks! This is helpful. Can you also make a video on 301, 302, 303, 304 errors? I struggle to understand the basics of these and how to fix them with Rankmath
Hello and thank you for your comment.
We do have a video discussing 301 & 302 Redirection here ruclips.net/video/yxmbcES_5CA/видео.html
For 303 and 304 redirect types, you can refer to this article: www.contentkingapp.com/academy/http-status-codes/
Hope that helps.
Thank you.
@@RankMath Thank you man! Appreciate it
Awesome video , easy to understand
Nice
Hello and thank you for your comment.
Glad that helped. We hope that your experience with plugin excels.
Please feel free to reach out to us again in case you need any other assistance.
Why do we need to disallow the files from crawlers? What's the disadvantage if they crawl anything and why should it be done??
Hello, and thank you for your comment.
The context in which files were disallowed in this tutorial is that they are supposed to be downloaded by the users after they fill out a form, typically subscription forms. If search engines were to crawl and index these file types, then these files would be available to download directly from search results, and users don't have to fill out the subscription forms.
To answer your second question, if search engines were to crawl everything on the site, including the internal resources, then they would end up causing an increased load on the website's server. Similarly, crawlers do have limited resources for each website, and we want them to be utilized for the most valuable pages on the website instead of thin content or pages that aren't allowed to be indexed.
Hope it helps!
Need more information by using star with example and with all types of use of star and it will implement.
Hello, and thanks for your comment,
With Rank Math you can achieve the star rating for your website using the following Schemas:
1. Book
2. Course
3. Event
4. Movie
5. Product
6. Recipe
7. Software Application
Here is a guide for your reference:
rankmath.com/kb/earning-star-ratings-in-serps/
Hope this helps.
Hi, what if the robots.tx is not writable?
Hello, thank you for your comment,
You can checkout our tutorial on about the topic: rankmath.com/kb/cant-edit-robots-txt/
Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php is causing a google error that robots.txt is blocking wp-admin/admin-ajax.php. How to fix this?
Hello,
Can you submit a support ticket here: rankmath.com/support/ so that our team can look into this closely?
Looking forward to helping you
@@RankMath I submitted the ticket. Thank you.
Ok thanks
To owner of
Google has started validating your fix of Page indexing issues on your site. Specifically, we are checking for ‘Discovered - currently not indexed’, which currently affects 248 pages.
Validation can take a few days; we will send you a message when the process is complete. You can monitor the progress of the test by following the link below.
Hello, thank your for your comment,
Sure let us know how it goes. If you need assistance, please feel free to open a support ticket so that our team can assist you in resolving this issue? Our support staff is available 24/7 to help you. rankmath.com/support/
Looking forward to helping you.
Thanks, my Feeds page have eaten me too deep
Hello, and thanks for your comment. You are always welcome. Feel free to get in touch if you have questions.
4:33
Hello, and thanks for your comment. Feel free to get in touch if you have questions.
I have no idea what you mean
Hello, thank you for your comment,
Do you have issues with robots.txt? Can you submit a support ticket here: rankmath.com/support/ so that our team can look into this closely?
Looking forward to helping you
There is no " Edit robots.txt " option right now. It's really annoying to say I'm not satisfied with your plugin. I must deactivate the plugin. Poor Service and documentation also. shame!
Hello, thank you for your comment,
Can you please ensure you're using Rank Math in the Advanced Mode? Here is the tutorial: rankmath.com/kb/advanced-mode/
In the mean time, you can submit a support ticket here: rankmath.com/support/ so that our team can look into this closely
Looking forward to helping you.