Robots.txt Feature in SpeedBoostr
The Robots.txt feature in our SpeedBoostr app helps you control how search engines (such as Google, Bing, and others) crawl and index your website.
What Are the Benefits?
- Prevent unnecessary pages from appearing in search results.
- Optimize your crawl budget so search engines focus on your most important pages.
- Reduce duplicate content issues by blocking low-value or duplicate pages.
- Help improve your website's overall SEO performance.
The SpeedBoostr Robots.txt tool allows you to manage robots.txt rules, ensuring that search engines crawl only the pages that are important for SEO. This gives you better control over your website's indexing and crawling behavior.
1. User-Agent (Meta Tags)
Here, you can select which search engine bot the rule should apply to.
Example:
•All bots (*) → Applies to all search engine bots.
• Googlebot →Applies only to Google.
• bingbot →→ Applies only to Bing.
•Other specific bots can also be selected.
2. Rule Type
There are two available options:
Allow
This tells the selected search engine bot that it is permitted to crawl the specified page or directory.
Disallow
This prevents the selected search engine bot from crawling the specified page or directory.
Important :Disallowing a page does not always guarantee that it will be removed from search results. If Google discovers the page through other sources (such as backlinks), the URL may still appear in search results even though its content is not crawled.
In simple terms:
- Allow =Search engines can crawl the page.
- Disallow Search engines are instructed not to crawl the page.
3. URL Path
In this field, you need to enter the path of the page you want to control.
After entering the URL path and selecting the desired rule type (Allow or Disallow), click the Add Rule button.
The selected rule will then be applied to that specific page.
Example:
- If you choose Allow, search engines will be permitted to crawl the page.
- If you chooseDisllow, search engines will be instructed not to crawl the page.
4. Sitemap (Optional)
Here you can add your sitemap URL.
A sitemap is an XML file that helps search engines understand which pages are available on your website and should be crawled and indexed.
In simple words, a sitemap acts as a roadmap for search engines.
Benefits of Adding a Sitemap
- Makes it easier for search engines to understand your website structure.
- Improves indexing of products and website pages
- Helps Google discover new pages faster.
Business Benefits of Robots.txt
1. Prevents Google from wasting resources crawling unnecessary pages.
2. Ensures crawl budget is spent on important pages.
3. Can improve indexing of product and collection pages.
4. Helps reduce duplicate content issues.
5. Provides better control over how search engines access your website.
Robots.txt File Section
In this section, you can view all the rules that have been created for search engines.You will be able to see:
- URL Path
- Status
- Time
- Rule Type
- Actions
When you click the Action arrow, you can view all rules configured under that user-agent. You can:
- See which URLs are allowed or disallowed.
- Edit existing rules.
- Delete existing rules.
Important Note
Be careful when configuring robots.txt rules. Blocking the wrong pages may prevent important products, collections, or content pages from appearing in search results. Always review your settings carefully before applying them.