Conductor
Try for free

Robots.txt example file

A lot of people look for example robots.txt files. Rightfully so, because robots.txt are easy to mess up.

In this article we'll cover a simple and a slightly more advanced example robots.txt file.

A simple robots.txt file

Here's an example of a simple robots.txt file that:

  • Allows all crawlers access
  • Lists the XML sitemap
User-agent: *
Disallow: 

Sitemap: https://www.example.com/sitemap.xml
Audit your robots.txt and find out right away!

A slightly more advanced robots.txt file

Here's an example of a slightly more advanced robots.txt file example that:

  • Prevents Googlebot from accessing /shoes/
  • Prevents Bingbot from accessing /socks/
  • Lists the XML sitemap
User-agent: googlebot
Disallow: /shoes/

User-agent: BingBot
Disallow: /socks/

Sitemap: https://www.example.com/sitemap.xml
Read the full Academy article to learn everything about Robots.txt for SEO: The Ultimate Guide
Share Article Details

Ready to maximize your website's potential?

Try Conductor Intelligence free for 30 days
TrustRadius logo
G2 logo
SoftwareReviews logo