Conductor
Try For Free

Robots.txt example file

A lot of people look for example robots.txt files. Rightfully so, because robots.txt are easy to mess up.

In this article we'll cover a simple and a slightly more advanced example robots.txt file.

A simple robots.txt file

Here's an example of a simple robots.txt file that:

  • Allows all crawlers access
  • Lists the XML sitemap
User-agent: * Disallow: Sitemap: https://www.example.com/sitemap.xml
Audit your robots.txt and find out right away!

A slightly more advanced robots.txt file

Here's an example of a slightly more advanced robots.txt file example that:

  • Prevents Googlebot from accessing /shoes/
  • Prevents Bingbot from accessing /socks/
  • Lists the XML sitemap
User-agent: googlebot Disallow: /shoes/ User-agent: BingBot Disallow: /socks/ Sitemap: https://www.example.com/sitemap.xml
Read the full Academy article to learn everything about Robots.txt for SEO: The Ultimate Guide
Share Article Details

Ready to maximize your website's potential?

Try Conductor Intelligence free for 30 days
TrustRadius logo
G2 logo
SoftwareReviews logo