How do I find my robots.txt file?
Finding your robots.txt file on the front-end
Crawlers will always look for your robots.txt file in the
root of your website, so for example:
https://www.contentkingapp.com/robots.txt. Navigate to your domain, and just add "
In case you want to update your robots.txt file: keep on reading.
Finding your robots.txt file on the back-end
If you're using a Content Management System (CMS), you may be able to manage it in there.
Your robots.txt in WordPress
If you're working on a WordPress website that hasn't gone live yet, and you find the following contents in your robots.txt:
Then check your settings under:
Reading and look for
Search Engine Visibility.
If the box
Discourage search engines from indexing this site is checked, WordPress will generate a virtual robots.txt preventing search engines from accessing the site.
In Magento 2, your robots.txt file is located here:
- There, edit the settings for
Main Websiteinstead of the
Default Store View, because the latter won’t allow you to change the robots.txt file.
- On the screen that follows, scroll down to
Search Engine Robots. There you can define your robots.txt content:
In Shopware, it's currently not possible to adjust your robots.txt file with the default installation.