WordPress uses a virtual robots.txt file

Posted in Articles

In the usual case, if you go to the home directory of your WordPress installation, you will not find a robots.txt file.  Yet however, if you navigate to the usual location of the robots.txt file in your browser, you will see something there — for example, say http://www.example.com/robots.txt (where “example” would be your own domain name).

And this “file” is seen by search engines.  This is because WordPress generates an virtual robots.txt file.  There is no actual “file”.  But the browser does see it as a file.

The search engine will see the virtual robots.txt file if and only if you do not have a real “robots.txt” file in the WordPress home directory.  If you do have an actual robots.txt in the WordPress home directory, then search engines would see that one instead.

To see which robots.txt file is seen by search engine, use your browsers address bar — for example, say http://www.example.com/robots.txt (where “example” would be your own domain name).  Search engine would see what you see there.

---