Q:
My report shows that robots.txt is one of the most frequently accessed files yet I don't see this file on my web server. I know the purpose of reports.txt but my web developer did not create one for the site. Can you suggest what might be happening?

A:
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Each spider of search engine will access robots.txt at first before access your page files, if robots.txt isn’t exist in your website, spider will index all your page files.  Even robots.txt is not exist in your website, a log entry will still be recorded in your web log files with 404 HTTP status code ( it mean the file is not exist).

Back To FAQ