I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.
I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but .txt still returns a 404.
Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?
I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.
I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but http://mysite/robots.txt still returns a 404.
Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?
Share Improve this question asked Feb 25, 2013 at 15:34 Mike DeckMike Deck 1171 silver badge8 bronze badges 2 |2 Answers
Reset to default 3First of all, in order for Wordpress to generate a robots.txt for you you must be using a non-default permalink structure. Make sure you've selected an option in the Settings > Permalinks menu.
Also, if a robots.txt file exists at your root directory it will override the setting in Wordpress. It looks like you already have a robots.txt file and that is the reason the wordpress setting is ignored.
You don't need to add robots.txt file to the root of your site. robots.txt file is generated in real time, when you visit http://mysite/robots.txt. The function, responsible for creation of this file, is do_robots.
If you wish to add your own directives, just write your hook for robots_txt filter, like this:
add_filter( 'robots_txt', 'wpse8170_my_robots_txt', 10, 2 );
function wpse8170_my_robots_txt( $output, $public ) {
if ( '0' != $public ) {
$output .= '
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
';
}
return $output;
}
<meta content="noindex,nofollow" name="robots">to the head section of each page which should accomplish my goal. I'd still like to know how to add a robots.txt too though. – Mike Deck Commented Feb 25, 2013 at 15:36robots.txtover FTP? WordPress doesn't use it anyway. – user26607 Commented Feb 25, 2013 at 15:44