There are times when it is preferable to keep a blog’s contents private from the prying eyes of search engines. Sometimes a webmaster will want to keep only certain parts of this blog private, while other times it is necessary to block the entire blog from search spiders.
There are two steps that you need to take in order to best ensure that the contents of your blog are ignored by search engines and kept out of the SERPS (search engine result pages).
Keep in mind that this article is only applicable to WordPress blogs hosted on private domains. If your blog is hosted on WordPress.com, these steps do not apply, and you will need to refer to the help section of WordPress.com for your situation.
Step one: In WordPress, navigate to settings> reading settings and scroll to the bottom of the page. Next to “Search Engine Visibility”, check the box that says “Discourage search engines from indexing this site”.
Keep in mind that it is up to each individual search engine to honor this request for privacy. Google will honor it, but Yahoo and many of the other secondary search engines may ignore it.
Step two: You need to edit your robots.txt file to effectively block search engines from crawling your site. WordPress generates a virtual robots.txt file that is not accessible from either WordPress itself, or the control panel of your hosting company.
The easiest way to access and modify your robots.txt file is to install a plugin that will allow you to access the files directly from the WordPress menu. I use and recommend WordPress SEO plugin, but there are many different plugins that will serve the same purpose.
The robots.txt file can be modified to block all search engines, or only specific search engines. Additionally, the contents of your entire blog can be blocked, or only certain sections that you wish not be crawled.
Keep in mind, again, that it is up to the individual search engines to honor the request for privacy. Google almost always does but Yahoo and other search engines sometimes ignore the request. Following these two steps is the safest way to ensure that the contents of your blog remain private.