For a variety of reasons, you may have some pages on your WordPress website which you need to prevent Google from indexing. There is a right way to do it and a wrong way. Maybe you’ve just received the nasty notice from Google regarding your robots.txt file and how you need to remove the “noindex” directives from within and have no idea how to proceed.
In any case, we’ve got you covered! Depending on your needs, you may be able to use a plugin or you can add some simple code that provides a nice and lightweight option.
Google Warning
Odds are that you’ve ended up here because Google sent you a warning message regarding your use of “noindex” in your robots.txt file. The contents of that email are as follows:
“Remove “noindex” statements from the robots.txt of https://example.com/
To owner of example.com,
Google has identified that your site’s robots.txt file contains the unsupported rule “noindex”.
This rule was never officially supported by Google and on September 1, 2019 it will stop working. Please see our help center to learn how to block pages from the Google index.”
While it may sounds very concerning, it’s really not a huge deal and we can solve this issue pretty easily. And take note: you won’t have any issues until at least September 1st, 2019. So as of the time of this writing, you have plenty of time to fix it. If you haven’t done anything by then, what will happen is that the specified pages will start to get indexed in Google.
The reason your are getting this email is because, like many websites, yours has added a “noindex” line to the robots.txt file to direct Google not to index specific pages on your site. Apparently, this was never an officially supported command. Google has been abiding by it, but at this point they’ve decided they won’t anymore.
I’ve heard it said that they are campaigning to make some kind of standard, though I’m not sure why they wouldn’t just make this the standard, as it’s so easy!
Easiest Solution: Plugins
It’s not all that hard to de-index pages using plugins! Particularly if you are already using Yoast SEO. Yoast actually has a feature built right in that allows you to de-index pages and posts. Just scroll down to the Yoast SEO section of your post from the backend, click the “Advanced” tab, and under “Allow search engines to show this post in search results?”, click the dropdown and select “No”.
Under “Allow search engines to show this post in search results”, select “No”. In this screenshot, we currently have the default selected, which is “Yes”.
If you use All-in-One SEO Pack, unfortunately you are out of luck. Frankly it boggles the mind, but that plugin has no option for de-indexing pages.
I’m sure there are other plugins that can do this, too, but I haven’t found any. Yoast is my recommendation! Or, you can add some lightweight code.
How to de-index Tags, Categories, and Other Taxonomies
Once again, I recommend using Yoast SEO for this, as the plugin has a built-in function to de-index these pages.
To de-index tags in WordPress:
- From the WP backend, navigate to Yoast SEO -> Settings.
- Expand “Categories & tags” and click “Tags”
- On the right, click the slider next to “Show tags in search results” to toggle it and de-index your tag pages.
That’s it!
Categories and other custom taxonomies will show up in the same place and you can manually de-index them in the exact same way.
While it may vary across websites, you may want to de-index tag and category pages because they tend to use up a huge portion of your crawl budget. This means that Google will spend less time indexing your other, more-valuable content in favor of these questionably-useful category archive pages.
Lightweight Option: Custom Function
It’s not too hard to add some lightweight code to your site that will de-index specific pages on your site. You’ll have to manually add the page/post IDs, but actually using it is not very hard, and it shouldn’t really add any load time to your site which is nice.
It’s fairly easy to add! All you need to do is find and edit the functions.php file from your theme. Note: you should really only be doing this if you are using a child theme. Otherwise, if your theme gets updated, these changes will be overwritten and you’ll have to start over. Also make sure you back everything up ahead of time, and generally you need to have some idea of what you’re doing.
Add this code before the final ?> at the end of the file, or just after everything if that doesn’t exist.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
//Make specific pages noindex - Not sure why All In One SEO Pack doesn't have this functionality, but here we are //Code taken from https://pagecrafter.com/ function add_noindex() { //posts to noindex - in this example, post IDs 1, 2, and 3 will be de-indexed $no_index = array(1,2,3); $post_id = get_the_ID(); if( in_array($post_id,$no_index) ) { ?><meta name="robots" content="noindex, nofollow"><?php } } add_filter("wp_head","add_noindex"); |
Then, replace the 1,2,3 in the $no_index array with your post ideas. In this example, posts 1, 2, and 3 will be de-indexed. Replace this with your post IDs, placing a comma between each.
To find the post ID in a page, just go to that page, click “edit page/post” from the toolbar (or navigate to it from the backend), and get the ID from the edit page URL. The ID is always included in that URL. For instance, this post that I’m editing right now has a URL of:
1 |
https://pagecrafter.com/wp-admin/post.php?post=3818&action=edit |
The post ID is 3818, so that’s the number you would add to the code. If you have other pages, find their IDs and add each one with a comma between.
Make sure you upload the new version of the file back to your site, and it should be all set!
Conclusion
It’s not generally too hard to de-index pages on your WordPress site properly. Unfortunately, robots.txt won’t work for this purpose anymore but we can easily get around that.
Google tends to take it in to their own hands to control the development of the internet, and this is the new example of that. I’m not sure exactly what was wrong with the old method and why they decided it was inadequate without offering an alternative, but here we are!
Let me know if you have any trouble with this.
One Comment on “How to De-index Individual Pages in WordPress”
Thanks for your article. The steps with screenshots are really clear for me to follow. There are some pages that I don’t want Google to index but don’t know how to do so. I often change the page status to Private. I can do deindex these pages with ease now.