3

were experiencing a strange issue with a wordpress sites meta robots tag. All pages have the following meta tag and we cant seem to remove it

<meta name="robots" content="noindex,follow"/>

We have unchecked "Discourage search engines from indexing this site" in Settings > Reading > Search Engine Visibility but it does nothing.

We are using the Yoast SEO plugin but even when this is disabled the still remains. In fact, we have tried disabling all plugins to check nothing was interfering with it.

We have setup our Robots.txt file as follows: 

User-agent: *
Disallow: 
Sitemap: http://speysidedistillery.co.uk/sitemap.xml

Im not sure if the Robots.txt takes precedent over the robots meta tag or not and there doesn't seem to be a definitive answer, as far as i can gather the most restrictive one will take precedent i.e in our case the meta tag.

This is mainly giving us issues with our google listing with the warning "A description for this result is not available because of this site's robots.txt" appearing instead of our sites description.

If worse comes to worst we can edit the wp_no_robots function in wp-includes/general-templates.php but would prefer to resolve this without editing the wp core files.

Any light anyone could shone on this would be great as we are at a loss, cheers

The site can be found at http://speysidedistillery.co.uk/

Christopher Craig
  • 51
  • 1
  • 1
  • 4
  • Allowing a site in robots.txt does not prevent pages from being blocked by robots meta tags. They are different things. Robots.txt tells the crawler whether or not to load the page in the first place. Robots meta tags tell search engines whether to index (or follow links on) pages that they have already loaded. – plasticinsect Jan 20 '15 at 21:31

10 Answers10

5

i Developed an E commerce site for a client and i was stacked with this issue for more than a week, all pages were indexed except for the Home Page.

After a long debugging of Yoast Seo Plugin, i got a solution for you guys.

SOLUTION The Homepage has a search page on it (for searching for products). By default, Yoast disables 'index' meta tag for Search Page and 404 Page. funny enough if your homepage has a search page or search functionality on it, your site may not be indexed by google or any other search engine.

To fix this default by Yoast SEO Plugin, follow the following steps:

STEP 1: Log into your cpanel and navigate to /public_html/yourwebsite.com/wp-content/themes/nameOfTheTheme/functions.php add this line of code to your theme's function.php file (at the top of the page)

 /**
  * EDITED BY OBOT ERNEST
  * REMOVES NOINDEX META TAG FROM HOME PAGE  & SEARCH PAGE IMPLEMENTED BY YOAST
 */


    add_filter('wpseo_robots', 'yoast_no_home_noindex', 999);
    function yoast_no_home_noindex($string= "") {
        if (is_home() || is_front_page() || is_search()) {
            $string= "index,follow";
        }
        return $string;
    }

After pasting the code, save and close file.

STEP 2 Note this second step is for those that are using any of the following SEO plugins (Yoast Premium, Yoast Free or wordpress-seo-premium plugin)

Navigate to: /public_html/youwebsite.com/wp-content/plugins/wordpress-seo-premium/frontend/class-frontend.php

replace the code at line 713 with this one below here:

    if ( is_search() ) {
$robots['index'] = 'index';
}
    if ( is_404() ) {
$robots['index'] = 'noindex';
}

Save file and close

Congratulations! you just made it. It working perfectly guys!

Obot Ernest
  • 412
  • 8
  • 19
4

If you are using Roots Bedrock, then check the value of WP_ENV in your .env file. If it is set to development then this automatically overrides the value of blog_public to always be false, meaning the "no robots meta" is always injected.

Simply change it to WP_ENV='production' in your .env file, save the changes and it will be fine.

codewithfeeling
  • 6,236
  • 6
  • 41
  • 53
  • You're a life saver, I just spent 3 hours looking at every single plugin / file and it was in the fact the .env set to development ... Thanks a lot – ThaoD5 May 04 '22 at 10:10
3

I found a solution that worked for me here

add_action( 'init', 'remove_wc_page_noindex' );
function remove_wc_page_noindex() {
    remove_action( 'wp_head', 'wc_page_noindex' );
}

This question appears to be on three threads:

  1. How do I stop Wordpress from inserting a noindex meta tag?
  2. Wordpress remove Robots Meta Tag noindex
  3. Cannot remove action noindex in wordpress
Das_Geek
  • 2,775
  • 7
  • 20
  • 26
Sean Sr
  • 94
  • 7
1

i looked into many solutions but was not able to update it, even updated the functions.php file by adding the below code snippet. You need to update the "class-frontend.php" but, I couldn't find that. Then I tried this and it worked for me.

add_filter('wpseo_robots', 'yoast_no_home_noindex', 999);
function yoast_no_home_noindex($string= "") {
    if (is_home() || is_front_page() || is_search()) {
        $string= "index,follow";
    }
    return $string;
}

go to appearance -> theme editor -> header.php

enter image description here enter image description here enter image description here enter image description here

irzum shahid
  • 181
  • 2
  • 12
0

You can try this Options->Privacy->Blog visibility. You have it set to block search engines.

Davit Huroyan
  • 302
  • 4
  • 16
  • Is this possibly for an older version of WP? Were using 4.0. As far as i can see the privacy setting are under Settings > Reading. I cant see an Options or pPrivacy tab anywhere – Christopher Craig Jan 20 '15 at 11:16
  • Yes you are right :) it was in older vresions in 4 version settings > reading Discourage search engines from indexing this site – Davit Huroyan Jan 21 '15 at 06:00
0

Check your header.php file inside the theme folder.

Zulhilmi Zainudin
  • 9,017
  • 12
  • 62
  • 98
0

I deleted yoast and reinstalled. While re installing it asks if to index, I choose yes this time. It worked. But luckily , for me I had not made much seo ,may be 1 or 2 pages the first time.

somnath balasubramaniam
  • 1
0

How to Remove WordPress Robots Meta Tag noindex, nofollow

We will show you how to remove meta name=’robots’ content=’noindex nofollow’ in WordPress using replace some code inside the admin panel, no plugin request.

Step 1 – Log into your cPanel

Well, you log inside the public_html directory, now you find a folder, name wp-includes, double click to open this folder. At the same time, there are so many folders and files, take a breath, and simply click to your keyboard Ctrl+f and type general-template.php.

Step 2 – Edit general-template.php File

Find this code inside general-template.php

function wp_no_robots() {
if ( get_option( 'blog_public' ) ) {
    echo "<meta name='robots' content='noindex,follow' />\n";
    return;
}

echo "<meta name='robots' content='noindex,nofollow' />\n";

}

Replace first with this code:

function wp_no_robots() {
if ( get_option( 'blog_public' ) ) {
    echo "<meta name='robots' content='index,follow' />\n";
    return;
}
echo "<meta name='robots' content='index,follow' />\n";

}

2 – Find this second code same file general-template.php:

This

function wp_sensitive_page_meta() {
?>
<meta name='robots' content='noindex,noarchive' />
<meta name='referrer' content='strict-origin-when-cross-origin' />
<?php

}

Replace to 

function wp_sensitive_page_meta() {
?>
<meta name='robots' content='index,archive' />
<meta name='referrer' content='strict-origin-when-cross-origin' />
<?php

}

Once that is complete, go back to the website check, change into Search Console, resubmit the URL you tried earlier. The URL inspection report should be void of all warnings and error messages, at least ones related to indexing and crawl ability, and you’ll be able to “Request Indexing,”.

I found a solution that worked 100% Read Full Details here

Dipankar
  • 1
0

Navigate to Yoast SEO -> Search Appearance. Choose tab Content Type or Taxonomies depending on the page you need indexed by google.

In my case, google can not index my tags page (/tag/tag-name)), So i switched Show Tags in search result? to Yes. It will change robots meta tag from:

<meta name="robots" content="noindex, follow" />

to

<meta name="robots" content="index, follow" />

Hope can help.

Bugatino
  • 73
  • 3
0
  1. Use the following code in robots.txt file:
User-agent: *
Allow: /

  1. Make Sure "Discourage search engines from indexing this site" checkbox is not checked under Settings-> Reading. if it is then mark it unchecked. enter image description here

  2. Lastly, This is the step that creating problem for me is inside Yoast SEO Plugin. Go to YOAST SEO-> Search appearance ->Content Types->Pages under these settings. Make sure Following Settings are on:

enter image description here

Clear your wordpress and browser cache. This steps helps in removing noindex from meta tag.

Sonia Verma
  • 41
  • 2