There are a series of weird behaviors which all seem to be stemming from a Content-Type header issue.
The site will be running fine but then the problem happens "randomly" throughout the day. But I can replicate the problem by doing the following:
- On server, clear HTCache from mod cache disk and flush cache from apache pagespeed module.
Test any page URL from my WordPress Site, on redbot.org/ which shows correct headers.
Content-Type: text/html; charset=UTF-8Go to any browser and load the feed url. mysite.com/en/feed/ The XML feed looks good. Or go to http://redbot.org/ and check the header of the feed, which looks good.
Content-Type: application/rss+xml; charset=UTF-8Go back and test the URL of any page on http://redbot.org/ . Now the Content-Type switched from text/html to rss+xml !
Content-Type: application/rss+xml; charset=UTF-8
This automatic switching of content-type makes In-App Browsers (IAB) like Facebook and Instagram go berzerk. Instead of the IAB rendering the html page, it loads the XML file for the RSS feed. The RSS Feed will be a random one, either the comments feed for some random page or the posts feed.
Multiple problems related to RSS Feed showing up instead of loading web site blog post url.
- Google pagespeed testing sometimes shows RSS feed. https://developers.google.com/speed/pagespeed/insights/
- Twitter Card Validator, Facebook Debugger Validator, whatsapp chat and LinkedIn validator not showing metadata image, title and description.
Doing this command in linux will show RSS feed instead of HTML of site
curl -A Twitterbot mysite.com/Testing my RSS feed (mysite.com/en/feed/) in https://validator.w3.org/feed/ sometimes shows the comment RSS feed instead of post RSS feed.
I've ruled out the following possible causes because I've either updated each item or checked the code for errors:
- Updated WordPress
- Updated Plugins
- Updated theme
- No longer inserting/Writing/Inlining CSS stylesheets directly into HTML page
- Posts are NOT custom post types. They are regular wordpress posts.
- Used Wordpress Native Text Editor to write posts to prevent RSS code errors generated by plugins like TinyMCE Editor.
- RSS Feed is validated. Tested with: validator.w3.org/feed/ and Feedvalidator.org
- With Apache PageSpeed Module disabled, the problem still occurs.
I've also tested by deactivating all plugins but one, polylang which makes the site bilingual. I can't disable polylang since without it, many posts wouldn't show up on the site.
What temporarily solves the problem for now: Manually clearing the HTCache always makes the Content-Type for regular URLs back to HTML/txt.
sudo htcacheclean -v -t -p /var/cache/apache2/mod_cache_disk -l1b -L1b
So I went ahead and disabled cache_disk and the Site has been working fine all day so far. Every time I load the RSS Feed url in the browser and then test the header of any web page at Redbot.org the header shows up correctly (html/txt)
sudo a2dismod cache_disk
sudo systemctl restart apache2
sudo systemctl stop apache-htcacheclean
I still do not comprehend what is the root of this problem in the first place. So I am bewildered and reaching out to you all for help.
Any ideas what can be causing such odd behavior and how to fix it?
