Is it possible with canonical URL for this pattern in htaccess: /a/*/id/uniqueid?

Question

A big problem is that I am not a programmer….! So I need to solve this with means within my own competence… I would be very happy for help!

I have an issue with a lot of duplicated URLs in the Google index and there are strong signs that it is causing SEO problems.

I don’t have duplicate links on the site itself, but as it once was set-up, for certain pages the system allows all sorts of variations in the URL. As long as is it has a specific article-id, the same content will be presented under an infinite number of URLs.

I guess the duplicates in Google's index has been growing over long time and is due to links gone wrong from other sites that links to mine. The problem is that the system have accepted the variations.

Here are examples of variations that exists in the Google index:

• site.com/a/Cow_Cat/id/5272
• site.com/a/cow_cat/id/5272
• site.com/a/cow…cat/id/5272
• site.com/a/cowcat/id/5272
• site.com/a/bird/id/5272

The first URL with mixed case is the one used site-wide and for now I have to live with it, it would take too long time to make a change to all lower case. I cannot make a manual effort via htaccess as it is a total of 300.000 articles. I believe there are 10 ‘s of thousands that have one or more duplicates.

My question is this:

Is it possible to create rules for canonical URLs in htaccess in order to make the above URLs to be handled as one as well as for the rest of the 300.000?

I e, is there a way to say that all URLs having

/a/*/id/uniqueid

should be seen as one = based only on the unique ID and not give any regard to the text expressed with the “*”?

My hope is that it would be possible to say that a certain pattern like above should only be differentiated by the last unique segment.

If it is not possible in htaccess, how would it be done with link rel="canonical" on each page, can the code include wildcards?

I should add that the majority of the duplicates are caused by incoming links being lower case where the site itself is using a mix. Would it be OK to assign a canonical URL only with lower case although the site itself is basically always using a mix of lower/upper case?

If this is possible, I would be very happy to be helped with how to do it!!!!

Jonas

---

Hi Michael! I am not an expert but this is how I think it could be done:

1) My problem is that the URLs have mixed cases and I cannot change that now.

2) If it is OK for the searchengines, it would be fine for me to make the canonical URL identical to the actual URLs with the difference that it was all lower case, that would solve approx 90% of the duplicates. I e this would be the used URL: site.com/a/Cow_Cat/id/5272 and this would be the canonical: site.com/a/cow_cat/id/5272. As I understand, that would be good SEO...or...?

My idea was NOT to change the address browser address bar (i e using 301 redirect) but rather just telling the search engines which URLs that are duplicates, as I understand, that can be done by defining a canonical URL either in htaccess (as a pattern - I hope) or as a tag on each page.

3) IF, it would be possible to find a wildcard solution...I am not sure if this is possible at all, but that would mean it was possible to NOT assign a specific canonical URL but rather a "group pattern", i e "Please search engine, see all URLs with this patter - having the unique identifier in the end - as if they are one and the same URL, you SE, decide which one you prefer": /a/*/id/uniqueid

Would that work? It will only work in htaccess if canonical URLs can be defined as a group where the group is defined as a pattern with a defined part as the unique id.

Is it possible when adding a tag for each page to say that "all URLs containing this unique id should be treated the same"? If that would work it would look something similar to this

link rel="canonical" /a/*/id/5272

I dont know if this syntax with wildcard exist but it would be nice : )

Answer 1

My advice would be to use 301 redirects, with URL rewriting. Ask your webmaster to place this in your apache config or virtual host config:

RewriteMap  lc int:tolower

Then inside your .htaccess file you can use the map ${lc:$1} to convert matches to lower case. Here, the $1 part is a match (backreference from brackets in a regex in the RewriteRule) and the ${lc: } part is just how you apply the lc (lowercase) function set up earlier. Here is an example of what you might want in your .htaccess file:

RewriteCond %{REQUEST_URI} [A-Z]      #this matches a url with any uppercase characters
RewriteRule (.*) /${lc:$1} [L,R=301]  #this makes it lowercase

As for matching the IDs, presuming your examples mean "always end with the ID" you could use a regex like:

^(.+/)(\d+))$

The first match (brackets) gets everything up to and including the forward slash before the ID, and the second part grabs the ID. We can then use it to point to a single, specific URL (like canonical, but with a 301).

If you do just want to use canonical tags, then you'll have to say what you're using code wise, but an example I use (so as not add tags to hundreds of individual pages, for instance) in PHP would be:

  if ($_SERVER["REDIRECT_URL"] != "") {
    $canonicalUrl = $_SERVER["SERVER_NAME"] . $_SERVER["REDIRECT_URL"];
  } else if ($_SERVER["REQUEST_URI"] != "") {
    $canonicalUrl = $_SERVER["SERVER_NAME"] . preg_replace('/^([^?]+)\?.*$/', "$1", $_SERVER['REQUEST_URI']);
  }

Here, the redirect URL is used if it's available, and if not the request uri is used. This code strips off the query string (this bold bit in http://www.mysite.com/a/blah/12345/?something=true). Of course you can add to this code to specify a custom path, not just taking off the query string, by playing with the regex.