Hello folks,
Lately, I have been dealing with search engine optimization (SEO) issues of my company's website and delved into everything about SEO from optimized content development to non-canonical url issues. I decided to keep my solutions written in my blog so that somebody else might benefit from these.
Problem brief: Search engines (esp. Google) expects mostly new content from each url. Non-uniqueness is used as a metric to measure quality of a website and some people call this TrustRank to emphasize the difference of this criteria from PageRank. Search engines may not always be intelligent enough (or perhaps they do not have time to (: ) to figure out the same page with varying url. These non-canonical versions of the same content need to be explicitly specified in order not to have trouble with our search engine rankings (SERPs). Most of the websites can usually be accessed via these URLs:
http://www.example.com
http://www.example.com/index.html
http://example.com
http://example.com/index.html
php-specific urls:
http://www.example.com/index.php
http://example.com/index.php
asp-specific urls:
http://www.example.com/index.asp
http://www.example.com/index.aspx
http://www.example.com/default.asp
http://www.example.com/default.aspx
http://example.com/index.asp
http://example.com/index.aspx
http://example.com/default.asp
http://example.com/default.aspx
It is possible to check common main page urls from this nice tool:
http://www.ragepank.com/redirect-check/
Search engines may not see these urls as the same page and infer that you do not have unique content in these urls. In addition, if external links to your website does not point the same url, your link profile is affected majorly (roughly half the number of links points to one url and the other points to another). The problem can be solved using 301 permanent redirects. Basically, you have the bots crawl and index only one version of each page thanks to permanent redirects in addition to transferring all the external link credit to the canonical version of each page. In Apache servers, .htaccess file (nameless file having this extension) is used to redirect pages via regular expressions. You can easily find solutions in Google if you search for "301 htaccess" and similar. In ASP.NET websites, the problem can be solved by placing the XML snippet below to Web.config files:
<configuration>
...
<rewriter>
...
<unless header="HTTP_HOST" match="www.example.com">
<redirect url="(^.*$)" to="http://www.example.com$1" permanent="true" />
</unless>
<if header="HTTP_HOST" match="index.html$" url="/index.html$">
<redirect to="http://www.example.com" permanent="true" />
< /if>
...
</rewriter>
...
</configuration>
I provided the code for redirecting non-www version of a website to the www version and index.html version to the root version. The others are intuitive. Please note that there are variations of this code due to the imported library in the web config file but I had trouble finding the solution for my version and wanted to share it.
Hope it helps.
