Robots.txt
Most search engine spiders only visit the places on a website where they have permission to crawl. When spiders access a site, they check a special file in the root directory of the web server called “robots.txt”. This file is a tool for blocking spiders from crawling and indexing duplicate content or parts of the site, but should be used with caution as it does not allow algorithmic metrics to pass through the pages that are blocked, thus compromising the SEO efforts.
For more information about the robots.txt file and correct usage please visit Robotstxt.org
Meta Noindex/Nofollow Directive
This method involves placing the below meta name directive in the header of the webpage. Linchpin SEO recommends utilizing this method for keeping sensitive or internal search result pages from being indexed in the search engines since it keeps pages out of the index but still allows algorithmic metrics to be passed through the pages.
This tells the robots not to index the page in their search results but if someone links to the page or if the page gains other algorithmic metrics, then let those metrics pass though the links on the page. This header code should be utilized on internal search results pages.
This tells the robots no to index the page in their search results and also not to let algorithmic metrics pass through the links on the page if the page gains any from other pages around the web. There has been studies to show that Google can and does follow links that contain the nofollow attribute to discover new content, but they do not allow metrics to pass through those links.
Canonical Tag
This is a suggestion to the search engines that a page actually resides on a different URL, and not to use the current URL that is being accessed for indexation and ranking. This can be done within a site or across sites.
Canonical tags address duplicate content issues. They allow the user to see the page, yet tell the search engines to apply metrics to the canonical version of that URL. Note that its important that pages that are canonicalized must be as close to replicas of each other as possible, or it could be seen as a manipulative tactic.
More information about canonical tags and their use can be found here
Bill is the Lead Linchpin at the company and focuses on SEO and content strategy. Bill’s ten-year background in digital marketing, SEO, and building online companies has helped him rise to strategic lead for some of the largest online brands and content websites in the world. Bill believes that at the core of a website's success sits the value that website provides it users.
Click here to learn more about LinchpinSEO's Lead Linchpin.
