Internet Marketing Ninjas Blog http://www.internetmarketingninjas.com/blog Internet marketing blog Wed, 22 Feb 2012 20:26:01 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 SEO Worst Practices http://www.internetmarketingninjas.com/blog/search-engine-optimization/seo-worst-practices/ http://www.internetmarketingninjas.com/blog/search-engine-optimization/seo-worst-practices/#comments Wed, 22 Feb 2012 15:08:49 +0000 Rick DeJarnette http://www.internetmarketingninjas.com/blog/?p=2748 read on »]]> SEOs love to talk and write about industry best practices, as well they should. To best serve their clients, following industry-established SEO best practices should always be the goal.

But what about those shady types on the fringes of the SEO community who advocate, hmmm, let’s call it a less reputable route? Not surprisingly, these SEO are also typically the same ones who claim they can absolutely get you to be the #1 spot in the search engine results pages (SERPs). Guaranteed, no less! There’s a reason the old adage, “If it’s too good to be true, it probably isn’t” remains so relevant in our lives.

If you run a business and are shopping for SEO services and consulting, if any of the following techniques are mentioned (and please do ask!), you can be confident that the consultant in question is following well-known (in professional circles) SEO Worst Practices. The results of such efforts used on your site will likely backfire, causing your website to be penalized (lowered in rank) or, if excessively egregious, perhaps even purged from the index. This applies to both Google and Bing.

Let’s take a look at a few top choices in SEO worst practices:

Keyword stuffing

If the grand plan for getting your site to #1 includes adding ~150 highly searched for (but largely irrelevant) terms and phrases into the <meta> keywords tag, not only is this poor form in terms of webpage spam, but it’s also a hopelessly out-of-date and obsolete technique. First of all, stuffing the <meta> keywords tag is a tactic that was old 10 years ago. Because so many sites used this tag in an attempt to increase their page rank for targeted keywords by repeating those words countless times (or, alternatively, expand the relevance of the page to keywords not otherwise used on the page), the search engines long ago abandoned using the <meta> keywords tag for keyword relevance. Google has come right out and stated it does not use the <meta> keywords tag for keyword relevance. Bing has taken a more nuanced position in that the tag is actually not really used today, but states there are hundreds of ranking factors that are considered, and one day, if the intentional spamming of this tag finally dies off due to neglect, it might eventually become useful again. Maybe. But not today, not soon, and no promises beyond that.

The act of keyword stuffing not only occurs in the <meta> keywords tag, but can also occur in <title> tags, <img> alt text, heading tags, anchor text, even sometimes boldly in plain body text!

The search engines crawl web pages and see what is in the code. They see the text within the <body> tag, as well as the page metadata. They see when a word is repeatedly used to excess, and they can mitigate any attempted beneficial manipulation in their page ranking assessment. Keyword stuffing is dumb, clumsy, ineffective, and amateurish SEO.

Keyword stuffing caveat

If the use of repeated terms is legitimate to the business of the page, the search engines will understand that and accommodate that in their search for what otherwise is web spam. For example, if you are an attorney who can help clients with income taxes, tax deductions, tax penalties from the IRS, interest on back taxes, business taxes, rules for excise taxes, estate tax planning (you see the point), the repetition of the word “tax” is not spam because the phrases in question are normal word usage, not artificially done for web spam.

The difference between this sample usage and keyword stuffing is intent. The search engines spend a huge amount of time and resources trying to parse legitimate from illegitimate intent. If you are incorrectly identified as a spammer and your site suddenly tanks in the rankings, and it’s not due to larger algorithm changes like Google’s series of Panda updates, then you can appeal a penalty in Google and/or Bing. Just note that if you were penalized for violations of the official webmaster guidelines of Google or Bing, your entire site must be cleaned up of all web spam techniques and republished before you apply for reconsideration.

Hidden content

Search engines don’t like seeing pages that have hidden content (content that’s crawlable in the code, but doesn’t show in the browser window). That’s considered to be the equivalent of telling the search engines, “Here, this extra bit of content is just for you.” They consider that to be a maliciously manipulative attempt to earn page ranking credit for material not actually shown in the page. As Martha Stewart might say, that’s not a good thing.

Webmasters use a multitude of easily detected techniques in the effort to hide content from display. They use code like <div style=”display: none;”> to hide entire passages of body text. They use style attributes to make text the same color as the background, rendering it invisible, or configure the font size to be so small that it’s unreadable. There are many such “tricks” used to stuff extra text in a page. Of course, the search engines see all of this coding and can interpret that it’s intended to be hidden from view. If the intent of the usage is malicious, penalties can ensue.

Hidden content caveat

The use of <input type=”hidden”> controls are not by themselves suspicious, as some controls are not revealed in the default view of a page. The issue is always intent. If passages of text are hidden, especially if they contain keyword stuffing as mentioned earlier, this is what raises the red flags for search engines.

Cloaking

Cloaking is where the web server uses the identity of the user agent making the request for the page to determine which version of the content is returned. For example, if an IE 6 user agent requests a page and then the Googlebot user agent requests the same page, but the content is different, that indicates there is user agent filtering (cloaking) occurring for the purposes of manipulating what content search engines see. The goal of malicious cloaking is always to artificially inflate the rank of the page the user sees. Cloaking can show users normal pages and serve search engine crawlers keyword-stuffed pages. Alternatively, cloaking can serve search engines nicely optimized, relevant pages but serve users junk sales pitches for illicit pharmaceuticals, porn, or other such content that would not rank otherwise for that query.

Cloaking caveat

Pages that filter for mobile browsers to show abbreviated or custom-formatted versions of the desktop page are not considered to be malicious cloaking. Again, it comes down to intent. Is the web server attempting to maliciously manipulate the search rankings for a given page? The use of cloaking on search engine user agents is not a good idea. The search engines will detect it and penalize a site employing that technique accordingly. Filtering user agents for mobile devices, as long as the content stays similar, is relevant to the search query, and useful to users, is not a problem.

Intent is key

All of these SEO worst practices are based on the intention to deceive, be it the search engine crawler or the human end user. As long as businesses create great sites that are of value to human visitors, have compelling content, and are easy to navigate (meaning they are also easy to crawl), they will be assessed accordingly. Those are the sites that earn backlinks and citations from other sites, and that is what is needed for white hat SEO.

]]> http://www.internetmarketingninjas.com/blog/search-engine-optimization/seo-worst-practices/feed/ 0 Using and Optimizing Images: Search and Social Optimization Cheat Sheets http://www.internetmarketingninjas.com/blog/search-engine-optimization/image-seo/ http://www.internetmarketingninjas.com/blog/search-engine-optimization/image-seo/#comments Mon, 20 Feb 2012 15:07:01 +0000 Ann Smarty http://www.internetmarketingninjas.com/blog/?p=2727 read on »]]> If you are running a website, there are most likely going to be plenty of images there.

While image-specific SEO is very-well explained in a few detailed guides, let’s try to create a very simple and easily-organized guide to using images properly:

1. Free Images You *Can* Use

First things first: let’s see where you can find free-to-use images online. That’s a popular misconception that you can actually use any image you find online as long as you credit the source.

Mind that you can only re-use images with a certain license; here’s a quick guide (here’s another a bit more advanced one) into the three types of the Creative Commons licenses that allow you to re-publish the images on your site under certain conditions:

License Icon You Can Re-use For Commercial Use? You Can Modify Credit
Attribution-NoDerivs License Attribution License Attribution-NoDerivs License Yes Yes No Yes (Required)
Attribution-NonCommercial-NoDerivs License Attribution License Attribution-NonCommercial-NoDerivs License No No
Attribution-ShareAlike License Attribution License Attribution-ShareAlike License Yes Yes

The two popular (and most effective) sources of Creative Commons images you can re-publish are:

2. Image File Name

An image file name is crucial when it comes to ranking an image in Google Image search results (though it’s not the only factor as well. See this case study on photo optimization). I’ve seen a huge boost of image search traffic each time I pick a good name for my image. Here’s what has always worked like a charm:

key-phrase.jpg

Mind that traditionally, search engines read a hyphen in URLs and file names as a “space” – that doesn’t mean Google won’t understand an underscore, an actual space or other characters there, but a hyphen is the most natural and straightforward way to go with.

3. Image ALT Text and Title

That’s another very popular question: what’s the difference between the ALT and the TITLE attributes when it comes to describing an image?

Most basically, here’s the difference:

Image “Alt” Attribute Image “Title” Attribute
Official rule of use Describes an image for search agents Gives *additional* information on what an image is about (when it’s required)
Screen readers (like JAWS or Orca) “Read” it Ignore it by default (it is mostly considered redundant*)
Browsers Google Chrome Is displayed when images are disabled Pops up when you hover over an image
FireFox Is displayed when images are disabled
Safari Is ignored
Opera Is displayed when images are disabled
IE Pops up when you hover over an image if notitle attribute is present

Conclusions:

  • (Very important!) Use ALT text to describe the image you are using;
  • Use title if you need to give additional information: do NOT duplicate it with alt text! (*accessibility rules only advise to use TITLE tags for abbreviations, forms, etc, i.e. where an explanation is really necessary);
  • (If there are many images on one page) Use different alt text throughout the page as it will be displayed as “text” in most browsers (when images are disabled) and in the email newsletter (when remote content is loaded on demand):

Image alt text displayed in an email

4. Image Size and Type

The good old rule has always been to keep your images less than 100K. My own rule of thumb: as long as I don’t sacrifice on the image quality, I make it the minimum size I can.

Google also recommends: “the less, the better”. Here are Google’s recommendations as to file types and compressors:

Best used for Recommended compressor
JPGs All photographic-style images jpegtran or jpegoptim
PNGs Logos, banners, etc (where you need transparent background) OptiPNG or PNGOUT
GIFs For very small / simple graphics (e.g. less than 10×10 pixels, or a color palette of less than 3 colors) & for animated images N/A
BMPs or TIFFs Don’t use

More great tools to try for any image file type you are using:

  • WP Smush.it – a WordPress plugin that uses Smush.it API to perform image optimization automatically. It does all essential image optimization tasks: optimizing JPEG compression, converting certain GIFs to indexed PNGs and stripping the un-used colors from indexed images (except for stripping JPEG meta data) automatically.
  • IrfanView (desktop) is an awesome free tool that helps you crop, (bulk-)optimize and (bulk-)rename images.

5. Image Thumbnails in Social Media

An image thumbnail generated with the snippet when someone shares your post on a Facebook or Google Plus wall is crucial when it comes to click-through and further shares.

While Google Plus is generally very smart at marking up your page and finding the best thumbnail to go with the update, Facebook seems to only rely on what you “point” to it. Besides, when using Facebook’s “Like” button, your readers have almost no control over the shared snippet and often an image that gets to your reader’s Facebook wall is absolutely random.

To ensure your beautiful, relevant and eye-catching images make it to your fans’ Facebook streams and get lots of attention, we are forced to use Open Graph Protocol to point Facebook to what needs to be grabbed from your page:

Open Graph

This WordPress Plugin makes integrating Open Graph very easy for WordPress bloggers. Also, this tool will help you identify how Facebook “sees” your page as well as refresh its cache.

Other “Obvious” Factors

The tips and tables above mostly list image-specific factors of making your images search- and social-friendlier. That doesn’t mean other commonsense practices don’t matter here:

  • Your images should be surrounded with relevant “text-based” content to rank well in image search results;
  • Your images should be located at powerful pages (in terms of link juice and on-page optimization).

Have I missed anything? Let’s help make is an actual “all-in-one” guide: add your image SEO tips in the comments!

For more useful SEO- and social-media-related content, don’t forget to follow us on Twitter and join us on Facebook!

]]> http://www.internetmarketingninjas.com/blog/search-engine-optimization/image-seo/feed/ 0 HOW TO: Make the Most of Your TWEET Button + Must-Have Checklist (WordPress) http://www.internetmarketingninjas.com/blog/social-media/how-to-make-the-most-of-your-tweet-button-must-have-checklist-wordpress/ http://www.internetmarketingninjas.com/blog/social-media/how-to-make-the-most-of-your-tweet-button-must-have-checklist-wordpress/#comments Thu, 16 Feb 2012 17:46:55 +0000 Ann Smarty http://www.internetmarketingninjas.com/blog/?p=2479 read on »]]> Back in 2010 Twitter partnered with Tweetmeme to create an official Tweet button. It resulted in lots of criticism as the official button lacked some obvious customization options (primarily, it forces the user to use Twitter’s official shortener. Besides, it has no color customization and is somewhat slow).

However the fact that Twitter partnered with the major player in that field to launch the official product has left us with almost no choice. Besides, the Tweet button does have quite a few great features (my favorite one: the ability to recommend your personal as well as your business accounts for people to follow).

Besides, what many people are likely to be unaware of: the Tweet button can be customized with the help of some of the hacks (listed below)

1. Use Your Own URL Shortener

Twitter Friendly Links is a great and absolutely easy-to-use WordPress plugin that lets you quickly create your own domain shortener. Just have it installed and activated, go to the plugin settings and configure the following:

  • Set the base URL: this option is *gold* if you have www in your default URLs. Making the base URL www-free will make your shortened URLs even shorter!
  • Set the Redirection type. I am using 301 redirect as a link-builder at heart: someone would want to use my tweet to quote or will copy-paste the URL from the URL directly from tweet and use it in a post. So I want this link not to be wasted.

Twitter Friendly Links

Now to use these pretty branded URLs with your Twitter button you’ll need this plugin: WP Tweet Button – it has plenty of cool features and the rest of the post will be dedicated to those features, but so far the most important option for us is the WP Tweet Button plugin ability to support Twitter-Friendly link. All you need to do is to select “Twitter-Friendly Links Plugin” from the drop-down on its Settings page:

Twitter Friendly Links

Now you are done: Readers are able to tweet your articles using the official Tweet button while using your pretty self-hosted URL shortener.

Twitter Friendly Links

Another option: YOURLS which is also fun but somewhat hard to set up.

2. Promote Your Guest Authors and Contributors

WP Tweet Button has another valuable option: you can set up your “default” Twitter username from the admin page but it will be overridden by the author’s Twitter name he specifies on his profile name.

WP Tweet Button

What does it mean?

The post author’s Twitter name will be included in the Tweet itself. Besides, the author’s Twitter username will be recommended for following along with your default username:

WP Tweet Button

Another great thing about the plugin is that it lets you specify the Tweet text as well as more accounts to “recommend” right on the “Edit post” level. This means you can get as creative as you want: add the author’s Twitter account to recommend, his custom hashtag, his name in the Tweet, etc:

Custom Tweet text

Here you go!

Custom Tweet text

3. *Smartly* Auto-Tweet Your Articles

Whether to auto-tweet your blog articles or not is up to you. I find it a great option for people on the go: blogging while traveling, scheduling articles for later, etc – in these cases, the ability to auto-tweet your posts that go live while you are away or busy is a life-saver.

Now, traditionally Twitterfeed is used to auto-tweet, but WP Tweet Button is so much better!

*Smartly* Auto-Tweet

  • Auto-tweet your own articles using your self-hosted URL shortener;
  • Auto-tweet your articles each time you update them.

The smart plugin even removed the default Twitter username to avoid you referencing yourself in your own Tweets.

Auto-tweet articles

So The Check-List Now!

If you choose to try the tips above, here’s an actionable and easy-to-follow to-do list for you:

Tool Settings
Twitter Friendly Links Install and activate
Set the base URL (www-free)
Set the Redirection type (301)
WP Tweet Button Install and activate
Select “Twitter-Friendly Links Plugin as your URL shortener
Set the default Twitter username to reference and recommend
Ask all your contributors to update their profiles to add their Twitter usernames (to reference them in the Tweets)
Authorize the plugin to auto-tweet your articles when you publish and/or update them

Don’t forget to follow our official Ninja Twitter account to have articles like this (and better) to be delivered right to your Twitter home page!

]]> http://www.internetmarketingninjas.com/blog/social-media/how-to-make-the-most-of-your-tweet-button-must-have-checklist-wordpress/feed/ 0 A Love Letter to the Site: Command http://www.internetmarketingninjas.com/blog/search-engine-optimization/the-site-command/ http://www.internetmarketingninjas.com/blog/search-engine-optimization/the-site-command/#comments Wed, 15 Feb 2012 21:44:12 +0000 Ninja Jen Van Iderstyne http://www.internetmarketingninjas.com/blog/?p=2705 read on »]]> We have a lot of tools here at Internet Marketing Ninjas. I mean A LOT. We have tools that crawl sites, compare competitors and create charts and tables full of useful data. I’m such an isolated geek – I actually called someone a “tool” the other day, forgetting that, in some circles, that’s actually an insult. But honestly, the tools our reporting team gets to work with are so awesome and comprehensive that occasionally it makes me a little nostalgic. It’s kind of the way I imagine some people still fondly remember rotary phones when looking at an iPhone. No, the old technology wasn’t better. But we didn’t know of better technology at the time and the old tools remind us of when progress was younger and we made the best of what we had.

That’s how I feel about a site: command.

Whenever I’m taking an initial look at a site, I run a few tools, absolutely, but either by habit or by instinct, one of the first things I still do is run a site: command in Google. It’s a simple string:

site:yoursite.com

And maybe I’m a relic, but when I started doing this work, I built links using Internet Explorer and Notepad. I used the linkfromdomain: command in MSN to find out where a site was linking out to (remember that?!). I looked at backlinks manually (which I still like to do) and I used the site: command every day. So maybe I have issues letting go, but I still get a lot of insight from that old trick.

Pages Indexed in Google

Obviously, a site: command tells me how many pages of a site Google has indexed. Most of the time this is background information that isn’t exactly earth shattering. But sometimes you’ll find Google has indexed more pages than a client thinks they have on their site. That’s a bad sign. That could mean duplicate URLS or indexed search results. Of course the opposite can also happen when, of 10,000 pages, Google has only indexed 3,000. That means something, too. The pages indexed figure doesn’t mean much when it’s right, but if it’s not what you anticipated, then it instantaneously reveals when something is probably wrong.

Title Tags

Next up for scrutiny are title tags. How are they laid out? Are they the same on every page? Do they balance keywords and branding? A site: command tells you all of that, which also tells you if someone who knows SEO has worked on these before. Title tags are your most powerful on-page tool, so using those well is one of the most important SEO decisions a webmaster makes. By looking at them, laid out in a site: command, we can tell if those are locked and loaded, of if they are a starting point for better optimization.

Meta Descriptions

Right below the title tags are the snippets which are often derived from meta descriptions.  Even though meta descriptions aren’t exactly a ranking factor in terms of keywords, they are important when they convert SERP impression into clicks. But sometimes they get mistreated. They may be keyword stuffed for no good reason or boiler-plated across a whole site. Sometimes they’re missing altogether. Depending on what else we find during an analysis, we may find that rewriting meta descriptions falls low on the SEO priority list. However, for the purposes of usability or conversions, we may want to get some revisions going ASAP.

URL Structures

You can also tell a lot about structure from a site: command. I look for things like subdomains, dynamic URLs and secure pages. You can also see tiny little things like underscores instead of hyphens or mixed case URLs. In some instances you can’t see the entire URLs because you see bread crumb links instead, which also helps you get a sense of what a site is doing with their architecture.

Google+ and Author

If a site is socially active, particularly in Google+, you can see that in a site: command now, too. This is a bit more of a recent development as is seeing an author’s picture show up next to URLs. Ok, so this may not be strategy-altering information. But it’s pretty cool. If someone has a thriving Google+ presence and a trusted author set up, we’re already off to a REALLY good start.

The Unexpected

I think this last category in a way encompasses every other one as well — I’m looking for the unexpected. Something that makes me go “Huh.” Maybe it’s the top pages that come up, or assets we can use in link building. Maybe it’s just a URL with no title or meta description. It could be anything. Ultimately the question you want to resolve is whether or not there is anything wildly out of the ordinary. Most of the time, nothing is going to jump out and slap you in the face with its incongruity. But when it does, it’s like wham, bam, thank you, site: command!

OK, so this may be a day late and a dozen roses short of a valentine, but it’s never too late for sentiment. And I know there are a lot of far superior tools for gaining insights, don’t worry, I use those too. A lot of them are based on recommendations from Ninjas Ann Smarty and Bonnie Stefanick, who are both tool gurus. Yet there’s still merit in the classics. A lot of the bigger, better, bad asser tools actually include site: commands in their complex series of functions. But whether your research methods are tried and true or new and shiny, if you don’t know what to make of the data in front of you, you may as well be empty-handed. It’s not so much how you gather your information, it’s how you interpret it and act on it that matters.

So what’s your favorite “old school” SEO method that you proudly (or with a little shame) still use today?

]]> http://www.internetmarketingninjas.com/blog/search-engine-optimization/the-site-command/feed/ 3 The Ultimate Guide to Blocking Your Content in Search http://www.internetmarketingninjas.com/blog/tips-and-tricks/the-ultimate-guide-to-blocking-your-content-in-search/ http://www.internetmarketingninjas.com/blog/tips-and-tricks/the-ultimate-guide-to-blocking-your-content-in-search/#comments Tue, 14 Feb 2012 18:18:30 +0000 Rick DeJarnette http://www.internetmarketingninjas.com/blog/?p=2690 read on »]]> We all work so hard to make sure all of our content is crawled and indexed by the search engines. So it’s ironic when sometimes we must also struggle to remove or prevent some otherwise private content from getting into the indexes.

The process of blocking content from search can be frustrating, removal can be slow, and the whole experience exasperating – especially if you don’t know what options you have. Let’s talk about the various options you have for both removing content from the search indexes and how to prevent it from being indexed in the first place.

Find all of the affected URLs

Before you leap into the URL removal process, look to see which URLs point to the content you want removed. Think in terms of reverse canonicalization. If the content is older, it might be indexed under multiple URLs, such as:

  • xyz.com/mystuff
  • xyz.com/mystuff/
  • www.xyz.com/mystuff
  • www.xyz.com/mystuff/
  • www.xyz.com/mystuff/Index.htm
  • www.xyz.com/mystuff/index.htm

and many other variations. Identify all of the URLs pointing to the content you want removed so you are ready to remove all references to it. For more information on canonicalization concepts, see this helpful post on canonicalization.

Remove indexed content from search

There are several ways to tell the search engines the content is no longer available. Let’s jump right in.

Remove it from the web server

The easiest way to remove content from the search indexes is simply to remove it from your site. When a search crawler comes back to your site to check the status of your published content, its request for the removed content will result in HTTP status 404 messages, which tells the crawler the file can’t be found. That result kicks off the automatic (albeit slow) process of removing the URL from the index.

Set the web server to return a 404 (or 410) for the URL

If you must leave the content on the server, you can configure the web server to still return either the 404 “File Not Found” or 410 “File Gone” message for the given URL. The process of configuring a specific, non-default HTTP status message for a URL on your web server depends upon the platform used. See your web server documentation for details. Note that this technique won’t work for non-HTML content, such as PDFs and Microsoft Word DOC files.

Permanently redirect a URL

Assigning a 301 (aka permanent) redirect to a URL tells the search crawler that the requested URL is no longer available and has been permanently replaced by a substitute (the URL receiving the redirect traffic).

All of the above methods take time for the results to take effect. They are dependent upon waiting for the search crawler to return to the site, request the affected URL in order to receive the actionable HTTP status code, and then for the search engine algorithm to eventually purge the content. If the issue is an emergency, such as when proprietary business or confidential personal information is accidentally exposed, you need immediate action to get that content purged. Here’s how to do that:

Use the search engines’ webmaster tools to remove specific pages

Both Google and Bing offer tools for requesting the immediate removal of indexed content. Before you can access them, you must be a registered user of Google Webmaster Tools and Bing Webmaster Center Tools (this alone is reason enough to register your site now before an urgent problem arises).

  • Google:
  1. Log in to Google Webmaster Tools and click Site configuration > Crawler access > URL removals tab.
  2. Click Create a new removal request, type or paste the URL to be removed, and then click Continue. Remember that URLs are case sensitive, so I recommend copying and pasting the URL to be removed.
  3. From the dropdown list, select the type of data removal you want (cache only, cache and SERP, or entire directory), and then click Submit Request. Your request will appear as a listing in the tool, where you can monitor the status of the request.
  • Bing (which includes organic SERPs in Yahoo!):
  1. Log in to Bing Webmaster Center Tools and click the Index tab > Block URLs.
  2. Select the type of data removal you want (click either Block URL and Cache or Block Cache).
  3. Select what to block (page only, directory, or entire site).
  4. Copy and paste the URL to be removed, click Next, click Confirm, and then click Finish.

Note that the search engine-provided URL removal tools are typically intended for urgently needed data removals. In addition to the above techniques, there are other ways to remove content that also proactively prevents it from be indexed in the first place. Let’s explore those.

Block URLs to prevent duplicate content in the index

The most commonly used method of managing search crawler access to your site’s content is to use Robots Exclusion Protocol (REP) directives. This can be achieved through several methodologies:

Use a robots.txt file on the site

The robots.txt file is a plain text file containing crawling exclusion directives aimed at one or more REP-compliant crawlers (or most commonly, generic directives applicable to all REP-compliant crawlers). When the file is uploaded to the domain (or subdomain) root of a website, will automatically be read by REP-compliant crawlers before any URLs are fetched (all major search engine crawlers are REP-complaint). If a targeted URL is blocked by a robots.txt directive, the URL is not fetched.

The robots.txt file (note that, by protocol, this file name always uses lower-case letters) enables webmasters to block crawlers from accessing one or more particular files in a directory, whole directories, or the entire site. (Note: Per Google, this is the only approved method for removing entire directories from their index.) It also supports wildcard characters to make it extremely versatile.

The most common robots.txt instruction targets all crawlers (referred to as “user-agents” in REP). It’s followed by a specific directive, such as blocking access to a file, directory, or the site. Sample robots.txt directive code for generic user-agents looks like this:

User-agent: *
Disallow: /private.htm
Disallow: /offlimits/

You can also use Allow directives to allow crawlers access to a specific file within an otherwise blocked directory, such as in the following example:

Allow: /offlimits/index-me.htm
Disallow: /offlimits/

Note: The Allow directive takes precedence in any logic conflicts between Allow and Disallow directives, so be careful. It’s an SEO best practice to isolate allowed and disallowed files on a per directory basis to eliminate confusion.

Wildcards in directives

The “*”represents all characters in URLs up to the point of usage, meaning that the following directive,

Disallow: *cars

would block crawler access to a variety of content such as:

  • /redcars.htm
  • /roadsters/blue-cars.htm
  • /cars/black-roadster.htm
  • /2012/cars/bmw

and so on. Note that asterisks are not needed as a wildcard suffix, as the directive, by default, applies to any child content underneath the listed location in robots.txt.

The “$” character is used to filter by file name extension, such as in the following sample:

Disallow: *.pdf$

The sample code blocks crawlers from accessing all URLs containing the file type “*.pdf”. By comparison, omitting the $ wildcard would block any file paths containing the string “.pdf”, such as /docs.pdf/newcars.htm.

Wildcards can create very powerful, wide-reaching directives. However, wildcard usage in robots.txt often contains logical coding errors, which can result in unintended crawler behavior. It is extremely common for search engines to receive complaints about wildly incomplete site crawls when in fact a misconfigured robots.txt file is actually to blame, and the crawlers were simply abiding by the directives listed.

Don’t attempt to hide confidential content with robots.txt

Some webmasters, in their effort to block search crawlers from accessing their business confidential files and directories, mistakenly list them in robots.txt. What they fail to realize is that the robots.txt file is always in the same location on a site, and is always available to be read, including by people. For example, let’s say you had a robots.txt file that contained the following code:

User-agent: *
Disallow: /private/
Disallow: /client-list.php
Disallow: /secrets/

You can rest assured that your competitors, the ones who know web technologies, are snooping around your site and will see these references. They will then attempt to browse to the listed files and directories to see what’s there, such as a client list or a business expansion plan. Listing such content in robots.txt is effectively advertising where you keep your confidential documents!

To block the snoopers from probing the depths of your website for business intelligence, you can protect the directory by restricting access to authenticated usernames with passwords. If the site structure or functionality prevents you from doing that, make sure you at least have an index page in the directory so the browser doesn’t return a directory listing showing all of the files up for grabs. You may even try renaming the directory to be blocked in robots.txt to a more innocuous name or burying it in a deep subdirectory (but first be sure any such change in the URL path won’t break any functionality within your site!).

Use a <meta> robots tag on the page

The <meta> tag (or “element” for you HTML grammarian purists) can be used with REP directives. These directives apply only to the page on which they appear. The following sample code demonstrates a common usage, in which the crawler is disallowed from both indexing the content and following any of the links on the page:

<meta name="robots" content="noindex, nofollow">

Note the name attribute uses the generic value “robots”, which is applicable to all REP-compliant crawlers. You can alternatively choose to specify the exact name of a user agent as well, such as googlebot or bingbot. If you do choose to specify individual user agents, be sure the name is exactly right, or the directive may be ignored by the targeted crawler. Any crawler not identified by a specific or a generic <meta> robots directive will default to crawling the page for purposes of potentially indexing its content and following its links.

The following values for the content attribute can be used in the <meta> robots tag:

Value Function Supported by
noindex Prevents the bot from indexing the contents of the page, but links on the page can be followed. Bing, Google
nofollow Prevents the bot from following the links on the page, but the page can be indexed Bing, Google
none Equivalent to “noindex, nofollow” Google
nosnippet Prevents the display of the descriptive snippet text for that page in the SERPs. Bing, Google
noarchive Prevents the display of a cache link for that page in the SERP. Bing, Google
nocache Same as noarchive. Bing
noodp Instructs the bot to not use a title and snippet from the Open Directory Project (ODP) for that page in the SERP. Bing, Google
notranslate Prevents translation of the page in the SERP. Google
noimageindex Prevents indexing of images on the page. Google
unavailable_after: [date/time] Prevents the page from showing in the SERPs after the specified date/time. The date/time data must be in RFC 850 format. Google

Note: The attribute and value data fields are not case-sensitive.

Use the HTTP header X-Robots-Tag on the web server

For non-HTML-based content, such as TXT, DOC, and PDF documents, there is no way to apply REP directives via <meta> robots tags to them. Assuming you don’t use robots.txt for this, you can instead set REP directives for individual URLs using the HTTP header X-Robots-Tag. This header uses the same content values as shown in the table above for the <meta> robots tag. The following is an example of a commonly used X-Robots-Tag header that applies to all REP-compliant crawlers:

X-Robots-Tag: noindex, nofollow

You can optionally identify a specific crawler for a directive, and pair that with a separate directive for all other crawlers not specified, as shown in the following sample:

X-Robots-Tag: googlebot: noindex, nofollow
X-Robots-Tag: otherbot: noindex

The process for implementing custom HTTP headers is dependent upon the web server platform used. Review your web server documentation for details.

REP methodology precedence

Generally speaking, it’s best to only use one REP method of controlling crawler access for your website. Redundant methods typically result in logic conflicts, crawler access problems, and indexing shortfalls, which can be difficult to resolve.

Note that from the search engine perspective, robots.txt blocking directives take precedence. This is because before a page on a site is accessed, the crawler first checks for the presence of a robots.txt file to see if access is blocked. If so, the page is not fetched. However, to read the directives in either <meta> robots tags or the HTTP header X-Robots-Tag, the page has to first be fetched. If blocking directives are found there, only then is the page discarded. As a result, this means that the URL of the page may get indexed, but no content from that page with blocking directives will be included in the index.

There is one caveat to the precedence of robots.txt directives: when a crawler is specifically given access in robots.txt with an Allow directive but then encounters a blocking directive in either <meta> robots or X-Robot-Tag, the blocking directive overrides the Allow directive.

Lastly, the use of REP directives not only identify what content is off-limits to crawlers, if a new REP directive appears that blocks content that has already been indexed, that content is purged from the index. For more information on the robots.txt protocol, see www.robotstxt.org.

Require authentication

Another method of preventing the search crawler from accessing content is to require authentication for access. If a password is required on a site, the search crawler will not be able to access its content. Note that using Secure HTTP (https) by itself (without requiring authentication) will not block the crawler. This is a common misunderstanding and is one of the reasons why so many duplicate pages are indexed by search.

Password protect a directory on the web server

Alternatively, instead of requiring authentication to use the site, a webmaster can put content in a password-protected directory on the server to prevent crawler access. This method can be used for web server administrator-related content.

Block dynamic URL parameters

For sites that use dynamic URL parameters to track referrer data to their pages, content duplication can become a significant problem. So to prevent URLs using specific URL parameters from being indexed, and thus avoid content duplication, you can tell the search engines via their webmaster tools to ignore indexing URLs using specified URL parameters. Here’s how:

  • In Google:
  1. Log in to Google Webmaster Tools and click Site configuration > URL parameters.
  2. Click Configure URL parameters, and then click Add parameter.
  3. Type the parameter name, select whether the parameter changes what the user sees in the page, and then click Save.
  • In Bing (which also covers organic content found in Yahoo!):
  1. Log in to Bing Webmaster Center Tools and click Index tab > URL Normalization.
  2. Click Add Parameter, type the parameter name, and then click Submit.

Be careful of what you add to these lists. If your site uses URL parameters to define the page contents rather than to track referrers, you could accidentally purge a large number of pages from the search index.

Canonicalization techniques

Canonicalization is the process of redirecting unwanted URL variants for a given page to that page’s designated primary URL. Canonicalization effectively blocks those URL variants from the search engine indexes by using 301 permanent redirects. I discuss canonicalization techniques in detail, including the use of the <link> rel=canonical tag and how to set up 301 redirects in the recent blog post, The Ultimate How-To Guide on 301 Redirects. This post is long enough as is. I’ll refer you there for those details. You’re welcome!

Ineffective methods

Lastly, I’ll briefly mention what doesn’t work. For years it was a given that text content embedded within images, Flash, and other non-text media on a webpage were deal-breakers. Well, crawlers have come a long way in recent years. But don’t misunderstand me – I am not saying you should feel free to put text content you want indexed within these types of media. They are still very difficult to crawl and parse for content, and search engine success rates are not great. That all said, very difficult is not impossible.

For crawling efficiency purposes, always spoon-feed your content to crawlers as pure, on-page text. But thanks to the use of optical character recognition (OCR) technologies and improvements in crawling JavaScripts and rich Internet application technologies, a portion of this once-lost content is today being crawled and indexed. As a result, you can’t depend upon these technologies to be impenetrable walls shielding content from the prying eyes of search. You can’t rely on it to work, and you can’t rely on it to fail. What a world we live in!

Getting content out of a search engine index can be a frustrating and time-consuming experience, but it can be done. By reviewing and implementing the techniques described above, you can get confidential content purged relatively quickly as well as prevent it from being indexed again.

Be careful out there. There’s very little that’s private anymore on the web.

]]> http://www.internetmarketingninjas.com/blog/tips-and-tricks/the-ultimate-guide-to-blocking-your-content-in-search/feed/ 2