Serious Robots.txt Misuse & High Impact Solutions

Some of the Internet's most important pages from many of the most linked-to domains, are blocked by a robots.txt file. Does your website misuse the robots.txt file, too? Find out how search engines really treat robots.txt blocked files, entertain yourself with a few seriously flawed implementation examples and learn how to avoid the same mistakes yourself.

The robots.txt protocol was established in 1994 as a way for webmasters to indicate which pages and directories should not be accessed by bots. To this day, respectable bots adhere to the entries in the file...but only to a point.

Your Pages Could Still Show Up in the SERPs

Bots that follow the instructions of the robots.txt file, including Google and the other big guys, won’t index the content of the page but they may still put the page in their index. We’ve all seen these limited listings in the Google SERPs. Below are two examples of pages that have been excluded using the robots.txt file yet still show up in Google.

Cisco Login Page

The below highlighted Cisco login page is blocked in the robots.txt file, but shows up with a limited listing on the second page of a Google search for ‘login’. Note that the Title Tag and URL are included in the listing. The only thing missing is the Meta Description or a snippet of text from the page.ReadMore

Original Article from:http://www.seomoz.org and the Author is Lindsay

Google Upgrades Punctuation Search Results

Google has updated its search results for queries involving punctuation marks and symbols. Searching for the symbol equivalent of a period, comma, carat, percent sign and more symbols now will return search results – though they aren't always exactly relevant, Google Operating System reported.

Punctuation marks now generating search results, generally where the Wikipedia page about the symbol ranks as the top link, include:

• . (full stop/period)
• , (comma)
• : (colon) – however, the colon that is a piece of the large intestine currently outranks the punctuation mark of the same name
• ; (semicolon)
• # (number sign)
• % (percent sign)
• @ (at sign)
• ^ (caret)
• ( ) { } [ ] (bracket) – parentheses, brackets and curly brackets are all combined into the same search result
• ~ (tilde)
• | (vertical bar)
• “ (quotation marks)
• < (less-than sign)
• > (greater-than sign)
• $ (dollar sign)

Other symbols and punctuation marks that Google can recognize and will return results for:

• ! (exclamation point)
• ‘ (apostrophe)
• & (ampersand)
• _ (underscore)
• - (minus sign)
• + (plus sign)
• = (equals sign)
• \ (backslash)
• / (slash) – however, guitarist Slash currently outranks the punctuation mark of the forward slash Read More

SOPA, PIPA On Hold, But a New Threat on Horizon

Late last week, announcements came down putting both PIPA and SOPA on hold. Nevada Senator Harry Reid has postponed the Protect I.P. vote, while House Judiciary Committee Chairman Lamar Smith announced the committee will “postpone consideration of the legislation until there is wider agreement on a solution.”

Meanwhile, Smith is again pushing his Protecting Children from Internet Pornographers Act of 2011; it has been amended, passed the required committees, and was put back on the Union Calendar December 16, 2011. Who wouldn’t want to protect children, you ask? They must be monsters! This is precisely the mentality that may see H.R.1981, known as the Data Retention Bill by protesters, pushed through, as it aims to force all U.S. Internet service providers to retain all customer IP addresses for 18 months.

Don’t count either PIPA or SOPA out for the count. Though this type of backlash is a politician’s worst nightmare in an election year, vested parties are none too pleased the bills didn’t pass and will continue to pressure lawmakers to enact anti-piracy legislation. ReadMore

Tablet Visitors Spent 54% More Than Smartphone Visitors in 2011

People using tablets spent $123 on average per purchase in 2011, 54 percent more than visitors using smartphones, according to an Adobe study.

"Mobile is no longer a one-size-fits-all-strategy. It's like saying you have an Internet strategy. You have to get more granular. Audiences are really different," John Mellor, VP of business development for Adobe's Digital Marketing Business, said in an interview. He recommends that retailers segment their customers to account for the different audiences and their behaviors. ReaMore