Web standards is generally a good thing. When done well, it can help to create lean pages, improve markup quality, provide better accessibility and can be easier to maintain (even while supporting IE6!)

One claim often made in the past to promote web standards is that following web standards (using CSS-based layouts, proper use of heading elements in markup, etc) will improve SEO.

Though this might be desirable, it is not the case; most sites on the web don’t follow web standards and most rank very high in their areas. Search engines are not going to penalize the 90+ % of sites that don’t follow web standards; that’s like shooting themselves in the foot!

As I describe in an earlier post on search engine ranking vs indexing while there are technical things one can do to ensure the site gets indexed well (e.g. URL canonicalization, good use of the <title /> element, etc), for ranking it is mostly about having good content that people will link to.

But as Matt Cutts, head of Google’s web spam team notes in this very short video, using CSS-based layouts or table-based layouts will have no bearing on search engine friendliness:

There are many good reasons to follow web standards principles, to use CSS-based layout and progressive enhancement but SEO, unfortunately, is not one of them.

Update 30 June 2009: A couple of decent articles recently talk about similar things:

1. SEO Best Practices: SEOmoz’s New Policies Based on Updated Correlation Data
2. The Triviality of On-Page HTML Tag Optimization
   

Why bother?

Visual designs often require hiding text in such a way that screen readers will still read the text and thus provide extra context for the screen reader user.

For example, a visual design may make the main navigation section of the page obvious and so a heading to announce it may be unnecessary or undesirable.

However, from the perspective of good document structure/semantics and accessibility, it would be good to have the heading in there.

Screen reader users, for example, often scan a page through having a list of headings being read out to them.

The problem of display:none in CSS?

display:none in CSS would seem ideal (plus some additional rules in an aural stylesheet, perhaps).

Unfortunately, screen readers are inconsistent in the way they handle display:none.

In addition, most do not yet support the aural stylesheet sufficiently.

Positioning text off the screen is more reliable

For years, I have been using an “off-top” technique whereby content can be moved off the top of the screen, in such a way that it will still be read out by screen readers. Here is an example:

.css-label {
    height:1px;
    left:0px;
    overflow:hidden;
    position:absolute;
    top:-500em;
    width:1px;
}

A simple example of using the above may be to provide a header to a section such as the navigation:

<h2 class="css-label">Navigation</h2>

Some use use an “off-left” approach instead, though it seems to cause the odd problems with earlier versions of Opera and Safari (any one using those anymore?)

That being said, a comment left on Gez Lemon’s post about display:none issues answers my question of whether off-top is better or not than off-left saying there may be some issues with off-top. I’ll need to look into that further, but either of those are certainly better than using display:none.

More information

For further reading, see the following:

• Hiding Content from View, AbilityNet, November 2007
• Screen Readers and display: none, Gez Lemon, Juicy Studio, 12 October, 2007

Will search engines see this as keyword stuffing?

A comment left on Gez Lemon’s post asked what would be the impact on search engine listing?

Comments are closed on that post, so I will try to answer it here, as it is something I have been wondering for a long time.

The concern from an SEO point of view is that search engines may think you are tricking them even though this is a legitimate techniques for a legitimate purpose. However, some people could use these techniques for keyword stuffing, thus putting this useful technique in jeopardy and making it harder to provide sufficient context to assistive technologies.

Search engines would likely have a hard time detecting this in an automated manner

But for search engines to detect this they would at least need to parse all the CSS. Then, they would have to looking for all the different permutation of these rules and be 100% sure this is being done for keyword stuffing and not for some other legitimate reason.

Sounds tricky (and easy to circumvent — e.g. by using JavaScript to set these styles, as it is only being hidden from visual user agents).

Or, search engines would need to implement an entire DOM and calculate where on a screen all the text would go. Some major web browsers struggle to do this properly, so search engines aren’t likely to do it too well either, I would think.

But they may use manual checks for better results; that would be good for legitimate use of this technique

However, while automation and throwing lots of servers at a problem may seem problematic (for now!), these things require human, subjective, interpretation. So why not have people check sites manually?

Google, for example, does have a little army of people to manually verify sites are not using unfair tricks. So I guess if they are concerned about this particular technique, they might be able to get software to go a certain distance and then flag up suspicious cases to be manually checked. Legitimate sites should then be okay.

More information

But don’t take this as advice! Search engines often change the way they work! So check out this article from SEOMoz, a prominent SEO web site: A Comprehensive Guide to Hidden Text & Search Engines (10 July, 2007).

without knowing the ultimate reason for doing the things they do, they [search engine optimiser professionals] don’t get it done correctly. You don’t create titles for titles’ sake. You don’t get links for links’ sake. Everything that we teach people to do in SEO has a purpose, and that purpose is not to make the search engines think our site is better than it is. The purpose is to actually make the site better than it is.

– Jill Whalen, Avoiding Clueless-Is As Clueless-Does SEO, Search engine land, September 27, 2007 (Emphasis is original)

In a way, this is obvious, but Whalen’s point is that many SEO people simply go through the motions and almost forget about creating good content; the technical on-page factors such as a good title, link text, etc are there because these are naturally important for good content, not solely for search engine optimisation.

As I mentioned in a previous post on SEO, “the task of getting good ranking is ultimately a business/marketing strategy: sites need to have compelling enough content for others to want to link to them.”

It reminds me of a few people I have talked to in recent months about SEO who say they would put extra title attributes (tooltips) on links, images, and even headings and paragraphs! To me that is excessive (and way out-dated); if it doesn’t help the end user, it is very likely not to be useful for search engines.

Yahoo is providing a Dynamic URL Rewriting tool

This tool will allow you to tell Yahoo which querystrings to ignore in a URL.

This is really good because one of the challenges for SEO has often been to ensure everyone always uses the same URL to avoid search engine dilution.

(E.g. somewhere.com/product/?id=123&src=affiliate1 is treated differently to somewhere.com/product/?id=123&src=affiliate2 by search engines — they don’t know (and can’t assume) that the “src” querystring parameter doesn’t (mostly likely) change the content. Links to a product page using such varied parameters means less weighting, or page dilution, which may affect ranking.)

Read more about this:

• A Yahoo blog post announcement
• Yahoo Site Explorer help on the topic

Yahoo’s tool is an excellent move. It needs the other big players to do similar things before we can be more relaxed on our URLs and their querystrings.

Microsoft Live Search announces a web master SEO tool

This tool sounds similar to google’s and will offer:

• Troubleshooting tools to ensure MSNBot is effectively crawling and indexing your site
• Sitemap creation, submission and ping tools
• Statistics about your website
• Consolidation of content submission resources
• New content and community resources

Read more in their announcement.

Good progress on both fronts.

(An irony is that it was Google’s Matt Cutts who mentioned these in one of his blog posts!)

I have seen IT tenders that requires a site that “must” appear on the first page of a search query. From a business perspective, this requirement is understandable. However, for an IT/web development/search company to somehow guarantee this is misleading and irresponsible. However, it is an opportunity to explain “natural search.”

Explaining Natural Search to business managers

I usually try to explain that natural search engine optimisation (SEO) involves at least two areas:

Search engine indexing

Getting search engines inside your site to understand the content

Search engine ranking

Determining how to rank the content

I then try to explain that the issues/techniques for each of these areas can be quite different (some may overlap though) and I often sum it as:

• Improving indexing is mostly a technical task
• Improving ranking is mostly a business/marketing strategy
• What might work now may not work in the future
• It takes time

Improving indexing; mostly a technical task

Technical things that web developers do typically helps increase the chance a site is indexed well. For example (in no particular order, and not a complete list!):

• Sufficiently clean URLs to avoid page weight dilution. This usually means trying to ensure that everyone uses the same link to a given page, with no variation in the way querystrings or paths are written (else the search engines will assume these are different pages)
• Good use of the <title> element as well as meta keywords and descriptions so that if a page does show up in the rankings, the summary information will be useful (these elements have little impact on ranking these days, but are useful for indexing).
• Proper use of redirects and other HTTP status codes to help search engines follow or not follow certain pages. For example, an HTTP 302 is for a temporary redirect and will not be followed by search engines. A 301 Permanent redirect will be.
• Sitemaps especially if your content is vast.
• etc

Side note about HTTP status codes

In some web technologies such as ASP/ASP.NET, a Response.Redirect issues an HTTP 302 and is the most common way to do redirects. Although headers can be set manually, there are no convenient methods for permanent redirects, and so developers often overlook this crucial difference.

It is especially important during site redesigns to use 301s to redirect old URLs to new ones. Otherwise, all those people linking to your old pages will be left out, and search engines won’t pass on the weighting/recognition of those links to the new pages.

Sometimes using proper 404 Not Found status codes on pages that query databases might be useful if you do not want the search engine to index that page (maybe your site no longer sells those products).

Or, similarly a 500 Internal Server Error is very important. For example, if there is a temporary glitch resulting in the page showing some error information, without the right status code, the search engine will index that content, even replacing your previously good indexed content!

What about using web standards?

Some web standards advocates will be surprised to read that I have not included the use of standards-compliant HTML markup and appropriate use of headers from a search engine perspective.

While these techniques are undoubtedly crucial for accessibility and forms the basis for any modern web development strategy, their value for search engine indexing or ranking is questionable (unfortunately), because spammers can easily abuse elements such as <h1>.

That being said, avoiding table-based layout and following web standards can help because:

• Standards help to minimize code bloat (some search engines limit how much of a page they will index, though increasingly less important it seems).
• A valid page does not hurt in ensuring a search engine can understand your content. An invalid page might be so invalid that even if it somehow renders okay, a technical program such as a search engine robot may struggle to make sense of it.
• Use of proper markup such as headers do help users, which can help with ranking indirectly (explained below)

There used to be a time when having content first and navigation last was helped too (something done reasonably easily with CSS, and to some extent with HTML layout tables too). But even this technique is less important. If something like HTML 5 becomes more prominent, then elements such as <nav> will make source code order less relevant (although search engines will still need to deal with abuse of those elements!)

Improving ranking; mostly a business/marketing strategy

To get good ranking however, there is typically very little a web developer can do. Ranking, these days, boils down to search engines trying to determine how popular your site is. The way they do that is through seeing how many links your page(s) get and the nature of those inbound links. (You can also provide internal links from some pages to others, and that can sometimes help, but most SEO experts seem to find that it is the external inbound links that are key.)

This means the task of getting good ranking is ultimately a business/marketing strategy: sites need to have compelling enough content for others to want to link to them.

There are of course always caveats or exceptions. For example, a new site on a very niche area may rank highly on those niche keywords (but the number of people searching for such niche words may be small too).

One of the few technical things that may help (though not necessarily “technical” as such) is training business/marketing to encourage those linking to your pages to use relevant text in those links, such as the title of the page, instead of “click here” or “more info.”

Providing content management systems that allow content creators to provide proper titles, keywords, descriptions, etc is also important.

What about link farms, keyword stuffing, etc?

So, you may have received lots of requests to join various link farms to help each other promote themselves. Search engines try to watch out for these things and only factor relevant in-bound links, by analysing the topics and keywords of the other site. Also, if the other site linking to you is itself determined to be popular, then your page’s weighting increases accordingly. It is kind of like search engines on the look out for who “votes” for your page.

Some people have tried to stuff key words everywhere in the content (I am surprised some developers even advocate use of the HTML title attribute in many places in the belief it will aide with search engine ranking!). Again, this misses the point that this technical trick is likely not to work (and actually make the site more noisy, especially for those using assistive technologies).

Trying to trick search engines isn’t worth it; you will likely get found out and deslisted. Building up your ranking and reputation will be difficult. For example, BMW was delisted from Google’s listings for providing different content to search engines and users. After changing their practices they were listed again, but few online businesses can afford such delisting.

Search engine companies such as Google provide guidelines that ultimately advise web masters not to trick search engines, but instead concentrate on creating sites for humans to consume; search engines will pick up on that and visit accordingly.

Here are some examples from Google:

Don’t fill your page with lists of keywords, attempt to “cloak” pages, or put up “crawler only” pages. If your site contains pages, links, or text that you don’t intend visitors to see, Google considers those links and pages deceptive and may ignore your site.

Don’t feel obligated to purchase a search engine optimization service. Some companies claim to “guarantee” high ranking for your site in Google’s search results. While legitimate consulting firms can improve your site’s flow and content, others employ deceptive tactics in an attempt to fool search engines. Be careful; if your domain is affiliated with one of these deceptive services, it could be banned from our index.

Don’t use images to display important names, content, or links. Our crawler doesn’t recognize text contained in graphics. Use ALT attributes if the main content and keywords on your page can’t be formatted in regular HTML.

– How can I create a Google-friendly site?, Google.com, accessed September 9, 2007

And:

Quality guidelines – basic principles

• Make pages for users, not for search engines. Don’t deceive your users or present different content to search engines…
• Avoid tricks intended to improve search engine rankings. … ask, “Does this help my users? Would I do this if search engines didn’t exist?”
• Don’t participate in link schemes designed to increase your site’s ranking or PageRank. … avoid links to web spammers … as your own ranking may be affected adversely by those links.

– Webmaster Guidelines, Google.com, accessed September 9, 2007

What works now may not work in the future

Search engine companies are always looking to improve their algorithms so what seems true today may not be the case tomorrow. They also try to guard their algorithms as much as possible so a lot of the above comes from trial and error.

I have often come across people who encourage techniques which are no longer as relevant as they may have once been. It is a rapidly changing area. For example, a useful search engine resource, SEOmoz, provides an excellent summary of ranking factors for 2007 and compares them to just two years ago, showing that even then factors have changed considerably.

Here is a part of their summary:

Top 10 Ranking Factors in 2005:

1. Title Tag
2. Anchor Text of Links
3. Keyword Use in Document Text
4. Accessibility of Document
5. Links to Document from Site-Internal Pages
6. Primary Subject Matter of Site
7. External Links to Linking Pages
8. Link Popularity of Site in Topical Community
9. Global Link Popularity of Site
10. Keyword Spamming

Top 10 Ranking Factors in 2007:

1. Keyword Use in Title Tag
2. Global Link Popularity of Site
3. Anchor Text of Inbound Link
4. Link Popularity within the Site’s Internal Link Structure
5. Age of Site
6. Topical Relevance of Inbound Links to Site
7. Link Popularity of Site in Topical Community
8. Keyword Use in Body Text
9. Global Link Popularity of Linking Site
10. Topical Relationship of Linking Page

For me, this is one of the most valuable documents on the web for determining how to approach an overall SEO strategy. While the factors may not be perfect, they give a remarkably concise and trustworthy view of what makes a site rank well at Google. I hope you all enjoy it as much as I have – please add your thoughts in the comments!

– Ranking Factors Version 2 Released, April 3, 2007, SEOmoz

It takes time

Businesses can understandably expect a new site to start ranking highly quickly, even if it is a prominent brand. However, the reality is that it takes time (and effort) to build up the critical mass and quality in-bound links. Some search engine companies, such as Google are even factoring in the age of the site into some of their ranking algorithms.

Other times, new content (e.g. a news-based site) even from a new site can rank highly quickly (temporarily or for a long time), especially on niche topics.

Bottom line though is that nothing is really guaranteed!

Terminology confuses matters?

“SEO” is probably a misleading phrase; you don’t optimise google (unless you work there!).

Maybe two terms should be used to help with communication: Search Engine Marketing (SEM) is already often used when talking about making a site compelling enough for others to link to it and improve ranking. So how about something like Search Engine Visibility (SEV) when talking about indexing?

Or maybe I contribute to the confusion by trying introducing another TLA!