Update 30 June 2009: Added a couple of links at the end of this post for additional information from some recent research/experimenting.
Update 19 July 2014: Translated into Swedish by Valeria Aleksandrova. Many thanks!
Web standards is generally a good thing. When done well, it can help to create lean pages, improve markup quality, provide better accessibility and can be easier to maintain (even while supporting IE6!)
One claim often made in the past to promote web standards is that following web standards (using CSS-based layouts, proper use of heading elements in markup, etc) will improve SEO.
Though this might be desirable, it is not the case; most sites on the web don’t follow web standards and most rank very high in their areas. Search engines are not going to penalize the 90+ % of sites that don’t follow web standards; that’s like shooting themselves in the foot!
As I describe in an earlier post on search engine ranking vs indexing while there are technical things one can do to ensure the site gets indexed well (e.g. URL canonicalization, good use of the <title /> element, etc), for ranking it is mostly about having good content that people will link to.
But as Matt Cutts, head of Google’s web spam team notes in this very short video, using CSS-based layouts or table-based layouts will have no bearing on search engine friendliness:
There are many good reasons to follow web standards principles, to use CSS-based layout and progressive enhancement but SEO, unfortunately, is not one of them.
Update 30 June 2009: A couple of decent articles recently talk about similar things: