Some search engines have limits on the amount of a page they index. Many of them leave a website when they hit javascript, or long segments of CSS. There are many good graphic elements of websites that ‘trip up’ search engine crawlers and lower a web page’s rank.

Web Design Goals

The first step is to create goals. Not every page needs to rank high. A website should have some pages created for human consumption. A website needs to sell. If the entire website is optimized for search engines, then the ‘conversion to buyer ratio’ will drop.

Many websites crate landing pages for the crawlers. These are often HTML pages, or have the CSS re-arranged so the crawlers can read the pages. A good web designer can create a landing page that mimics the rest of the website, but is still optimized for websites.

Foundation of SEO

To understand SEO, you need to know that web crawlers work on mathematical algorithms. These do not change unless the search engine changes them. There are a certain set of rules. There are no magic formulas, and no one can get an edge. Those who follow the rules, rank high.

Advancement Of Code

AT the moment, optimized means using Plain Old Semantic HTML (POSH). This combined with user defined, Web Ontology Language (OWL) applications such as W3C validation. However, I don’t recommend getting caught up on the Ontology language and creating Website Parse Templates. You’ll run into the same problem as CSS causes, or the first Content Management Systems caused.

• Websites do not read CSS meta tags
• The ontology language is just tags – no content on the page
• Too much coding before content
• Content does not appear on the page, it is just a tag

The solutions between ranking high, generating traffic, and controlling web content are not easy to find.

Solution: Distributed Web Crawling

Google uses thousands of individual computers to crawl the Web. Newer projects use a collaboration platform at enlists volunteers crawl the web. LookSmart is the largest search engine to use this technique, which powers its Grub distributed web-crawling project. After downloading crawled web pages, they are sent to the central servers. This is a twist on Open Source search engines: Grub, Nutch, Sphinx, Zettair, etc.

Solution: User Tracking (Desktop Search)

Another solution on the horizon is indexing information collected while tracking a user’s internet use. The information is sent to search engines and indexed. Desktop search engines include Google Desktop/toolbar, Ask.com, Windows Search.

The conclusion, is to keep an open mind when designing websites. It is more important than ever to make websites W3C. Start exploring the optimization techniques for the new generation of search engines. These Tools are still in the development stage, and have not evolved to the Internet - but they will.