Spiders, Spiders everywhere

Not only webmasters need to understand spiders, if you are undertaking an SEO Campaign you should know something about them too.

Search engines use spiders or robots to index websites that are posted on the WWW. If your website is indexed this means that when someone types in a search query your website will come up in the result pages. On what page it is shown is another story completely. This is what SEO is about. However this is not only good news for your website. Search engines also benefit. The more correct and accurate a search engine’s index is the more people will use them to find information.

How do spiders work you may ask

Search engine spiders are nothing more than an automated program running in a search engine. Their job is to visit websites. Their purpose is reading content, Meta tags, Title tags and HTML tags. They also follow links that are placed on a website. They take back all the information they gather and index it.

The minute you submit your website to a search engine for publication the spiders start trolling it. They read what is on your web pages and return what they find to a central repository where all data is indexed. They keep coming back to your site periodically to see if anything has changed. The frequency they do this is determined by moderators. But, if your site is static chances are you will eventually stop being trolled. If your site is active, interesting and packed with fresh content you should get trolled more frequently.

It takes spiders some time to do their job. On average it takes a spider about six weeks to index an entire website. Even though they can index over a million pages per day, be warned not to expect results over night.

This is because the process that goes into indexing and converting web pages into entries on search engine pages is very complex. It involves data warehousing and information retrieval process and these vary from search engine to search engine.

more later....

remember, Pigs DO Fly