the amazing crawlers

Nope I’m not talking about Spiderman, but if the internet had superheroes, they would probably be these “crawlers” I’m talking about! Crawlers, also known as “spiders” or “robots”, forms an important aspect of search engine technology. These invisible programs are sent out constantly by search engines such as Google and Bing to crawl all over billions of websites. They do so to collect information from the web to build a searchable index for the search engine we use to seek information.

Here’s a useful video by Google that talked about crawlers (or spiders as they call it):

Quoting Matt, basically they “start by crawling a few web pages, then they follow the links on those pages and fetch the pages they point to, and follow all the links on those pages and fetch the pages they link to, and so on, until they’ve indexed a big chunk of the web.”



These programs determine whether or not some information you churn out on your webpage gets indexed by the search engines. So the smart thing as a marketeer to do is to 1) make sure crawlers get to see what you want them to see, 2) do not lead them to places that you do not need them to be!

How to optimise your web to be “crawler-friendly”?

A simple search on Google shows many websites that actually teaches us how to make a website crawler-friendly. Some basic ones include:

2) Simplifying URLs: Crawlers, like us humans likes URLs which looks logical. URLs that has many numbers and weird symbols doesn’t make any sense to crawlers and are less likely to be crawled and indexed properly. Instead, URLs which are named properly makes navigation easy, benefiting both the crawlers AND the readers.

3) Using Alt text: Sadly, crawlers are blind to images. So in order for them to know what your images are about, they read the alternative text you set for the images. Hence it is important to name your images on the site relevant to its content!

3) Being careful of rich media: Text still plays a big role in reaching out to crawlers. They may have some trouble reading rich media files such as Javascript, Flash or silverlight. However, if removing them completely means making your site less interesting, then it might be a bad idea to do so. Furthermore, Google has improved their methods of indexing flash files.

On the other hand, if you have a site that includes many different (unecessary) pages, you might want to block them from being read by crawlers! This would allow the pages that you want to be featured to be indexed by the search engines. This can be done by using robots.txt files to prevent crawlers from going to certain pages. Here’s a great guide that teaches us how to create such files!

From what I see, crawlers are hard working creatures that follows links and absorb almost everything you present on the web. Making our website crawler friendly- directing them to pages you want them to index and stopping them from going to pages you do not want them to go to definitely helps in rising up in the search rankings!

Was this post helpful? Did I miss anything out? Comment and let me know!



“Half the money I spend on advertising is wasted; the trouble is I don’t know which half.”

John Wanamaker


Have you ever “Googled” yourself (your own name) to see what kind of information pops up? Or spent a day writing an emotionally-charged blog entry, published, and “Googled” it to see whether you can find it among the first few pages? If you did, DON’t WORRY — it’s a perfectly normal behaviour (it even has a term called Egosurfing); but If you haven’t, you should probably do it, but do avoid admitting to it to appear less narcissistic.

As vain as it may sound, “Googling” themselves should be an essential activity for digital marketers, to check where their company lies on the pages of the search engine. Why so?

It’s hard to imagine a world without the search engine. Whenever we’re looking for certain information, the search engine is the first place we’ll go to. More importantly, it plays a huge role in consumer decision making process. Whether it’s buying a book, clothes or even a car, the first place they would probably go to would be the search engine to type out their query. In today’s world, businesses would probably lose out if they do not establish a presence in the online world, and part of that “presence” would be whether they appear on the top of the search list.


MRSIEW RECOMMENDS (some search engines to consider)

1) Google

Google Logo 2010
What is Google? Please Google it to find out (I’m kidding, but if you do not know what is Google, it makes not sense that you’re reading this entry)

Why Google? The word “Google” has almost become synonymous with the word “search”, and they have astoundingly made up more than 40% of all internet traffic (link).

Will Google sustain future?  As mighty as Google sounds, it is without its criticisms (link). However, It’s hardly foreseeable that any search engine as efficient will appear and take over in the near future. But as rapidly changing as technology goes, it’s really hard to tell.

mrsiew’s verdict:

This verdict might be a little biased since I’ve already started the entry equating Google to search. I usually start my day with Google, whenever i boot up the computer and open my Chrome browser. It’s so convenient and powerful that i don’t even bother looking at another search engine site. Sometimes I wonder if the results that come up are the most relevant ones? But usually I don’t really think too much.

What do you think?

2) Bing

What is Bing? Brainchild of Microsoft. Of course the technology giant wants a piece of the search engine market share.

mrsiew’s verdict:

I don’t know how to feel about it. It’s automatically configured as the designated search engine in my school computers (just like how Internet Explorer is the default internet browser). It gives me good search results, but intuitively I would start comparing it with Google and can’t help but to feel that it’s slower, and gives less meaningful results.

What do you think?

3) Yahoo!

What is Yahoo!? A wildly popular website in the 90s which curated a list of useful and popular weblinks. The size of their list grew so much they became a searchable directory. However, they did not work on their search engine until 2002, which was mostly outsourced.

mrsiew’s verdict:

I remembered playing with Yahoo! in primary school with so many cool links to click on. But in terms of search, sad to say, it’s not really relevant anymore. The search results are disappointing compared to Google. I experimented by typing “Happy” in both Yahoo and Google and Google seemed to be able to read my mind. The first thing that came up was the Happy video by Pharrell Williams which was exactly what I was expecting rather than a Yahoo! news about a hongkong actress I’ve never heard of.

What do you think?

Of course there are many other search engines including popular ones in different countries such as Baidu (Chinese) and Naver (Korean). However, Google is one of the biggest so far, and as a starter, marketeers should aim to have their pages noticeable in Google search results.

How to stay “searchable”? Search Engine Marketing

After understanding the importance of search in digital marketing, what’s next? Marketeers can either:
1) Engage in SEO – optimising your web content to be search friendly, or
2) Pay to get searched
3) Do absolutely nothing (and get busy with other work)

Option 1: Search Engine Optimisation (SEO)

In simpler words, it is to use tactics to improve the change that the site will appear high in the organic search results page. Google has actually come up with a handbook to help you get started. It may be a little time consuming to go through the steps and takes some effort in re-configuring your web design, but since it’s free — why not give it a try?

Option 2: Paid search

Paid search was actually quite a genius idea came up by Bill Gross in one of  his venture, However, after a series of unfortunate events, Google has taken over as the biggest revenue generating search engine raking in more than $50 billion revenue in advertisements ( $50,000,000,000 — that’s a lot of zeroes) The author John Batelle has has managed to document the story in a very interesting way in his book in his book (The Search: How Google and It’s Rivals Rewrote the Rules of Business and Transformed Our Culture). You’d want to read it if you enjoyed watching “The Social Network”.

How does it work actually? It’s similar to paying for TV ads or newspaper ads, but a little different. Instead of paying for a space on a high traffic website to place your banner ads, paid search works on the basis of keywords. Let’s say you are marketing for a second-hand car dealer and you want to expand your online presence in order to attract more customers. What you can do is to bid for keywords like “cars” or “second hand cars” or “mercedes”, and search engines would rank your web link up the pages, so whenever people search for such keywords, the first result they see would be your second-hand car dealing webpage!

Such a system of keyboard bidding is termed “Adwords” by Google who has come up with an efficient bidding system for companies who wishes to have their sites rise up in the Google ranking. Here’s a link to a simple infographic by Wordstream that explains how Adwords Work: link

Option 3: Do nothing

Of course, marketeers can do absolutely nothing about search engine marketing and keep themselves busy with other activities. That is if you’re able to find a way to get people coming to your site skipping the search process.

mrsiew says: GO SEARCH OR GO HOME! there’s no excuse to be lazy if you want your company/brand to be seen. It may take quite some effort and the results could only surface gradually; but if you put in effort, at least one day you won’t be surprised when your name appears on the first page of Google!

Update: This post gives just a really brief introduction to search, and there are still many layers of the technology to be discussed! If there’s anything you might want to know more or provide information on about SEARCH, please do not hesitate to leave a comment!