If you want in-depth information, don’t ‘just Google it’

When we need information on any topic, our natural tendency is to 'just Google it'. But Google's algorithms are designed in a way that limits one's choices. The best information is often hidden in the deep web, says Manoj Agarwal, who has worked with Yahoo Search and XoXoday

If you are among the myriads of people who depend mainly on Google while searching for information, you should be aware that Google (and other similar search engines) tell you only what you search for.

The search results are heavily dependent upon the keywords you search, how many pages you really go through in the Google search results, and so on. Google also ranks these pages in the search results based on certain algorithms to cater to a large audience. This sequence of results may not be the best for your requirement.

Even when some of your required information is available on web, Google may not have ranked those pages for various reasons. Or its page rank might be so low, and you generally leave after looking at the first search result page of Google. Many times, you don't know the right keyword which will give you relevant results in Google.

Information served up by Google is a vicious circle of popular links. The more times something is viewed on Google, the more likely it is to pop up in a search, and the more it pops up in a search, the more likely it is to be viewed. So, you have this feedback loop that results in a rapid disappearance of a lot of less 'popular' but relevant pages and information.

The ''surface web'' that search engines like Google have indexed is just the tip of the information pyramid. Search engine content is a commodity - the same information available is to everyone, so there's no real advantage.

Estimates say that no single search engine indexes more than 15-20 per cent of the content available on the web.

Beyond this surface web is a deep web, estimated to be hundreds of times larger, much better in quality, and growing faster than the surface web. Essentially it refers to content that is not indexed and therefore invisible in most internet searches. Some deep web content examples are intranets, gated websites, and access-controlled websites.

It is this deep web where one can find high quality, researched and unique content. This is where the best information can be found – information that leads to true insights, sound decisions and more creative thinking.

Search engines like Google are amazing but we need to constantly make them better and better with relevant and unique content. One should seek information from 'real sources' like newspapers or a magazine or a book, or some information from the locals, any news clip and much more. Once you digitize this information very well and Google starts crawling that information, then you can rely on Google for such information. Till then, someone must feed relevant high-quality content to Google from scratch.

Most of the 'Google-able' web content is created from existing web content from Google search results. This vicious circle of content feeds to Google results into a mess of duplicated information from one website to another.

Personalisation and recommendation algorithms in search engines are another issue leading to shallow information feeds. Google thinks it knows so much about you that it feeds you only the information it decides you would be interested in. While these algorithms do save you time, they are taking the serendipity out of the mix. Unfortunately, this way the brain is slowly getting into a design pattern and people reduce thinking beyond what is fed by these algorithms. The issue gets back to the old divide between predictability and novelty.

We like comfort zones and convenience. We stick with certain products and, especially in the internet age, want our choices delivered quickly and efficiently. However, there are times when the unexpected pays huge dividends, whether it's taking a chance on a new dish, reading a new author or simply getting lost.

I am a great believer in serendipity. Many times, while reading a newspaper, a magazine, a book, or in human conversations one gets to think beyond what one is searching for. Because you learn things by accident that you would never find if you were looking for them. It's like the information finds us rather than we are looking out for it.

Insights in this article have been drawn through my experience of working with Yahoo Search and Xoxoday. Xoxoday help customers discover and book unique experiences or activities. I generally ask this question from interview candidates: ''Where and how will you keep finding such unique experiences?'' And the answer is mostly the same: ''We will Google it.''

This is the biggest issue when you are building a discovery engine whose underlying principle is another search engine like Google. There is not enough and relevant content about experiences which is available in digital format. People generally don't realize Google won't throw up relevant results if there is not enough substance in the web.

A lot of high quality content is available outside Google in the form of physical newspapers, magazines, intranets, locals, etc, which is just neglected. The result is a vicious circle of duplicate information across the web as the feeder and output is similar.