The what and why of data warehousing

Here’s a possible requirement you may have. You want to know, say, which five of your company’s products are fetching the best profit margins, how imports will affect sales in the next two quarters, how to identify stocks of these products with the best of your retailers, how much credit to offer buyers, and whether the plant can increase production to meet higher demand created by the new advertising campaign.

If you are working with an average-sized Indian company, some of the information will be with the finance department, some with the sales department, some with manufacturing, some with the economic advisor’s office, and so on. It would take you days if not weeks to collect all the data you need, process it, and come to any useful conclusion.

What would your first reaction be? It’d probably be: "I want to have all that information on the hard disk in my desktop computer. Then I will do the analysis and decide."

Now imagine what would happen if everybody in a large organisation said the same thing ("I want all information in my own hard disk")? The organisation would collapse with the weight of the data in its system – indiscriminate and duplicated hundreds of times, yet far from integrated and useful across the organisation. Each individual would be adding to the data, changing and processing it on his own machine, and very soon the files would stop resembling one another.

It’s not merely the cost of maintaining large volumes of data that is in question. Nor even the slowing down of a network when there is an overload. It’s a question of how effectively the information is used by every concerned manager. And the integrity of all that information.

That’s where data warehousing comes in. Very briefly, it lets various departments store the data they generate internally, send up some portions to a centralised storage system, and then query the entire system for decisions that the departments have to take. It recognises two things: one, a significant part of the data in an organisation needs to be stored, processed and integrated in a central place, and, two: that decision-makers and implementers must get relevant information without delay.

You can also have data warehousing within a department – linking and supporting the information resources and requirements of, say, the marketing department, or manufacturing, or finance. In technical jargon, these function-specific data warehouses are also called data marts.

The shape a data warehouse takes is determined by the decisions that you want to take. If you are not clear about the purpose of the data warehousing, spend more time crystallising the answer to this question - which decisions need to be facilitated? That will give you the objective of the data warehousing project.

If you rush this decision, you might, after some months, have to abandon the project, and start all over again. So go over this question very carefully.