Do You Really Need a Data Warehouse?

A lot of executives, once their data systems get large enough, wonder how they can sift through all their data at a faster pace.

And they think to themselves, “Maybe we should get a Data Warehouse. That ought to speed things up. And then, I can do . . . trend analysis!!”

Unfortunately, most writings on Data Warehousing discuss matters of technical jargon, such as comparing ETL tools, Dimensional Data Modeling best practices, Inmon vs. Kimball method, and other concepts that most executives could care less about.

The executive, then, having learned nothing useful after 2 hours of research, turns to an IT manager and asks how long it will take to build a Data Warehouse – often expecting an answer like, “2 weeks”. Instead, after hearing “6 months” in response, a thud is heard throughout the office as the executive’s jaw drops and hits the carpet.

Allow me to save your jaw from hitting the ground. Here’s what you need to know – 98% lingo-free.

A Data Warehouse is a different kind of database, derived from (and an extension of) an existing database. It is useful for retrieving and summarizing lots of data very fast.

A Data Warehouse requires a time and effort to get working right. Two months absolute minimum, 6-12 months minimum for enterprise-level. If you can live with fast (and not very fast) reporting, there is another option.

A Low-Cost Alternative

There is no official name for this; I call it the “Semi Data Warehouse” (SDW).

Here are some quick facts about the SDW:

The SDW is a hybrid of the Database and Data Warehouse concepts.
The SDW is still an extension on top of the current Database.
The SDW still requires planning, programming, and sometimes extra servers.
The SDW offers more powerful reporting abilities than a regular database.
The SDW is not as powerful or feature-rich as a full-scale Data Warehouse.
The SDW’s big advantage lies in the cost and time savings:
- A Semi Data Warehouse requires 1-2 weeks to create and start using.
- A Full Data Warehouse requires 2-12 months to create and start using.

The Semi Data Warehouse uses a de-normalized data format that makes it fast and easy to select data into reports. Regular databases store data in a format called 3rd Normal Form (3NF). This gives the database tremendous flexibility to expand, however data is stored in many different places and re-assembling that data for a report requires a lot of time. SDWs create a pre-gathered set of data, from those many different places, and so reports can be built and refreshed much faster.

Again, the SDW will give you fast (and not very fast) reporting. A large, 3NF-database report may take 10 minutes to run; the same report from a SDW may take 1 minute; a Data Warehouse may take 2 seconds. If the SDW solution is fast enough, you do not need a Data Warehouse. If it is not fast enough, you do need a Data Warehouse.