what the heck is a data warehouse

Gartner says that data warehouse is the next big thing. Oracle is big into this space. It is something that many customers are interested in. It begs the questions, what the heck is data warehouse and why should you care?

The key words that are usually associated with data warehouse are consolidation, business intelligence, single server, data mining, and analytics. What I can gather is that data warehousing is a mechanism for gathering key metrics about the business on one server so that you can analyze trends, changes, or requirements of your business. Isn’t this what your database does for you anyway? Well, not really. You use the database to store HR data for payroll and benefits. You use a different instance to store inventory and accounts reveivable. Yet another for manufacturing or shipping. Some of these are transactional in nature with entries happening every minute. Others are batch operations to generate paychecks every week, two weeks, or once a month.

Unfortunately, most companies have different departments that manage this data. Accounting and finance deal with things money related. Manufacturing deals with inventory, catalogs, and part lists. HR deals with people in the company. Sales deals with customer information and retail web sites. Data warehouse tries to integrate these departments and aggregate much of the data so that an executive can look at things like marketing defects and customer satisfication issues as well as inventory turns and the cost of shipping and storing parts. If our company, for example, generates plastic parts for our product and that product is starting to have a higher rate of returns and customer complaints, it might be of benefit to look at our suppliers for plastic as well as our die machine that molds the plastic parts. If the die machine is old or does not work properly with a new additive from our supplier, we need to find this out. The inventory and parts list repository will not show this.

My understanding is that you can currently create a star schema that crosses many tables in one database instance. This allows you to create materialized views that cross department boundaries. Most modern databases can cache and optimize these views. When data is updated in your inventory system, theoretically it is copies to your data warehouse and the materialized view that is in your data warehouse is located is updated. This does not negatively impact your inventory system other than copying the data to a remote system. I dont’ truly understand the need for data cubes and how they apply to this technology. It is something that I need to do more research on.

Data mining is another concept that comes up as part of a warehouse. It is typically said in conjunction with modeling and predictive analytics. Regression analysis and linear models appear to be important as well as deviation from predictions. I remember some of this from control systems but I am sure that the technology on keeping a pendilum swinging upside down is different than that of keeping inventory minimized and product quality maximized.

ETL is another feature that seems to be important. This piece typically consists of a point to point connection to a data stream and data translation in some way. If something changes in your mainframe, it is copied into your warehouse and converted from the language that it is stored into a format that works in a database. It also has the ability to convert data from one representation to another. For example, sales in Europe might be stored in Euros. If we want to aggregate data into a warehouse we might want to standardize on US Dollars as it is copied.

The biggest problems that I have seen for building a data warehouse are not technical but political. The key issues are around funding, data ownership, and data access. No one wants to fund something for the whole company out of their budget. Since IT is typically a cost center and not a revenue center, it is difficult for IT to being this type of initiative. What usually happens is that you get a mini-warehouse between two departments or data shared in a spreadsheet that is not available to anyone else. Some times you do see an analytics tool that runs on top of a manufacturing system but only runs at night since the system is tuned for high rate of transactions during the day and not batch operations. Mixing the two has traditionally caused problems with one or both functions.

What I need to figure out moving forward with this is:
1) how do cubes relate to data warehousing and what value does it bring.
2) what impact is there on copying data to a data warhouse from my existing system
3) is there a good example that shows a tangible benefit to building a warehouse over building integration between two or three systems
4) who is the end user and a typical use case for a warehouse? Is it just used by the higher level managers or by mid tier managers to run their operations?

Post Views: 1,371

404 thoughts on “what the heck is a data warehouse”

  81. Thank you a lot for providing individuals with such a memorable opportunity to discover important secrets from here. It’s always so terrific and as well , stuffed with a good time for me personally and my office colleagues to visit your site the equivalent of three times every week to study the newest guidance you will have. Of course, I’m actually fascinated for the eye-popping advice you give. Selected 2 facts on this page are undeniably the finest we have all had.

    Reply

  82. I in addition to my pals were checking the good information and facts located on your web page while all of the sudden got an awful feeling I had not thanked the web site owner for those tips. The ladies are already as a consequence glad to read through them and have now in fact been tapping into those things. Thank you for simply being simply thoughtful and for opting for certain high-quality guides millions of individuals are really eager to be aware of. Our own sincere regret for not saying thanks to sooner.

    Reply

  89. I am glad for commenting to make you understand of the outstanding discovery my cousin’s daughter enjoyed going through yuor web blog. She realized a wide variety of issues, which included what it is like to possess an amazing helping nature to have certain people without hassle learn a number of multifaceted topics. You really did more than people’s expectations. Thanks for supplying these beneficial, trusted, explanatory and also cool tips about the topic to Emily.

    Reply

  90. I simply wished to thank you so much all over again. I am not sure what I would have tried without the tricks discussed by you directly on this concern. Certainly was an absolute hard setting in my view, nevertheless taking a look at a new specialised strategy you treated that made me to leap with happiness. Now i’m happy for this information as well as expect you are aware of a powerful job you are always getting into training people through a web site. I am certain you haven’t got to know any of us.

    Reply

  91. I together with my guys were analyzing the best information and facts located on your web blog and then suddenly developed an awful suspicion I never thanked the site owner for those tips. Those guys appeared to be absolutely warmed to read all of them and now have really been tapping into them. Thanks for simply being well helpful as well as for choosing varieties of remarkable resources millions of individuals are really eager to learn about. My very own sincere regret for not expressing appreciation to sooner.

    Reply

  92. Thank you for your whole labor on this blog. Debby takes pleasure in making time for investigation and it is simple to grasp why. My partner and i know all concerning the lively form you produce functional tips through your web site and in addition welcome response from some other people on that article so our girl has been starting to learn a whole lot. Enjoy the rest of the new year. You are always conducting a good job.

    Reply

  112. I truly wanted to write a small word to be able to appreciate you for the remarkable concepts you are posting at this website. My long internet investigation has at the end of the day been honored with pleasant facts and techniques to exchange with my partners. I ‘d assume that most of us readers actually are quite blessed to be in a really good network with very many wonderful people with interesting methods. I feel rather grateful to have seen your entire website page and look forward to many more excellent minutes reading here. Thanks again for a lot of things.

    Reply

  113. I precisely needed to say thanks once more. I am not sure the things that I would’ve implemented in the absence of the type of basics revealed by you concerning my question. It was actually a very troublesome case for me personally, however , seeing the very skilled form you processed that took me to cry for fulfillment. Extremely happy for this advice and in addition sincerely hope you recognize what a great job you are undertaking educating the rest by way of your webblog. I am sure you’ve never come across all of us.

    Reply

  114. I simply had to say thanks again. I do not know the things that I would have achieved in the absence of these creative concepts documented by you about such area. It was a depressing setting in my view, nevertheless witnessing your expert tactic you treated it forced me to weep for gladness. Now i’m grateful for the support and as well , wish you are aware of a great job that you are providing teaching some other people with the aid of your blog post. I’m certain you haven’t got to know all of us.

    Reply

  115. I want to express appreciation to you for rescuing me from this type of condition. Just after surfing through the online world and finding techniques which were not helpful, I figured my entire life was well over. Existing minus the solutions to the problems you have resolved by way of your website is a critical case, and the kind which might have in a wrong way damaged my career if I had not come across the blog. Your primary capability and kindness in controlling all areas was helpful. I don’t know what I would have done if I hadn’t come upon such a stuff like this. I’m able to now look ahead to my future. Thanks so much for this expert and sensible guide. I won’t hesitate to endorse the website to any person who should have tips on this problem.

    Reply

  122. I want to express my thanks to the writer just for rescuing me from such a incident. Right after checking through the world-wide-web and seeing concepts which are not powerful, I believed my life was over. Being alive minus the answers to the issues you have sorted out through this short article is a crucial case, and those which could have negatively damaged my career if I had not come across your web page. Your own understanding and kindness in playing with every item was crucial. I am not sure what I would have done if I hadn’t discovered such a subject like this. I’m able to now look ahead to my future. Thanks for your time so much for your professional and effective guide. I will not think twice to recommend the sites to any individual who should get assistance on this subject.

    Reply

  123. I’m just writing to make you know of the incredible encounter our child undergone visiting your site. She mastered plenty of issues, which include what it’s like to possess an amazing coaching mindset to get many people with ease learn about a variety of tricky things. You undoubtedly exceeded readers’ desires. Thank you for supplying those invaluable, dependable, edifying and even fun tips about this topic to Evelyn.

    Reply

  124. Thank you for all your hard work on this web site. My mother take interest in carrying out research and it is obvious why. I notice all relating to the powerful medium you deliver good tactics via this web blog and therefore inspire contribution from website visitors on this article so our princess is now becoming educated a great deal. Enjoy the rest of the new year. You have been carrying out a remarkable job.

    Reply

  125. Thank you so much for providing individuals with an extremely superb chance to read critical reviews from this site. It is often so ideal and as well , packed with a good time for me and my office mates to search the blog at the least three times in a week to learn the newest secrets you will have. And lastly, I am just usually impressed for the stunning ideas you serve. Selected two areas on this page are essentially the most effective I have had.

    Reply

  132. I must voice my gratitude for your kind-heartedness supporting people who should have help on this important issue. Your very own dedication to getting the message throughout ended up being exceedingly powerful and has always made people much like me to achieve their aims. Your new warm and helpful tips and hints indicates this much to me and a whole lot more to my mates. Thanks a lot; from all of us.

    Reply

  133. I would like to express my appreciation to this writer for rescuing me from this particular setting. Right after researching through the search engines and meeting concepts that were not powerful, I figured my entire life was over. Living without the approaches to the difficulties you have fixed all through your main short article is a crucial case, as well as the kind which might have badly damaged my career if I hadn’t discovered your web page. Your competence and kindness in controlling almost everything was crucial. I am not sure what I would have done if I hadn’t come across such a thing like this. I’m able to at this time look forward to my future. Thanks so much for the reliable and results-oriented guide. I won’t be reluctant to endorse your web site to any individual who needs recommendations on this subject.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *