The cost of business intelligence (or not)

I have sat in on a couple of meetings with financial departments and listened to the need for reporting. This seems to be the topic of the year. The struggle is how to do this and how to fix what has been done in the past.

First, let’s look at what has been done in the past. The key reporting tool historically has been something like Discoverer. Everyone I talk to has a love/hate relationship with this tool. They love the information that comes out of the tool. They hate it because it makes concurrent manager more complex to run and drives up your hardware and license cost by increasing the horsepower needed to run your general ledger. The key problem with this solution is that it does not scale. The more reports that you run, the more overloaded your system becomes. The more you use it, the more difficult it becomes to keep up and closing your general ledger takes longer and longer time. One company that tried to reduce their close time quickly realized that they had written custom interfaces that pulled data out of E-Biz, put the information into a spreadsheet, performed a calculation and manually entered the results of the calculation back into E-Biz. When they initialized the project they quickly realized that it wasn’t the process that was broken, it was the data that was invalid. Over the years, errors accumulated and got bigger and bigger and eventually hit the bottom line. One division that always appeared to be profitable turned out to be loosing money on a regular basis. An error in a calculation that added instead of subtracting two cells showed a profit and took money from another department.

If we take this methodology to the extreme, you get people who do nothing but come into work and look at the nightly runs of data extraction, verify that the results look good, and manually key the information into E-Biz to reduce the close time. You get hundreds of people monitoring thousands of calculations and an IT department that does nothing but handle failed jobs and bad data being pulled from a variety of sources.

Let’s look at the polar opposite of this. Let’s look at someone who created their own interfaces so that they can capture data as it is entered then push the data into E-Biz. The reason to capture the data is to reduce the Discoverer runs and concurrent jobs. The data is pushed into a spreadsheet as it is entered into E-Biz and updates are run nightly to verify that all of the data was properly collected. This solves the processing power issue in E-Biz but does not solve the overall picture.

Both of these solutions hamstring organizations because it ties custom code either in the forms entry or spreadsheet calculations. When it comes time to upgrade the E-Biz package or install a patch, testing becomes a nightmare. If a new feature is needed, like handling a new currency or language, multiple lines of code need to be changed in different places. If not all places are changed, errors can creep into the system without an ability to trace these errors.

The real solution to this problem is two fold. The first is to have a common API to pull and push data through E-Biz. This can be done through SOA or WebServices. If a financial element is pushed into E-Biz, the currency type is pushed as well. The conversion is done in the WSDL and not in the spreadsheet. The second solution is to have a data warehouse that is populated on a regular basis separate from the E-Biz services. Rather than run Discoverer against E-Biz, you run OBIEE/Analytics against the data warehouse and not have to have as large as a database under E-Biz. This solution also allows you to pull data from alternate repositories that are not Oracle solutions. If you patch or upgrade E-Biz, you change the API interface and tell it how to populate the data appropriately. If you patch or upgrade your third party application, you change the API interface to understand the data change. This allows you to centralize your changes and not have to change things in multiple places.

Politically, this solution is not a valid solution. Given that knowledge is power, giving up the calculation in your spreadsheet or giving up headcount to verify a calculation for your department means loss of budget and resources. Typically a solution like this involves an additional expense, significant consulting and manpower to implement the solution, and loss of power for individuals. What this does is give you a single source of truth and control over who sees what and how much detail that they can get. A solution like this typically requires direction from executive management and investment of resources to document processes and procedures. This can be a scary thing at times and expose stuff that some people want to keep hidden.

I realize that this is a little off topic from what I typically write about but I have heard the same message multiple times from different customers in different industries.