I related a story to a customer today that surprised them. At OpenWorld there was a contest between a newly graduated DBA and a 15-year experienced DBA. The problem was a performance issue with a database that was not trivial to find. The experienced DBA started executing scripts and launching querries from the command line. The new DBA used OEM and got to the same result in just slighly less time than the experienced DBA. It wasn’t the end result that they expected. They thought I was going to say that OEM would be better, faster, and could slice bread. The message that I wanted to get across was that they could tier their support. They did not always need to involve the lead DBA in everything but could give the junior staff rights to do the mundane tasks. The senior DBAs could get more involved with the line of business and make their contribution more significant.
Surprisingly, this spawned a discussion that they don’t want junior DBAs when then spun into how difficult it would be to upgrade their databases from 8i and 9i to 10g. They have too many custom scripts and home grown notification systems. Why would they go to a vendor provided system? Well, I showed them the management interface for a host, the database, the applicaiton server, and PeopleSoft. Going with a vendor provided system meant that they did not have to write customer scripts for the new database. It also meant that they did not need to involve an app server admin or a PeopleSoft admin. They did not need to learn the PeopleTools management interface to be able to see what is wrong with their installation. In the end, they got it. They understood that they are not senior in all aspects of the applications that they support. They also understood that a vendor provided system allows you to look at stuff and not be an expert.
I am surprised that a theme this month always comes back to build vs buy. The cost of the scripts that they wrote not only improved the quality of a product that they purchased (a database) but it locked them into a specific version. They don’t want to upgrade from 9.2.0.5 to 9.2.0.8 because a couple of their scripts broke when they upgraded on a test system. It was better for them to remain on an unsupported version than it was to change their scripts. It was also easier to not upgrade because it would be too much testing and overtime hours. I understand keeping cost at a minimum but at what level does this cost bubble up? If something breaks or a new vulnerability comes out, they will eventually need to upgrade. If they do not upgrade, they risk the system coming down or being comprimised.
In my opinion, when the cost of upgrading to the latest patch involves testing of customized scripts and extensions, you need to start questioning the value of the customizations. There is a difference between job security and stagnation but it is a fine line at some locations.