Data Protection Down Pat

Yesterday I did something that some would say was an incredible waste of time. Others say that it would be one of those mastercard moments, you know “priceless”. It turns out that our local major league soccer team won the playoff games and qualified for the MLS Cup Championship. The game was scheduled for Pizza Hut Park in Dallas which is 250 miles north of here.

Let me set the stage here. My wife is a big soccer fan. She works for a men’s soccer league in town. She plays on two or three teams. We played together for a while on a co-ed team before I was advised not to play any more if I wanted to walk when I got older. My oldest son played for a while with us as well. My daughter played for a while before she figured out that she did not like running that much. Swimming and volleyball are more her sport. My youngest son plays and his mom is the coach. Her family is also big into soccer so much so that they go out of town almost every weekend for a tournament or a game. I have been volunteering for the Houston Dynamos all season as a stats keeper for home games. I have sat in the stands for three games and watch as my wife kept stats so both of us know the Dynamo staff and a few of the players.

Did I say we have a vested interest in soccer? Well, the championship game was yesterday in Dallas. We paid $45 per ticket to go to the game. We got up at 8am to make a 2:30pm game. Yes, 4 1/2 hours of driving to get to the field with some time to tailgate before the game. The game was tied at the end, 0-0. After a 15 minute overtime still 0-0. After a second 15 minute overtime, 1-1. After penalty kicks, 4-3.

When I looked at my daughter who was wearing her orange team shirt with the team scarf wrapped around her and her team hat signed by all of the starters she was smiling. She knew the importance of the day. She new that she would remember this season for a long time. After the game she wanted a copy of the special release of the Houston Chronicle pronouncing “Dynamo Wins” so that she could put it up in her school locker next to the picture of the top two stars with their arms around her.

10 hours driving in the car: $175

5 minutes while we get a speeding ticket: $125

3 hours at the game: $180 (tickets)

2 minutes of smiles, yelling, and cheering: priceless

some things are worth it.

What the heck is BerkeleyDB? Origionally it was SleepyCat but was acquired by Oracle a few months back. Ok, that dosen’t answer the question, it only gives the history of the product. What it is typically used for is an embedded database. There are two flavors, Berkeley DB and Berkeley DB Java Edition. The java edition is entirely written in Java and can run on any machine that runs a java virtual machine. On top of the Berkeley DB is an XML engine that allows you to query the database using XML queries. The database is not relational. It is not object oriented. It uses key-value pairs to find the data. Values are simply payloads and can be upto 4G in length. A database table can be upto 256TB in size.

This software is different than a traditional database in that it is compiled into the application with a set of libraries. There may be many threads of control but no barriers between the threads. It is assumed that the program will maintain consistency and separation of duties. There are three access methods to data through the database engine: btree, hash, queue, and renco. The interfaces to save, change, and read records are the same. The access method and storage procedures are different for each access method.

SQL and Berkeley DB have similar actions but are performed differently. The concepts are the same but the actions are different. In SQL, for example, a database is a collection of tables which contains rows. Different elements from tables can be combined with an AND command to join data. In Berkeley DB, the database is an element of an environment. Instead of a row, there is a record which is a key-data pair. To combine different elements from different databases a join function is used passing the database parameters. Transactions, indexes, and backup have similar contexts between both systems.

Berkeley DB is very fast as far as transactions are concerned since everything is done in memory and copied to disk. A typical in memory or cached disk configuration can support 200 million non transaction reads per second or 250 thousand transactions per second. Transaction commit sync to IO can support 30 to 60 transactions per second. A sync to the operating system is about 50 thousand transactions per second and commit with no sync is 90 thousand transactions per second sustained.

The key reasons to choose Berkeley DB is performance, concurrency, memory footprint, and cost. Since this is an embedded library, a higher degree of knowledge is necessary. Since it does not use typical SQL commands but library calls, the knowledge is specific to Berkeley DB and not database in general.

Ok, some times I am simple minded. I need to write this down so that I don’t forget this. If I put it in a blog I should be able to access when I need it (hopefully). Oracle offers a variety of database technologies. The databases that exist are:

• BerkeleyDB – free
• TimesTen – embedded memory database. It can sync with the other types of database and effectively be a cache front end to the other databases.
• Lite Edition – cost. used for mobile devices with a database on the back end.

to compare these three http://www.oracle.com/database/embedded-compare.html

• Express Edition – free. no CPU limit but runs on 1 CPU, 1G Ram, 4G data, support through online forum.
• Standard Edition One – minimal cost. 1-2 CPUs, no memory limits. support available. Priced per CPU or per user
• Standard Edition – 1-4 CPUs, can be clustered with RAC. no memory limits. support available. Prices per CPU or per user
• Personal Edition – minimal cost. single user environment, no processor limit
• Enterprise Edition – 4+ CPUs, no memory limits. support available, can be clustered with RAC. Prices per CPU or per user

The extended features and options supported and not supported are

Feature Database	Berkeley DB	Times Ten	Express Edition	Standard Edition One	Standard Edition	Personal Edition	Enterprise Edition
Data Guard	No	No	No	No	No	No	Yes
Rolling Upgrades	No	No	No	No	No	No	Yes
Fast start recovery	No	No	No	No	No	No	Yes
Schema reorg/redefine	No	No	No	No	No	No	Yes
Flashback Query	No	No	Yes	Yes	Yes	Yes	Yes
Flashback	No	No	No	No	No	No	Yes
RAC	No	No	No	No	Yes (free)	No	Yes (cost)
Security and Auditing	No	No	No	No	No	No	Yes
XML support	Yes	No	Yes	Yes	Yes	Yes	Yes
.NET and Active Dir	?	Yes	Yes	Yes	Yes	Yes	Yes
Management Packs	No	No	No	No	No	No	Yes
Enterprise Manager	No	No	Yes	Yes	Yes	Yes	Yes
Streams	No	No	No	No	No	No	Yes
Replication	Yes	Yes	Yes	Yes	Yes	Yes	Advanced
Spatial	No	No	No	No	No	No	Yes
ETL – Warehouse Builder	No	No	No	Std (free)	Std (free)	Std (free)	Enterprise (cost)

http://www.oracle.com/database/product_editions.html

For a change I will review a book. The book is from Oracle Press. High Availability with RAC, Flashback and Data Guard. Matthew Hart and Scott Jesse.

Book outline

1. Illustrated Downtime Scenarios

This chapter is an introductory overview of what can go wrong. Unfortunately, this chapter is a little light and not complete. It is a good introduction and quick read.

2. RDBMS features for availability

This chapter introduces us to Enterprise Manager (EM). It is mostly screen shots of the EM that comes with a database. It does describe the differences between spfile and init.ora very well. The chapter then goes on and describes different types of partitioning. “Table and index partitioning refers to the ability within Oracle to organize a single table of data into multiple logical parts that can be maintained and treated independently”. “Partitioning is usually a maintenance-to-performance trade-off: maintaining partitions require more DBA footwork, but typically provides better performance”. “There are three distinct types of partitions for tables: range, hash, and list”. “The range partition divvies up data into segments that are separated by ranges of values, such as dates”. “List partitioning occurs by grouping specific values to be placed in each partition. Thus you can group data logically that does not have a natural range grouping, and control which partition is used”. Hash partitioning splits the table based on a hash value of the data.

You can also lay out a database for availability. Some key elements to make this happen are: index-organized tables and materialized views. The chapter does have good code samples to show you how to implement these. It also goes into how to modify an existing table for performance and availability.

The rest of the chapter appears to be a catch all for availability. Yes, it is important to talk about the scheduler and the resource manager because things can run away and you need to know how to stop this. I’m not sure why logminer is included in this chapter other than it works with redo logs which helps with recovery. The chapter does have a good logminer tutorial using SQL commands. The same is true for the section on transportable tablespaces.

3. Tuning your database for availability

This chapter mainly focuses on 10g new features. Some of the key elements are MMON, AWR (Automatic workload repository), and ADDM (automatic database diagnostic monitor). The first half of the chapter does a good job of introducing these concepts and examples of how to look at these elements. Next the chapter goes on to talk about SQL Tuning advisor, memory advisor, and ASMM (advanced shared memory management). Finally the chapter ends up talking about ASM (automatic storage management). All of these discussions are good introductory material that introduces concepts, command lines, and reasons why you would use each feature.  

4. RAC Setup and configuration

This chapter starts out talking about cluster ready services. This is a good introduction to how RAC works. I like the configuration recommendations but the book makes assumptions that the target is a Linux/Unix system and does not point out how it would be different on other operating systems. Overall, this is a good introduction to RAC installation. It touches the base concepts and goes through a typical install.

5. Database administration in a RAC environment

This chapter describes some of the RAC specific parameters that are unique to a RAC environment. It also goes into a good description of RAC background processes and what each does. The tutorial goes through and shows how to enable archiving and flashback. The chapter ends up with a description of how to add or remove nodes in a RAC installation.

6. Utility Computing: Application as Services

7. Oracle Data Guard: Surviving the disaster

8. Backup and recovery for high-availability

9. Oracle Flashback: surviving user-induced trauma

10. Oracle streams for high availability

11. Oracle net configuration for failover

I don’t know why I keep coming back to this topic but I do. Risk taking is something that everyone is forced to do at some point. Heck, driving in your car is a risk. Walking to the mailbox is a risk. Not much of one but still a risk. Tonight I watched the school that I went to (Texas A&M) compete against Oklahoma in football. The game was very close but it all came down to the last minute. The Bob Stoopes, the Oklahoma coach, took the clock from 3:30 down to 1:30 but still needed half a yard to run out the clock. Easy, right. Anyone can get one yard. Unfortunately, they had not gotten a first down in the fourth quarter and they were on their own 30 yard line. If they went with the odds, they were not going to get another first down and the A&M kicker can make it from 50 yards out. Given that Oklahoma only had a one point lead, it looked like a huge risk. The announcers were saying that they would not make this call. They ran a play and made it but the coach called a timeout and stopped the play before it was successful. The second time that they ran it, the defense serged and almost stopped the quarterback. The key word here is almost. They stopped him at the line of scrimmage but he bounced off and tried again. When he fell down he fell forward and gained the one yard needed.

After the game Stoops was interviewed and said that the wind and the crowd noise was against them. The punt would have ended up short and would have given A&M just a little farther with a minute and a half to play. Given that failure would have meant the game, what is the risk. The risk is loosing the game. Given that Oklahoma was ranked in the top twenty and so was A&M, how bad would it be loosing by two points on the opponents field. Would they have dropped out of the top twenty? Probably not. Would the coach have been fired? Probably not. This isn’t one of their major rivals and they don’t have much of a chance to win the conference title.

In my opinion, it was a calculated risk. Would I have made the same call? I doubt it. I don’t take large risks like that. I focus on winning and not the bigger picture most of the time. Is that a bad thing? Probably not.

I have a friend that I recommended for a job years ago. He interviewed well and got a job in a different group than me. I kept telling him to become good at one thing. It dosen’t matter what that one thing was as long as he was the best in the office, area, and company. Once he obtained that status, everything else would fall in line with that. In my opinion being able to say that absolutely that something is the right thing to do truly is the right thing to do is one of the best emotions I can think of. For me, it takes being the best at something to have that comfort level and confidence. My friend decided he could make more money by investing in old houses and renovating them. His job suffered. He was never happy doing what he was hired to do. He always go sub par ratings because he spent too much time tracking stocks and looking at real estate. Fortunately for him, he realized this and is now buying and selling houses. Would I have the courage to drop out of a steady paycheck and totally change careers? Probably. I did this when I went back to graduate school. Looking back I made the same mistakes my friend made. I didn’t finish my Ph.D. because I was too distracted making money consulting and not doing my research. I got a reputation of someone who took too long to finish projects and my ideas were just a step behind. Does this mean that I can’t get a doctorate? No. It means that I screwed up in my attempt and didn’t even take my own advice.

Would I have gone for it up one point, a half a yard to go on my own 30? Looking in the mirror I look at every morning I don’t think that I have that much confidence or comfort. Ask me again in six months after being at Oracle for a whole year. I might answer the question differently. I just need to find that one thing that will make me a hero and build that confidence and comfort.

I have been reading through some of the presentations from OpenWorld. Some are nothing more than representation of typical Oracle marketing and sales presentations. Some are true gems. One got me thinking about selection of hardware and software. The presentation can be found at http://www28.cplan.com/cbo_export/PS_S284441_284441_139-1_FIN_v6.pdf

The premise behind the presentation is that Dell configures a server that allows it to provision a database node into a RAC environment and make it part of the data warehouse cluster. One of the things that the author debates is the merits of deploying the latest and greatest hardware and software. The advantage is that you get the latest bells and whistles and newest features. Unfortunately you also get the least stable code test matrix. A new driver might be needed to support some new hardware feature and it might have an untested data path that happens when you least want it to. This brings me back to the age old question. If I were a CIO, what policies would I mandate? When I purchase a new car I never buy the latest and greatest. I did this years ago when I got a Miatta. The car had some bugs and problems with it but I got lucky. My wife purchased a Olds Silouette the first year that it was out. They never figured out how to get the brightness on the video screen to work properly and we were stuck with a VHS player that could not be converted to play our DVD library. The automatic door never wanted to close and stay closed. I think that I would mandate being a release behind the vendors or go with a qualified configuration that someone else that I know has tested and figured out what the problems are. If I were a CIO, I would follow the configuration that Dell proposed at OpenWorld. It appears to be well thought out, well tested, and I could call one of the IT directors at Dell for a site visit. I’m sure that the Dell sales rep could arrange this since I would be purchasing from them hopefully in large volumes. (Hey, if I am going to be a pretent CIO, it is going to be for a large and successful company).

Cutting edge is one thing. I always believe in learning things before everyone else. It helps me keep my edge. Oh well, I guess I better get back to reading the 11g database release information as well as the rest of the OpenWorld presentations that I find interesting.

we did an exercise the other day that surprised me. If you get 2-3 people and put a single word or pair of words then have everyone write down words that match. You know the game. I say birds. You say bees. I say red. You say room because you watched the Shining the other night and everyone in the room questions your sanity. The word that we had was trust. The exercise is to list 10-15 words that are synonyms for the word trust. Once you are finished match your words with theirs and see how many matches you come up with. We did this with a room of 30 people broken into 10 groups and the most exact matches that we came up with was one group with one match.

Next time someone comes in your office and says that they can improve your productivity, what do they really mean? I will be more carefull with my words in the future.

so with the keynote today, Oracle announced that it is supporting Red Hat and backporting bug fixes and patches to previous and older versions of the operating system. The VP of Engineering at Yahoo used a phrase that brought back memories. The term “legacy systems” to me always meant Dell and IBM hardware. Today, that means Solaris Sparc machines. With this announcement I wonder how this will change the industry. Since Red Hat makes the majority of their revenue with service and Oracle is giving away the Linux kernel, where will Red Hat make money?

The argument that Larry brought forth that patches and bug fixes go into the new kernel for new and previous kernel versions. This is an interesting concept. The ability to change your patch system from Red Hat to Oracle is amazing. To make this happen you need to go to linux.oracle.com and download the up2date rpm. This rpm is configured to go to oracle.com. This is a great way of getting adoption and migration of existing Red Hat customers to switch to Oracle.

This should be very interesting.

If you don’t know about Data Guard, it is something that everyone should know about. A few tips and tricks from the session

1. network bandwidth is important. There are a bunch of tuning metrics that can be done at the TCP layer level
2. low latency, low problems. high latency, problems start to creep in.
3. async redo transport – no async buffer to fill up. LNS can fall behind and will go into archive log if it is too slow.
4. gap resolution – increase the number of arch processes from 2 to 4. Also use MAX_CONNECTIONS to 2. this reduces the overall time needed to resolve a gap by transferring an archive log using multiple processes.
5. if the gap between primary and physical gets very large, use incremental backup to speed up gap resolution. Stop redo apply process, determine the current SCN of physical, take incremental backup of primary, catalog and recovery the incremental to physical, restart redo.
6. automate archive log management. move backups to a physical standby. use RMAN to setup the archive log retention. requires flash recovery area on all databases
7. this is slightly different for SQL apply. For 10.1, it required a manual purge applied to archive logs. For 10R2 they are automatic. This can be disabled on 10R2 but why?

switchover

• multiple standby configurations is not that difficult and works well. It is recommended to define standby machines on the standby just incase it becomes the primary.This does require more redo being shipped between systems.
• switchover stops all xactions and the redo log is drained. Once the log is cleared, the switchover will happen. You can make a physical standby in a logical switchover reinitiate itself as a physical standby but this is a manual process.
• always try to failover to your physical standby. If you must switchover to a logical, use flashback to recovery primary and process redo logs to bring it back in sync.
• switchover should not take more than a few minutes. you can speed this up by using real time apply.
• in 10r2, DB_ROLE_CHANGE can fire a trigger when roles change

If I want to run a report on a standby I can suspend redo apply, flash back the database to a point in time, and run the report. This will allow you to write data to the database then flush the changed when you flash it back to a restore point and turn on the redo apply.

Like I said, there is a good reason to understand Data Guard. It is a very powerful tool that allows you to offload your production server and have a live standby backup of your system.

Ok, I haven’t exactly kept up with blogging during the conference. I just figured out how to connect with my laptop from the conference hall. I am currently sitting in the keynote address from Sun. I miss Scott McNealy as a speaker. Johnathan Schwarts is interesting but not as good as Scott. It is interesting to note that he has updated his wardrobe and is starting to look like a CEO.

“Our [Sun’s] business model is to drive change and innovation into the data center”. Interesting concept. How do you run a $26B company with this concept? Not sure about that but I guess it is how you run a $13B corporation.

“A massive global buildout is underway”. Every business is growing. On average 4% growth in GDP for countries is expected. Some grow more, some grow less. Faster, better computers are not necessarily a good thing. It takes less computer and less capital to run the same investment. It talkes about $8M for the hardware to run infrastructure for a $13B company. Some companies are growing faster than Moore’s Law because they are responding to consumers that demand interaction. Some examples are EBay, Major League Baseball, ExxonMobil. These companies need very large data centers that require significantly more horsepower. This is where Sun is targeting because the traditional customer will eventually turn to service suppliers to get servers in their offices.

Sun is focusing on four verticals. Software, servers, storage, and services.

glassfish.dev.java.net – need to look at this…..GlassFish is the name for the open source development
project
for
building a Java EE 5 application server. It is based on the source code
for Sun Java System Application Server PE 9 donated by Sun Microsystems
and TopLink persistence code donated by Oracle. The question is how does it correlate to Tomcat or the Oracle Apps server

“Solaris crossed the 6M install base. 70% of these are on x64/x86”. This begs the question of why Sparc and why Sun continues to develop these chips. The Linux variant on ubuntu now runs on Sparc so this might add some lifetime to the chipset.

“There are times when our hardware will be the most innovative, there are times that our software will be the most innovative”. This is an interesting concept. It is also a great statement because the hardware group typically lags in delivery of product while someone else leapfrogs the technology. Using this strategy is interesting.

The $1k rebate from PG&E is an interesting sales concept. This successfully focuses the fact that other vendors consume too much power. If a customer has a competitors server running in California, PG&E will give the customer $1k rebate if they replace this with a Sun server. Interesting idea. I wonder how they structured this deal and how we could potentially replicate it with Entergy or Reliant.

The new concept…. 40 foot shipping container full of Sun equipment. Project blackbox. 250 systems/container. 2 Petabytes of tape. Up and running in five minutes.  Interesting packaging. To connect it requires three phase power, network connection, and hot and cold water connections. The racks are water cooled with chillers between the sideways mounted racks and has a center walkway for servicing. This is interesting and my guess is that everyone else will start delivering this as a solution within a few years.

more later….