Data Protection Down Pat

For this blog entry, I will be reviewing a book.
Oracle Database 10g High Availability with RAC, Flashback & Data Guard by Matthew Hart and Scott Jesse

The book is organized well with part 1 being logical availability, part 2 RAC, part 3 disaster planning and part 4 being the distributed database. The book starts out talking about high availability as an unreachable concept that typically requires a large budget. The author compares the computing grid to the utility grid. I sure hope that the grid that they are speaking of isn’t the power grid because it is down on a regular basis at my house. I have been looking at a UPS because ours is down for at least five minutes a month, every month. More during rainstorms.

I usually don’t mind a book that dives straight into the details but putting code on page 4 seems a bit much for me. The code isn’t explained very well but it is used as an example for the chapter. The code looks very simple, create tablespace, create a user, connect, create table and populate. The sample that they give stating that there is no window for a backup except on the weekends is a little hard to believe. The example on accounts receivable comsuming resources is a totally believable problem. The rest of the chapter just details more and more potential failures. Unfortunately, this chapter is not comprehensive and does not recommend solutions or processes to stop the problems. They also don’t detail the impact of the problems but implies that there are significant problems if the failure mode happens.

more later…..

Ok, it isn’t truly day 1 but it is my first day. Today is Sunday which is the second day of the tutorials. I’ve been to JavaOne but it wasn’t nearly this large. The conference is spread across Moscone West, South, and the Hilton hotel. It looks like I will be doing a substantial amount of walking between the conference center and the Hilton. I’ve already done a bit of walking today to find out what is where and what hands on labs that I can sneak into.

I’ve already gotten kicked out of one session. It was funny because I went in and did 99% of the lab and jumped out to answer a phone call. I got tagged going back into the room but they let me go back in to get my stuff and leave. The hands on tracks are very interesting. Very open format with lectures between labs. I like this idea.

I don’t think that I have seen this many laptops in one place in my life. The open access systems are all laptops. The hands on labs are all laptops. They all have 19 inch screens attached to them. I wonder what these machines will do once this event is finished. My guess is that they are leased and will go to the next big trade show. When I was with Sun we deployed a large server and a series of JavaStations. I wonder what the cost tradeoff now is for laptops and java consoles.

I have been trying to plan my schedule. It is challenging because they won’t let Oracle employees register. We have to wait until 10 minutes into the talk and take a seat if there is one. I understand but it does make planning and scheduling difficult. The biggest problem that I am having is deciding on what I want to be when I grow up. I’ve always adopted the philosophy of doing one or two things very well, making a name for myself, and moving on to the next topic. Oracle seems to have an information page setup for all of the products that is driven by product marketing. I’ve used this tactic before. I might need to write a series of tutorials or possibly a book to bring up my levels of expertise.

more later….

Today has been an unusual day. I didn’t have any meetings. I didn’t have any conference calls. I didn’t have anywhere that I was required to be. Very unlike the rest of the week. First, let’s recap the events leading upto today.

Monday:
 – 6am work out. must start the week out properly
 – 7:45 in office for meeting with team
 – 8:30 meetings start to plan out how to accomplish tasks for the week
 – 11:30 skip out of office and head to customer site
 – 1:00 meeting at customer site, lasts until 2:30
 – 2:30 follow up to meeting, write follow up emails, finish some documentation, do some research
 – 4:30 drive home to make soccer practice for the kids
 – 5:30 soccer practice
 – 7pm dinner
 – 7:30 finish up project on the planet Mercury (3rd grade project)
 – 8:30 start hearding kids to bed
 – 9:30 finished with bedtime, get to read paper, check on email again, watch last quarter of Monday Night Football

Tuesday
 – 6:00 work out. wow, two days back to back
 – 8:30 doctors appointment
 – 10:30 did it really take two hours at the doctor? finished reading book
 – 11:30 customer meeting/event
 – 1:30 event rained out
 – 2:30 research and training, prep demo system
 – 6pm dinner, did we really have dinner two nights in a row? wow.
 – 7pm play games, watch really bad kids tv show
 – 8:30 start hearding kids to bed
 – 9:30 finish with bedtime, check on email again, watch tv show
 – 10:45 read boring technical manuals instead of sleeping pill

Weds
 – 7:30 breakfast with the family, unusual event. I’m usually gone by this time
 – 8:00 check email, follow up on correspondance
 – 9:00 customer meeting
 – 11:30 back in office, pick up salad at desk and read email while working
 – 1pm internal meetings to plan customer architecture presentation
 – 4:30 – drive home, tired from all day meetings
 – 6pm – dinner with kids, wife playing soccer
 – 7pm – cub scouts at our house
 – 9pm – finished cleaning up, everyone to bed
 – 10pm – check email, read until time to go to bed

Thurs
 – breakfast with the family, two days in a row. I will get spoiled if this keeps up.
 – nothing planned. could go to golf tournament, looks like rain. tough to justify “skipping out on work” for tournament.
 – long range planning
 – research
 – prep for presentations next week
 – prep for presentations tomorrow
 – get up to date on expenses, plan training
 – training, playing with demo systems.

Tomorrow
 – work out in morning
 – paperwork in office
 – customer meeting and presentation
 – conference call to set next weeks schedule.

total time at customer sites: 11 hours
number of customers visited in person: 7
number of customers contacted via phone and email: 17
number of phone calls with customers: 13
number of internal phone calls: 15
number of emails with customers: 27
number of internal emails: 8
time spent on blogging: 1 hour
time spent on fantasy football: 1 1/2 hours + 5 minutes trash talking

Days like today are rare. I wanted to record what I did in a week so that I could look back and see if this was typical, unusual, productive, or waste of time.

pat

My kids and I are constantly quoting movies. Some movies have more relevance than others and some are more quotable as well. We have been doing this for years. Every Christmas we watch It’s A Wonderful Life and every February 2nd we watch Groundhogs Day. I realize that they are corny movies but they help build context. Two of our favorite movies are It’s a Mad, Mad, Mad, Mad World and What’s Up Doc. These movies are movies that you can watch with the kids.

At work we joke about movies like Caddy Shack and The Godfather, neither of which are appropriate for kids. To me, the importance for these touchstones is a commonality. We try to build similar commonalities with cutomers. Some of them are contrived and some of them are really shared experiences. I’ve been talking to some of Oracle’s long time customers and they really like some of the events that we host. I’m not talking about the lunch and learn lectures or speaking events at hotels. I am talking about events like plays, volunteer events, or even golf tournaments. Years ago I got tickets to the Shell Open and sat on the 16th green with one of my friends/customers. We still talk about it. It was very interesting watching the players round the corner and see how each of them made their approach shots to the hole.

I guess that a common theme is starting to repeat itself, history. History is what makes us good at what we do. History is what makes us have a common ground to share more experiences. History is what lets people trust each other. If I was right the last ten times, there is a good chance that I will be right again. If I bent the truth the last two times and it cost you money, it will probably cost you money again.

My only hope is that I am building up enough history and karma so that I will be trustworthy. I know that part of the Boy Scout oath is a scout will be trustworthy. I guess this is what it means.

Does everyone think in terms of risk and reward. It seems like most people that I have talked to lately are greatly focused on risk and reward. I understand if a manager is looking at investing money in a product and needs to figure out if the cost of the software is worth the risk of not deploying it. I struggle with justification of software. Ok, I understand the need to get a word processor and presentation software package. Is it worth the $200 or so dollars to get Word and Powerpoint or is downloading StarOffice adequate? If I send a thank you note, can I send it via email or do I really need to get a card and send it through the US mail? At what point does being too cheap really get in the way of success? At what point does doing what is required cost you in lost opportunity or perception? Would going a little farther really make difference? I keep coming back to a story that I heard about Larry Bird the basketball player. To become truly great he would spend hours practicing one shot and one shot only. After weeks and weeks of making the same shot, he would move slightly to the left or right on the court and repeat the training.

Does the same apply to non-sports activities? Can I read documentation, play with software, give presentations, crate a prototype deployment and truly know enough about a product to be an expert? Do I need to do this multiple times to truly see it from different angles? What are my cost and risk for cutting corners? If I use StarOffice to create a document, what am I loosing? If I use a public domain database do I sacrifice functionality or features when compared to using a commercial version? (Ok, I know the answer to this one and can even explain it in gory detail).

I wish I knew how to analyze risk enough to make the informed decisions. One of my daughters friends is constantly analyzing risk and reward. She has been doing this since she was able to talk. “If I don’t make my bed, what will happen?” “If I don’t eat my peas, what are the consequences?” I’m constantly amazed by her rationalization that the peas are not worth the bowl full of ice cream. Not making her bed is not a fair trade off for an hour of television. My friends finally found the breaking point now that she wants a cell phone. It is amazing what she will compromise to get her phone back since it is her lifeline to her friends. I’ll stop here because I don’t want to get started on how old someone should be to have a phone with them.

Risk vs reward? I guess that this is what separates really good decision makers from really bad ones.

 

 

I understand the concept of delivering a message to the audience. I also understand of not wanting to create enemies by saying the wrong thing to the wrong person. For example, if I say that a new business process or software package improves your life by making a daily task take 40% less time. If you are a DBA, your response is — great. I hate typing in these commands or explaining my scripts to the latest new guy in the office. If the DBA’s manager is in the room, it translates into —- great, his time just got free so I can assign this project that has been on hold. If an executive is in the room, it translates to — great, I can deny that new hire request or cut staff by one person. If all three are in the room, no one says anything because they see it from the other persons perspective and it gets translated into fear across the board because the DBA will loose his job, the manager will loose headcount, and the executive is now held accountable for reducing expenses to pay for this software. Net result is that nothing gets done and everything stays the same.

I go to lunch with a group of administrators that I worked with years ago. It is amazing how slowly things change. Projects that I worked on two years ago are still being rolled out. The manager that I locked horns with has run two more people out of the group. A replacement has not been hired to fill my position. And most surprisingly, my accounts still work on most of the machines that I managed. I haven’t tried going to superuser because for some reason that crosses a line for me. For some reason, me logging into the university web server is not a violation of morals and principals. I can update a web page that has my name on it (and my personal content) but running sudo crosses the line for me. Dosen’t it seem that I should not have a web page at this institution? I was a student at one university and graduated last December. It is now October and I still have account access on most machines. I was an employee at a different university two years ago and I still have an LDAP entry as well as home directory access on most IT machines. Is it me or is this typical? If I were the IT manager, LDAP and Kerberos would be the first thing that I implement and restrict access to only those that should have access. When you leave the group or organization, your account is disabled.

One thing that I am getting really good at of late is doing a demo of a subset of our products. I really like having a corporate demo site managed by corporate. When I was at Sun, we started to have a set of software demos so that we could show the portal and ldap servers in action. At Oracle, the product suite is much larger. The rack of demo machines consists of a few hundred machines that are all refreshed daily from a golden image. I like the ability to schedule a demo and go to a customer site so that we can drive the software from my laptop. I’ve done two of them in the past few weeks and have been to a couple of training classes that used the demo suite to explain the products. I guess I am a visual learner and do much better seeing the product and not bullets on a presentation.

I guess I should have asked first before I started rambling; who am I talking to? I sure hope I didn’t offend you with my ramblings. If you are a former co-worker and don’t like being made fun of; get over it. Bring it up at lunch next week. If you are reading this to find out about Oracle software, this probably isn’t the place to learn. I’ll throw out tidbits here and there but this is more of a high level moral discussion and relevancy discussion. If this is my manager, forget that you read any of this. If anyone else, why are you reading this blog? What are you hoping to get out of my ramblings?

pat

I was talking with one of my neighbors the other day. We both have kids and the subject of work and kids came up. When your kids ask what you do at work how do you answer them? When I was at a hardware company, the answer was simple….. I play with computers so that I can help other people who want to use them and make it easier for them to use the stuff we sell. My youngest thought that I played Nintendo all day and wanted to be just like his dad. The only thing that he heard was that I play with computers all day. I must be an expert at every computer game on earth.

Now that they are older and hopefully wiser, my story has changed a bit. The last time they asked me what I do at work all day I answered ….. I talk on the phone for a few hours, I read a lot of books and documents, I experiment with different configurations and combinations of software, and I go to meetings and talk about software. I can just imagine what they think that I do all day. Putting on their filter, this is probably what they hear…. I talk smack about fantasy football all morning with my friends and associates. When my manager catches me and tells me to get to work I have to read boring and dusty old manuals. I still play with computers but now it is trying to combine different games for multiplayer arena smackdowns.

I have also been talking to some former coleagues that still work at universities and they see me as a demo jockey. All they perceive that I do is go from one customer to another and demo stuff. Yesterday I did my first demo in years. It actually went well. It is truly hard to understand the grid management software until you see it in action. I think that I have a good handle on this software and understand how it works. When I was at Rice we used Nagios for a monitoring and management system. It was interesting but it has some severe limitations on time delays and sampling skews. After playing with database monitoring tools and operating system extensions, I think I could replace a Nagios system with the Grid Enterprise Management system. One of the significant advantages is that it would show a relationship between correlated systems and heiarchy of hardware to deliver a service. I like the concept of going to an applicaiton server and looking at a breakdown of the response time too see if it was the DNS server, Apache web server, or LDAP server that took the most time to reply.

I think I know how to talk to former coleagues and potential customers on the capabilities of our products and what I do on a day to day basis. I can almost even explain it to someone who is trying to get a job in our group. If I could just figure out how to explain it to my kids my life would be perfect.

So last week was a bit of a let down. I have been to Orlando a few times and stayed in bad and good hotels. I have had good and bad service. This past week I got a mix of both. The hotel that I stayed in was a Disney property. The hotel was loud, filled with little kids, and not very happy people. Most of the people staying in the hotel were bright and cheerful in the morning but not loving the world come the end of the day. They all had the look of “just shoot me now. Don’t make me ride the teacups again and I won’t hurl on you”.  The staff at the hotel wasn’t much better either.
One of the guys said that it was a little wierd waking up to a bright and cheerful mouse telling you that it was going to be a bright and sunny day in the magic kingdom. His comment was that it was either a cruel joke by someone he reported to or he was at a Disney property again.

I thought that once I got back to Houston, things would be normal again. I went into the office yesterday with the hopes of working out and visiting two customer sites. I got in the office at 6:45 and half my co-workers were there and had been there since 6am. I got sidetracked and only got to visit one customer (all day long) and tried to repair a bad sequence of events that showed me what really bad service was. When I left at 6pm, half the office was still there. Many of them stayed until 9pm. When I got in this morning at 6:45, half the office was there again today. When I left at 6pm, they were still there. Is it me or do the people I work with work long hours.

This begs the question…. is it better to put in the hours or is it better to do what is required to be successful? Yes, I could work three times harder and do some really awsome projects. Yes, I could read another four or five hours of technical manuals a week. Yes, I could get better demos working on my laptop. I just have to ask myself, is it worth it? I would rather read a good novel so that I pay attention to the technical manuals. I would rather spend 30 more minutes at the gym rather than 30 more minutes in front of my laptop. I would rather spend 30 minutes having dinner with my family than really knowing the demo backwards and forwards. In my experience, the really great demo comes from doing it a few hundred times and not customizing what everyone else has done. The demos on my laptop will never be as good as the ones on the demo servers in Austin. I just need to walk through them a few more times and get it down properly.

I’m hoping that at my age, I know what needs to get done and do it right the first time. I think that I did today and only had three things on my to do list at the end of the day. I can wake up early tomorrow and take care of them before driving in to the office again.

this week will be an interesting study in service. today i was in new orleans and got a chance to see life without services. we made a few customer calls yesterday and helped build a house through habitat for humanity. man, talk about no services. there was electricity but no running water in the new houses. we had to cart buckets of water to clean brushed and paint rollers. toilets were porta potties. very primitive.

the airport is open but just getting back to normal. next week should be the turning point with the saints home opener.

at the other extreme is my meetings in Orlando. when i got to the airport i was met by 9 people to help me get on a bus. one was singing and humming, all were smiling, and the bus driver told jokes to the kids on the bus. the bus had movies for the short trip to the hotels.

it will be an interesting trip. my main focus will be on how well i am served and how i can translatee it to our customers.

to be continued…

>>> begin cynic mode <<<

Am I missing something? If I have a production system in one city and a development team or production system in another city why not make them disaster recovery failover for each other? For example, I have a system in Houston that runs my production xyz service. I have a company that I acquired that runs a production jkf service. Why not create a new instance on each of the databases and use dataguard between the two systems?

Given that the 10g database contains DataGuard with either physical or logical backup, the only cost in this configuration would be to add more memory and disk to the two systems and make sure that we have a good network connection between the two systems. Is the problem that no one is running 10g? No, not really. I have seen a bunch of companies that are running on 10g. Is the problem that no one has enough memory or cpu on these systems? Ok, I will give you that memory might be a problem. Most processors are only running 50-60% so there is enough CPU power. Is there a problem with the network? Since most IT shops don’t want to talk about what their border routers traffic looks like, the most likely problem appears to be network bandwidth.

Given that this is a bundled option with 10g, I don’t see why more and more people are not using DataGuard as a disaster recovery and business continuity option. With the current implementation, you don’t even have to copy all of the tables, just part of them. It seems like it would be easy to figure out which ones are super critical and would cost money for the corporation and make sure they are up and recoverable quickly.

>>> cynicism off <<<<