distributed and high availability

I have had a discussion at a number of customers of how to expand their existing footprint. They are regional companies that have merged or acquired another company. The new organization is typically an international company. The question comes up how to let the users from Singapore, Sydney, or Scotland see the financials, inventory, and hr data without taking forever to perform standard operations.

The first method that I always recommend is to take inventory of what they have and how it is configured and accessed. If you have central servers in a major metropolitan area in the US, make sure that you have high speed access to your intranet. This means something larger than a T1 connection for most companies. Once you have the high speed connection, make sure that you have two data centers. It does not matter where the remote data center is, just make sure that it isn’t in the same building or server room. If you have a site failure because the power company goes down or a storm floods the city, put your alternate data center in another city. If you are looking at supporting operations in Europe, Asia, or China, why not put an alternate data center there? Typical reasons to stop people from putting data centers there are legal, logistical, and political. Some countries do have restrictions on what data can be exported from the country. France, for example, does not let HR data leave the country. Employee data for French citizens must stay in France. If you have a large data center in Asia, you might want to put your data center in cities like Singapore because they have advanced and reliable network infrastructure. Putting it in China using the publically available network might lead to problems or random outages. The same is true for many companies in Africa.

Once you have alternate sites available, make sure that you have two network connections into and out of each site. If one network goes down, the second network connection provides backup. Many companies in the US only have one internet connection because maintaining two high speed connections can be expensive. Many of the services allow you to pay for an all you can eat service where you only pay for the bandwidth that you use. This is an excellent idea but it does not allow for predictable cost. If you generate an event that causes a significant amount of traffic, like a new product release or a new marketing event, your network cost can go through the roof. Many companies pay for a high speed fast pipe like an OC3 or OC12 and supplement this with a variable network connection. The variable network connection is the standby failover line and is only used in event of failure or substantial congestion on the high speed fast pipe.

Once you have the network plumbing working, you need to look at latency between the sites. The latency needs to be guaranteed less than 300ms, typically less than 150ms for the Oracle suite of applications to properly work. Typical network speeds around the world on an uncongested network is on the order of 85ms. Network latency inside a data center is typically on the order of 10ms. If you can fall into the 30 to 50ms range, things will work properly and you typically will not notice that you are at a different data center. When latency jumps to 100ms plus, users will start to complain of slow applicaitons. Realize for each web page interaction there are four to five round trips on the network to transfer the web page. This means that a half a second per web page before it starts to paint. This delay is noticable from the user.

The discussions that I have had recently are how do I split my financials and HR data between two sites. If I have operations in Europe and operations in the US, can I have the European data in London and my US data in Houston? The answer is yes and no. Yes, I can have my Oracle Apps running in both locations. Yes, I can have a database that stores the Europe data on a database server in London. Yes, I can have a database that stores US data in Houston. The big problem is how do I run two financial systems and merge these repositories? How do I convert between Euros and Dollars? How do I report expenses in multiple currencies and convert this data on a time basis based on when I exchange money with my bank? How do I adhere to the Tax laws in the US and the Tax laws in the UK? There is more to consider than just can I store my data in multiple locations.

You can have Oracle Apps running in two different sites and have two different databases in the two sites as well. The number two is not special, this could be twenty or twelve just as easy. The difficulty comes about in merging this data into one corporate view. If, for example, I want to look for a part in our inventory, I want to look first locally then internationally. The search can be done through Oracle Apps (PeopleSoft, Seibel, JD Edwards, Hyperion, etc.) with a regional restriction or with a global search. The regional restriction will look locally. The global search will look in multiple tables or data partitions. You can replicate the European data to the US using DataGuard and the US data to Europe the same way. This will give you two database tables at each site with one table being the primary and the second being a logical or physical standby as well. The standby data can be accessed in read-only mode and changes can be pushed across using a streams interface to the primary. The drawback to this is that 9i and 10g restrict database availability for databases that are opened with read only. The redo apply stalls until the database is closed. The 11g database allows you to have the database open and have the redo apply processed to this standby repository.

The distribution of applicaiton internationally is a complex problem. There are many alternatives or options to make it work. There are many problems and issues that you need to think about beyond simple data synchronization. This is a topic that I will probably visit a few more times.