Outsourcing your Oracle E-Business Suite environment - Part IV

In the previous articles in this series about outsourcing E-Business Suite environments I discussed the question why, the more technical aspects and the supporting services. This article will cover the topics availability, disaster recovery and security.

Availability

Load Balancing - Load Balancing techniques are used for two reasons: To increase availability and to increase capacity.

Application Server
On the application server level of E-Business Suite, one can choose to implement two kinds of load balancing: Forms based (11i) and HTTP based (11i and r12). In order to run forms based load balancing, one server needs to be designated as Forms Metrics Server, the remaining servers will be Forms Metrics Clients. On each of the servers the Forms Server needs to be running. All clients will be connected to the Forms Metrics Server, which diverts the connection to either of the forms servers in the application tier. This is a basic form of load balancing, because it is based on a round-robin principle. The alternative is HTTP Load Balancing, which requires Forms Servlet Mode to be in use. In this configuration, the forms server is running through a servlet, which runs under the http server. Therefore, there is no need to use other ports than the port the http server is using. A side effect is increased security. Another requirement is to use a separate load balancer. This load balancer will be configured with a central IP address (resolvable by a DNS name). The connections will be spread over the available application servers. This can be done on a round-robin principle, but can also be based on actual system load, like with Cisco ACE technology. One thing to keep in mind, which is of great importance when implementing load balancing is to ensure that Session Stickiness/Persistence (IP or cookie based) is implemented.
It is also possible to implement DNS based load balancing. In this configuration, the DNS server has got multiple IP addresses for the same hostname and will randomly reply a possible IP address to any DNS request for an application server. This requires some additional configuration on the Application Tier, like JServ load balancing and defining OProcMgr nodes (web nodes) in your environment (using the context editor).
Refer to metalink note 217368.1 for implementing load balancing for E-Business Suite

Database Server
Increasing availability on the database server can be established using Real Applications Cluster technology. Implementing Real Applications Cluster will require Oracle Clusterware, and takes some of the system resources for cluster node intercommunication. However, the increase in availability will more than compensate for this. I do need to say that no matter how much I appreciate RAC, it adds to the complexity of your infrastructure and you will definitely need resources in your team or at your hosting provider (depending on who is going to manage/maintain the E-Business Suite) capable of managing and maintaining Real Applications Cluster environments. Otherwise you may end up with a system with lower uptime compared to a single instance database environment.

Disaster Recovery
Dataguard – This topic adds up to the previous one, availability. If your hosting provider can provide multiple data centers, it might be worthwhile investigating the possibilities of implementing DataGuard. With E-Business Suite you can establish a physical standby database on a remote location that can be switched over to when the primary database fails for whatever reasons. For more information on implementing DataGuard ee Metalink Notes 216212.1 and 403347.1 for Release 11i and Metalink Note 452056.1 for Release 12.

Data Replication – If Dataguard is not feasible, you should at least be given the possibilities for data replication. Make sure your hosting partner is able to replicate your business-critical data to a remote data center, so you, or your hosting partner can rebuild the entire environment on another location in case of a site failure.

For any DR solution, you should be able to quantify the maximum downtime you find acceptable. In order to be able to do this, it may be required to estimate the cost of downtime per time-unit. Take into consideration that your company should have the resources to sustain the damage of this downtime. Make sure your hosting partner can live up to the required level of service. After all, it is your business-critical data.

Security
Talking about business-critical data, you don’t want anyone that is not supposed to be there strolling around in your environment. Therefore it is of greatest importance that your hosting partner can guarantee the highest level of security. The International Standards Organization (ISO) has a certification for this: ISO-27001:2005. Ask for this certification at your partner of choice. This ensures that your partner has been audited on a number of controls dealing with information security.
Various security issues need to be covered when you want to outsource your environment. This can be issues that you as a customer may require, but it can also be issues that are required by your hosting partner. It is essential to work these out before starting your contract, or you may be surprised your partner doesn’t provide a solution you want to be implemented because of their tight security, or you will become personably responsible for security measures your partner can or will not provide.

Hardening – your systems need to be secure enough by themselves. This means that e.g. you don’t want to allow direct root access from a remote location, sudo lists to limit the amount of users that may become root, sudo command lists, to limit the commands that can be run with root privileges, limiting the services to those that are necessary to run the environment.

External Access – Remote OS access should be regulated by an external authentication system like an Active Directory system. If this is not possible, you should require vLANs that separate your environment(s) from others. It can also be arranged with Access Control Lists, but they should be combined with an AD solution.Intrusion Detection – No matter how secure your environment is hosted, it will always be possible for someone to try and attack your environment. For this, it is important to have an Intrusion Detection system in place. This system alarms when it detects uncommon activities on the environment, indicating someone or some program is trying to attack the system.

E-Business Suite Security - Ask your outsourcing partner what they would do to secure your E-Business Suite. It is of great importance. You might have a secure Operating System, but your application that runs on it must be secure as well, because it might be located at a remote site, in a hosting center, along with various other applications from who knows where. Security may very well be the top priority on your list. Refer to Metalink Note 189367.1 - Best practices for securing Oracle E-Business Suite for directions.

I hope to publish a questionnaire to ask a potential outsourcing partner as a conclusion to this series.

Further Tuning of OC4J Containers in SOA Suite 10.1.3.3

In one of my recent posts I discussed a performance problem with OC4J Containers being restarted in SOA Suite. This week I had to go back, the environment had become a lot more stable, however similar problems still occurred.
Conclusion: The system was more stable, however the cause of the problem had not been taken away yet, so further investigation was needed.

This time I did some more investigation in tuning java options (opmn.xml) and increased the heap space to 2GB, hoping this would work out, however to no result: The problems persisted.

Then, my eyes fell on the error_log of the http daemon. I wondered what the cause of all of those "oc4j_socket_recvfull timed out" messages could mean...

There is a number of Metalink Notes about this issue, however the solutions I checked were not solving our issues.

Then I realized that the environment was deployed on AIX 5.3. I have worked for IBM and with AIX for a couple of years. I have seen similar issues on other environments (like Oracle E-Business Suite and RAC) and I still had a note somewhere about network parameter tuning on AIX. I checked these on the server, and found and fixed the following parameters, which seemed to have solved the issues:

rfc1323 - This defaults to 0 on AIX, and should be set to 1. By default, the TCP window size is limited to 65536 bytes (64 K) but can be set higher if the rfc1323 value is set to 1 (see tcp_recvspace value).

tcp_sendspace - The TCP Send Buffer. This defaults to 16384 bytes and should set to a higher value for Oracle Application Server (OHS). I set it to 266140.

tcp_recvspace - The TCP Receive Buffer. This also defaults to 16384 and should be set to a higher value for OHS. I set it to 266140, similar to tcp_sendspace. In combination with rfc1323, this enables a connection to negotiate a larger TCP window. If you are sending data through adapters that have large MTU sizes (32 K or 64 K for example), TCP streaming performance might not be optimal unless this option is enabled because a single packet will consume the entire TCP window size. Therefore, TCP is unable to stream multiple packets as it will have to wait for a TCP acknowledgment and window update from the receiver for each packet.


tcp_nodelayack - Value should be set to 1 for Oracle applications. The tcp_nodelayack option prompts TCP to send an immediate acknowledgement, rather than the usual 200 ms delay. Sending an immediate acknowledgement might add a little more overhead, but in some cases, greatly improves performance. Performance problems have been seen when TCP delays sending an acknowledgement for 200 ms, because the sender is waiting on an acknowledgment from the receiver and the receiver is waiting on more data from the sender. This might result in low streaming throughput.

Outsourcing your E-Business Suite environment - Part III

In this third article about outsourcing E-Business environments I will cover the supporting Services that you may need for your environment.
Read the first and second article in this series on Outsourcing your E-Business Suite

Supporting Services
Oracle E-Business Suite is a large application suite with many functionalities. These functionalities sometimes depend on external services that it needs in order to provide functionality to a user or an organization. The Oracle E-Business Suite has a three tier model, i.e. a database, an application and a client tier. The connectivity between the tiers is of essential importance. Therefore, a number of measures need to be taken in order to provide services to enable connectivity according to requirements one is setting for their E-Business Suite.

DNS
Domain Name System - DNS is of great importance if you want your hostname resolution taking place centrally. However, your hosting provider can give you two options: to use their general DNS Servers, or to establish a “local” or customer dedicated DNS service. The latter will mean that a server (preferably more) in your infrastructure needs to be designated as a DNS server, and this DNS server needs to be maintained. One of the key questions here is whether you want your application servers to be able to resolve addresses within the companies own network. Your companies network will be – in a way – extended to the hosting partner, by means of a VPN or WAN connection. The question is whether your application servers need to be able to resolve host names in your own network. If that is the case, you will probably need your own DNS servers. You can dedicate specific servers for this task, but usually a shared server will suffice.

SMTP
Simple Mail Transfer Protocol - Oracle E-Business Suite is often used with Oracle Workflow. One of the major advantages of Oracle Workflow is that tasks can be assigned to people, and they can automatically be notified that specific tasks are waiting to be completed. Usually, this notification is done through the Workflow Mailer. As the name says, this service uses e-mail functionality for notification. If you are using Workflow Mailer, you will require a SMTP server to be available to the application server, in order to be able to send e-mails.Again, it is possible to have dedicated servers for this, but the SMTP service can be established on a server that for example also provides DNS Service.

DHCP
Dynamic Host Configuration Protocol - Usually DHCP is not in use for an Oracle E-Business Suite. The reason for me to mention the DHCP service is because when you are discussing the options with your possible partner of choice, they will ask you whether you need it. In my opinion, the answer should be no. The database servers will not use DHCP, neither the Application servers. Any other server should have a fixed IP Address, otherwise it would be a workstation, in my humble opinion.

SSL
Secure Sockets Layer - Unless you have a dedicated VPN or WAN connection, you should demand a kind of secure connection for your E-Business Suite. This can be done through SSL Accelerators, or by setting up SSL in the E-Business Suite. When you have a VPN connection, the connection is secured already, as is the case with a WAN connection, because both of them are extensions on your current network. Still, having SSL implemented for your production E-Business Suite is never a bad idea. With hackers around almost every corner these days, you can never be sure enough that the information that is exchanged between your clients and application servers is secured and encrypted.

AD
Active Directory – This supporting service can be used in two ways. One way is to configure Active Directory Services to control who can log on to the operating systems, the other way is to control access to the database and applications. AD services can provide access control to many users in your environment, and can even be used in a centralized, i.e. shared service configuration. This means that your hosting provider could use AD to control access for all of its customers, including you. By configuring the users in AD, the provider can grant and deny access to certain servers or parts of its infrastructure. The question whether you as a customer want this, is a whole different issue, but I will come back to this in my next article.

NTP
Network Time Protocol – Especially when you have a multi-node environment or cluster technology like Real Application Clusters, it can be of great importance that all of your servers have exactly synchronized time. In order to provide this, one of the servers in your infrastructure, or again a shared server in the hosting center of your partner should be configured as a NTP server. All of the servers in your infrastructure should be synchronizing their system clocks to this server via a small piece of software. It is also possible to use Time Servers on the Internet, but then your servers need to have external internet access (see outbound web services)

Inbound Web Services
This service enables accessibility from the outside internet. In many cases, your E-Business Suite will be used from within your own company infrastructure. There is a number of cases where you would have parties from outside your company needing access to your environment, for example iProcurement, Oracle Time and Labor, iStore, etc. may require your E-Business Suite to be available from the Internet, if the parties accessing your E-Business Suite don’t have access to your own network. Usually, the server providing this service is placed in a so-called Demilitarized Zone (DMZ) about which later in this article.

Outbound Web Services
If your servers providing the E-Business Suite application need access to internet, you will need Outbound Web Service. For example, you have need to configure your application server to access Metalink services for updates, or to upload configuration data which is used for the Oracle Configuration Manager. In these case your servers need access to Internet.

DMZ
Demilitarized Zone – This is an area in the network that has a lower level of security, in order to facilitate access from the Internet. In a normal situation the servers of your infrastructure are only available to your own organization and not to the outside world. If your requirements are such that access from the Internet should be possible, you would need a zone that is accessible from the internet (usually via a SSL port: 443), which holds the servers and applications that provide the necessary services. Usually these servers are a bit more hardened than the other servers, which are located behind one or more additional firewalls, i.e. a safer zone.

Most of the above supporting services will be needed to provide an Oracle E-Business Suite environment, regardless whether you outsource or not. However, when you consider outsourcing, you should have a very clear view on these aspects, all in one. Everything needs to be clear before you start an outsourcing project. Adding a service to your environment should never be a problem with any outsourcing partner, but it may be defined as a change to the project/contract, which will end up in additional (read: more) costs. Covering these at the beginning of the contract will save you money at the end of the day.
I know the above list is not yet complete. Aspects like availability, load balancing, disaster recovery and, last but not least, security have not yet been covered, though they may be even more important to your environment. I will cover these in my next article.

OC4J Containers getting restarted in OAS

In the last weeks I have been called by a number of customers having problems with their newly installed Oracle Application Server, or rather Oracle SOA Suite.
I found more than once an issue with the timeouts on response times of these OC4J Containers.
All of the situations I was faced with were SOA Suites that had just been taken into production. The behaviour differed a little from case to case, but nevertheless the solution to the problem was found in the same solution for each of them.
OPMN manages the OC4J containers in Application Server (10g, that is). Because of this, it tries to ping all of the containers in order to check whether they are still alive. If the ping doesn't get returned quickly enough, it restarts the OC4J container and writes an error into the opmn.log file located under $ORACLE_HOME/opmn/logs. This can lead to various problems: If OPMN restarts the HTTP Server, the website will temporarily be unavailable which can be rather disturbing. If other OC4J containers get restarted, it can lead to various other errors like HTTP-500 (Internal Server Error) or other problems.
The solution is to tune the OC4J container ping parameters in the opmn.xml file found under $ORACE_HOME/opmn/conf:

For the http server, look for the tag process-type id="HTTP_Server" module-id="OHS". Add the following after this tag:

<process-set id="HTTP_Server" restart-on-death="true" numprocs="1">
<start timeout="300" retry="3"/>
<stop timeout="300"/>
<restart timeout="300" retry="3"/>
<ping timeout="60" interval="600"/>
</process-set>

Next, look for the tag within the HTTP Server definition and add the following lines:

<category id="ping-parameters">
<data id="ping-url" value="/"/>
</category>
<category id="restart-parameters">
<data id="reverseping-timeout" value="345"/>
<data id="no-reverseping-failed-ping-limit" value="3"/>
<data id="reverseping-failed-ping-limit" value="6"/>
</category>

For all other OC4J containers in the file, insert the same lines you added between the tags for the HTTP Server.
This will increase the timeout for the response of the OC4J container, giving it a little more time.

Outsourcing your E-Business Suite environment - Part II

In the previous article in this series I pointed out why E-Business Suite is a perfect application to outsource.
This article will cover more of some technical aspects to give attention to while selecting an outsourcing partner.

Hardware Requirements - Your partner of choice should be able to meet the requirements you have set for your environment. Given the assumption that you have a running environment in-house, you don't want to be faced with a platform migration during an outsourcing project. It might very well be possible that you do want to migrate to another platform, but in general, an outsourcing project should not be the primary reason for this. There should be other reasons to migrate to another platform, and it might very well be more sensible to perform the platform migration in house, before starting the outsourcing project. The reason I am taking this approach, is because platform-migrating and outsourcing are two too big changes to your environment to combine in one project. Besides, If your partner of choice is able to provide a similar platform, the direct need for migrating may become obsolete, because they have better resources.

CPU - Many hosting providers, nowadays, offer blade technology. Often, the choices are limited to either Windows or Linux based solutions. The consequence is that the amount of processors is limited, often to 2 (be it single, dual or quad core). For smaller to mid-size environments this usually is no problem, however, whith larger environments environments can become quite complex. It must be said that with the newer CPU technologies, more powerful servers are achieved and the larger server solutions (like SUN Enterprise Servers, larger IBM pSeries, HP SuperDomes) become less in demand.

Network Interfaces - Especially with blade technology, it is of great importance to be able to influence your hosting partner when designing the network interface usage. Take an Oracle E-Business Suite environment running Real Application Clusters database technology. You would probably require a

• Management LAN
• Server LAN
• Interconnect LAN
• Storage LAN (if you are using NAS)

Most blades have a maximum of 4 network interfaces. Many hosting providers will tell you their default configuration is to team two sets of two interfaces, netting you to two network interfaces available to your applications. If you are using as storage LAN (NAS), you want as much bandwidth as possible, especially for your database servers. The Interconnect LAN must be non-routed and dedicated to the Interconnect traffic between the RAC nodes. The only option you have left is to share the Management vLAN and Server vLAN with the Storage vLAN over one Interface. In my humble opinion something you don't want.
Therefore, it might be necessary to move away from teamed interfaces and have dedicated interfaces. The fact that you are running RAC means you will have redundancy on a solution level, and the necessity for redundancy on NIC level will become less.

OS Requirements
Linux - In my humble opinion, for small to mid-sized environments I would definitely choose Linux (especially Oracle Enterprise Linux), because it is Oracle's platform of choice and because Oracle can support both application and OS. With business critical environments like E-Business Suite you don't want to spend time going from OS support provider to application support provider and vice versa, just because they are fingerpointing one another. Having the support centralized is one of the key features Oracle can provide when choosing Oracle Enterprise Linux.
However, do make sure your hosting provider is offering you the right type of Linux. Make sure you check the certification matrix, when your partner is offering you a standard Operating System Solution. One of the important details today is that 64-bit OS'es are offered frequently, however, for example E-Business Suite 11i only runs on a 32-bit OS, or at least a 32-bit kernel. It is possible to use 64-bit OS for the database, but not for the application. Things are a little different with E-Business Suite R12. R12 is certified against almost any 64-bit OS. No need to worry there.

Unix Same story here as above with Linux. Make sure your (or the proposed) architecture is in line with certification. Too many different flavors of Unix are available and there is too much to write about in a single article like this one. Certification is the key.

Windows To be honest, I have never seen an E-Business Suite running on Windows in my life, so writing about it would not be honest. What I do know is that you cannot just take a Windows Server and install Oracle E-Business Suite on it. It requires a couple of additional software packages in order to run.

Storage
NAS Especially when considering Linux, NAS is the way to go, when you want to have multiple application servers. I am a huge fan of SAN, however, when it comes to the application server, there is a lack of support on shared/clustered filesystems for the application tier of Oracle E-Business Suite. In fact, the only file system (as far as I am aware) that is certified with Oracle E-Business Suite is NFS, which is used in NAS technology. So, for sharing your APPL_TOP, do not consider OCFS, GFS or any other shared filesystem other than NFS. Oracle will help you when you run into trouble, but when problems cannot be reproduced on regular file systems, you will be diverted to the manufacturer of the file system. In production, I would never want to run on an architecture that is not fully certified and supported by the application.

SAN Probably the fastest storage solution available still. If your hosting provider is able to deliver SAN, use it for the database tier. First of all, when running on blade technology, this will save you an Ethernet port, which you can then use for different purposes. Second, performance is generally better with SAN (at least, that is according to my experience).

Backup / Recovery
NearLine Storage Your partner of choice should be able to provide remote storage. One of the key features of hosting your environment is to outsource the responsibility for data recovery. But one should never assume. Please take time to understand what your partner of choice is offering on retention policies, offloading backup data to a remote location, whether they have a remote datacenter at all and how they are dealing with connections between the datacenters.
If your partner of choice hase multiple datacenters, it may be possible to designate one of them as disaster recovery location. You will have the choice of leaving the Disaster Recovery into the hands of your hosting partner, or building your own solution for Disaster Recovery. Most of the time the hosting partner (or rather outsourcing partner) will, or should have expertise on building, maintaining and managing advanced topologies for Oracle E-Business Suite.

Disaster Recovery Establishing Oracle's Maximum Availability Architecture for E-Business Suite can be required to meet the availability and disaster recovery requirements. Your hosting partner should have multiple datacenters and have an idea about connectivity to the other datacenter when a disaster should occur. Creating MAA in one datacente doesn't make sense.

This all might seem trivial, but it is of greatest importance to cover before selecting your partner.

In the next article, I will discuss the surrounding systems, or supporting services.

Outsourcing your E-Business Suite environment - Part I

Many, most smaller, companies do want to have all the benefits that the Oracle E-Business Suite can provide, however don't have the necessary resources to run, maintain and manage this complex suite of applications. A typical way to overcome these issues in general is to outsource this kind of solutions to a partner who is specialized in this matter. This can be done in various ways:

• Remote Management
  This way, the environment is kept inside the premises of the customer, but is managed remotely by a third party
• Hosting
  This way, the environment is hosted outside the premises of the customer, and optionally managed by a third party.

Hosting can be a good solution if you don't have enough resources to house your environment yourself, however a good deal of things need to be covered when selecting a hosting partner.

In the coming series of articles I will try to cover as many of these options as possible in order to make it easier to outsource your environment.

This first article will deal with the question why you would want to outsource.

The Oracle e-Business Suite is a complex suite of applications. It is composed out of several components, each requiring their own experience. For the bigger companies it may not be a problem hiring a team of professionals to maintain and manage this kind of environments. However, smaller companies often have an IT department that can do the daily management of the general IT infrastructure, but having a dedicated E-Business Suite DBA, or rather a couple of them (for continuity of service) can be too costly. Nevertheless, the E-Business Suite often is a business critical application.

Oracle E-Business Suite is a great application to be managed remotely. The architecture suits this approach seemlesly. It is designed to be implemented at large companies that often have multiple sites or even multinationals. Therefore, offshoring the E-Business Suite should not be a problem in itself, because it is designed to work in a multi site environment.

It is virtually impossible to understand, let alone master every aspect of the E-Business Suite, simply because it is too big a piece of software. That in itself implies that the E-Business Suite is applicable to almost any kind of business. One of the major advantages of outsourcing the E-Business Suite is that, if you choose the right partner, you can leverage the knowledge of all of your partner’s application administrators in order to ensure a smoothly running application in your organization. Be it Functional or Technical Application Administrators, they provide services for other companies as well, and consolidate this knowledge in their service offering. It is cost-efficient to outsource your environment, because you will only pay for the service you need or want. Besides, outsourcing partners have resources in place to maximize availability of service. Things you will have to pay for in full when you decide not to outsource.

Key factors of success when considering or selecting an outsourcing partner are trust, security, technical understanding, functional understanding, connectivity, continuity of service, and not to forget: performance. In addition to this, you need to understand yourself what your application is about. There is a whole lot of questions to ask your potential parters in order to get a good understanding of them being a good fit for you or not. All of these questions need answers. Only then you can make a solid choice whether to outsource and to whom.

In my next article(s) I will discuss some more of the technical questions you may need to consider for outsourcing.